Changeset 15748 for main/waeup.kofa/trunk/src/waeup/kofa/utils

Timestamp:

4 Nov 2019, 10:15:44 (5 years ago)

Author:

uli

Message:

Bad workaround to cope with overcrowded catalogs.

Zope does not cope well with catalogs of more than 500.000 elements.
Reindexing the catalog then means to put all objects into memory before
saving them - this can blow everything.

We therefore use a dirty hack to replace the updateIndex method of
a catalog with a more careful function, that, however, should not be
run in production mode. No new items should be added during the run.

In the long run, we certainly need something more sustainable.

File:

• main/waeup.kofa/trunk/src/waeup/kofa/utils/helpers.py (modified) (2 diffs)

main/waeup.kofa/trunk/src/waeup/kofa/utils/helpers.py

@@ -27 +27 @@
 import shutil
 import tempfile
+import transaction
 import grok
 from cStringIO import StringIO
@@ -956 +957 @@
     uids = get_catalog_docids(cat)
     print("Found %s entries..." % len(uids))
-    for docid in uids:
+    for n, docid in enumerate(uids):
         ob = uidutil.getObject(docid)
         cat.index_doc(docid, ob)
+        # indexes can become huge. commit changes every 5000th round to
+        # keep the memory footprint of catalogs `updateIndex` manageable
+        if not n % 5000:
+            transaction.commit()
     d2 = datetime.datetime.now()
     print("Finished. %s" % (d2 - d1))