Changeset 15748

Timestamp:

4 Nov 2019, 10:15:44 (5 years ago)

Author:

uli

Message:

Bad workaround to cope with overcrowded catalogs.

Zope does not cope well with catalogs of more than 500.000 elements.
Reindexing the catalog then means to put all objects into memory before
saving them - this can blow everything.

We therefore use a dirty hack to replace the updateIndex method of
a catalog with a more careful function, that, however, should not be
run in production mode. No new items should be added during the run.

In the long run, we certainly need something more sustainable.

Location:

main/waeup.kofa/trunk/src/waeup/kofa

Files:

• hostels/hostel.py (modified) (1 diff)
• utils/helpers.py (modified) (2 diffs)

main/waeup.kofa/trunk/src/waeup/kofa/hostels/hostel.py

@@ -345 +345 @@
             if 'bed_id' not in cat.keys():
                 nothing_to_do = False
+                # replace original `updateIndex` method
+                def updateIndexReplacement(index):
+                    reindex_cat(cat)
+                cat._updateIndex = cat.updateIndex
+                cat.updateIndex = updateIndexReplacement
+                # setup catalog
                 cat[u'bed_id'] = FieldIndex(field_name=u'bed_id')
+                cat.updateIndex = cat._updateIndex  # undo changes
                 logger.info(
                     '%s: bed_id index added to beds_catalog.'

main/waeup.kofa/trunk/src/waeup/kofa/utils/helpers.py

@@ -27 +27 @@
 import shutil
 import tempfile
+import transaction
 import grok
 from cStringIO import StringIO
@@ -956 +957 @@
     uids = get_catalog_docids(cat)
     print("Found %s entries..." % len(uids))
-    for docid in uids:
+    for n, docid in enumerate(uids):
         ob = uidutil.getObject(docid)
         cat.index_doc(docid, ob)
+        # indexes can become huge. commit changes every 5000th round to
+        # keep the memory footprint of catalogs `updateIndex` manageable
+        if not n % 5000:
+            transaction.commit()
     d2 = datetime.datetime.now()
     print("Finished. %s" % (d2 - d1))