Batch Processing via Browser **************************** Preliminaries: We define a function that looks up a form with several submit buttons for the one with a given value (this functionality is missing in zope.testbrowser): >>> def lookup_submit_value(name, value, browser): ... """Find a button with a certain value.""" ... for num in range(0, 100): ... try: ... button = browser.getControl(name=name, index=num) ... if button.value.endswith(value): ... return button ... except IndexError: ... break ... return None Create a site: >>> from waeup.ikoba.app import Company >>> getRootFolder()['app'] = Company() >>> from zope.component.hooks import setSite >>> setSite(getRootFolder()['app']) Create a datacenter storage path: >>> import os >>> import tempfile >>> dc_path = tempfile.mkdtemp() Log in: >>> from zope.testbrowser.testing import Browser >>> browser = Browser() >>> browser.addHeader('Authorization', 'Basic mgr:mgrpw') >>> browser.handleErrors = False Set datacenter path and deselect moving old data: >>> browser.open('http://localhost/app') >>> browser.getLink('Data Center').click() >>> browser.getLink('Edit settings').click() >>> browser.getControl(name='newpath').value = dc_path >>> browser.getControl(name='move').value = False >>> browser.getControl(name='save').click() Set non-usable datacenter path: >>> browser.getLink('Edit settings').click() >>> browser.getControl(name='newpath').value = '/' >>> browser.getControl(name='save').click() >>> 'Given storage path cannot be used' in browser.contents True >>> browser.getControl('Back to Data Center').click() Batch processing users ====================== >>> browser.open('http://localhost/app/datacenter') Prepare a CSV file for users: >>> open('users.csv', 'wb').write( ... """name,title,public_name,email,phone,roles ... uli,Uli Fouquet,Chief Developer,uli@abc.de,+49-234-567,[] ... henrik,Henrik Bettermann,Admin,henrik@abc.de,+49-234-567,"['waeup.PortalManager', 'waeup.ImportManager']" ... """) Upload the file: >>> import cStringIO >>> browser.getLink('Upload data').click() >>> filecontents = cStringIO.StringIO( ... open('users.csv', 'rb').read()) >>> filewidget = browser.getControl(name='uploadfile:file') >>> filewidget.add_file(filecontents, 'text/plain', 'users.csv') >>> browser.getControl(name='SUBMIT').click() Step 1: start batch processing: >>> browser.getLink('Process data').click() >>> browser.getLink('Switch maintenance mode').click() >>> button = lookup_submit_value( ... 'select', 'users_zope.mgr.csv', browser) >>> button.click() Step 2: select a processor and mode: >>> importerselect = browser.getControl(name='importer') >>> importerselect.getControl('User Processor').selected = True >>> modeselect = browser.getControl(name='mode') >>> modeselect.getControl(value='create').selected = True >>> browser.getControl('Proceed to step 3').click() Step 3: Fix headerlines We get informed that there are no problems with the current header: >>> print browser.contents >> browser.getControl('Perform import').disabled False >>> browser.getControl('Perform import').click() Step 4: See import results The import was successful: >>> print browser.contents >> henrik = getRootFolder()['app']['users']['henrik'] >>> henrik.roles ['waeup.ProductsOfficer', 'waeup.PortalManager', 'waeup.ImportManager'] Pending files ============= When an error occurs during an import, two files are generated: a CSV file with finished files and a CSV file with pending data. Both are stored in the appropriate subdirectories in datacenter. We try to create faculties, from which one already exists. Go to datacenter page: >>> browser.open('http://localhost/app/datacenter') Prepare a CSV file for users: >>> open('newusers.csv', 'wb').write( ... """name,title,public_name,email,phone,roles ... isouaba,Issoufou Abba Moussa,Chief Developer,isouaba@abc.de,+49-234-567,[] ... henrik,Henrik Bettermann,Admin,henrik@abc.de,+49-234-567,"['waeup.PortalManager', 'waeup.ImportManager']" ... anne,Anne Palina,,anne@abc.de,+49-234-567,"['waeup.Nonsense']" ... """) Upload the file: >>> import cStringIO >>> browser.getLink('Upload data').click() >>> filecontents = cStringIO.StringIO( ... open('newusers.csv', 'rb').read()) >>> filewidget = browser.getControl(name='uploadfile:file') >>> filewidget.add_file(filecontents, 'text/plain', 'newusers.csv') >>> browser.getControl(name='SUBMIT').click() Since we now have a user with waeup.ImportManager role, an email has been sent: >>> print browser.contents >> browser.getLink('Process data').click() >>> browser.getLink('Switch maintenance mode').click() >>> button = lookup_submit_value( ... 'select', 'newusers_zope.mgr.csv', browser) >>> button.click() Step 2: select a processor and mode: >>> importerselect = browser.getControl(name='importer') >>> importerselect.getControl('User Processor').selected = True >>> modeselect = browser.getControl(name='mode') >>> modeselect.getControl(value='create').selected = True >>> browser.getControl('Proceed to step 3').click() Step 3: Fix headerlines As there should be no problem with the headers, we can immediately perfom the import: >>> browser.getControl('Perform import').click() Two lines could not be imported: >>> print browser.contents >> pending_file = dc_path + '/newusers_zope.mgr.create.pending.csv' >>> print open(pending_file).read() name,roles,title,public_name,phone,email,--ERRORS-- henrik,"['waeup.PortalManager', 'waeup.ImportManager']",Henrik Bettermann,Admin,+49-234-567,henrik@abc.de,This object already exists. Skipping. anne,['waeup.Nonsense'],Anne Palina,,+49-234-567,anne@abc.de,roles: invalid role >>> finished_file = dc_path + '/finished/newusers_zope.mgr.create.finished.csv' >>> print open(finished_file).read() name,roles,title,public_name,phone,email isouaba,,Issoufou Abba Moussa,Chief Developer,+49-234-567,isouaba@abc.de The finished-file contains the dataset we could import, while the pending file contains the dataset that failed, appended by an error message. Fixing the pending file ----------------------- We 'edit' the pending file by removing anne and replacing henrik by gbenga and finish the import this way: >>> open(dc_path + '/newusers_zope.mgr.create.pending.csv', 'wb').write( ... """name,title,public_name,email,phone,roles ... gbenga,Jason Bamidele,Chief Developer,gbenga@abc.de,+49-234-567,[] ... """) Step 1: start batch processing: >>> browser.open('http://localhost/app/datacenter') >>> browser.getLink('Process data').click() >>> browser.getLink('Switch maintenance mode').click() >>> button = lookup_submit_value( ... 'select', 'newusers_zope.mgr.create.pending.csv', browser) >>> button.click() Step 2: select a processor and mode: >>> importerselect = browser.getControl(name='importer') >>> importerselect.getControl('User Processor').selected = True >>> modeselect = browser.getControl(name='mode') >>> modeselect.getControl(value='create').selected = True >>> browser.getControl('Proceed to step 3').click() Step 3/4: Fix headerlines and import: As there should be no problem with the headers, we can immediately perfom the import: >>> browser.getControl('Perform import').click() This time everything should work: >>> print browser.contents >> sorted(os.listdir(dc_path)) ['deleted', 'finished', 'logs', 'unfinished'] >>> os.listdir(dc_path + '/unfinished') [] >>> sorted(os.listdir(dc_path + '/finished')) ['newusers_zope.mgr.create.finished.csv', 'newusers_zope.mgr.csv', 'users_zope.mgr.create.finished.csv', 'users_zope.mgr.csv'] Processed (finished) Files ========================== >>> browser.open('http://localhost/app/datacenter/processed') >>> 'download?filename=finished/users_zope.mgr.create.finished.csv' in browser.contents True Log Files ========= >>> browser.open('http://localhost/app/datacenter/logs') >>> 'datacenter.log' in browser.contents True >>> browser.getControl('Show', index=0).click() >>> browser.getControl('Back', index=0).click() >>> browser.getControl('Back to Data Center').click() >>> 'Storage path:' in browser.contents True Clean up: >>> import shutil >>> shutil.rmtree(dc_path)