Batch processing via browser
****************************

Preliminaries:

We define a function that looks up a form with several submit buttons
for the one with a given value (this functionality is missing in
zope.testbrowser):

    >>> def lookup_submit_value(name, value, browser):
    ...   """Find a button with a certain value."""
    ...   for num in range(0, 100):
    ...     try:
    ...       button = browser.getControl(name=name, index=num)
    ...       if button.value.endswith(value):
    ...         return button
    ...     except IndexError:
    ...       break
    ...   return None

Create a site:

    >>> from waeup.ikoba.app import Institution
    >>> getRootFolder()['app'] = Institution()
    >>> from zope.component.hooks import setSite
    >>> setSite(getRootFolder()['app'])

Create a datacenter storage path:

    >>> import os
    >>> import tempfile
    >>> dc_path = tempfile.mkdtemp()

Log in:

    >>> from zope.testbrowser.testing import Browser
    >>> browser = Browser()
    >>> browser.addHeader('Authorization', 'Basic mgr:mgrpw')
    >>> browser.handleErrors = False

Set datacenter path and deselect moving old data:

    >>> browser.open('http://localhost/app')
    >>> browser.getLink('Data Center').click()
    >>> browser.getLink('Edit settings').click()
    >>> browser.getControl(name='newpath').value = dc_path
    >>> browser.getControl(name='move').value = False
    >>> browser.getControl(name='save').click()

Set non-usable datacenter path:

    >>> browser.getLink('Edit settings').click()
    >>> browser.getControl(name='newpath').value = '/'
    >>> browser.getControl(name='save').click()
    >>> 'Given storage path cannot be used.' in browser.contents
    True
    >>> browser.getControl('Back to Data Center').click()


Batch processing users
======================

    >>> browser.open('http://localhost/app/datacenter')

Prepare a CSV file for users:

    >>> open('users.csv', 'wb').write(
    ... """name,title,public_name,email,phone,roles
    ... uli,Uli Fouquet,Chief Developer,uli@abc.de,+49-234-567,[]
    ... henrik,Henrik Bettermann,Admin,henrik@abc.de,+49-234-567,"['waeup.PortalManager', 'waeup.ImportManager']"
    ... """)

Upload the file:

    >>> import cStringIO
    >>> browser.getLink('Upload data').click()
    >>> filecontents = cStringIO.StringIO(
    ...   open('users.csv', 'rb').read())
    >>> filewidget = browser.getControl(name='uploadfile:file')
    >>> filewidget.add_file(filecontents, 'text/plain', 'users.csv')
    >>> browser.getControl(name='SUBMIT').click()

Step 1: start batch processing:

    >>> browser.getLink('Process data').click()
    >>> button = lookup_submit_value(
    ...   'select', 'users_zope.mgr.csv', browser)
    >>> button.click()

Step 2: select a processor and mode:

    >>> importerselect = browser.getControl(name='importer')
    >>> importerselect.getControl('User Processor').selected = True
    >>> modeselect = browser.getControl(name='mode')
    >>> modeselect.getControl(value='create').selected = True
    >>> browser.getControl('Proceed to step 3').click()

Step 3: Fix headerlines

We get informed that there are no problems with the current header:

    >>> print browser.contents
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    ...
    Header fields OK
    ...

The submit button is enabled:

    >>> browser.getControl('Perform import').disabled
    False

    >>> browser.getControl('Perform import').click()

Step 4: See import results

The import was successful:

    >>> print browser.contents
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    ...Successfully processed 2 rows...
    ...Batch processing finished...
    ...File:...users_zope.mgr.csv...

User henrik has got the global roles:

   >>> henrik = getRootFolder()['app']['users']['henrik']
   >>> henrik.roles
   ['waeup.ProductsOfficer', 'waeup.PortalManager', 'waeup.ImportManager']

Pending files
=============

When an error occurs during an import, two files are generated: a CSV
file with finished files and a CSV file with pending data. Both are
stored in the appropriate subdirectories in datacenter. We try to
create faculties, from which one already exists.

Go to datacenter page:

    >>> browser.open('http://localhost/app/datacenter')

Prepare a CSV file for users:

    >>> open('newusers.csv', 'wb').write(
    ... """name,title,public_name,email,phone,roles
    ... isouaba,Issoufou Abba Moussa,Chief Developer,isouaba@abc.de,+49-234-567,[]
    ... henrik,Henrik Bettermann,Admin,henrik@abc.de,+49-234-567,"['waeup.PortalManager', 'waeup.ImportManager']"
    ... """)

Upload the file:

    >>> import cStringIO
    >>> browser.getLink('Upload data').click()
    >>> filecontents = cStringIO.StringIO(
    ...   open('newusers.csv', 'rb').read())
    >>> filewidget = browser.getControl(name='uploadfile:file')
    >>> filewidget.add_file(filecontents, 'text/plain', 'newusers.csv')
    >>> browser.getControl(name='SUBMIT').click()

Since we now have a user with waeup.ImportManager role, an email has been sent:
    >>> print browser.contents
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    ...
    ...All import managers have been notified by email...
    ...

Step 1: start batch processing:

    >>> browser.getLink('Process data').click()
    >>> button = lookup_submit_value(
    ...   'select', 'newusers_zope.mgr.csv', browser)
    >>> button.click()

Step 2: select a processor and mode:

    >>> importerselect = browser.getControl(name='importer')
    >>> importerselect.getControl('User Processor').selected = True
    >>> modeselect = browser.getControl(name='mode')
    >>> modeselect.getControl(value='create').selected = True
    >>> browser.getControl('Proceed to step 3').click()

Step 3: Fix headerlines

As there should be no problem with the headers, we can immediately
perfom the import:

    >>> browser.getControl('Perform import').click()

One line could not be imported:

    >>> print browser.contents
    <!DOCTYPE html PUBLIC...
    ...
    ...Processing of 1 rows failed...
    ...Successfully processed 1 rows...
    ...

Now there are two files as a result in datacenter storage's root and
``finished`` dirs:

    >>> pending_file = dc_path + '/newusers_zope.mgr.create.pending.csv'
    >>> print open(pending_file).read()
    name,roles,title,public_name,phone,email,--ERRORS--
    henrik,"['waeup.PortalManager', 'waeup.ImportManager']",Henrik Bettermann,Admin,+49-234-567,henrik@abc.de,This object already exists. Skipping.

    >>> finished_file = dc_path + '/finished/newusers_zope.mgr.create.finished.csv'
    >>> print open(finished_file).read()
    name,roles,title,public_name,phone,email
    isouaba,[],Issoufou Abba Moussa,Chief Developer,+49-234-567,isouaba@abc.de

The finished-file contains the dataset we could import, while the
pending file contains the dataset that failed, appended by an error
message.


Fixing the pending file
-----------------------

We 'edit' the pending file (setting code to ``FAC5`` and title
appropriately, and removing the --ERROR-- column) and finish the
import this way:

    >>> open(dc_path + '/newusers_zope.mgr.create.pending.csv', 'wb').write(
    ... """name,title,public_name,email,phone,roles
    ... gbenga,Jason Bamidele,Chief Developer,gbenga@abc.de,+49-234-567,[]
    ... """)

Step 1: start batch processing:

    >>> browser.open('http://localhost/app/datacenter')
    >>> browser.getLink('Process data').click()
    >>> button = lookup_submit_value(
    ...   'select', 'newusers_zope.mgr.create.pending.csv', browser)
    >>> button.click()

Step 2: select a processor and mode:

    >>> importerselect = browser.getControl(name='importer')
    >>> importerselect.getControl('User Processor').selected = True
    >>> modeselect = browser.getControl(name='mode')
    >>> modeselect.getControl(value='create').selected = True
    >>> browser.getControl('Proceed to step 3').click()

Step 3/4: Fix headerlines and import:

As there should be no problem with the headers, we can immediately
perfom the import:

    >>> browser.getControl('Perform import').click()

This time everything should work:

    >>> print browser.contents
    <!DOCTYPE html PUBLIC...
    ...
    ...Successfully processed 1 rows...
    ...

    >>> sorted(os.listdir(dc_path))
    ['deleted', 'finished', 'logs', 'unfinished']

    >>> os.listdir(dc_path + '/unfinished')
    []

    >>> sorted(os.listdir(dc_path + '/finished'))
    ['newusers_zope.mgr.create.finished.csv', 'newusers_zope.mgr.csv', 'users_zope.mgr.create.finished.csv', 'users_zope.mgr.csv']

Processed (finished) Files
==========================

    >>> browser.open('http://localhost/app/datacenter/processed')
    >>> 'download?filename=finished/users_zope.mgr.create.finished.csv' in browser.contents
    True

Log Files
=========

    >>> browser.open('http://localhost/app/datacenter/logs')
    >>> 'datacenter.log' in browser.contents
    True
    >>> browser.getControl('Show', index=0).click()
    >>> browser.getControl('Back', index=0).click()
    >>> browser.getControl('Back to Data Center').click()
    >>> 'Storage path:' in browser.contents
    True


Clean up:

    >>> import shutil
    >>> shutil.rmtree(dc_path)
