Index: in/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batching.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batching.txt	(revision 12990)
+++ 	(revision )
@@ -1,556 +1,0 @@
-:mod:`waeup.ikoba.utils.batching` -- Batch processing
-****************************************************
-
-Batch processing is much more than pure data import.
-
-Overview
-========
-
-Basically, it means processing CSV files in order to mass-create,
-mass-remove, or mass-update data.
-
-So you can feed CSV files to processors, that are part of
-the batch-processing mechanism.
-
-Processors
-----------
-
-Each CSV file processor
-
-* accepts a single data type identified by an interface.
-
-* knows about the places inside a site (Company) where to store,
-  remove or update the data.
-
-* can check headers before processing data.
-
-* supports the mode 'create', 'update', 'remove'.
-
-* creates log entries (optional)
-
-* creates csv files containing successful and not-successful processed
-  data respectively.
-
-Output
-------
-
-The results of processing are written to loggers, if a logger was
-given. Beside this new CSV files are created during processing:
-
-* a pending CSV file, containing datasets that could not be processed
-
-* a finished CSV file, containing datasets successfully processed.
-
-The pending file is not created if everything works fine. The
-respective path returned in that case is ``None``.
-
-The pending file (if created) is a CSV file that contains the failed
-rows appended by a column ``--ERRROR--`` in which the reasons for
-processing failures are listed.
-
-The complete paths of these files are returned. They will be in a
-temporary directory created only for this purpose. It is the caller's
-responsibility to remove the temporay directories afterwards (the
-datacenters distProcessedFiles() method takes care for that).
-
-It looks like this::
- 
-     -----+      +---------+
-    /     |      |         |              +------+
-   | .csv +----->|Batch-   |              |      |
-   |      |      |processor+----changes-->| ZODB |
-   |  +------+   |         |              |      |
-   +--|      |   |         +              +------+
-      | Mode +-->|         |                 -------+
-      |      |   |         +----outputs-+-> /       |
-      |  +----+->+---------+            |  |.pending|
-      +--|Log |  ^                      |  |        |
-         +----+  |                      |  +--------+
-           +-----++                     v
-           |Inter-|                  ----------+
-           |face  |                 /          |
-           +------+                | .finished |
-                                   |           |
-                                   +-----------+
-
-
-Creating a batch processor
-==========================
-
-We create an own batch processor for an own datatype. This datatype
-must be based on an interface that the batcher can use for converting
-data.
-
-Founding Stoneville
--------------------
-
-We start with the interface:
-
-    >>> from zope.interface import Interface
-    >>> from zope import schema
-    >>> class ICave(Interface):
-    ...   """A cave."""
-    ...   name = schema.TextLine(
-    ...     title = u'Cave name',
-    ...     default = u'Unnamed',
-    ...     required = True)
-    ...   dinoports = schema.Int(
-    ...     title = u'Number of DinoPorts (tm)',
-    ...     required = False,
-    ...     default = 1)
-    ...   owner = schema.TextLine(
-    ...     title = u'Owner name',
-    ...     required = True,
-    ...     missing_value = 'Fred Estates Inc.')
-    ...   taxpayer = schema.Bool(
-    ...     title = u'Payes taxes',
-    ...     required = True,
-    ...     default = False)
-
-Now a class that implements this interface:
-
-    >>> import grok
-    >>> class Cave(object):
-    ...   grok.implements(ICave)
-    ...   def __init__(self, name=u'Unnamed', dinoports=2,
-    ...                owner='Fred Estates Inc.', taxpayer=False):
-    ...     self.name = name
-    ...     self.dinoports = 2
-    ...     self.owner = owner
-    ...     self.taxpayer = taxpayer
-
-We also provide a factory for caves. Strictly speaking, this not
-necessary but makes the batch processor we create afterwards, better
-understandable.
-
-    >>> from zope.component import getGlobalSiteManager
-    >>> from zope.component.factory import Factory
-    >>> from zope.component.interfaces import IFactory
-    >>> gsm = getGlobalSiteManager()
-    >>> cave_maker = Factory(Cave, 'A cave', 'Buy caves here!')
-    >>> gsm.registerUtility(cave_maker, IFactory, 'Lovely Cave')
-
-Now we can create caves using a factory:
-
-    >>> from zope.component import createObject
-    >>> createObject('Lovely Cave')
-    <Cave object at 0x...>
-
-This is nice, but we still lack a place, where we can place all the
-lovely caves we want to sell.
-
-Furthermore, as a replacement for a real site, we define a place where
-all caves can be stored: Stoneville! This is a lovely place for
-upperclass cavemen (which are the only ones that can afford more than
-one dinoport).
-
-We found Stoneville:
-
-    >>> stoneville = dict()
-
-Everything in place.
-
-Now, to improve local health conditions, imagine we want to populate
-Stoneville with lots of new happy dino-hunting natives that slept on
-the bare ground in former times and had no idea of
-bathrooms. Disgusting, isn't it?
-
-Lots of cavemen need lots of caves.
-
-Of course we can do something like:
-
-    >>> cave1 = createObject('Lovely Cave')
-    >>> cave1.name = "Fred's home"
-    >>> cave1.owner = "Fred"
-    >>> stoneville[cave1.name] = cave1
-
-and Stoneville has exactly
-
-    >>> len(stoneville)
-    1
-
-inhabitant. But we don't want to do this for hundreds or thousands of
-citizens-to-be, do we?
-
-It is much easier to create a simple CSV list, where we put in all the
-data and let a batch processor do the job.
-
-The list is already here:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner,taxpayer
-    ... Barneys Home,2,Barney,1
-    ... Wilmas Asylum,1,Wilma,1
-    ... Freds Dinoburgers,10,Fred,0
-    ... Joeys Drive-in,110,Joey,0
-    ... """)
-
-All we need, is a batch processor now.
-
-    >>> from waeup.ikoba.utils.batching import BatchProcessor
-    >>> from waeup.ikoba.interfaces import IGNORE_MARKER
-    >>> class CaveProcessor(BatchProcessor):
-    ...   util_name = 'caveprocessor'
-    ...   grok.name(util_name)
-    ...   name = 'Cave Processor'
-    ...   iface = ICave
-    ...   location_fields = ['name']
-    ...   factory_name = 'Lovely Cave'
-    ...
-    ...   def parentsExist(self, row, site):
-    ...     return True
-    ...
-    ...   def getParent(self, row, site):
-    ...     return stoneville
-    ...
-    ...   def entryExists(self, row, site):
-    ...     return row['name'] in stoneville.keys()
-    ...
-    ...   def getEntry(self, row, site):
-    ...     if not self.entryExists(row, site):
-    ...       return None
-    ...     return stoneville[row['name']]
-    ...
-    ...   def delEntry(self, row, site):
-    ...     del stoneville[row['name']]
-    ...
-    ...   def addEntry(self, obj, row, site):
-    ...     stoneville[row['name']] = obj
-    ...
-    ...   def updateEntry(self, obj, row, site, filename):
-    ...     # This is not strictly necessary, as the default
-    ...     # updateEntry method does exactly the same
-    ...     for key, value in row.items():
-    ...       if value != IGNORE_MARKER:
-    ...         setattr(obj, key, value)
-
-If we also want the results being logged, we must provide a logger
-(this is optional):
-
-    >>> import logging
-    >>> logger = logging.getLogger('stoneville')
-    >>> logger.setLevel(logging.DEBUG)
-    >>> logger.propagate = False
-    >>> handler = logging.FileHandler('stoneville.log', 'w')
-    >>> logger.addHandler(handler)
-
-Create the fellows:
-
-    >>> processor = CaveProcessor()
-    >>> result = processor.doImport('newcomers.csv',
-    ...                   ['name', 'dinoports', 'owner', 'taxpayer'],
-    ...                    mode='create', user='Bob', logger=logger)
-    >>> result
-    (4, 0, '/.../newcomers.finished.csv', None)
-
-The result means: four entries were processed and no warnings
-occured. Furthermore we get filepath to a CSV file with successfully
-processed entries and a filepath to a CSV file with erraneous entries.
-As everything went well, the latter is ``None``. Let's check:
-
-    >>> sorted(stoneville.keys())
-    [u'Barneys Home', ..., u'Wilmas Asylum']
-
-The values of the Cave instances have correct type:
-
-    >>> barney = stoneville['Barneys Home']
-    >>> barney.dinoports
-    2
-
-which is a number, not a string.
-
-Apparently, when calling the processor, we gave some more info than
-only the CSV filepath. What does it all mean?
-
-While the first argument is the path to the CSV file, we also have to
-give an ordered list of headernames. These replace the header field
-names that are actually in the file. This way we can override faulty
-headers.
-
-The ``mode`` paramter tells what kind of operation we want to perform:
-``create``, ``update``, or ``remove`` data.
-
-The ``user`` parameter finally is optional and only used for logging.
-
-We can, by the way, see the results of our run in a logfile if we
-provided a logger during the call:
-
-    >>> print open('stoneville.log').read()
-    processed: newcomers.csv, create mode, 4 lines (4 successful/ 0 failed), ... s (... s/item)
-
-
-We cleanup the temporay dir created by doImport():
-
-    >>> import shutil
-    >>> import os
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-As we can see, the processing was successful. Otherwise, all problems
-could be read here as we can see, if we do the same operation again:
-
-    >>> result = processor.doImport('newcomers.csv',
-    ...                   ['name', 'dinoports', 'owner', 'taxpayer'],
-    ...                    mode='create', user='Bob', logger=logger)
-    >>> result
-    (4, 4, '/.../newcomers.finished.csv', '/.../newcomers.pending.csv')
-
-This time we also get a path to a .pending file.
-
-The log file will tell us this in more detail:
-
-    >>> print open('stoneville.log').read()
-    processed: newcomers.csv, create mode, 4 lines (4 successful/ 0 failed), ... s (... s/item)
-    processed: newcomers.csv, create mode, 4 lines (0 successful/ 4 failed), ... s (... s/item)
-
-
-This time a new file was created, which keeps all the rows we could not
-process and an additional column with error messages:
-
-    >>> print open(result[3]).read()
-    owner,name,taxpayer,dinoports,--ERRORS--
-    Barney,Barneys Home,1,2,This object already exists. Skipping.
-    Wilma,Wilmas Asylum,1,1,This object already exists. Skipping.
-    Fred,Freds Dinoburgers,0,10,This object already exists. Skipping.
-    Joey,Joeys Drive-in,0,110,This object already exists. Skipping.
-
-This way we can correct the faulty entries and afterwards retry without
-having the already processed rows in the way.
-
-We also notice, that the values of the taxpayer column are returned as
-in the input file. There we wrote '1' for ``True`` and '0' for
-``False`` (which is accepted by the converters).
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-We can also tell to ignore some cols from input by passing
-``--IGNORE--`` as col name:
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             '--IGNORE--', '--IGNORE--'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (4, 0, '...', None)
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-If something goes wrong during processing, the respective --IGNORE--
-cols won't be populated  in the resulting pending file:
-
-    >>> result = processor.doImport('newcomers.csv', ['name', 'dinoports',
-    ...                             '--IGNORE--', '--IGNORE--'],
-    ...                    mode='create', user='Bob')
-    >>> result
-    (4, 4, '...', '...')
-
-    >>> print open(result[3], 'rb').read()
-    name,dinoports,--ERRORS--
-    Barneys Home,2,This object already exists. Skipping.
-    Wilmas Asylum,1,This object already exists. Skipping.
-    Freds Dinoburgers,10,This object already exists. Skipping.
-    Joeys Drive-in,110,This object already exists. Skipping.
-
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-
-
-Updating entries
-----------------
-
-To update entries, we just call the batchprocessor in a different
-mode:
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (4, 0, '...', None)
-
-Now we want to tell, that Wilma got an extra port for her second dino:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... Wilmas Asylum,2,Wilma
-    ... """)
-
-    >>> wilma = stoneville['Wilmas Asylum']
-    >>> wilma.dinoports
-    1
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-We start the processor:
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                    'dinoports', 'owner'], mode='update', user='Bob')
-    >>> result
-    (1, 0, '...', None)
-
-    >>> wilma = stoneville['Wilmas Asylum']
-    >>> wilma.dinoports
-    2
-
-Wilma's number of dinoports raised.
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-If we try to update an unexisting entry, an error occurs:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... NOT-WILMAS-ASYLUM,2,Wilma
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (1, 1, '/.../newcomers.finished.csv', '/.../newcomers.pending.csv')
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-    
-Also invalid values will be spotted:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... Wilmas Asylum,NOT-A-NUMBER,Wilma
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (1, 1, '...', '...')
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-We can also update only some cols, leaving some out. We skip the
-'dinoports' column in the next run:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,owner
-    ... Wilmas Asylum,Barney
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name', 'owner'],
-    ...                             mode='update', user='Bob')
-    >>> result
-    (1, 0, '...', None)
-
-    >>> wilma.owner
-    u'Barney'
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-We can however, not leave out the 'location field' ('name' in our
-case), as this one tells us which entry to update:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... 2,Wilma
-    ... """)
-
-    >>> processor.doImport('newcomers.csv', ['dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    Traceback (most recent call last):
-    ...
-    FatalCSVError: Need at least columns 'name' for import!
-
-This time we get even an exception!
-
-Generally, empty strings are considered as ``None``:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... "Wilmas Asylum","","Wilma"
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (1, 0, '...', None)
-
-    >>> wilma.dinoports
-    2
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-We can tell to set dinoports to ``None`` although this is not a
-number, as we declared the field not required in the interface:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... "Wilmas Asylum","XXX","Wilma"
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob', ignore_empty=False)
-    >>> result
-    (1, 0, '...', None)
-
-    >>> wilma.dinoports is None
-    True
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-Removing entries
-----------------
-
-In 'remove' mode we can delete entries. Here validity of values in
-non-location fields doesn't matter because those fields are ignored.
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... "Wilmas Asylum","ILLEGAL-NUMBER",""
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='remove', user='Bob')
-    >>> result
-    (1, 0, '...', None)
-
-    >>> sorted(stoneville.keys())
-    [u'Barneys Home', "Fred's home", u'Freds Dinoburgers', u'Joeys Drive-in']
-
-Oops! Wilma is gone.
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-Clean up:
-
-    >>> import os
-    >>> os.unlink('newcomers.csv')
-    >>> os.unlink('stoneville.log')
Index: /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batchprocessing.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batchprocessing.txt	(revision 12990)
+++ /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batchprocessing.txt	(revision 12991)
@@ -1,293 +1,556 @@
-Batch processing via browser
-****************************
-
-Preliminaries:
-
-We define a function that looks up a form with several submit buttons
-for the one with a given value (this functionality is missing in
-zope.testbrowser):
-
-    >>> def lookup_submit_value(name, value, browser):
-    ...   """Find a button with a certain value."""
-    ...   for num in range(0, 100):
-    ...     try:
-    ...       button = browser.getControl(name=name, index=num)
-    ...       if button.value.endswith(value):
-    ...         return button
-    ...     except IndexError:
-    ...       break
-    ...   return None
-
-Create a site:
-
-    >>> from waeup.ikoba.app import Company
-    >>> getRootFolder()['app'] = Company()
-    >>> from zope.component.hooks import setSite
-    >>> setSite(getRootFolder()['app'])
-
-Create a datacenter storage path:
-
+:mod:`waeup.ikoba.utils.batching` -- Batch processing
+****************************************************
+
+Batch processing is much more than pure data import.
+
+Overview
+========
+
+Basically, it means processing CSV files in order to mass-create,
+mass-remove, or mass-update data.
+
+So you can feed CSV files to processors, that are part of
+the batch-processing mechanism.
+
+Processors
+----------
+
+Each CSV file processor
+
+* accepts a single data type identified by an interface.
+
+* knows about the places inside a site (Company) where to store,
+  remove or update the data.
+
+* can check headers before processing data.
+
+* supports the mode 'create', 'update', 'remove'.
+
+* creates log entries (optional)
+
+* creates csv files containing successful and not-successful processed
+  data respectively.
+
+Output
+------
+
+The results of processing are written to loggers, if a logger was
+given. Beside this new CSV files are created during processing:
+
+* a pending CSV file, containing datasets that could not be processed
+
+* a finished CSV file, containing datasets successfully processed.
+
+The pending file is not created if everything works fine. The
+respective path returned in that case is ``None``.
+
+The pending file (if created) is a CSV file that contains the failed
+rows appended by a column ``--ERRROR--`` in which the reasons for
+processing failures are listed.
+
+The complete paths of these files are returned. They will be in a
+temporary directory created only for this purpose. It is the caller's
+responsibility to remove the temporay directories afterwards (the
+datacenters distProcessedFiles() method takes care for that).
+
+It looks like this::
+ 
+     -----+      +---------+
+    /     |      |         |              +------+
+   | .csv +----->|Batch-   |              |      |
+   |      |      |processor+----changes-->| ZODB |
+   |  +------+   |         |              |      |
+   +--|      |   |         +              +------+
+      | Mode +-->|         |                 -------+
+      |      |   |         +----outputs-+-> /       |
+      |  +----+->+---------+            |  |.pending|
+      +--|Log |  ^                      |  |        |
+         +----+  |                      |  +--------+
+           +-----++                     v
+           |Inter-|                  ----------+
+           |face  |                 /          |
+           +------+                | .finished |
+                                   |           |
+                                   +-----------+
+
+
+Creating a batch processor
+==========================
+
+We create an own batch processor for an own datatype. This datatype
+must be based on an interface that the batcher can use for converting
+data.
+
+Founding Stoneville
+-------------------
+
+We start with the interface:
+
+    >>> from zope.interface import Interface
+    >>> from zope import schema
+    >>> class ICave(Interface):
+    ...   """A cave."""
+    ...   name = schema.TextLine(
+    ...     title = u'Cave name',
+    ...     default = u'Unnamed',
+    ...     required = True)
+    ...   dinoports = schema.Int(
+    ...     title = u'Number of DinoPorts (tm)',
+    ...     required = False,
+    ...     default = 1)
+    ...   owner = schema.TextLine(
+    ...     title = u'Owner name',
+    ...     required = True,
+    ...     missing_value = 'Fred Estates Inc.')
+    ...   taxpayer = schema.Bool(
+    ...     title = u'Payes taxes',
+    ...     required = True,
+    ...     default = False)
+
+Now a class that implements this interface:
+
+    >>> import grok
+    >>> class Cave(object):
+    ...   grok.implements(ICave)
+    ...   def __init__(self, name=u'Unnamed', dinoports=2,
+    ...                owner='Fred Estates Inc.', taxpayer=False):
+    ...     self.name = name
+    ...     self.dinoports = 2
+    ...     self.owner = owner
+    ...     self.taxpayer = taxpayer
+
+We also provide a factory for caves. Strictly speaking, this not
+necessary but makes the batch processor we create afterwards, better
+understandable.
+
+    >>> from zope.component import getGlobalSiteManager
+    >>> from zope.component.factory import Factory
+    >>> from zope.component.interfaces import IFactory
+    >>> gsm = getGlobalSiteManager()
+    >>> cave_maker = Factory(Cave, 'A cave', 'Buy caves here!')
+    >>> gsm.registerUtility(cave_maker, IFactory, 'Lovely Cave')
+
+Now we can create caves using a factory:
+
+    >>> from zope.component import createObject
+    >>> createObject('Lovely Cave')
+    <Cave object at 0x...>
+
+This is nice, but we still lack a place, where we can place all the
+lovely caves we want to sell.
+
+Furthermore, as a replacement for a real site, we define a place where
+all caves can be stored: Stoneville! This is a lovely place for
+upperclass cavemen (which are the only ones that can afford more than
+one dinoport).
+
+We found Stoneville:
+
+    >>> stoneville = dict()
+
+Everything in place.
+
+Now, to improve local health conditions, imagine we want to populate
+Stoneville with lots of new happy dino-hunting natives that slept on
+the bare ground in former times and had no idea of
+bathrooms. Disgusting, isn't it?
+
+Lots of cavemen need lots of caves.
+
+Of course we can do something like:
+
+    >>> cave1 = createObject('Lovely Cave')
+    >>> cave1.name = "Fred's home"
+    >>> cave1.owner = "Fred"
+    >>> stoneville[cave1.name] = cave1
+
+and Stoneville has exactly
+
+    >>> len(stoneville)
+    1
+
+inhabitant. But we don't want to do this for hundreds or thousands of
+citizens-to-be, do we?
+
+It is much easier to create a simple CSV list, where we put in all the
+data and let a batch processor do the job.
+
+The list is already here:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner,taxpayer
+    ... Barneys Home,2,Barney,1
+    ... Wilmas Asylum,1,Wilma,1
+    ... Freds Dinoburgers,10,Fred,0
+    ... Joeys Drive-in,110,Joey,0
+    ... """)
+
+All we need, is a batch processor now.
+
+    >>> from waeup.ikoba.utils.batching import BatchProcessor
+    >>> from waeup.ikoba.interfaces import IGNORE_MARKER
+    >>> class CaveProcessor(BatchProcessor):
+    ...   util_name = 'caveprocessor'
+    ...   grok.name(util_name)
+    ...   name = 'Cave Processor'
+    ...   iface = ICave
+    ...   location_fields = ['name']
+    ...   factory_name = 'Lovely Cave'
+    ...
+    ...   def parentsExist(self, row, site):
+    ...     return True
+    ...
+    ...   def getParent(self, row, site):
+    ...     return stoneville
+    ...
+    ...   def entryExists(self, row, site):
+    ...     return row['name'] in stoneville.keys()
+    ...
+    ...   def getEntry(self, row, site):
+    ...     if not self.entryExists(row, site):
+    ...       return None
+    ...     return stoneville[row['name']]
+    ...
+    ...   def delEntry(self, row, site):
+    ...     del stoneville[row['name']]
+    ...
+    ...   def addEntry(self, obj, row, site):
+    ...     stoneville[row['name']] = obj
+    ...
+    ...   def updateEntry(self, obj, row, site, filename):
+    ...     # This is not strictly necessary, as the default
+    ...     # updateEntry method does exactly the same
+    ...     for key, value in row.items():
+    ...       if value != IGNORE_MARKER:
+    ...         setattr(obj, key, value)
+
+If we also want the results being logged, we must provide a logger
+(this is optional):
+
+    >>> import logging
+    >>> logger = logging.getLogger('stoneville')
+    >>> logger.setLevel(logging.DEBUG)
+    >>> logger.propagate = False
+    >>> handler = logging.FileHandler('stoneville.log', 'w')
+    >>> logger.addHandler(handler)
+
+Create the fellows:
+
+    >>> processor = CaveProcessor()
+    >>> result = processor.doImport('newcomers.csv',
+    ...                   ['name', 'dinoports', 'owner', 'taxpayer'],
+    ...                    mode='create', user='Bob', logger=logger)
+    >>> result
+    (4, 0, '/.../newcomers.finished.csv', None)
+
+The result means: four entries were processed and no warnings
+occured. Furthermore we get filepath to a CSV file with successfully
+processed entries and a filepath to a CSV file with erraneous entries.
+As everything went well, the latter is ``None``. Let's check:
+
+    >>> sorted(stoneville.keys())
+    [u'Barneys Home', ..., u'Wilmas Asylum']
+
+The values of the Cave instances have correct type:
+
+    >>> barney = stoneville['Barneys Home']
+    >>> barney.dinoports
+    2
+
+which is a number, not a string.
+
+Apparently, when calling the processor, we gave some more info than
+only the CSV filepath. What does it all mean?
+
+While the first argument is the path to the CSV file, we also have to
+give an ordered list of headernames. These replace the header field
+names that are actually in the file. This way we can override faulty
+headers.
+
+The ``mode`` paramter tells what kind of operation we want to perform:
+``create``, ``update``, or ``remove`` data.
+
+The ``user`` parameter finally is optional and only used for logging.
+
+We can, by the way, see the results of our run in a logfile if we
+provided a logger during the call:
+
+    >>> print open('stoneville.log').read()
+    processed: newcomers.csv, create mode, 4 lines (4 successful/ 0 failed), ... s (... s/item)
+
+
+We cleanup the temporay dir created by doImport():
+
+    >>> import shutil
     >>> import os
-    >>> import tempfile
-    >>> dc_path = tempfile.mkdtemp()
-
-Log in:
-
-    >>> from zope.testbrowser.testing import Browser
-    >>> browser = Browser()
-    >>> browser.addHeader('Authorization', 'Basic mgr:mgrpw')
-    >>> browser.handleErrors = False
-
-Set datacenter path and deselect moving old data:
-
-    >>> browser.open('http://localhost/app')
-    >>> browser.getLink('Data Center').click()
-    >>> browser.getLink('Edit settings').click()
-    >>> browser.getControl(name='newpath').value = dc_path
-    >>> browser.getControl(name='move').value = False
-    >>> browser.getControl(name='save').click()
-
-Set non-usable datacenter path:
-
-    >>> browser.getLink('Edit settings').click()
-    >>> browser.getControl(name='newpath').value = '/'
-    >>> browser.getControl(name='save').click()
-    >>> 'Given storage path cannot be used' in browser.contents
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+As we can see, the processing was successful. Otherwise, all problems
+could be read here as we can see, if we do the same operation again:
+
+    >>> result = processor.doImport('newcomers.csv',
+    ...                   ['name', 'dinoports', 'owner', 'taxpayer'],
+    ...                    mode='create', user='Bob', logger=logger)
+    >>> result
+    (4, 4, '/.../newcomers.finished.csv', '/.../newcomers.pending.csv')
+
+This time we also get a path to a .pending file.
+
+The log file will tell us this in more detail:
+
+    >>> print open('stoneville.log').read()
+    processed: newcomers.csv, create mode, 4 lines (4 successful/ 0 failed), ... s (... s/item)
+    processed: newcomers.csv, create mode, 4 lines (0 successful/ 4 failed), ... s (... s/item)
+
+
+This time a new file was created, which keeps all the rows we could not
+process and an additional column with error messages:
+
+    >>> print open(result[3]).read()
+    owner,name,taxpayer,dinoports,--ERRORS--
+    Barney,Barneys Home,1,2,This object already exists. Skipping.
+    Wilma,Wilmas Asylum,1,1,This object already exists. Skipping.
+    Fred,Freds Dinoburgers,0,10,This object already exists. Skipping.
+    Joey,Joeys Drive-in,0,110,This object already exists. Skipping.
+
+This way we can correct the faulty entries and afterwards retry without
+having the already processed rows in the way.
+
+We also notice, that the values of the taxpayer column are returned as
+in the input file. There we wrote '1' for ``True`` and '0' for
+``False`` (which is accepted by the converters).
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+We can also tell to ignore some cols from input by passing
+``--IGNORE--`` as col name:
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             '--IGNORE--', '--IGNORE--'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (4, 0, '...', None)
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+If something goes wrong during processing, the respective --IGNORE--
+cols won't be populated  in the resulting pending file:
+
+    >>> result = processor.doImport('newcomers.csv', ['name', 'dinoports',
+    ...                             '--IGNORE--', '--IGNORE--'],
+    ...                    mode='create', user='Bob')
+    >>> result
+    (4, 4, '...', '...')
+
+    >>> print open(result[3], 'rb').read()
+    name,dinoports,--ERRORS--
+    Barneys Home,2,This object already exists. Skipping.
+    Wilmas Asylum,1,This object already exists. Skipping.
+    Freds Dinoburgers,10,This object already exists. Skipping.
+    Joeys Drive-in,110,This object already exists. Skipping.
+
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+
+
+Updating entries
+----------------
+
+To update entries, we just call the batchprocessor in a different
+mode:
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (4, 0, '...', None)
+
+Now we want to tell, that Wilma got an extra port for her second dino:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... Wilmas Asylum,2,Wilma
+    ... """)
+
+    >>> wilma = stoneville['Wilmas Asylum']
+    >>> wilma.dinoports
+    1
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+We start the processor:
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                    'dinoports', 'owner'], mode='update', user='Bob')
+    >>> result
+    (1, 0, '...', None)
+
+    >>> wilma = stoneville['Wilmas Asylum']
+    >>> wilma.dinoports
+    2
+
+Wilma's number of dinoports raised.
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+If we try to update an unexisting entry, an error occurs:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... NOT-WILMAS-ASYLUM,2,Wilma
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (1, 1, '/.../newcomers.finished.csv', '/.../newcomers.pending.csv')
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+    
+Also invalid values will be spotted:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... Wilmas Asylum,NOT-A-NUMBER,Wilma
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (1, 1, '...', '...')
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+We can also update only some cols, leaving some out. We skip the
+'dinoports' column in the next run:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,owner
+    ... Wilmas Asylum,Barney
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name', 'owner'],
+    ...                             mode='update', user='Bob')
+    >>> result
+    (1, 0, '...', None)
+
+    >>> wilma.owner
+    u'Barney'
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+We can however, not leave out the 'location field' ('name' in our
+case), as this one tells us which entry to update:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... 2,Wilma
+    ... """)
+
+    >>> processor.doImport('newcomers.csv', ['dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    Traceback (most recent call last):
+    ...
+    FatalCSVError: Need at least columns 'name' for import!
+
+This time we get even an exception!
+
+Generally, empty strings are considered as ``None``:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... "Wilmas Asylum","","Wilma"
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (1, 0, '...', None)
+
+    >>> wilma.dinoports
+    2
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+We can tell to set dinoports to ``None`` although this is not a
+number, as we declared the field not required in the interface:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... "Wilmas Asylum","XXX","Wilma"
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob', ignore_empty=False)
+    >>> result
+    (1, 0, '...', None)
+
+    >>> wilma.dinoports is None
     True
-    >>> browser.getControl('Back to Data Center').click()
-
-
-Batch processing users
-======================
-
-    >>> browser.open('http://localhost/app/datacenter')
-
-Prepare a CSV file for users:
-
-    >>> open('users.csv', 'wb').write(
-    ... """name,title,public_name,email,phone,roles
-    ... uli,Uli Fouquet,Chief Developer,uli@abc.de,+49-234-567,[]
-    ... henrik,Henrik Bettermann,Admin,henrik@abc.de,+49-234-567,"['waeup.PortalManager', 'waeup.ImportManager']"
-    ... """)
-
-Upload the file:
-
-    >>> import cStringIO
-    >>> browser.getLink('Upload data').click()
-    >>> filecontents = cStringIO.StringIO(
-    ...   open('users.csv', 'rb').read())
-    >>> filewidget = browser.getControl(name='uploadfile:file')
-    >>> filewidget.add_file(filecontents, 'text/plain', 'users.csv')
-    >>> browser.getControl(name='SUBMIT').click()
-
-Step 1: start batch processing:
-
-    >>> browser.getLink('Process data').click()
-    >>> button = lookup_submit_value(
-    ...   'select', 'users_zope.mgr.csv', browser)
-    >>> button.click()
-
-Step 2: select a processor and mode:
-
-    >>> importerselect = browser.getControl(name='importer')
-    >>> importerselect.getControl('User Processor').selected = True
-    >>> modeselect = browser.getControl(name='mode')
-    >>> modeselect.getControl(value='create').selected = True
-    >>> browser.getControl('Proceed to step 3').click()
-
-Step 3: Fix headerlines
-
-We get informed that there are no problems with the current header:
-
-    >>> print browser.contents
-    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
-    ...
-    Header fields OK
-    ...
-
-The submit button is enabled:
-
-    >>> browser.getControl('Perform import').disabled
-    False
-
-    >>> browser.getControl('Perform import').click()
-
-Step 4: See import results
-
-The import was successful:
-
-    >>> print browser.contents
-    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
-    ...Successfully processed 2 rows...
-    ...Batch processing finished...
-    ...File:...users_zope.mgr.csv...
-
-User henrik has got the global roles:
-
-   >>> henrik = getRootFolder()['app']['users']['henrik']
-   >>> henrik.roles
-   ['waeup.ProductsOfficer', 'waeup.PortalManager', 'waeup.ImportManager']
-
-Pending files
-=============
-
-When an error occurs during an import, two files are generated: a CSV
-file with finished files and a CSV file with pending data. Both are
-stored in the appropriate subdirectories in datacenter. We try to
-create faculties, from which one already exists.
-
-Go to datacenter page:
-
-    >>> browser.open('http://localhost/app/datacenter')
-
-Prepare a CSV file for users:
-
-    >>> open('newusers.csv', 'wb').write(
-    ... """name,title,public_name,email,phone,roles
-    ... isouaba,Issoufou Abba Moussa,Chief Developer,isouaba@abc.de,+49-234-567,[]
-    ... henrik,Henrik Bettermann,Admin,henrik@abc.de,+49-234-567,"['waeup.PortalManager', 'waeup.ImportManager']"
-    ... anne,Anne Palina,,anne@abc.de,+49-234-567,"['waeup.Nonsense']"
-    ... """)
-
-Upload the file:
-
-    >>> import cStringIO
-    >>> browser.getLink('Upload data').click()
-    >>> filecontents = cStringIO.StringIO(
-    ...   open('newusers.csv', 'rb').read())
-    >>> filewidget = browser.getControl(name='uploadfile:file')
-    >>> filewidget.add_file(filecontents, 'text/plain', 'newusers.csv')
-    >>> browser.getControl(name='SUBMIT').click()
-
-Since we now have a user with waeup.ImportManager role, an email has been sent:
-    >>> print browser.contents
-    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
-    ...
-    ...All import managers have been notified by email...
-    ...
-
-Step 1: start batch processing:
-
-    >>> browser.getLink('Process data').click()
-    >>> button = lookup_submit_value(
-    ...   'select', 'newusers_zope.mgr.csv', browser)
-    >>> button.click()
-
-Step 2: select a processor and mode:
-
-    >>> importerselect = browser.getControl(name='importer')
-    >>> importerselect.getControl('User Processor').selected = True
-    >>> modeselect = browser.getControl(name='mode')
-    >>> modeselect.getControl(value='create').selected = True
-    >>> browser.getControl('Proceed to step 3').click()
-
-Step 3: Fix headerlines
-
-As there should be no problem with the headers, we can immediately
-perfom the import:
-
-    >>> browser.getControl('Perform import').click()
-
-Two lines could not be imported:
-
-    >>> print browser.contents
-    <!DOCTYPE html PUBLIC...
-    ...
-    ...Processing of 2 rows failed...
-    ...Successfully processed 1 rows...
-    ...
-
-Now there are two files as a result in datacenter storage's root and
-``finished`` dirs:
-
-    >>> pending_file = dc_path + '/newusers_zope.mgr.create.pending.csv'
-    >>> print open(pending_file).read()
-    name,roles,title,public_name,phone,email,--ERRORS--
-    henrik,"['waeup.PortalManager', 'waeup.ImportManager']",Henrik Bettermann,Admin,+49-234-567,henrik@abc.de,This object already exists. Skipping.
-    anne,['waeup.Nonsense'],Anne Palina,<IGNORE>,+49-234-567,anne@abc.de,roles: invalid role
-
-    >>> finished_file = dc_path + '/finished/newusers_zope.mgr.create.finished.csv'
-    >>> print open(finished_file).read()
-    name,roles,title,public_name,phone,email
-    isouaba,<IGNORE>,Issoufou Abba Moussa,Chief Developer,+49-234-567,isouaba@abc.de
-
-The finished-file contains the dataset we could import, while the
-pending file contains the dataset that failed, appended by an error
-message.
-
-
-Fixing the pending file
------------------------
-
-We 'edit' the pending file by removing anne and replacing henrik by gbenga 
-and finish the import this way:
-
-    >>> open(dc_path + '/newusers_zope.mgr.create.pending.csv', 'wb').write(
-    ... """name,title,public_name,email,phone,roles
-    ... gbenga,Jason Bamidele,Chief Developer,gbenga@abc.de,+49-234-567,[]
-    ... """)
-
-Step 1: start batch processing:
-
-    >>> browser.open('http://localhost/app/datacenter')
-    >>> browser.getLink('Process data').click()
-    >>> button = lookup_submit_value(
-    ...   'select', 'newusers_zope.mgr.create.pending.csv', browser)
-    >>> button.click()
-
-Step 2: select a processor and mode:
-
-    >>> importerselect = browser.getControl(name='importer')
-    >>> importerselect.getControl('User Processor').selected = True
-    >>> modeselect = browser.getControl(name='mode')
-    >>> modeselect.getControl(value='create').selected = True
-    >>> browser.getControl('Proceed to step 3').click()
-
-Step 3/4: Fix headerlines and import:
-
-As there should be no problem with the headers, we can immediately
-perfom the import:
-
-    >>> browser.getControl('Perform import').click()
-
-This time everything should work:
-
-    >>> print browser.contents
-    <!DOCTYPE html PUBLIC...
-    ...
-    ...Successfully processed 1 rows...
-    ...
-
-    >>> sorted(os.listdir(dc_path))
-    ['deleted', 'finished', 'logs', 'unfinished']
-
-    >>> os.listdir(dc_path + '/unfinished')
-    []
-
-    >>> sorted(os.listdir(dc_path + '/finished'))
-    ['newusers_zope.mgr.create.finished.csv', 'newusers_zope.mgr.csv', 'users_zope.mgr.create.finished.csv', 'users_zope.mgr.csv']
-
-Processed (finished) Files
-==========================
-
-    >>> browser.open('http://localhost/app/datacenter/processed')
-    >>> 'download?filename=finished/users_zope.mgr.create.finished.csv' in browser.contents
-    True
-
-Log Files
-=========
-
-    >>> browser.open('http://localhost/app/datacenter/logs')
-    >>> 'datacenter.log' in browser.contents
-    True
-    >>> browser.getControl('Show', index=0).click()
-    >>> browser.getControl('Back', index=0).click()
-    >>> browser.getControl('Back to Data Center').click()
-    >>> 'Storage path:' in browser.contents
-    True
-
-
-Clean up:
-
-    >>> import shutil
-    >>> shutil.rmtree(dc_path)
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+Removing entries
+----------------
+
+In 'remove' mode we can delete entries. Here validity of values in
+non-location fields doesn't matter because those fields are ignored.
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... "Wilmas Asylum","ILLEGAL-NUMBER",""
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='remove', user='Bob')
+    >>> result
+    (1, 0, '...', None)
+
+    >>> sorted(stoneville.keys())
+    [u'Barneys Home', "Fred's home", u'Freds Dinoburgers', u'Joeys Drive-in']
+
+Oops! Wilma is gone.
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+Clean up:
+
+    >>> import os
+    >>> os.unlink('newcomers.csv')
+    >>> os.unlink('stoneville.log')
Index: /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batchprocessing_browser.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batchprocessing_browser.txt	(revision 12991)
+++ /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batchprocessing_browser.txt	(revision 12991)
@@ -0,0 +1,293 @@
+Batch processing via browser
+****************************
+
+Preliminaries:
+
+We define a function that looks up a form with several submit buttons
+for the one with a given value (this functionality is missing in
+zope.testbrowser):
+
+    >>> def lookup_submit_value(name, value, browser):
+    ...   """Find a button with a certain value."""
+    ...   for num in range(0, 100):
+    ...     try:
+    ...       button = browser.getControl(name=name, index=num)
+    ...       if button.value.endswith(value):
+    ...         return button
+    ...     except IndexError:
+    ...       break
+    ...   return None
+
+Create a site:
+
+    >>> from waeup.ikoba.app import Company
+    >>> getRootFolder()['app'] = Company()
+    >>> from zope.component.hooks import setSite
+    >>> setSite(getRootFolder()['app'])
+
+Create a datacenter storage path:
+
+    >>> import os
+    >>> import tempfile
+    >>> dc_path = tempfile.mkdtemp()
+
+Log in:
+
+    >>> from zope.testbrowser.testing import Browser
+    >>> browser = Browser()
+    >>> browser.addHeader('Authorization', 'Basic mgr:mgrpw')
+    >>> browser.handleErrors = False
+
+Set datacenter path and deselect moving old data:
+
+    >>> browser.open('http://localhost/app')
+    >>> browser.getLink('Data Center').click()
+    >>> browser.getLink('Edit settings').click()
+    >>> browser.getControl(name='newpath').value = dc_path
+    >>> browser.getControl(name='move').value = False
+    >>> browser.getControl(name='save').click()
+
+Set non-usable datacenter path:
+
+    >>> browser.getLink('Edit settings').click()
+    >>> browser.getControl(name='newpath').value = '/'
+    >>> browser.getControl(name='save').click()
+    >>> 'Given storage path cannot be used' in browser.contents
+    True
+    >>> browser.getControl('Back to Data Center').click()
+
+
+Batch processing users
+======================
+
+    >>> browser.open('http://localhost/app/datacenter')
+
+Prepare a CSV file for users:
+
+    >>> open('users.csv', 'wb').write(
+    ... """name,title,public_name,email,phone,roles
+    ... uli,Uli Fouquet,Chief Developer,uli@abc.de,+49-234-567,[]
+    ... henrik,Henrik Bettermann,Admin,henrik@abc.de,+49-234-567,"['waeup.PortalManager', 'waeup.ImportManager']"
+    ... """)
+
+Upload the file:
+
+    >>> import cStringIO
+    >>> browser.getLink('Upload data').click()
+    >>> filecontents = cStringIO.StringIO(
+    ...   open('users.csv', 'rb').read())
+    >>> filewidget = browser.getControl(name='uploadfile:file')
+    >>> filewidget.add_file(filecontents, 'text/plain', 'users.csv')
+    >>> browser.getControl(name='SUBMIT').click()
+
+Step 1: start batch processing:
+
+    >>> browser.getLink('Process data').click()
+    >>> button = lookup_submit_value(
+    ...   'select', 'users_zope.mgr.csv', browser)
+    >>> button.click()
+
+Step 2: select a processor and mode:
+
+    >>> importerselect = browser.getControl(name='importer')
+    >>> importerselect.getControl('User Processor').selected = True
+    >>> modeselect = browser.getControl(name='mode')
+    >>> modeselect.getControl(value='create').selected = True
+    >>> browser.getControl('Proceed to step 3').click()
+
+Step 3: Fix headerlines
+
+We get informed that there are no problems with the current header:
+
+    >>> print browser.contents
+    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+    ...
+    Header fields OK
+    ...
+
+The submit button is enabled:
+
+    >>> browser.getControl('Perform import').disabled
+    False
+
+    >>> browser.getControl('Perform import').click()
+
+Step 4: See import results
+
+The import was successful:
+
+    >>> print browser.contents
+    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+    ...Successfully processed 2 rows...
+    ...Batch processing finished...
+    ...File:...users_zope.mgr.csv...
+
+User henrik has got the global roles:
+
+   >>> henrik = getRootFolder()['app']['users']['henrik']
+   >>> henrik.roles
+   ['waeup.ProductsOfficer', 'waeup.PortalManager', 'waeup.ImportManager']
+
+Pending files
+=============
+
+When an error occurs during an import, two files are generated: a CSV
+file with finished files and a CSV file with pending data. Both are
+stored in the appropriate subdirectories in datacenter. We try to
+create faculties, from which one already exists.
+
+Go to datacenter page:
+
+    >>> browser.open('http://localhost/app/datacenter')
+
+Prepare a CSV file for users:
+
+    >>> open('newusers.csv', 'wb').write(
+    ... """name,title,public_name,email,phone,roles
+    ... isouaba,Issoufou Abba Moussa,Chief Developer,isouaba@abc.de,+49-234-567,[]
+    ... henrik,Henrik Bettermann,Admin,henrik@abc.de,+49-234-567,"['waeup.PortalManager', 'waeup.ImportManager']"
+    ... anne,Anne Palina,,anne@abc.de,+49-234-567,"['waeup.Nonsense']"
+    ... """)
+
+Upload the file:
+
+    >>> import cStringIO
+    >>> browser.getLink('Upload data').click()
+    >>> filecontents = cStringIO.StringIO(
+    ...   open('newusers.csv', 'rb').read())
+    >>> filewidget = browser.getControl(name='uploadfile:file')
+    >>> filewidget.add_file(filecontents, 'text/plain', 'newusers.csv')
+    >>> browser.getControl(name='SUBMIT').click()
+
+Since we now have a user with waeup.ImportManager role, an email has been sent:
+    >>> print browser.contents
+    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+    ...
+    ...All import managers have been notified by email...
+    ...
+
+Step 1: start batch processing:
+
+    >>> browser.getLink('Process data').click()
+    >>> button = lookup_submit_value(
+    ...   'select', 'newusers_zope.mgr.csv', browser)
+    >>> button.click()
+
+Step 2: select a processor and mode:
+
+    >>> importerselect = browser.getControl(name='importer')
+    >>> importerselect.getControl('User Processor').selected = True
+    >>> modeselect = browser.getControl(name='mode')
+    >>> modeselect.getControl(value='create').selected = True
+    >>> browser.getControl('Proceed to step 3').click()
+
+Step 3: Fix headerlines
+
+As there should be no problem with the headers, we can immediately
+perfom the import:
+
+    >>> browser.getControl('Perform import').click()
+
+Two lines could not be imported:
+
+    >>> print browser.contents
+    <!DOCTYPE html PUBLIC...
+    ...
+    ...Processing of 2 rows failed...
+    ...Successfully processed 1 rows...
+    ...
+
+Now there are two files as a result in datacenter storage's root and
+``finished`` dirs:
+
+    >>> pending_file = dc_path + '/newusers_zope.mgr.create.pending.csv'
+    >>> print open(pending_file).read()
+    name,roles,title,public_name,phone,email,--ERRORS--
+    henrik,"['waeup.PortalManager', 'waeup.ImportManager']",Henrik Bettermann,Admin,+49-234-567,henrik@abc.de,This object already exists. Skipping.
+    anne,['waeup.Nonsense'],Anne Palina,<IGNORE>,+49-234-567,anne@abc.de,roles: invalid role
+
+    >>> finished_file = dc_path + '/finished/newusers_zope.mgr.create.finished.csv'
+    >>> print open(finished_file).read()
+    name,roles,title,public_name,phone,email
+    isouaba,<IGNORE>,Issoufou Abba Moussa,Chief Developer,+49-234-567,isouaba@abc.de
+
+The finished-file contains the dataset we could import, while the
+pending file contains the dataset that failed, appended by an error
+message.
+
+
+Fixing the pending file
+-----------------------
+
+We 'edit' the pending file by removing anne and replacing henrik by gbenga 
+and finish the import this way:
+
+    >>> open(dc_path + '/newusers_zope.mgr.create.pending.csv', 'wb').write(
+    ... """name,title,public_name,email,phone,roles
+    ... gbenga,Jason Bamidele,Chief Developer,gbenga@abc.de,+49-234-567,[]
+    ... """)
+
+Step 1: start batch processing:
+
+    >>> browser.open('http://localhost/app/datacenter')
+    >>> browser.getLink('Process data').click()
+    >>> button = lookup_submit_value(
+    ...   'select', 'newusers_zope.mgr.create.pending.csv', browser)
+    >>> button.click()
+
+Step 2: select a processor and mode:
+
+    >>> importerselect = browser.getControl(name='importer')
+    >>> importerselect.getControl('User Processor').selected = True
+    >>> modeselect = browser.getControl(name='mode')
+    >>> modeselect.getControl(value='create').selected = True
+    >>> browser.getControl('Proceed to step 3').click()
+
+Step 3/4: Fix headerlines and import:
+
+As there should be no problem with the headers, we can immediately
+perfom the import:
+
+    >>> browser.getControl('Perform import').click()
+
+This time everything should work:
+
+    >>> print browser.contents
+    <!DOCTYPE html PUBLIC...
+    ...
+    ...Successfully processed 1 rows...
+    ...
+
+    >>> sorted(os.listdir(dc_path))
+    ['deleted', 'finished', 'logs', 'unfinished']
+
+    >>> os.listdir(dc_path + '/unfinished')
+    []
+
+    >>> sorted(os.listdir(dc_path + '/finished'))
+    ['newusers_zope.mgr.create.finished.csv', 'newusers_zope.mgr.csv', 'users_zope.mgr.create.finished.csv', 'users_zope.mgr.csv']
+
+Processed (finished) Files
+==========================
+
+    >>> browser.open('http://localhost/app/datacenter/processed')
+    >>> 'download?filename=finished/users_zope.mgr.create.finished.csv' in browser.contents
+    True
+
+Log Files
+=========
+
+    >>> browser.open('http://localhost/app/datacenter/logs')
+    >>> 'datacenter.log' in browser.contents
+    True
+    >>> browser.getControl('Show', index=0).click()
+    >>> browser.getControl('Back', index=0).click()
+    >>> browser.getControl('Back to Data Center').click()
+    >>> 'Storage path:' in browser.contents
+    True
+
+
+Clean up:
+
+    >>> import shutil
+    >>> shutil.rmtree(dc_path)
Index: in/waeup.ikoba/trunk/src/waeup/ikoba/doctests/browser.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/browser.txt	(revision 12990)
+++ 	(revision )
@@ -1,368 +1,0 @@
-Browsing Ikoba
-*************
-
-Here we visit all parts of a Ikoba portal using a browser.
-
-Company
-==========
-
-We can watch universities in the browser.
-
-We create an company object and put into the ZODB root::
-
-  >>> root = getRootFolder()
-  >>> list(root)
-  []
-
-  >>> from waeup.ikoba.app import Company
-  >>> u = Company()
-  >>> root['mycompany'] = u
-  >>> list(root)
-  [u'mycompany']
-
-  >>> from zope.component.hooks import setSite
-  >>> setSite(root['mycompany'])
-
-To make sure, we can 'watch' pages, we first have to initialize out
-test browser::
-
-  >>> from zope.testbrowser.testing import Browser
-  >>> browser = Browser()
-
-Let's get the default view of a company::
-
-  >>> browser.open('http://localhost/mycompany')
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Welcome to WAeUP.Ikoba...
-  ...
-
-We can change to German::
-  >>> browser.getLink('de').click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Anfragen...
-  ...
-
-And then change back to English by clicking on the third link containing 'en'
-behind 'Anfragen' and 'Einloggen'::
-  >>> browser.getLink('en', index=2).click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Enquiries...
-  ...
-
-The contact form for anonymous users is called 'Enquiries'::
-
-  >>> browser.open('http://localhost/mycompany')
-  >>> browser.getLink('Enquiries').click()
-  >>> browser.getControl('Send').click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Required input is missing...
-  ...
-
-  >>> browser.getControl(name='form.fullname').value = "Bob Tester"
-  >>> browser.getControl(name='form.email_from').value = "xx@yy.zz"
-  >>> browser.getControl(name='form.body').value = u'test message'
-  >>> browser.getControl('Send').click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Your message has been sent...
-  ...
-
-Registered users with an email address can request a password change:
-
-  >>> root['mycompany']['users'].addUser('forgetful', 'secret',title='Bob Forgetful',
-  ...                               description='A forgetful user', email='aa@aa.ng')
-  >>> browser.open('http://localhost/mycompany/changepw')
-  >>> browser.getControl(name="form.email").value = 'aa@aa.ng'
-  >>> browser.getControl("Send login credentials").click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...An email with your user name and password has been sent to aa@aa.ng...
-
-Now we login as manager::
-
-  >>> browser.addHeader('Authorization', 'Basic mgr:mgrpw')
-  >>> browser.handleErrors = False
-
-We can then get an edit view of the configuration container::
-
-  >>> browser.open('http://localhost/mycompany/configuration')
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...<form action="http://localhost/mycompany/configuration/@@index"
-  ...
-
-The edit form contains the default value for the company name::
-
-  >>> 'Sample Company' in browser.contents
-  True
-
-We can perform several actions on the edit form::
-
-  >>> browser.getControl("Save", index=0).click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Form has been saved...
-  ...
-
-  >>> browser.open('http://localhost/mycompany/configuration')
-  >>> browser.getControl("Update plugins").click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Plugins were updated. See log file for details...
-  ...
-
-The default frontpage HTML has been saved in a dictionary
-and is properly rendered on the frontpage of the portal:
-
-  >>> browser.open('http://localhost/mycompany')
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...<h1>Welcome to WAeUP.Ikoba...</h1>
-  ...
-
-The German part is really not being rendered:
-
-  >>> 'Willkommen' in browser.contents
-  False
-
-
-If we change to German so that the German part of frontpage.rst is rendered:
-
-  >>> browser.open('http://localhost/mycompany//@@change_language?lang=de')
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...<h1>Willkommen auf WAeUP.Ikoba...</h1>
-  ...
-
-The English part is really not being rendered:
-
-  >>> 'Welcome' in browser.contents
-  False
-
-Switch back to English:
-
-  >>> browser.open('http://localhost/mycompany//@@change_language?lang=en')
-
-
-Officers
-========
-
-  >>> browser.open('http://localhost/mycompany')
-  >>> browser.getLink('Officers').click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Officers...
-  ...
-
-We can add users:
-
-  >>> browser.getLink("Add user").click()
-  >>> browser.getControl(name="form.name").value = 'bob'
-  >>> browser.getControl(name="form.title").value = 'Bob The User'
-  >>> browser.getControl(name="password").value = 'secret'
-  >>> browser.getControl(name="control_password").value = 'secret'
-  >>> browser.getControl(name="form.email").value = 'xx@yy.zz'
-  >>> browser.getControl(name="form.phone.country").value = ['+234']
-  >>> browser.getControl(name="form.phone.area").value = '123'
-  >>> browser.getControl(name="form.phone.ext").value = '45678'
-  >>> browser.getControl("Add user").click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...<td>bob</td>
-  ...
-
-We can edit user bob:
-
-  >>> browser.getControl("Manage", index=0).click()
-  >>> browser.getControl("Save", index=0).click()
-  >>> browser.getControl("Cancel", index=0).click()
-
-We can add site roles which are then displayed on the user container page.
-Since the test browser does not use javascript, we have to add site roles
-manually by setting the roles attribute:
-
-Since JS is not supported by the test browser, the default role has been removed
-when clicking the Save button.
-
-  >>> print root['mycompany']['users']['bob'].roles
-  []
-  >>> root['mycompany']['users']['bob'].roles = ['waeup.UsersManager']
-  >>> print root['mycompany']['users']['bob'].roles
-  ['waeup.UsersManager']
-  >>> browser.open('http://localhost/mycompany/users')
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...<td nowrap>Users Manager <br /></td>
-  ...
-
-Users can't be added twice:
-
-  >>> browser.open('http://localhost/mycompany/users/add')
-  >>> browser.getControl(name="form.name").value = 'bob'
-  >>> browser.getControl(name="form.title").value = 'Bob The User'
-  >>> browser.getControl(name="password").value = 'secret'
-  >>> browser.getControl(name="control_password").value = 'secret'
-  >>> browser.getControl(name="form.email").value = 'xx@yy.zz'
-  >>> browser.getControl(name="form.phone.country").value = ['+234']
-  >>> browser.getControl(name="form.phone.area").value = '123'
-  >>> browser.getControl(name="form.phone.ext").value = '45678'
-  >>> browser.getControl("Add user").click()
-  >>> 'The userid chosen already exists' in browser.contents
-  True
-
-Users can be deleted:
-
-  >>> browser.open('http://localhost/mycompany/users')
-  >>> browser.getControl("Remove", index=0).click()
-  >>> 'User account bob successfully deleted' in browser.contents
-  True
-
-
-
-Contact Form
-============
-
-Let's enter the contact form::
-
-  >>> browser.open('http://localhost/mycompany/contactadmin')
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Contact
-  ...
-
-We fill the form  (this will send a real message to
-contact@waeup.org)::
-
-  >>> browser.open('http://localhost/mycompany/contactadmin')
-  >>> browser.getControl(name='form.body').value = "test message"
-  >>> browser.getControl('Send').click()
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
-  ...Your message has been sent...
-  ...
-
-If this test fails, chances are, that the local machine has no SMTP
-server installed.
-
-
-Data Center
-===========
-
-The data center helps us uploading files for later import or similar.
-
-  >>> browser.open('http://localhost/mycompany')
-  >>> browser.getLink('Data Center').click()
-
-Setting the file path
----------------------
-
-A datacenter stores files in a path in filesystem. By default this is
-a directory in the sources:
-
-  >>> print browser.contents
-  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
-  ...
-  <b>Storage path:</b> <span>.../parts/test/datacenter</span>
-  ...
-
-Going to 'Settings` we can change the path:
-
-  >>> browser.getLink('Edit settings').click()
-  >>> pathsetting = browser.getControl(name='newpath')
-
-We create a directory and set it as our upload path:
-
-  >>> import os
-  >>> cwd = os.getcwd()
-  >>> uploadpath = os.path.join(cwd, 'testfiles')
-  >>> os.mkdir(uploadpath)
-  >>> pathsetting.value = uploadpath
-
-And submit the form:
-
-  >>> browser.getControl(name='save').click()
-
-We clean up the set directory path, as there might be some files be
-copied some files from installation:
-
-  >>> files = os.listdir(uploadpath)
-  >>> for filename in files:
-  ...   if os.path.isdir(os.path.join(uploadpath, filename)):
-  ...     continue
-  ...   os.unlink(os.path.join(uploadpath, filename))
-
-We also remove any existing 'accesscodes' subdir:
-
-  >>> import shutil
-  >>> for filename in files:
-  ...   if not os.path.isdir(os.path.join(uploadpath, filename)):
-  ...     continue
-  ...   if filename != 'accesscodes':
-  ...     continue
-  ...   shutil.rmtree(os.path.join(uploadpath, filename))
-
-The new upload directory is now empty, except from the logs and other
-standard dirs, which are created automatically:
-
-  >>> sorted(os.listdir(uploadpath))
-  ['deleted', 'finished', 'logs', 'unfinished']
-
-
-Uploading files
----------------
-
-Now we can upload files. Most interesting files might be CSV files,
-that can be imported lateron. We create a CSV file containing faculty
-descriptions:
-
-  >>> open('faculties.csv', 'wb').write(
-  ... """code,title,title_prefix
-  ... FA,Arts,faculty
-  ... FS,Sciences,faculty
-  ... """)
-
-Now we can upload this file. To do this, we first go to the upload
-page:
-
-  >>> browser.getLink('Upload data').click()
-
-and enter the appropriate data in the form:
-
-  >>> filewidget = browser.getControl(name='uploadfile:file')
-  >>> filewidget
-  <Control name='uploadfile:file' type='file'>
-
-A sidenote for developers: by marking the filewidget with the
-``:file`` extension, we tell Zope to handle this field as a file
-widget.
-
-  >>> import cStringIO
-  >>> filecontents = cStringIO.StringIO(
-  ...   open('faculties.csv', 'rb').read())
-  >>> filewidget.add_file(filecontents, 'text/plain', 'myfaculties.csv')
-
-  >>> browser.getControl(name='SUBMIT').click()
-
-The file was indeed uploaded, with the current userid inserted:
-
-  >>> sorted(os.listdir(uploadpath))
-  ['deleted', 'finished', 'logs', 'myfaculties_zope.mgr.csv', 'unfinished']
-
-
-
-Importing a CSV file
---------------------
-
-The import of CSV files is described in batchprocessing.txt.
-
-
-Clean up:
-
-  >>> import os
-  >>> import shutil
-  >>> shutil.rmtree(uploadpath)
-
Index: in/waeup.ikoba/trunk/src/waeup/ikoba/doctests/converters.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/converters.txt	(revision 12990)
+++ 	(revision )
@@ -1,861 +1,0 @@
-:mod:`waeup.ikoba.utils.converters` -- Converters
-************************************************
-
-.. module:: waeup.ikoba.utils.converters
-
-Converters for :mod:`zope.schema` based data.
-
-.. contents::
-
-
-.. :NOdoctest:
-.. :NOlayer: waeup.ikoba.testing.IkobaUnitTestLayer
-
-
-Constants
-=========
-
-.. attribute:: NONE_STRING_VALUE
-
-   This value determines, which string represents
-   'non-values'. Current setting is:
-
-      >>> from waeup.ikoba.utils.converters import NONE_STRING_VALUE
-      >>> NONE_STRING_VALUE
-      ''
-
-   If this value is found during conversion from strings, the value to
-   set will become ``None``.
-
-
-Adapters
-========
-
-The :mod:`waeup.ikoba.utils.converters` module is basically a collection of
-adapters for field types of :mod:`zope.schema`.
-
-:class:`Converter`
-------------------
-
-.. class:: Converter(field)
-
-   Base class for converter classes. You cannot create working
-   instances of this class. Instead derive from it and implement the
-   missing classes.
-
-   `field` should be an object implementing some interface from
-   :mod:`zope.schema` like :class:`zope.schema.TextLine`,
-   `zope.schema.Choice` or similar.
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: _convertValueFromString(string)
-
-       Raises :exc:`NotImplementedError`.
-
-       Called by :meth:`fromString()` to convert a given string to a
-       value appropriate for the applied field type. The string might
-       be ``None`` or ``missing_value`` from field definition.
-
-       Override this method in derived classes to get a working
-       converter.
-
-   .. method:: _convertValueToString(value)
-
-       Raises :exc:`NotImplementedError`.
-
-       Called by :meth:`toString()` to convert a given value to a
-       string suitable to be stored for instance in CSV files.
-
-       Override this method in derived classes to get a working
-       converter.
-
-   .. method:: fromString(string[, strict=True])
-
-       Compute a value for the context field from the given
-       `string`. Be aware that this might fail for several reasons,
-       for instance because the string cannot be converted to an
-       integer (if the applied field is an Integer field), etc.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-   .. method:: toString(value[, strict=True])
-
-       Generate a string from ``value``, suitable to be stored in CSV
-       files or similar. This method does preliminary checks, handles
-       defaults and ``missing_value`` stuff and then calls
-       :meth:`_convertValueToString(value)` to do the actual
-       conversion.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-       
-
-
-:class:`TextConverter`
-----------------------
-
-.. class:: TextConverter(field)
-
-   Create a converter for text fields.
-
-   `field` must be an object implementing
-   ``zope.schema.interfaces.IText``. This is true for instance for
-   :class:`zope.schema.TextLine` fields and most other, purely
-   text-related field types.
-
-   Instances of this class are available as adapters from :mod:`IText`
-   to :mod:`ISchemaTypeConverter`.
-
-   .. method:: adapts(zope.schema.IText)
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: fromString(string[, strict=True])
-
-       Compute a unicode string value from the given `string`. Be
-       aware that this might fail if no conversion of the delivered
-       data is possible.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-   .. method:: toString(value[, strict=True])
-
-       Return `value` as string or ``None``.
-
-       If `value` is ``None`` (or field's `missing_value`), ``None``
-       is returned.
-
-       If the value is not valid for the applied field an exception
-       might be raised.
-
-       Use `strict` to enable/disable validations of input data.
-
-
-:class:`BoolConverter`
-----------------------
-
-.. class:: BoolConverter(field)
-
-   Create a converter for boolean fields.
-
-   `field` must be an object implementing
-   ``zope.schema.interfaces.IBool``.
-
-   Instances of this class are available as adapters from :mod:`IBool`
-   to :mod:`ISchemaTypeConverter`.
-
-   .. method:: adapts(zope.schema.IBool)
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: fromString(string[, strict=True])
-
-       Compute an integer number value from the given `string`. In
-       general '1', 'true', and 'yes' are interpreted as ``True``,
-       ``None`` is interpreted as 'no value' and all other values are
-       interpreted as ``True``.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-   .. method:: toString(value[, strict=True])
-
-       Return `value` as string or ``None``.
-
-       If `value` is ``None`` (or field's `missing_value`), ``None``
-       is returned. Otherwise ``0`` or ``1`` are returned.
-
-       If the value is not valid for the applied field (i.e. not an
-       integer or `missing_value`) an exception
-       might be raised.
-
-       Use `strict` to enable/disable validations of input data.
-
-
-:class:`IntConverter`
----------------------
-
-.. class:: IntConverter(field)
-
-   Create a converter for integer fields.
-
-   `field` must be an object implementing
-   ``zope.schema.interfaces.IInt``.
-
-   Instances of this class are available as adapters from :mod:`IInt`
-   to :mod:`ISchemaTypeConverter`.
-
-   .. method:: adapts(zope.schema.IInt)
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: fromString(string[, strict=True])
-
-       Compute an integer number value from the given `string`. Be
-       aware that this might fail if no conversion of the delivered
-       data is possible.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-   .. method:: toString(value[, strict=True])
-
-       Return `value` as string or ``None``.
-
-       If `value` is ``None`` (or field's `missing_value`), ``None``
-       is returned.
-
-       If the value is not valid for the applied field (i.e. not an
-       integer or `missing_value`) an exception
-       might be raised.
-
-       Use `strict` to enable/disable validations of input data.
-
-
-
-:class:`ChoiceConverter`
-------------------------
-
-.. class:: ChoiceConverter(field)
-
-   Create a converter for choice fields.
-
-   `field` must be an object implementing
-   ``zope.schema.interfaces.IChoice``.
-
-   Instances of this class are available as adapters from :mod:`IChoice`
-   to :mod:`ISchemaTypeConverter`.
-
-   .. method:: adapts(zope.schema.IChoice)
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: fromString(string[, strict=False])
-
-       Compute a value from the given `string`. Be aware that this
-       might fail if no conversion of the delivered data is possible.
-
-       As choices generally require a certain set of allowed values,
-       only these values be allowed, in their tokenized form. The
-       converter takes the given string as token and then tries to
-       generate the appropriate value out of it (see examples below).
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-       As an invalid value will result in a `KeyError`, additional
-       validation checks are disabled with this converter. This speeds
-       up things as expensive lookups are avoided.
-
-   .. method:: toString(value[, strict=False])
-
-       Return `value` as string or ``None``.
-
-       If `value` is ``None`` (or field's `missing_value`), ``None``
-       is returned.
-
-       If the value is not valid for the applied field (i.e. not an
-       integer or `missing_value`) an exception
-       might be raised.
-
-       Use `strict` to enable/disable validations of input data.
-
-       As an invalid value will result in a `ValueError`, additional
-       validation checks are disabled with this converter. This speeds
-       up things as expensive lookups are avoided.
-
-
-:class:`DateConverter`
-----------------------
-
-.. autoclass:: waeup.ikoba.utils.converters.DateConverter
-   :members:
-   :inherited-members:
-
-
-Examples
-========
-
-Text field converters
----------------------
-
-We create a field that also implements ``IText``, a regular
-``TextLine`` field:
-
-    >>> from zope.schema import TextLine
-    >>> field = TextLine(title=u'Some Title',
-    ...                  default=u'Default Value',
-    ...                  required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`TextConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import TextConverter
-    >>> converter = TextConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.TextConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('blah')
-    u'blah'
-
-And back:
-
-    >>> converter.toString(u'blah')
-    'blah'
-
-Okay, not very surprising. But the field definitions can help also
-deliver values, if the given value is missing:
-
-    >>> converter.fromString(None)
-    u'Default Value'
-
-``None`` is not an acceptable value for fields which are required but
-provide no default:
-
-    >>> field = TextLine(title=u'Some Title',
-    ...                  required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-If we want to avoid this type of exception (and risk non-applicable
-data to be stored), we can use the ``strict`` parameter of
-``fromString()``:
-
-    >>> converter.fromString(None, strict=False) is None
-    True 
-
-
-If a field is not required, we will get the ``missing_value`` type in
-same case:
-
-    >>> field = TextLine(title=u'Some Title',
-    ...                  required=False)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None) is None
-    True
-
-We can also set a value that indicates missing data (``None`` by default):
-
-    >>> field = TextLine(title=u'Some Title',
-    ...                  required=False,
-    ...                  missing_value=u'<NO-VALUE>')
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    u'<NO-VALUE>'
-
-And put it back:
-
-    >>> converter.toString(u'<NO-VALUE>') is None
-    True
-
-Bool field converters
----------------------
-
-We create a field that also implements ``IBool``, a regular
-``Bool`` field:
-
-    >>> from zope.schema import Bool
-    >>> field = Bool(title=u'Some truth',
-    ...             default=True,
-    ...             required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`BoolConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import BoolConverter
-    >>> converter = BoolConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.BoolConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('yes')
-    True
-
-    >>> converter.fromString('Yes')
-    True
-
-    >>> converter.fromString('yEs')
-    True
-
-    >>> converter.fromString('1')
-    True
-
-    >>> converter.fromString('True')
-    True
-
-    >>> converter.fromString('0')
-    False
-
-    >>> converter.fromString('no')
-    False
-
-    >>> converter.fromString('false')
-    False
-
-
-And back:
-
-    >>> converter.toString(True)
-    '1'
-
-    >>> converter.toString(False)
-    '0'
-
-Okay, not very surprising. But the field definitions can help also
-deliver values, if the given value is missing:
-
-    >>> converter.fromString(None)
-    True
-
-``None`` is not an acceptable value for fields which are required but
-provide no default:
-
-    >>> field = Bool(title=u'Some Truth',
-    ...             required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-If we want to avoid this type of exception (and risk non-applicable
-data to be stored), we can use the ``strict`` parameter of
-``fromString()``:
-
-    >>> converter.fromString(None, strict=False) is None
-    True
-
-The same for the inverse operation:
-
-    >>> converter.toString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-    >>> converter.toString(None, strict=False) is None
-    True
-
-
-If a field is not required, we will get the ``missing_value`` type in
-same case:
-
-    >>> field = Bool(title=u'Some Title',
-    ...             required=False)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None) is None
-    True
-
-    >>> converter.toString(None) is None
-    True
-
-
-Int field converters
---------------------
-
-We create a field that also implements ``IInt``, a regular
-``Int`` field:
-
-    >>> from zope.schema import Int
-    >>> field = Int(title=u'Some number',
-    ...             default=12,
-    ...             required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`IntConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import IntConverter
-    >>> converter = IntConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.IntConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('666')
-    666
-
-And back:
-
-    >>> converter.toString(666)
-    '666'
-
-Okay, not very surprising. But the field definitions can help also
-deliver values, if the given value is missing:
-
-    >>> converter.fromString(None)
-    12
-
-``None`` is not an acceptable value for fields which are required but
-provide no default:
-
-    >>> field = Int(title=u'Some Title',
-    ...             required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-If we want to avoid this type of exception (and risk non-applicable
-data to be stored), we can use the ``strict`` parameter of
-``fromString()``:
-
-    >>> converter.fromString(None, strict=False) is None
-    True
-
-The same for the inverse operation:
-
-    >>> converter.toString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-    >>> converter.toString(None, strict=False) is None
-    True
-
-
-If a field is not required, we will get the ``missing_value`` type in
-same case:
-
-    >>> field = Int(title=u'Some Title',
-    ...             required=False)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None) is None
-    True
-
-    >>> converter.toString(None) is None
-    True
-
-
-Choice field converters
------------------------
-
-Before choices really work in unit tests, we have to make sure, that
-the following adapters are registered (this is normally done during
-startup automatically):
-
-    >>> from zc.sourcefactory.browser.source import FactoredTerms
-    >>> from zc.sourcefactory.browser.token import (
-    ...   fromString, fromUnicode, fromInteger, fromPersistent)
-    >>> from zope.component import getGlobalSiteManager
-    >>> gsm = getGlobalSiteManager()
-    >>> gsm.registerAdapter(FactoredTerms)
-    >>> gsm.registerAdapter(fromString)
-    >>> gsm.registerAdapter(fromUnicode)
-    >>> gsm.registerAdapter(fromInteger)
-    >>> gsm.registerAdapter(fromPersistent)
-
-We create a field that also implements ``IChoice``, a regular
-``Choice`` field:
-
-    >>> from zope.schema import Choice
-    >>> field = Choice(title=u'Some number',
-    ...                default=u'a',
-    ...                values=['a', 'b', 'c'],
-    ...                required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`ChoiceConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import ChoiceConverter
-    >>> converter = ChoiceConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.ChoiceConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('a')
-    'a'
-
-or values back to strings:
-
-    >>> converter.toString('a')
-    'a'
-
-Non-listed values will not be accepted:
-
-    >>> converter.fromString('nonsense')
-    Traceback (most recent call last):
-    ...
-    LookupError: nonsense
-
-Also non-string values will work:
-
-    >>> from zope.schema import Choice
-    >>> field = Choice(title=u'Some number',
-    ...                default=2,
-    ...                values=[1, 2, 3],
-    ...                required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString('1')
-    1
-
-If we define an own source, then it will cause no problems:
-
-    >>> from zc.sourcefactory.basic import BasicSourceFactory
-    >>> class CustomSource(BasicSourceFactory):
-    ...   def getValues(self):
-    ...     return [1,2,3]
-    ...    
-    ...   def getTitle(self, value):
-    ...     return "Number %s" % (value,)
-
-    >>> field = Choice(title=u'Some number',
-    ...                default=2,
-    ...                source=CustomSource(),
-    ...                required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString('1')
-    1
-
-Note, that when asking for conversion of Choice fields, you must
-deliver the token value of the source/vocabulary. This might be the
-same as the actual value as string, but it it might be different,
-depending of the kind of source/vocabulary used.
-
-Furthermore, when writing 'exporters', i.e. components that turn a
-value into a string for usage in CSV files, then you have to save the
-token value of a term.
-
-Using the :meth:`toString()` method will deliver the token for us:
-
-    >>> converter.toString(1)
-    '1'
-
-    >>> from waeup.ikoba.interfaces import SimpleIkobaVocabulary
-    >>> field = Choice(
-    ...   title = u'Favourite Dish',
-    ...   default = 0,
-    ...   vocabulary = SimpleIkobaVocabulary(
-    ...        ('N/A', 0), ('Pizza', 1),
-    ...        ('Cake', 2), ('Muffins', 3)),
-    ...   required = True,
-    ...   )
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString('0')
-    0
-
-    >>> converter.toString(1)
-    '1'
-
-A more complex (but still realistic example) for conversion of
-:mod:`zope.schema.Choice` fields follows. Here we have a special kind
-of object, the `Cave` class, and get their `name` attribute as token.
-
-    >>> class Cave(object):
-    ...   name = 'Home sweet home'
-
-    >>> def tokenizer(cave):
-    ...   return cave.name
-
-The tokenizer has to be registered as an Cave-to-IToken adapter to
-keep :mod:`zc.sourcefactory` happy:
-
-    >>> from zc.sourcefactory.interfaces import IToken
-    >>> from zope.component import getGlobalSiteManager
-    >>> gsm = getGlobalSiteManager()
-    >>> gsm.registerAdapter(tokenizer, required=(Cave,), provided=IToken)
-
-Now we finally can create two objects of `Cave` and make a field, that
-gives the choice between these two objects:
-
-    >>> obj1 = Cave()
-    >>> obj1.name = 'Wilma'
-
-    >>> obj2 = Cave()
-    >>> obj2.name = 'Fred'
-
-    >>> from zc.sourcefactory.basic import BasicSourceFactory
-    >>> class CustomSource(BasicSourceFactory):
-    ...   def getValues(self):
-    ...     return [obj1, obj2]
-    ...    
-    ...   def getTitle(self, value):
-    ...     return value.name
-
-    >>> field = Choice(title=u'Some cave',
-    ...                default=obj1,
-    ...                source=CustomSource(),
-    ...                required=True)
-
-We get a converter for this field in the usual way. The
-:meth:`waeup.ikoba.utils.converters.fromString()` method will return one of
-the objects if we feed it with ``'Wilma'`` or ``'Fred'``:
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> result = converter.fromString('Wilma')
-    >>> result
-    <Cave object at 0x...>
-
-    >>> result is obj1, result.name
-    (True, 'Wilma')
-
-If we use the converter the other way round, it will call the
-tokenizer above and deliver ``'Wilma'`` or ``'Fred'`` as they are the
-tokens of the objects in question:
-
-    >>> converter.toString(obj2)
-    'Fred'
-
-Date field converters
----------------------
-
-We create a field that also implements ``IDate``, a regular
-``Date`` field:
-
-    >>> from zope.schema import Date
-    >>> from datetime import date
-    >>> field = Date(title=u'Some date',
-    ...             default=date(2010, 4, 1),
-    ...             required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`IntConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import DateConverter
-    >>> converter = DateConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.DateConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('2007-04-23')
-    datetime.date(2007, 4, 23)
-
-And back:
-
-    >>> converter.toString(date(2007,4,23))
-    '2007-04-23'
-
-As string input we also accept 'DD/MM/YYYY':
-
-    >>> converter.fromString('13/05/1945')
-    datetime.date(1945, 5, 13)
-
-and some more bizarre formats:
-
-    >>> converter.fromString('1945/05/13')
-    datetime.date(1945, 5, 13)
-
-    >>> converter.fromString('1945/05/13 12:12:34 GMT+1')
-    datetime.date(1945, 5, 13)
-
-    >>> converter.fromString('1945/5/6')
-    datetime.date(1945, 5, 6)
-
-    >>> converter.fromString('06.05.1945')
-    datetime.date(1945, 5, 6)
-
-    >>> converter.fromString('6.5.1945')
-    datetime.date(1945, 5, 6)
-
-while other date formats will mean trouble:
-
-    >>> converter.fromString('first of July 1960')
-    Traceback (most recent call last):
-    ...
-    ValueError: Cannot convert to date: first of July 1960. Use YYYY-MM-DD.
-
-Okay, not very surprising. But the field definitions can help also
-deliver values, if the given value is missing:
-
-    >>> converter.fromString(None)
-    datetime.date(2010, 4, 1)
-
-``None`` is not an acceptable value for fields which are required but
-provide no default:
-
-    >>> field = Date(title=u'Some Title',
-    ...             required=True)
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-If we want to avoid this type of exception (and risk non-applicable
-data to be stored), we can use the ``strict`` parameter of
-``fromString()``:
-
-    >>> converter.fromString(None, strict=False) is None
-    True
-
-The same for the inverse operation:
-
-    >>> converter.toString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-    >>> converter.toString(None, strict=False) is None
-    True
-
-
-If a field is not required, we will get the ``missing_value`` type in
-same case:
-
-    >>> field = Date(title=u'Some Title',
-    ...             required=False)
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None) is None
-    True
-
-    >>> converter.toString(None) is None
-    True
Index: /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/pages.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/pages.txt	(revision 12991)
+++ /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/pages.txt	(revision 12991)
@@ -0,0 +1,368 @@
+Browsing Ikoba
+*************
+
+Here we visit all parts of a Ikoba portal using a browser.
+
+Company
+==========
+
+We can watch universities in the browser.
+
+We create an company object and put into the ZODB root::
+
+  >>> root = getRootFolder()
+  >>> list(root)
+  []
+
+  >>> from waeup.ikoba.app import Company
+  >>> u = Company()
+  >>> root['mycompany'] = u
+  >>> list(root)
+  [u'mycompany']
+
+  >>> from zope.component.hooks import setSite
+  >>> setSite(root['mycompany'])
+
+To make sure, we can 'watch' pages, we first have to initialize out
+test browser::
+
+  >>> from zope.testbrowser.testing import Browser
+  >>> browser = Browser()
+
+Let's get the default view of a company::
+
+  >>> browser.open('http://localhost/mycompany')
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Welcome to WAeUP.Ikoba...
+  ...
+
+We can change to German::
+  >>> browser.getLink('de').click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Anfragen...
+  ...
+
+And then change back to English by clicking on the third link containing 'en'
+behind 'Anfragen' and 'Einloggen'::
+  >>> browser.getLink('en', index=2).click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Enquiries...
+  ...
+
+The contact form for anonymous users is called 'Enquiries'::
+
+  >>> browser.open('http://localhost/mycompany')
+  >>> browser.getLink('Enquiries').click()
+  >>> browser.getControl('Send').click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Required input is missing...
+  ...
+
+  >>> browser.getControl(name='form.fullname').value = "Bob Tester"
+  >>> browser.getControl(name='form.email_from').value = "xx@yy.zz"
+  >>> browser.getControl(name='form.body').value = u'test message'
+  >>> browser.getControl('Send').click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Your message has been sent...
+  ...
+
+Registered users with an email address can request a password change:
+
+  >>> root['mycompany']['users'].addUser('forgetful', 'secret',title='Bob Forgetful',
+  ...                               description='A forgetful user', email='aa@aa.ng')
+  >>> browser.open('http://localhost/mycompany/changepw')
+  >>> browser.getControl(name="form.email").value = 'aa@aa.ng'
+  >>> browser.getControl("Send login credentials").click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...An email with your user name and password has been sent to aa@aa.ng...
+
+Now we login as manager::
+
+  >>> browser.addHeader('Authorization', 'Basic mgr:mgrpw')
+  >>> browser.handleErrors = False
+
+We can then get an edit view of the configuration container::
+
+  >>> browser.open('http://localhost/mycompany/configuration')
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...<form action="http://localhost/mycompany/configuration/@@index"
+  ...
+
+The edit form contains the default value for the company name::
+
+  >>> 'Sample Company' in browser.contents
+  True
+
+We can perform several actions on the edit form::
+
+  >>> browser.getControl("Save", index=0).click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Form has been saved...
+  ...
+
+  >>> browser.open('http://localhost/mycompany/configuration')
+  >>> browser.getControl("Update plugins").click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Plugins were updated. See log file for details...
+  ...
+
+The default frontpage HTML has been saved in a dictionary
+and is properly rendered on the frontpage of the portal:
+
+  >>> browser.open('http://localhost/mycompany')
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...<h1>Welcome to WAeUP.Ikoba...</h1>
+  ...
+
+The German part is really not being rendered:
+
+  >>> 'Willkommen' in browser.contents
+  False
+
+
+If we change to German so that the German part of frontpage.rst is rendered:
+
+  >>> browser.open('http://localhost/mycompany//@@change_language?lang=de')
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...<h1>Willkommen auf WAeUP.Ikoba...</h1>
+  ...
+
+The English part is really not being rendered:
+
+  >>> 'Welcome' in browser.contents
+  False
+
+Switch back to English:
+
+  >>> browser.open('http://localhost/mycompany//@@change_language?lang=en')
+
+
+Officers
+========
+
+  >>> browser.open('http://localhost/mycompany')
+  >>> browser.getLink('Officers').click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Officers...
+  ...
+
+We can add users:
+
+  >>> browser.getLink("Add user").click()
+  >>> browser.getControl(name="form.name").value = 'bob'
+  >>> browser.getControl(name="form.title").value = 'Bob The User'
+  >>> browser.getControl(name="password").value = 'secret'
+  >>> browser.getControl(name="control_password").value = 'secret'
+  >>> browser.getControl(name="form.email").value = 'xx@yy.zz'
+  >>> browser.getControl(name="form.phone.country").value = ['+234']
+  >>> browser.getControl(name="form.phone.area").value = '123'
+  >>> browser.getControl(name="form.phone.ext").value = '45678'
+  >>> browser.getControl("Add user").click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...<td>bob</td>
+  ...
+
+We can edit user bob:
+
+  >>> browser.getControl("Manage", index=0).click()
+  >>> browser.getControl("Save", index=0).click()
+  >>> browser.getControl("Cancel", index=0).click()
+
+We can add site roles which are then displayed on the user container page.
+Since the test browser does not use javascript, we have to add site roles
+manually by setting the roles attribute:
+
+Since JS is not supported by the test browser, the default role has been removed
+when clicking the Save button.
+
+  >>> print root['mycompany']['users']['bob'].roles
+  []
+  >>> root['mycompany']['users']['bob'].roles = ['waeup.UsersManager']
+  >>> print root['mycompany']['users']['bob'].roles
+  ['waeup.UsersManager']
+  >>> browser.open('http://localhost/mycompany/users')
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...<td nowrap>Users Manager <br /></td>
+  ...
+
+Users can't be added twice:
+
+  >>> browser.open('http://localhost/mycompany/users/add')
+  >>> browser.getControl(name="form.name").value = 'bob'
+  >>> browser.getControl(name="form.title").value = 'Bob The User'
+  >>> browser.getControl(name="password").value = 'secret'
+  >>> browser.getControl(name="control_password").value = 'secret'
+  >>> browser.getControl(name="form.email").value = 'xx@yy.zz'
+  >>> browser.getControl(name="form.phone.country").value = ['+234']
+  >>> browser.getControl(name="form.phone.area").value = '123'
+  >>> browser.getControl(name="form.phone.ext").value = '45678'
+  >>> browser.getControl("Add user").click()
+  >>> 'The userid chosen already exists' in browser.contents
+  True
+
+Users can be deleted:
+
+  >>> browser.open('http://localhost/mycompany/users')
+  >>> browser.getControl("Remove", index=0).click()
+  >>> 'User account bob successfully deleted' in browser.contents
+  True
+
+
+
+Contact Form
+============
+
+Let's enter the contact form::
+
+  >>> browser.open('http://localhost/mycompany/contactadmin')
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Contact
+  ...
+
+We fill the form  (this will send a real message to
+contact@waeup.org)::
+
+  >>> browser.open('http://localhost/mycompany/contactadmin')
+  >>> browser.getControl(name='form.body').value = "test message"
+  >>> browser.getControl('Send').click()
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"...
+  ...Your message has been sent...
+  ...
+
+If this test fails, chances are, that the local machine has no SMTP
+server installed.
+
+
+Data Center
+===========
+
+The data center helps us uploading files for later import or similar.
+
+  >>> browser.open('http://localhost/mycompany')
+  >>> browser.getLink('Data Center').click()
+
+Setting the file path
+---------------------
+
+A datacenter stores files in a path in filesystem. By default this is
+a directory in the sources:
+
+  >>> print browser.contents
+  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  ...
+  <b>Storage path:</b> <span>.../parts/test/datacenter</span>
+  ...
+
+Going to 'Settings` we can change the path:
+
+  >>> browser.getLink('Edit settings').click()
+  >>> pathsetting = browser.getControl(name='newpath')
+
+We create a directory and set it as our upload path:
+
+  >>> import os
+  >>> cwd = os.getcwd()
+  >>> uploadpath = os.path.join(cwd, 'testfiles')
+  >>> os.mkdir(uploadpath)
+  >>> pathsetting.value = uploadpath
+
+And submit the form:
+
+  >>> browser.getControl(name='save').click()
+
+We clean up the set directory path, as there might be some files be
+copied some files from installation:
+
+  >>> files = os.listdir(uploadpath)
+  >>> for filename in files:
+  ...   if os.path.isdir(os.path.join(uploadpath, filename)):
+  ...     continue
+  ...   os.unlink(os.path.join(uploadpath, filename))
+
+We also remove any existing 'accesscodes' subdir:
+
+  >>> import shutil
+  >>> for filename in files:
+  ...   if not os.path.isdir(os.path.join(uploadpath, filename)):
+  ...     continue
+  ...   if filename != 'accesscodes':
+  ...     continue
+  ...   shutil.rmtree(os.path.join(uploadpath, filename))
+
+The new upload directory is now empty, except from the logs and other
+standard dirs, which are created automatically:
+
+  >>> sorted(os.listdir(uploadpath))
+  ['deleted', 'finished', 'logs', 'unfinished']
+
+
+Uploading files
+---------------
+
+Now we can upload files. Most interesting files might be CSV files,
+that can be imported lateron. We create a CSV file containing faculty
+descriptions:
+
+  >>> open('faculties.csv', 'wb').write(
+  ... """code,title,title_prefix
+  ... FA,Arts,faculty
+  ... FS,Sciences,faculty
+  ... """)
+
+Now we can upload this file. To do this, we first go to the upload
+page:
+
+  >>> browser.getLink('Upload data').click()
+
+and enter the appropriate data in the form:
+
+  >>> filewidget = browser.getControl(name='uploadfile:file')
+  >>> filewidget
+  <Control name='uploadfile:file' type='file'>
+
+A sidenote for developers: by marking the filewidget with the
+``:file`` extension, we tell Zope to handle this field as a file
+widget.
+
+  >>> import cStringIO
+  >>> filecontents = cStringIO.StringIO(
+  ...   open('faculties.csv', 'rb').read())
+  >>> filewidget.add_file(filecontents, 'text/plain', 'myfaculties.csv')
+
+  >>> browser.getControl(name='SUBMIT').click()
+
+The file was indeed uploaded, with the current userid inserted:
+
+  >>> sorted(os.listdir(uploadpath))
+  ['deleted', 'finished', 'logs', 'myfaculties_zope.mgr.csv', 'unfinished']
+
+
+
+Importing a CSV file
+--------------------
+
+The import of CSV files is described in batchprocessing.txt.
+
+
+Clean up:
+
+  >>> import os
+  >>> import shutil
+  >>> shutil.rmtree(uploadpath)
+
Index: in/waeup.ikoba/trunk/src/waeup/ikoba/index.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/index.txt	(revision 12990)
+++ 	(revision )
@@ -1,30 +1,0 @@
-Welcome to waeup documentation!
-*******************************
-
-Contents:
-
-.. toctree::
-   :maxdepth: 2
-
-   README
-   doc/developernotes
-
-API documentation
-=================
-
-.. toctree::
-   :maxdepth: 1
-
-   app
-   catalog
-   accesscodes/api.txt
-   jambtables/api.txt
-   company/api.txt
-   utils/api.txt
-
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
Index: in/waeup.ikoba/trunk/src/waeup/ikoba/utils/api.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/utils/api.txt	(revision 12990)
+++ 	(revision )
@@ -1,19 +1,0 @@
-:mod:`waeup.ikoba.utils` -- misc. helpers
-****************************************
-
-.. module:: waeup.ikoba.utils
-
-Components and other stuff of general interest.
-
-Submodules
-==========
-
-The :mod:`waeup.ikoba.utils` module provides the following submodules:
-
-.. toctree::
-   :maxdepth: 1
-
-   helpers.txt
-   batching.txt
-   importexport.txt
-   converters.txt
Index: in/waeup.ikoba/trunk/src/waeup/ikoba/utils/importexport.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/utils/importexport.txt	(revision 12990)
+++ 	(revision )
@@ -1,145 +1,0 @@
-Importing and Exporting Things
-******************************
-
-The waeup package tries to support all special object types to be
-exportable and importable.
-
-.. :doctest:
-.. :layer: waeup.ikoba.testing.IkobaUnitTestLayer
-
-As imports and exports of data are a cruical point when doing updates
-or to enable data backups, this support must be considered a serious
-topic for the base functionality of the whole software.
-
-Exports and imports are implemented using the Zope component framework
-capabilities. This means we need some adapters and utilities, which we
-register, by grokking the whole package. This is done on startup
-automatically, but in doctests (like this one) it is done by the
-testlayer.
-
-Furthermore we need something to import/export. As only pickable
-classes are importable/exportable and these have to be defined in some
-globally reachable module, we first define some silly object in
-filesystem and adjust the path to lookup modules:
-
-    >>> import os
-    >>> import sys
-    >>> os.mkdir('stoneage')
-    >>> old_sys_path = sys.path
-    >>> open(os.path.join('stoneage', '__init__.py'), 'wb').write(
-    ... """
-    ... class Cave(object):
-    ...   name = 'Unnamed cave'
-    ...   dinoports = 2
-    ... """)
-    >>> sys.path.append(os.getcwd())
-
-Now we can import stoneage.Cave objects.
-
-
-Exporting XML
-=============
-
-One of the major formats we support for data export is XML.
-
-When we want to export an object, we can choose the destination
-file. If we do not give a file to write to, we get a StringIO object,
-i.e. an in-memory file.
-
-We can export caves. First we create a cave:
-
-    >>> from stoneage import Cave
-    >>> mycave = Cave()
-    >>> mycave.name = 'Freds Cave'
-    >>> mycave.dinoports = 1
-
-Exporting plain XML
--------------------
-
-Now let's export it to XML. To do this we first ask the framework for
-an exporter::
-
-    >>> from waeup.ikoba.interfaces import IIkobaXMLExporter
-    >>> exporter = IIkobaXMLExporter(mycave)
-    >>> exporter
-    <waeup.ikoba.utils.importexport.XMLExporter object at 0x...>
-
-All exporters provide an ``export(<obj>)`` method::
-
-    >>> result = exporter.export()
-
-The result is an object of type StringIO. We therefore use its
-`read()` method to get the actual XML data::
-
-    >>> print result.read()
-    <?xml version="1.0" encoding="utf-8" ?>...
-
-
-Generating XML files
---------------------
-
-We can also let the results be written to a file::
-
-    >>> result = exporter.export('myexport.xml')
-    >>> print open('myexport.xml', 'rb').read()
-    <?xml version="1.0" encoding="utf-8" ?>
-    <pickle>
-      <object>
-    ...
-    </pickle>
-
-
-Importing XML
-=============
-
-We can generate objects from XML.
-
-We change the name of the cave:
-
-    >>> mycave.name = 'Another name'
-
-Now we create an importer for that object:
-
-    >>> from waeup.ikoba.interfaces import IIkobaXMLImporter
-    >>> importer = IIkobaXMLImporter(mycave)
-
-Importing from filenames
-------------------------
-
-We can use a filepath to import from the denoted file::
-
-    >>> new_obj = importer.doImport('myexport.xml')
-
-The object created is indeed a cave::
-
-    >>> new_obj
-    <stoneage.Cave object at 0x...>
-
-    >>> new_obj.name
-    'Freds Cave'
-
-The new object is really a new object::
-
-    >>> new_obj is mycave
-    False
-
-Importing from filelike objects
--------------------------------
-
-We can also pass a file-like object instead of a filepath::
-
-    >>> filelike_obj = open('myexport.xml', 'rb')
-    >>> new_obj = importer.doImport(filelike_obj)
-
-The object created is indeed a faculty::
-
-    >>> new_obj
-    <stoneage.Cave object at 0x...>
-
-Clean up::
-
-    >>> import os
-    >>> import shutil
-    >>> sys.path = old_sys_path
-    >>> os.unlink('myexport.xml')
-    >>> shutil.rmtree('stoneage')
