Index: /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batching.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batching.txt	(revision 12990)
+++ /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/batching.txt	(revision 12990)
@@ -0,0 +1,556 @@
+:mod:`waeup.ikoba.utils.batching` -- Batch processing
+****************************************************
+
+Batch processing is much more than pure data import.
+
+Overview
+========
+
+Basically, it means processing CSV files in order to mass-create,
+mass-remove, or mass-update data.
+
+So you can feed CSV files to processors, that are part of
+the batch-processing mechanism.
+
+Processors
+----------
+
+Each CSV file processor
+
+* accepts a single data type identified by an interface.
+
+* knows about the places inside a site (Company) where to store,
+  remove or update the data.
+
+* can check headers before processing data.
+
+* supports the mode 'create', 'update', 'remove'.
+
+* creates log entries (optional)
+
+* creates csv files containing successful and not-successful processed
+  data respectively.
+
+Output
+------
+
+The results of processing are written to loggers, if a logger was
+given. Beside this new CSV files are created during processing:
+
+* a pending CSV file, containing datasets that could not be processed
+
+* a finished CSV file, containing datasets successfully processed.
+
+The pending file is not created if everything works fine. The
+respective path returned in that case is ``None``.
+
+The pending file (if created) is a CSV file that contains the failed
+rows appended by a column ``--ERRROR--`` in which the reasons for
+processing failures are listed.
+
+The complete paths of these files are returned. They will be in a
+temporary directory created only for this purpose. It is the caller's
+responsibility to remove the temporay directories afterwards (the
+datacenters distProcessedFiles() method takes care for that).
+
+It looks like this::
+ 
+     -----+      +---------+
+    /     |      |         |              +------+
+   | .csv +----->|Batch-   |              |      |
+   |      |      |processor+----changes-->| ZODB |
+   |  +------+   |         |              |      |
+   +--|      |   |         +              +------+
+      | Mode +-->|         |                 -------+
+      |      |   |         +----outputs-+-> /       |
+      |  +----+->+---------+            |  |.pending|
+      +--|Log |  ^                      |  |        |
+         +----+  |                      |  +--------+
+           +-----++                     v
+           |Inter-|                  ----------+
+           |face  |                 /          |
+           +------+                | .finished |
+                                   |           |
+                                   +-----------+
+
+
+Creating a batch processor
+==========================
+
+We create an own batch processor for an own datatype. This datatype
+must be based on an interface that the batcher can use for converting
+data.
+
+Founding Stoneville
+-------------------
+
+We start with the interface:
+
+    >>> from zope.interface import Interface
+    >>> from zope import schema
+    >>> class ICave(Interface):
+    ...   """A cave."""
+    ...   name = schema.TextLine(
+    ...     title = u'Cave name',
+    ...     default = u'Unnamed',
+    ...     required = True)
+    ...   dinoports = schema.Int(
+    ...     title = u'Number of DinoPorts (tm)',
+    ...     required = False,
+    ...     default = 1)
+    ...   owner = schema.TextLine(
+    ...     title = u'Owner name',
+    ...     required = True,
+    ...     missing_value = 'Fred Estates Inc.')
+    ...   taxpayer = schema.Bool(
+    ...     title = u'Payes taxes',
+    ...     required = True,
+    ...     default = False)
+
+Now a class that implements this interface:
+
+    >>> import grok
+    >>> class Cave(object):
+    ...   grok.implements(ICave)
+    ...   def __init__(self, name=u'Unnamed', dinoports=2,
+    ...                owner='Fred Estates Inc.', taxpayer=False):
+    ...     self.name = name
+    ...     self.dinoports = 2
+    ...     self.owner = owner
+    ...     self.taxpayer = taxpayer
+
+We also provide a factory for caves. Strictly speaking, this not
+necessary but makes the batch processor we create afterwards, better
+understandable.
+
+    >>> from zope.component import getGlobalSiteManager
+    >>> from zope.component.factory import Factory
+    >>> from zope.component.interfaces import IFactory
+    >>> gsm = getGlobalSiteManager()
+    >>> cave_maker = Factory(Cave, 'A cave', 'Buy caves here!')
+    >>> gsm.registerUtility(cave_maker, IFactory, 'Lovely Cave')
+
+Now we can create caves using a factory:
+
+    >>> from zope.component import createObject
+    >>> createObject('Lovely Cave')
+    <Cave object at 0x...>
+
+This is nice, but we still lack a place, where we can place all the
+lovely caves we want to sell.
+
+Furthermore, as a replacement for a real site, we define a place where
+all caves can be stored: Stoneville! This is a lovely place for
+upperclass cavemen (which are the only ones that can afford more than
+one dinoport).
+
+We found Stoneville:
+
+    >>> stoneville = dict()
+
+Everything in place.
+
+Now, to improve local health conditions, imagine we want to populate
+Stoneville with lots of new happy dino-hunting natives that slept on
+the bare ground in former times and had no idea of
+bathrooms. Disgusting, isn't it?
+
+Lots of cavemen need lots of caves.
+
+Of course we can do something like:
+
+    >>> cave1 = createObject('Lovely Cave')
+    >>> cave1.name = "Fred's home"
+    >>> cave1.owner = "Fred"
+    >>> stoneville[cave1.name] = cave1
+
+and Stoneville has exactly
+
+    >>> len(stoneville)
+    1
+
+inhabitant. But we don't want to do this for hundreds or thousands of
+citizens-to-be, do we?
+
+It is much easier to create a simple CSV list, where we put in all the
+data and let a batch processor do the job.
+
+The list is already here:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner,taxpayer
+    ... Barneys Home,2,Barney,1
+    ... Wilmas Asylum,1,Wilma,1
+    ... Freds Dinoburgers,10,Fred,0
+    ... Joeys Drive-in,110,Joey,0
+    ... """)
+
+All we need, is a batch processor now.
+
+    >>> from waeup.ikoba.utils.batching import BatchProcessor
+    >>> from waeup.ikoba.interfaces import IGNORE_MARKER
+    >>> class CaveProcessor(BatchProcessor):
+    ...   util_name = 'caveprocessor'
+    ...   grok.name(util_name)
+    ...   name = 'Cave Processor'
+    ...   iface = ICave
+    ...   location_fields = ['name']
+    ...   factory_name = 'Lovely Cave'
+    ...
+    ...   def parentsExist(self, row, site):
+    ...     return True
+    ...
+    ...   def getParent(self, row, site):
+    ...     return stoneville
+    ...
+    ...   def entryExists(self, row, site):
+    ...     return row['name'] in stoneville.keys()
+    ...
+    ...   def getEntry(self, row, site):
+    ...     if not self.entryExists(row, site):
+    ...       return None
+    ...     return stoneville[row['name']]
+    ...
+    ...   def delEntry(self, row, site):
+    ...     del stoneville[row['name']]
+    ...
+    ...   def addEntry(self, obj, row, site):
+    ...     stoneville[row['name']] = obj
+    ...
+    ...   def updateEntry(self, obj, row, site, filename):
+    ...     # This is not strictly necessary, as the default
+    ...     # updateEntry method does exactly the same
+    ...     for key, value in row.items():
+    ...       if value != IGNORE_MARKER:
+    ...         setattr(obj, key, value)
+
+If we also want the results being logged, we must provide a logger
+(this is optional):
+
+    >>> import logging
+    >>> logger = logging.getLogger('stoneville')
+    >>> logger.setLevel(logging.DEBUG)
+    >>> logger.propagate = False
+    >>> handler = logging.FileHandler('stoneville.log', 'w')
+    >>> logger.addHandler(handler)
+
+Create the fellows:
+
+    >>> processor = CaveProcessor()
+    >>> result = processor.doImport('newcomers.csv',
+    ...                   ['name', 'dinoports', 'owner', 'taxpayer'],
+    ...                    mode='create', user='Bob', logger=logger)
+    >>> result
+    (4, 0, '/.../newcomers.finished.csv', None)
+
+The result means: four entries were processed and no warnings
+occured. Furthermore we get filepath to a CSV file with successfully
+processed entries and a filepath to a CSV file with erraneous entries.
+As everything went well, the latter is ``None``. Let's check:
+
+    >>> sorted(stoneville.keys())
+    [u'Barneys Home', ..., u'Wilmas Asylum']
+
+The values of the Cave instances have correct type:
+
+    >>> barney = stoneville['Barneys Home']
+    >>> barney.dinoports
+    2
+
+which is a number, not a string.
+
+Apparently, when calling the processor, we gave some more info than
+only the CSV filepath. What does it all mean?
+
+While the first argument is the path to the CSV file, we also have to
+give an ordered list of headernames. These replace the header field
+names that are actually in the file. This way we can override faulty
+headers.
+
+The ``mode`` paramter tells what kind of operation we want to perform:
+``create``, ``update``, or ``remove`` data.
+
+The ``user`` parameter finally is optional and only used for logging.
+
+We can, by the way, see the results of our run in a logfile if we
+provided a logger during the call:
+
+    >>> print open('stoneville.log').read()
+    processed: newcomers.csv, create mode, 4 lines (4 successful/ 0 failed), ... s (... s/item)
+
+
+We cleanup the temporay dir created by doImport():
+
+    >>> import shutil
+    >>> import os
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+As we can see, the processing was successful. Otherwise, all problems
+could be read here as we can see, if we do the same operation again:
+
+    >>> result = processor.doImport('newcomers.csv',
+    ...                   ['name', 'dinoports', 'owner', 'taxpayer'],
+    ...                    mode='create', user='Bob', logger=logger)
+    >>> result
+    (4, 4, '/.../newcomers.finished.csv', '/.../newcomers.pending.csv')
+
+This time we also get a path to a .pending file.
+
+The log file will tell us this in more detail:
+
+    >>> print open('stoneville.log').read()
+    processed: newcomers.csv, create mode, 4 lines (4 successful/ 0 failed), ... s (... s/item)
+    processed: newcomers.csv, create mode, 4 lines (0 successful/ 4 failed), ... s (... s/item)
+
+
+This time a new file was created, which keeps all the rows we could not
+process and an additional column with error messages:
+
+    >>> print open(result[3]).read()
+    owner,name,taxpayer,dinoports,--ERRORS--
+    Barney,Barneys Home,1,2,This object already exists. Skipping.
+    Wilma,Wilmas Asylum,1,1,This object already exists. Skipping.
+    Fred,Freds Dinoburgers,0,10,This object already exists. Skipping.
+    Joey,Joeys Drive-in,0,110,This object already exists. Skipping.
+
+This way we can correct the faulty entries and afterwards retry without
+having the already processed rows in the way.
+
+We also notice, that the values of the taxpayer column are returned as
+in the input file. There we wrote '1' for ``True`` and '0' for
+``False`` (which is accepted by the converters).
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+We can also tell to ignore some cols from input by passing
+``--IGNORE--`` as col name:
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             '--IGNORE--', '--IGNORE--'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (4, 0, '...', None)
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+If something goes wrong during processing, the respective --IGNORE--
+cols won't be populated  in the resulting pending file:
+
+    >>> result = processor.doImport('newcomers.csv', ['name', 'dinoports',
+    ...                             '--IGNORE--', '--IGNORE--'],
+    ...                    mode='create', user='Bob')
+    >>> result
+    (4, 4, '...', '...')
+
+    >>> print open(result[3], 'rb').read()
+    name,dinoports,--ERRORS--
+    Barneys Home,2,This object already exists. Skipping.
+    Wilmas Asylum,1,This object already exists. Skipping.
+    Freds Dinoburgers,10,This object already exists. Skipping.
+    Joeys Drive-in,110,This object already exists. Skipping.
+
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+
+
+Updating entries
+----------------
+
+To update entries, we just call the batchprocessor in a different
+mode:
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (4, 0, '...', None)
+
+Now we want to tell, that Wilma got an extra port for her second dino:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... Wilmas Asylum,2,Wilma
+    ... """)
+
+    >>> wilma = stoneville['Wilmas Asylum']
+    >>> wilma.dinoports
+    1
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+We start the processor:
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                    'dinoports', 'owner'], mode='update', user='Bob')
+    >>> result
+    (1, 0, '...', None)
+
+    >>> wilma = stoneville['Wilmas Asylum']
+    >>> wilma.dinoports
+    2
+
+Wilma's number of dinoports raised.
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+If we try to update an unexisting entry, an error occurs:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... NOT-WILMAS-ASYLUM,2,Wilma
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (1, 1, '/.../newcomers.finished.csv', '/.../newcomers.pending.csv')
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+    
+Also invalid values will be spotted:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... Wilmas Asylum,NOT-A-NUMBER,Wilma
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (1, 1, '...', '...')
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+We can also update only some cols, leaving some out. We skip the
+'dinoports' column in the next run:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,owner
+    ... Wilmas Asylum,Barney
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name', 'owner'],
+    ...                             mode='update', user='Bob')
+    >>> result
+    (1, 0, '...', None)
+
+    >>> wilma.owner
+    u'Barney'
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+We can however, not leave out the 'location field' ('name' in our
+case), as this one tells us which entry to update:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... 2,Wilma
+    ... """)
+
+    >>> processor.doImport('newcomers.csv', ['dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    Traceback (most recent call last):
+    ...
+    FatalCSVError: Need at least columns 'name' for import!
+
+This time we get even an exception!
+
+Generally, empty strings are considered as ``None``:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... "Wilmas Asylum","","Wilma"
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob')
+    >>> result
+    (1, 0, '...', None)
+
+    >>> wilma.dinoports
+    2
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+We can tell to set dinoports to ``None`` although this is not a
+number, as we declared the field not required in the interface:
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... "Wilmas Asylum","XXX","Wilma"
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='update', user='Bob', ignore_empty=False)
+    >>> result
+    (1, 0, '...', None)
+
+    >>> wilma.dinoports is None
+    True
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+Removing entries
+----------------
+
+In 'remove' mode we can delete entries. Here validity of values in
+non-location fields doesn't matter because those fields are ignored.
+
+    >>> open('newcomers.csv', 'wb').write(
+    ... """name,dinoports,owner
+    ... "Wilmas Asylum","ILLEGAL-NUMBER",""
+    ... """)
+
+    >>> result = processor.doImport('newcomers.csv', ['name',
+    ...                             'dinoports', 'owner'],
+    ...                    mode='remove', user='Bob')
+    >>> result
+    (1, 0, '...', None)
+
+    >>> sorted(stoneville.keys())
+    [u'Barneys Home', "Fred's home", u'Freds Dinoburgers', u'Joeys Drive-in']
+
+Oops! Wilma is gone.
+
+Clean up:
+
+    >>> shutil.rmtree(os.path.dirname(result[2]))
+
+
+Clean up:
+
+    >>> import os
+    >>> os.unlink('newcomers.csv')
+    >>> os.unlink('stoneville.log')
Index: /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/converters.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/converters.txt	(revision 12990)
+++ /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/converters.txt	(revision 12990)
@@ -0,0 +1,861 @@
+:mod:`waeup.ikoba.utils.converters` -- Converters
+************************************************
+
+.. module:: waeup.ikoba.utils.converters
+
+Converters for :mod:`zope.schema` based data.
+
+.. contents::
+
+
+.. :NOdoctest:
+.. :NOlayer: waeup.ikoba.testing.IkobaUnitTestLayer
+
+
+Constants
+=========
+
+.. attribute:: NONE_STRING_VALUE
+
+   This value determines, which string represents
+   'non-values'. Current setting is:
+
+      >>> from waeup.ikoba.utils.converters import NONE_STRING_VALUE
+      >>> NONE_STRING_VALUE
+      ''
+
+   If this value is found during conversion from strings, the value to
+   set will become ``None``.
+
+
+Adapters
+========
+
+The :mod:`waeup.ikoba.utils.converters` module is basically a collection of
+adapters for field types of :mod:`zope.schema`.
+
+:class:`Converter`
+------------------
+
+.. class:: Converter(field)
+
+   Base class for converter classes. You cannot create working
+   instances of this class. Instead derive from it and implement the
+   missing classes.
+
+   `field` should be an object implementing some interface from
+   :mod:`zope.schema` like :class:`zope.schema.TextLine`,
+   `zope.schema.Choice` or similar.
+
+   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
+
+   .. method:: _convertValueFromString(string)
+
+       Raises :exc:`NotImplementedError`.
+
+       Called by :meth:`fromString()` to convert a given string to a
+       value appropriate for the applied field type. The string might
+       be ``None`` or ``missing_value`` from field definition.
+
+       Override this method in derived classes to get a working
+       converter.
+
+   .. method:: _convertValueToString(value)
+
+       Raises :exc:`NotImplementedError`.
+
+       Called by :meth:`toString()` to convert a given value to a
+       string suitable to be stored for instance in CSV files.
+
+       Override this method in derived classes to get a working
+       converter.
+
+   .. method:: fromString(string[, strict=True])
+
+       Compute a value for the context field from the given
+       `string`. Be aware that this might fail for several reasons,
+       for instance because the string cannot be converted to an
+       integer (if the applied field is an Integer field), etc.
+
+       Use `strict` to enable/disable validations of transformed
+       data.
+
+   .. method:: toString(value[, strict=True])
+
+       Generate a string from ``value``, suitable to be stored in CSV
+       files or similar. This method does preliminary checks, handles
+       defaults and ``missing_value`` stuff and then calls
+       :meth:`_convertValueToString(value)` to do the actual
+       conversion.
+
+       Use `strict` to enable/disable validations of transformed
+       data.
+       
+
+
+:class:`TextConverter`
+----------------------
+
+.. class:: TextConverter(field)
+
+   Create a converter for text fields.
+
+   `field` must be an object implementing
+   ``zope.schema.interfaces.IText``. This is true for instance for
+   :class:`zope.schema.TextLine` fields and most other, purely
+   text-related field types.
+
+   Instances of this class are available as adapters from :mod:`IText`
+   to :mod:`ISchemaTypeConverter`.
+
+   .. method:: adapts(zope.schema.IText)
+
+   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
+
+   .. method:: fromString(string[, strict=True])
+
+       Compute a unicode string value from the given `string`. Be
+       aware that this might fail if no conversion of the delivered
+       data is possible.
+
+       Use `strict` to enable/disable validations of transformed
+       data.
+
+   .. method:: toString(value[, strict=True])
+
+       Return `value` as string or ``None``.
+
+       If `value` is ``None`` (or field's `missing_value`), ``None``
+       is returned.
+
+       If the value is not valid for the applied field an exception
+       might be raised.
+
+       Use `strict` to enable/disable validations of input data.
+
+
+:class:`BoolConverter`
+----------------------
+
+.. class:: BoolConverter(field)
+
+   Create a converter for boolean fields.
+
+   `field` must be an object implementing
+   ``zope.schema.interfaces.IBool``.
+
+   Instances of this class are available as adapters from :mod:`IBool`
+   to :mod:`ISchemaTypeConverter`.
+
+   .. method:: adapts(zope.schema.IBool)
+
+   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
+
+   .. method:: fromString(string[, strict=True])
+
+       Compute an integer number value from the given `string`. In
+       general '1', 'true', and 'yes' are interpreted as ``True``,
+       ``None`` is interpreted as 'no value' and all other values are
+       interpreted as ``True``.
+
+       Use `strict` to enable/disable validations of transformed
+       data.
+
+   .. method:: toString(value[, strict=True])
+
+       Return `value` as string or ``None``.
+
+       If `value` is ``None`` (or field's `missing_value`), ``None``
+       is returned. Otherwise ``0`` or ``1`` are returned.
+
+       If the value is not valid for the applied field (i.e. not an
+       integer or `missing_value`) an exception
+       might be raised.
+
+       Use `strict` to enable/disable validations of input data.
+
+
+:class:`IntConverter`
+---------------------
+
+.. class:: IntConverter(field)
+
+   Create a converter for integer fields.
+
+   `field` must be an object implementing
+   ``zope.schema.interfaces.IInt``.
+
+   Instances of this class are available as adapters from :mod:`IInt`
+   to :mod:`ISchemaTypeConverter`.
+
+   .. method:: adapts(zope.schema.IInt)
+
+   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
+
+   .. method:: fromString(string[, strict=True])
+
+       Compute an integer number value from the given `string`. Be
+       aware that this might fail if no conversion of the delivered
+       data is possible.
+
+       Use `strict` to enable/disable validations of transformed
+       data.
+
+   .. method:: toString(value[, strict=True])
+
+       Return `value` as string or ``None``.
+
+       If `value` is ``None`` (or field's `missing_value`), ``None``
+       is returned.
+
+       If the value is not valid for the applied field (i.e. not an
+       integer or `missing_value`) an exception
+       might be raised.
+
+       Use `strict` to enable/disable validations of input data.
+
+
+
+:class:`ChoiceConverter`
+------------------------
+
+.. class:: ChoiceConverter(field)
+
+   Create a converter for choice fields.
+
+   `field` must be an object implementing
+   ``zope.schema.interfaces.IChoice``.
+
+   Instances of this class are available as adapters from :mod:`IChoice`
+   to :mod:`ISchemaTypeConverter`.
+
+   .. method:: adapts(zope.schema.IChoice)
+
+   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
+
+   .. method:: fromString(string[, strict=False])
+
+       Compute a value from the given `string`. Be aware that this
+       might fail if no conversion of the delivered data is possible.
+
+       As choices generally require a certain set of allowed values,
+       only these values be allowed, in their tokenized form. The
+       converter takes the given string as token and then tries to
+       generate the appropriate value out of it (see examples below).
+
+       Use `strict` to enable/disable validations of transformed
+       data.
+
+       As an invalid value will result in a `KeyError`, additional
+       validation checks are disabled with this converter. This speeds
+       up things as expensive lookups are avoided.
+
+   .. method:: toString(value[, strict=False])
+
+       Return `value` as string or ``None``.
+
+       If `value` is ``None`` (or field's `missing_value`), ``None``
+       is returned.
+
+       If the value is not valid for the applied field (i.e. not an
+       integer or `missing_value`) an exception
+       might be raised.
+
+       Use `strict` to enable/disable validations of input data.
+
+       As an invalid value will result in a `ValueError`, additional
+       validation checks are disabled with this converter. This speeds
+       up things as expensive lookups are avoided.
+
+
+:class:`DateConverter`
+----------------------
+
+.. autoclass:: waeup.ikoba.utils.converters.DateConverter
+   :members:
+   :inherited-members:
+
+
+Examples
+========
+
+Text field converters
+---------------------
+
+We create a field that also implements ``IText``, a regular
+``TextLine`` field:
+
+    >>> from zope.schema import TextLine
+    >>> field = TextLine(title=u'Some Title',
+    ...                  default=u'Default Value',
+    ...                  required=True)
+
+Now we can get a converter for this field by explicitly creating a
+:class:`TextConverter` instance:
+
+    >>> from waeup.ikoba.utils.converters import TextConverter
+    >>> converter = TextConverter(field)
+
+Or we can just grab a registered adapter:
+
+    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
+    >>> converter = ISchemaTypeConverter(field)
+
+This will select the correct converter for us automatically.
+
+    >>> converter
+    <waeup.ikoba.utils.converters.TextConverter object at 0x...>
+
+Now we can convert strings to this type:
+
+    >>> converter.fromString('blah')
+    u'blah'
+
+And back:
+
+    >>> converter.toString(u'blah')
+    'blah'
+
+Okay, not very surprising. But the field definitions can help also
+deliver values, if the given value is missing:
+
+    >>> converter.fromString(None)
+    u'Default Value'
+
+``None`` is not an acceptable value for fields which are required but
+provide no default:
+
+    >>> field = TextLine(title=u'Some Title',
+    ...                  required=True)
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString(None)
+    Traceback (most recent call last):
+    ...
+    RequiredMissing
+
+If we want to avoid this type of exception (and risk non-applicable
+data to be stored), we can use the ``strict`` parameter of
+``fromString()``:
+
+    >>> converter.fromString(None, strict=False) is None
+    True 
+
+
+If a field is not required, we will get the ``missing_value`` type in
+same case:
+
+    >>> field = TextLine(title=u'Some Title',
+    ...                  required=False)
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString(None) is None
+    True
+
+We can also set a value that indicates missing data (``None`` by default):
+
+    >>> field = TextLine(title=u'Some Title',
+    ...                  required=False,
+    ...                  missing_value=u'<NO-VALUE>')
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString(None)
+    u'<NO-VALUE>'
+
+And put it back:
+
+    >>> converter.toString(u'<NO-VALUE>') is None
+    True
+
+Bool field converters
+---------------------
+
+We create a field that also implements ``IBool``, a regular
+``Bool`` field:
+
+    >>> from zope.schema import Bool
+    >>> field = Bool(title=u'Some truth',
+    ...             default=True,
+    ...             required=True)
+
+Now we can get a converter for this field by explicitly creating a
+:class:`BoolConverter` instance:
+
+    >>> from waeup.ikoba.utils.converters import BoolConverter
+    >>> converter = BoolConverter(field)
+
+Or we can just grab a registered adapter:
+
+    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
+    >>> converter = ISchemaTypeConverter(field)
+
+This will select the correct converter for us automatically.
+
+    >>> converter
+    <waeup.ikoba.utils.converters.BoolConverter object at 0x...>
+
+Now we can convert strings to this type:
+
+    >>> converter.fromString('yes')
+    True
+
+    >>> converter.fromString('Yes')
+    True
+
+    >>> converter.fromString('yEs')
+    True
+
+    >>> converter.fromString('1')
+    True
+
+    >>> converter.fromString('True')
+    True
+
+    >>> converter.fromString('0')
+    False
+
+    >>> converter.fromString('no')
+    False
+
+    >>> converter.fromString('false')
+    False
+
+
+And back:
+
+    >>> converter.toString(True)
+    '1'
+
+    >>> converter.toString(False)
+    '0'
+
+Okay, not very surprising. But the field definitions can help also
+deliver values, if the given value is missing:
+
+    >>> converter.fromString(None)
+    True
+
+``None`` is not an acceptable value for fields which are required but
+provide no default:
+
+    >>> field = Bool(title=u'Some Truth',
+    ...             required=True)
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString(None)
+    Traceback (most recent call last):
+    ...
+    RequiredMissing
+
+If we want to avoid this type of exception (and risk non-applicable
+data to be stored), we can use the ``strict`` parameter of
+``fromString()``:
+
+    >>> converter.fromString(None, strict=False) is None
+    True
+
+The same for the inverse operation:
+
+    >>> converter.toString(None)
+    Traceback (most recent call last):
+    ...
+    RequiredMissing
+
+    >>> converter.toString(None, strict=False) is None
+    True
+
+
+If a field is not required, we will get the ``missing_value`` type in
+same case:
+
+    >>> field = Bool(title=u'Some Title',
+    ...             required=False)
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString(None) is None
+    True
+
+    >>> converter.toString(None) is None
+    True
+
+
+Int field converters
+--------------------
+
+We create a field that also implements ``IInt``, a regular
+``Int`` field:
+
+    >>> from zope.schema import Int
+    >>> field = Int(title=u'Some number',
+    ...             default=12,
+    ...             required=True)
+
+Now we can get a converter for this field by explicitly creating a
+:class:`IntConverter` instance:
+
+    >>> from waeup.ikoba.utils.converters import IntConverter
+    >>> converter = IntConverter(field)
+
+Or we can just grab a registered adapter:
+
+    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
+    >>> converter = ISchemaTypeConverter(field)
+
+This will select the correct converter for us automatically.
+
+    >>> converter
+    <waeup.ikoba.utils.converters.IntConverter object at 0x...>
+
+Now we can convert strings to this type:
+
+    >>> converter.fromString('666')
+    666
+
+And back:
+
+    >>> converter.toString(666)
+    '666'
+
+Okay, not very surprising. But the field definitions can help also
+deliver values, if the given value is missing:
+
+    >>> converter.fromString(None)
+    12
+
+``None`` is not an acceptable value for fields which are required but
+provide no default:
+
+    >>> field = Int(title=u'Some Title',
+    ...             required=True)
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString(None)
+    Traceback (most recent call last):
+    ...
+    RequiredMissing
+
+If we want to avoid this type of exception (and risk non-applicable
+data to be stored), we can use the ``strict`` parameter of
+``fromString()``:
+
+    >>> converter.fromString(None, strict=False) is None
+    True
+
+The same for the inverse operation:
+
+    >>> converter.toString(None)
+    Traceback (most recent call last):
+    ...
+    RequiredMissing
+
+    >>> converter.toString(None, strict=False) is None
+    True
+
+
+If a field is not required, we will get the ``missing_value`` type in
+same case:
+
+    >>> field = Int(title=u'Some Title',
+    ...             required=False)
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString(None) is None
+    True
+
+    >>> converter.toString(None) is None
+    True
+
+
+Choice field converters
+-----------------------
+
+Before choices really work in unit tests, we have to make sure, that
+the following adapters are registered (this is normally done during
+startup automatically):
+
+    >>> from zc.sourcefactory.browser.source import FactoredTerms
+    >>> from zc.sourcefactory.browser.token import (
+    ...   fromString, fromUnicode, fromInteger, fromPersistent)
+    >>> from zope.component import getGlobalSiteManager
+    >>> gsm = getGlobalSiteManager()
+    >>> gsm.registerAdapter(FactoredTerms)
+    >>> gsm.registerAdapter(fromString)
+    >>> gsm.registerAdapter(fromUnicode)
+    >>> gsm.registerAdapter(fromInteger)
+    >>> gsm.registerAdapter(fromPersistent)
+
+We create a field that also implements ``IChoice``, a regular
+``Choice`` field:
+
+    >>> from zope.schema import Choice
+    >>> field = Choice(title=u'Some number',
+    ...                default=u'a',
+    ...                values=['a', 'b', 'c'],
+    ...                required=True)
+
+Now we can get a converter for this field by explicitly creating a
+:class:`ChoiceConverter` instance:
+
+    >>> from waeup.ikoba.utils.converters import ChoiceConverter
+    >>> converter = ChoiceConverter(field)
+
+Or we can just grab a registered adapter:
+
+    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
+    >>> converter = ISchemaTypeConverter(field)
+
+This will select the correct converter for us automatically.
+
+    >>> converter
+    <waeup.ikoba.utils.converters.ChoiceConverter object at 0x...>
+
+Now we can convert strings to this type:
+
+    >>> converter.fromString('a')
+    'a'
+
+or values back to strings:
+
+    >>> converter.toString('a')
+    'a'
+
+Non-listed values will not be accepted:
+
+    >>> converter.fromString('nonsense')
+    Traceback (most recent call last):
+    ...
+    LookupError: nonsense
+
+Also non-string values will work:
+
+    >>> from zope.schema import Choice
+    >>> field = Choice(title=u'Some number',
+    ...                default=2,
+    ...                values=[1, 2, 3],
+    ...                required=True)
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString('1')
+    1
+
+If we define an own source, then it will cause no problems:
+
+    >>> from zc.sourcefactory.basic import BasicSourceFactory
+    >>> class CustomSource(BasicSourceFactory):
+    ...   def getValues(self):
+    ...     return [1,2,3]
+    ...    
+    ...   def getTitle(self, value):
+    ...     return "Number %s" % (value,)
+
+    >>> field = Choice(title=u'Some number',
+    ...                default=2,
+    ...                source=CustomSource(),
+    ...                required=True)
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString('1')
+    1
+
+Note, that when asking for conversion of Choice fields, you must
+deliver the token value of the source/vocabulary. This might be the
+same as the actual value as string, but it it might be different,
+depending of the kind of source/vocabulary used.
+
+Furthermore, when writing 'exporters', i.e. components that turn a
+value into a string for usage in CSV files, then you have to save the
+token value of a term.
+
+Using the :meth:`toString()` method will deliver the token for us:
+
+    >>> converter.toString(1)
+    '1'
+
+    >>> from waeup.ikoba.interfaces import SimpleIkobaVocabulary
+    >>> field = Choice(
+    ...   title = u'Favourite Dish',
+    ...   default = 0,
+    ...   vocabulary = SimpleIkobaVocabulary(
+    ...        ('N/A', 0), ('Pizza', 1),
+    ...        ('Cake', 2), ('Muffins', 3)),
+    ...   required = True,
+    ...   )
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString('0')
+    0
+
+    >>> converter.toString(1)
+    '1'
+
+A more complex (but still realistic example) for conversion of
+:mod:`zope.schema.Choice` fields follows. Here we have a special kind
+of object, the `Cave` class, and get their `name` attribute as token.
+
+    >>> class Cave(object):
+    ...   name = 'Home sweet home'
+
+    >>> def tokenizer(cave):
+    ...   return cave.name
+
+The tokenizer has to be registered as an Cave-to-IToken adapter to
+keep :mod:`zc.sourcefactory` happy:
+
+    >>> from zc.sourcefactory.interfaces import IToken
+    >>> from zope.component import getGlobalSiteManager
+    >>> gsm = getGlobalSiteManager()
+    >>> gsm.registerAdapter(tokenizer, required=(Cave,), provided=IToken)
+
+Now we finally can create two objects of `Cave` and make a field, that
+gives the choice between these two objects:
+
+    >>> obj1 = Cave()
+    >>> obj1.name = 'Wilma'
+
+    >>> obj2 = Cave()
+    >>> obj2.name = 'Fred'
+
+    >>> from zc.sourcefactory.basic import BasicSourceFactory
+    >>> class CustomSource(BasicSourceFactory):
+    ...   def getValues(self):
+    ...     return [obj1, obj2]
+    ...    
+    ...   def getTitle(self, value):
+    ...     return value.name
+
+    >>> field = Choice(title=u'Some cave',
+    ...                default=obj1,
+    ...                source=CustomSource(),
+    ...                required=True)
+
+We get a converter for this field in the usual way. The
+:meth:`waeup.ikoba.utils.converters.fromString()` method will return one of
+the objects if we feed it with ``'Wilma'`` or ``'Fred'``:
+
+    >>> converter = ISchemaTypeConverter(field)
+    >>> result = converter.fromString('Wilma')
+    >>> result
+    <Cave object at 0x...>
+
+    >>> result is obj1, result.name
+    (True, 'Wilma')
+
+If we use the converter the other way round, it will call the
+tokenizer above and deliver ``'Wilma'`` or ``'Fred'`` as they are the
+tokens of the objects in question:
+
+    >>> converter.toString(obj2)
+    'Fred'
+
+Date field converters
+---------------------
+
+We create a field that also implements ``IDate``, a regular
+``Date`` field:
+
+    >>> from zope.schema import Date
+    >>> from datetime import date
+    >>> field = Date(title=u'Some date',
+    ...             default=date(2010, 4, 1),
+    ...             required=True)
+
+Now we can get a converter for this field by explicitly creating a
+:class:`IntConverter` instance:
+
+    >>> from waeup.ikoba.utils.converters import DateConverter
+    >>> converter = DateConverter(field)
+
+Or we can just grab a registered adapter:
+
+    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
+    >>> converter = ISchemaTypeConverter(field)
+
+This will select the correct converter for us automatically.
+
+    >>> converter
+    <waeup.ikoba.utils.converters.DateConverter object at 0x...>
+
+Now we can convert strings to this type:
+
+    >>> converter.fromString('2007-04-23')
+    datetime.date(2007, 4, 23)
+
+And back:
+
+    >>> converter.toString(date(2007,4,23))
+    '2007-04-23'
+
+As string input we also accept 'DD/MM/YYYY':
+
+    >>> converter.fromString('13/05/1945')
+    datetime.date(1945, 5, 13)
+
+and some more bizarre formats:
+
+    >>> converter.fromString('1945/05/13')
+    datetime.date(1945, 5, 13)
+
+    >>> converter.fromString('1945/05/13 12:12:34 GMT+1')
+    datetime.date(1945, 5, 13)
+
+    >>> converter.fromString('1945/5/6')
+    datetime.date(1945, 5, 6)
+
+    >>> converter.fromString('06.05.1945')
+    datetime.date(1945, 5, 6)
+
+    >>> converter.fromString('6.5.1945')
+    datetime.date(1945, 5, 6)
+
+while other date formats will mean trouble:
+
+    >>> converter.fromString('first of July 1960')
+    Traceback (most recent call last):
+    ...
+    ValueError: Cannot convert to date: first of July 1960. Use YYYY-MM-DD.
+
+Okay, not very surprising. But the field definitions can help also
+deliver values, if the given value is missing:
+
+    >>> converter.fromString(None)
+    datetime.date(2010, 4, 1)
+
+``None`` is not an acceptable value for fields which are required but
+provide no default:
+
+    >>> field = Date(title=u'Some Title',
+    ...             required=True)
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString(None)
+    Traceback (most recent call last):
+    ...
+    RequiredMissing
+
+If we want to avoid this type of exception (and risk non-applicable
+data to be stored), we can use the ``strict`` parameter of
+``fromString()``:
+
+    >>> converter.fromString(None, strict=False) is None
+    True
+
+The same for the inverse operation:
+
+    >>> converter.toString(None)
+    Traceback (most recent call last):
+    ...
+    RequiredMissing
+    >>> converter.toString(None, strict=False) is None
+    True
+
+
+If a field is not required, we will get the ``missing_value`` type in
+same case:
+
+    >>> field = Date(title=u'Some Title',
+    ...             required=False)
+    >>> converter = ISchemaTypeConverter(field)
+    >>> converter.fromString(None) is None
+    True
+
+    >>> converter.toString(None) is None
+    True
Index: /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/helpers.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/helpers.txt	(revision 12990)
+++ /main/waeup.ikoba/trunk/src/waeup/ikoba/doctests/helpers.txt	(revision 12990)
@@ -0,0 +1,228 @@
+:mod:`waeup.ikoba.utils.helpers` -- Helpers for Ikoba
+***************************************************
+
+.. module:: waeup.ikoba.utils.helpers
+
+Helper functions for Ikoba.
+
+.. :doctest:
+
+:func:`remove_file_or_directory`
+================================
+
+.. function:: remove_file_or_directory(path)
+
+   Removes a file or directory given by a path. We can remove files:
+
+     >>> import os
+     >>> import tempfile
+     >>> old_location = os.getcwd()
+     >>> new_location = tempfile.mkdtemp()
+     >>> os.chdir(new_location)
+
+     >>> from waeup.ikoba.utils.helpers import remove_file_or_directory
+     >>> open('blah', 'wb').write('nonsense')
+     >>> 'blah' in os.listdir('.')
+     True
+
+     >>> remove_file_or_directory('blah')
+     >>> 'blah' in os.listdir('.')
+     False
+
+   We can remove directories:
+
+     >>> os.mkdir('blah')
+     >>> 'blah' in os.listdir('.')
+     True
+
+     >>> remove_file_or_directory('blah')
+     >>> 'blah' in os.listdir('.')
+     False
+
+
+:func:`copy_filesystem_tree`
+============================
+
+.. function:: ccopy_filesystem_tree(src_path, dst_path[, overwrite=False[, del_old=False]])
+
+   Copies the contents of an (existing) directory to another
+   (existing) directory.
+
+   :param src_path: filesystem path to copy from
+   :type  src_path: string
+   :param dst_path: filesystem path to copy to
+   :type  dst_path: string
+   :keyword overwrite: Whether exiting files with same names should be
+                     overwritten.
+   :type  overwrite: bool
+   :keyword del_old: Whether old contents in destination path should be 
+                   removed.
+   :type  del_old: bool
+   :return: List of non-copied files
+ 
+   Both directories must exist.
+
+   Unix hidden files and directories (starting with '.') are not
+   processed by this function.
+
+   Without any further parameters, we can copy complete file trees:
+
+     >>> os.mkdir('src')
+     >>> os.mkdir('dst')
+     >>> open(os.path.join('src', 'blah'), 'wb').write('nonsense')
+
+     >>> from waeup.ikoba.utils.helpers import copy_filesystem_tree
+     >>> result = copy_filesystem_tree('src', 'dst')
+
+   As a result we get a list of non-copied files:
+
+     >>> result
+     []
+
+   The created file was indeed copied:
+
+     >>> 'blah' in os.listdir('dst')
+     True
+
+   Hidden files (i.e. such starting with a dot) are not copied:
+
+     >>> open(os.path.join('src', '.blah'), 'wb').write('nonsense')
+     >>> result = copy_filesystem_tree('src', 'dst')
+     >>> '.blah' in os.listdir('dst')
+     False
+
+   This function supports some keyword parameters as explained below.
+
+Using ``overwrite``
+-------------------
+
+Boolean. If set to ``True``, any existing and same named files and
+directories in the destination dir are overwritten with copies from
+the source. Default is `False`.
+
+Normally, existing same named files in the destination are not
+overwritten:
+
+    >>> open(os.path.join('src', 'blah'), 'wb').write('newnonsense')
+    >>> result = copy_filesystem_tree('src', 'dst')
+    >>> open(os.path.join('dst', 'blah'), 'rb').read()
+    'nonsense'
+
+Instead the filename is added to the result (a list of non-copied
+files):
+
+    >>> result
+    ['blah']
+
+If, however, we use `overwrite`, the existing file will be
+overwritten:
+
+    >>> result = copy_filesystem_tree('src', 'dst', overwrite=True)
+    >>> open(os.path.join('dst', 'blah'), 'rb').read()
+    'newnonsense'
+
+    >>> result
+    []
+
+This also works for complete directories:
+
+    >>> os.mkdir(os.path.join('src', 'mydir'))
+    >>> os.mkdir(os.path.join('dst', 'mydir'))
+    >>> open(os.path.join(
+    ...   'src', 'mydir', 'blah'), 'wb').write('srcblah')
+    >>> open(os.path.join(
+    ...   'dst', 'mydir', 'blah'), 'wb').write('dstblah')
+
+    >>> result = copy_filesystem_tree('src', 'dst', overwrite=True)
+    >>> open(os.path.join('dst', 'mydir', 'blah'), 'rb').read()
+    'srcblah'
+
+
+Using ``del_old``
+-----------------
+
+Boolean. If set to ``True``, any copied files and directories will be
+removed from the src dir. Default is `False`.
+
+    >>> result = copy_filesystem_tree('src', 'dst', overwrite=True,
+    ...                                           del_old=True)
+    >>> os.listdir('src')
+    ['.blah']
+
+All files and directories are removed from src, except the hidden file
+we created in the beginning.
+
+
+Clean up:
+
+    >>> import shutil
+    >>> shutil.rmtree(new_location)
+    >>> os.chdir(old_location)
+    
+
+
+:func:`get_inner_HTML_part()`
+=============================
+
+.. function:: get_inner_HTML_part(html_code)
+
+   Get the 'inner' part out of a piece of HTML code.
+
+   Helper function mainly to extract 'real content' from already
+   rendered forms.
+
+   The term 'inner part' here means the ``<form>`` part of an HTML
+   snippet. If this cannot be found, we look for a ``<body>`` part and
+   if this cannot be found as well, we simply return the whole input.
+
+   If a ``<form>`` part can be found in an HTML snippet, this is
+   returned with all preceeding/following stuff stripped:
+
+     >>> from waeup.ikoba.utils.helpers import get_inner_HTML_part
+     >>> print get_inner_HTML_part("""<html>
+     ... <head>
+     ... </head>
+     ... <BLANKLINE>
+     ... <body>
+     ... <form action="http://localhost/mycompany/faculties/TF/add"
+     ...       method="post" class="edit-form"
+     ...       enctype="multipart/form-data">
+     ...   <h1>Add a department</h1>
+     ... </form>
+     ... </body>
+     ... </html>
+     ... """)
+     <BLANKLINE>
+     <form action="http://localhost/mycompany/faculties/TF/add"
+           method="post" class="edit-form"
+           enctype="multipart/form-data">
+     <BLANKLINE>
+       <h1>Add a department</h1>
+     </form>
+     <BLANKLINE>
+     <BLANKLINE>
+
+   If there is no ``<form>`` part, try to find any ``<body>`` part:
+
+     >>> print get_inner_HTML_part("""<html>
+     ... <head>
+     ... </head>
+     ... <BLANKLINE>
+     ... <body>
+     ...  <div>Some content</div>
+     ... </body>
+     ... </html>
+     ... """)
+     <BLANKLINE>
+      <div>Some content</div>
+     <BLANKLINE>
+
+   If there is also no ``<body>`` tag, we return the input as-is:
+
+     >>> print get_inner_HTML_part("""<div>
+     ...  <div>Some content</div>
+     ... </div>
+     ... """)
+     <div>
+      <div>Some content</div>
+     </div>
Index: in/waeup.ikoba/trunk/src/waeup/ikoba/utils/batching.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/utils/batching.txt	(revision 12989)
+++ 	(revision )
@@ -1,556 +1,0 @@
-:mod:`waeup.ikoba.utils.batching` -- Batch processing
-****************************************************
-
-Batch processing is much more than pure data import.
-
-Overview
-========
-
-Basically, it means processing CSV files in order to mass-create,
-mass-remove, or mass-update data.
-
-So you can feed CSV files to processors, that are part of
-the batch-processing mechanism.
-
-Processors
-----------
-
-Each CSV file processor
-
-* accepts a single data type identified by an interface.
-
-* knows about the places inside a site (Company) where to store,
-  remove or update the data.
-
-* can check headers before processing data.
-
-* supports the mode 'create', 'update', 'remove'.
-
-* creates log entries (optional)
-
-* creates csv files containing successful and not-successful processed
-  data respectively.
-
-Output
-------
-
-The results of processing are written to loggers, if a logger was
-given. Beside this new CSV files are created during processing:
-
-* a pending CSV file, containing datasets that could not be processed
-
-* a finished CSV file, containing datasets successfully processed.
-
-The pending file is not created if everything works fine. The
-respective path returned in that case is ``None``.
-
-The pending file (if created) is a CSV file that contains the failed
-rows appended by a column ``--ERRROR--`` in which the reasons for
-processing failures are listed.
-
-The complete paths of these files are returned. They will be in a
-temporary directory created only for this purpose. It is the caller's
-responsibility to remove the temporay directories afterwards (the
-datacenters distProcessedFiles() method takes care for that).
-
-It looks like this::
- 
-     -----+      +---------+
-    /     |      |         |              +------+
-   | .csv +----->|Batch-   |              |      |
-   |      |      |processor+----changes-->| ZODB |
-   |  +------+   |         |              |      |
-   +--|      |   |         +              +------+
-      | Mode +-->|         |                 -------+
-      |      |   |         +----outputs-+-> /       |
-      |  +----+->+---------+            |  |.pending|
-      +--|Log |  ^                      |  |        |
-         +----+  |                      |  +--------+
-           +-----++                     v
-           |Inter-|                  ----------+
-           |face  |                 /          |
-           +------+                | .finished |
-                                   |           |
-                                   +-----------+
-
-
-Creating a batch processor
-==========================
-
-We create an own batch processor for an own datatype. This datatype
-must be based on an interface that the batcher can use for converting
-data.
-
-Founding Stoneville
--------------------
-
-We start with the interface:
-
-    >>> from zope.interface import Interface
-    >>> from zope import schema
-    >>> class ICave(Interface):
-    ...   """A cave."""
-    ...   name = schema.TextLine(
-    ...     title = u'Cave name',
-    ...     default = u'Unnamed',
-    ...     required = True)
-    ...   dinoports = schema.Int(
-    ...     title = u'Number of DinoPorts (tm)',
-    ...     required = False,
-    ...     default = 1)
-    ...   owner = schema.TextLine(
-    ...     title = u'Owner name',
-    ...     required = True,
-    ...     missing_value = 'Fred Estates Inc.')
-    ...   taxpayer = schema.Bool(
-    ...     title = u'Payes taxes',
-    ...     required = True,
-    ...     default = False)
-
-Now a class that implements this interface:
-
-    >>> import grok
-    >>> class Cave(object):
-    ...   grok.implements(ICave)
-    ...   def __init__(self, name=u'Unnamed', dinoports=2,
-    ...                owner='Fred Estates Inc.', taxpayer=False):
-    ...     self.name = name
-    ...     self.dinoports = 2
-    ...     self.owner = owner
-    ...     self.taxpayer = taxpayer
-
-We also provide a factory for caves. Strictly speaking, this not
-necessary but makes the batch processor we create afterwards, better
-understandable.
-
-    >>> from zope.component import getGlobalSiteManager
-    >>> from zope.component.factory import Factory
-    >>> from zope.component.interfaces import IFactory
-    >>> gsm = getGlobalSiteManager()
-    >>> cave_maker = Factory(Cave, 'A cave', 'Buy caves here!')
-    >>> gsm.registerUtility(cave_maker, IFactory, 'Lovely Cave')
-
-Now we can create caves using a factory:
-
-    >>> from zope.component import createObject
-    >>> createObject('Lovely Cave')
-    <Cave object at 0x...>
-
-This is nice, but we still lack a place, where we can place all the
-lovely caves we want to sell.
-
-Furthermore, as a replacement for a real site, we define a place where
-all caves can be stored: Stoneville! This is a lovely place for
-upperclass cavemen (which are the only ones that can afford more than
-one dinoport).
-
-We found Stoneville:
-
-    >>> stoneville = dict()
-
-Everything in place.
-
-Now, to improve local health conditions, imagine we want to populate
-Stoneville with lots of new happy dino-hunting natives that slept on
-the bare ground in former times and had no idea of
-bathrooms. Disgusting, isn't it?
-
-Lots of cavemen need lots of caves.
-
-Of course we can do something like:
-
-    >>> cave1 = createObject('Lovely Cave')
-    >>> cave1.name = "Fred's home"
-    >>> cave1.owner = "Fred"
-    >>> stoneville[cave1.name] = cave1
-
-and Stoneville has exactly
-
-    >>> len(stoneville)
-    1
-
-inhabitant. But we don't want to do this for hundreds or thousands of
-citizens-to-be, do we?
-
-It is much easier to create a simple CSV list, where we put in all the
-data and let a batch processor do the job.
-
-The list is already here:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner,taxpayer
-    ... Barneys Home,2,Barney,1
-    ... Wilmas Asylum,1,Wilma,1
-    ... Freds Dinoburgers,10,Fred,0
-    ... Joeys Drive-in,110,Joey,0
-    ... """)
-
-All we need, is a batch processor now.
-
-    >>> from waeup.ikoba.utils.batching import BatchProcessor
-    >>> from waeup.ikoba.interfaces import IGNORE_MARKER
-    >>> class CaveProcessor(BatchProcessor):
-    ...   util_name = 'caveprocessor'
-    ...   grok.name(util_name)
-    ...   name = 'Cave Processor'
-    ...   iface = ICave
-    ...   location_fields = ['name']
-    ...   factory_name = 'Lovely Cave'
-    ...
-    ...   def parentsExist(self, row, site):
-    ...     return True
-    ...
-    ...   def getParent(self, row, site):
-    ...     return stoneville
-    ...
-    ...   def entryExists(self, row, site):
-    ...     return row['name'] in stoneville.keys()
-    ...
-    ...   def getEntry(self, row, site):
-    ...     if not self.entryExists(row, site):
-    ...       return None
-    ...     return stoneville[row['name']]
-    ...
-    ...   def delEntry(self, row, site):
-    ...     del stoneville[row['name']]
-    ...
-    ...   def addEntry(self, obj, row, site):
-    ...     stoneville[row['name']] = obj
-    ...
-    ...   def updateEntry(self, obj, row, site, filename):
-    ...     # This is not strictly necessary, as the default
-    ...     # updateEntry method does exactly the same
-    ...     for key, value in row.items():
-    ...       if value != IGNORE_MARKER:
-    ...         setattr(obj, key, value)
-
-If we also want the results being logged, we must provide a logger
-(this is optional):
-
-    >>> import logging
-    >>> logger = logging.getLogger('stoneville')
-    >>> logger.setLevel(logging.DEBUG)
-    >>> logger.propagate = False
-    >>> handler = logging.FileHandler('stoneville.log', 'w')
-    >>> logger.addHandler(handler)
-
-Create the fellows:
-
-    >>> processor = CaveProcessor()
-    >>> result = processor.doImport('newcomers.csv',
-    ...                   ['name', 'dinoports', 'owner', 'taxpayer'],
-    ...                    mode='create', user='Bob', logger=logger)
-    >>> result
-    (4, 0, '/.../newcomers.finished.csv', None)
-
-The result means: four entries were processed and no warnings
-occured. Furthermore we get filepath to a CSV file with successfully
-processed entries and a filepath to a CSV file with erraneous entries.
-As everything went well, the latter is ``None``. Let's check:
-
-    >>> sorted(stoneville.keys())
-    [u'Barneys Home', ..., u'Wilmas Asylum']
-
-The values of the Cave instances have correct type:
-
-    >>> barney = stoneville['Barneys Home']
-    >>> barney.dinoports
-    2
-
-which is a number, not a string.
-
-Apparently, when calling the processor, we gave some more info than
-only the CSV filepath. What does it all mean?
-
-While the first argument is the path to the CSV file, we also have to
-give an ordered list of headernames. These replace the header field
-names that are actually in the file. This way we can override faulty
-headers.
-
-The ``mode`` paramter tells what kind of operation we want to perform:
-``create``, ``update``, or ``remove`` data.
-
-The ``user`` parameter finally is optional and only used for logging.
-
-We can, by the way, see the results of our run in a logfile if we
-provided a logger during the call:
-
-    >>> print open('stoneville.log').read()
-    processed: newcomers.csv, create mode, 4 lines (4 successful/ 0 failed), ... s (... s/item)
-
-
-We cleanup the temporay dir created by doImport():
-
-    >>> import shutil
-    >>> import os
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-As we can see, the processing was successful. Otherwise, all problems
-could be read here as we can see, if we do the same operation again:
-
-    >>> result = processor.doImport('newcomers.csv',
-    ...                   ['name', 'dinoports', 'owner', 'taxpayer'],
-    ...                    mode='create', user='Bob', logger=logger)
-    >>> result
-    (4, 4, '/.../newcomers.finished.csv', '/.../newcomers.pending.csv')
-
-This time we also get a path to a .pending file.
-
-The log file will tell us this in more detail:
-
-    >>> print open('stoneville.log').read()
-    processed: newcomers.csv, create mode, 4 lines (4 successful/ 0 failed), ... s (... s/item)
-    processed: newcomers.csv, create mode, 4 lines (0 successful/ 4 failed), ... s (... s/item)
-
-
-This time a new file was created, which keeps all the rows we could not
-process and an additional column with error messages:
-
-    >>> print open(result[3]).read()
-    owner,name,taxpayer,dinoports,--ERRORS--
-    Barney,Barneys Home,1,2,This object already exists. Skipping.
-    Wilma,Wilmas Asylum,1,1,This object already exists. Skipping.
-    Fred,Freds Dinoburgers,0,10,This object already exists. Skipping.
-    Joey,Joeys Drive-in,0,110,This object already exists. Skipping.
-
-This way we can correct the faulty entries and afterwards retry without
-having the already processed rows in the way.
-
-We also notice, that the values of the taxpayer column are returned as
-in the input file. There we wrote '1' for ``True`` and '0' for
-``False`` (which is accepted by the converters).
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-We can also tell to ignore some cols from input by passing
-``--IGNORE--`` as col name:
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             '--IGNORE--', '--IGNORE--'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (4, 0, '...', None)
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-If something goes wrong during processing, the respective --IGNORE--
-cols won't be populated  in the resulting pending file:
-
-    >>> result = processor.doImport('newcomers.csv', ['name', 'dinoports',
-    ...                             '--IGNORE--', '--IGNORE--'],
-    ...                    mode='create', user='Bob')
-    >>> result
-    (4, 4, '...', '...')
-
-    >>> print open(result[3], 'rb').read()
-    name,dinoports,--ERRORS--
-    Barneys Home,2,This object already exists. Skipping.
-    Wilmas Asylum,1,This object already exists. Skipping.
-    Freds Dinoburgers,10,This object already exists. Skipping.
-    Joeys Drive-in,110,This object already exists. Skipping.
-
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-
-
-Updating entries
-----------------
-
-To update entries, we just call the batchprocessor in a different
-mode:
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (4, 0, '...', None)
-
-Now we want to tell, that Wilma got an extra port for her second dino:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... Wilmas Asylum,2,Wilma
-    ... """)
-
-    >>> wilma = stoneville['Wilmas Asylum']
-    >>> wilma.dinoports
-    1
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-We start the processor:
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                    'dinoports', 'owner'], mode='update', user='Bob')
-    >>> result
-    (1, 0, '...', None)
-
-    >>> wilma = stoneville['Wilmas Asylum']
-    >>> wilma.dinoports
-    2
-
-Wilma's number of dinoports raised.
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-If we try to update an unexisting entry, an error occurs:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... NOT-WILMAS-ASYLUM,2,Wilma
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (1, 1, '/.../newcomers.finished.csv', '/.../newcomers.pending.csv')
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-    
-Also invalid values will be spotted:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... Wilmas Asylum,NOT-A-NUMBER,Wilma
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (1, 1, '...', '...')
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-We can also update only some cols, leaving some out. We skip the
-'dinoports' column in the next run:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,owner
-    ... Wilmas Asylum,Barney
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name', 'owner'],
-    ...                             mode='update', user='Bob')
-    >>> result
-    (1, 0, '...', None)
-
-    >>> wilma.owner
-    u'Barney'
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-We can however, not leave out the 'location field' ('name' in our
-case), as this one tells us which entry to update:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... 2,Wilma
-    ... """)
-
-    >>> processor.doImport('newcomers.csv', ['dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    Traceback (most recent call last):
-    ...
-    FatalCSVError: Need at least columns 'name' for import!
-
-This time we get even an exception!
-
-Generally, empty strings are considered as ``None``:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... "Wilmas Asylum","","Wilma"
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob')
-    >>> result
-    (1, 0, '...', None)
-
-    >>> wilma.dinoports
-    2
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-We can tell to set dinoports to ``None`` although this is not a
-number, as we declared the field not required in the interface:
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... "Wilmas Asylum","XXX","Wilma"
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='update', user='Bob', ignore_empty=False)
-    >>> result
-    (1, 0, '...', None)
-
-    >>> wilma.dinoports is None
-    True
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-Removing entries
-----------------
-
-In 'remove' mode we can delete entries. Here validity of values in
-non-location fields doesn't matter because those fields are ignored.
-
-    >>> open('newcomers.csv', 'wb').write(
-    ... """name,dinoports,owner
-    ... "Wilmas Asylum","ILLEGAL-NUMBER",""
-    ... """)
-
-    >>> result = processor.doImport('newcomers.csv', ['name',
-    ...                             'dinoports', 'owner'],
-    ...                    mode='remove', user='Bob')
-    >>> result
-    (1, 0, '...', None)
-
-    >>> sorted(stoneville.keys())
-    [u'Barneys Home', "Fred's home", u'Freds Dinoburgers', u'Joeys Drive-in']
-
-Oops! Wilma is gone.
-
-Clean up:
-
-    >>> shutil.rmtree(os.path.dirname(result[2]))
-
-
-Clean up:
-
-    >>> import os
-    >>> os.unlink('newcomers.csv')
-    >>> os.unlink('stoneville.log')
Index: in/waeup.ikoba/trunk/src/waeup/ikoba/utils/converters.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/utils/converters.txt	(revision 12989)
+++ 	(revision )
@@ -1,861 +1,0 @@
-:mod:`waeup.ikoba.utils.converters` -- Converters
-************************************************
-
-.. module:: waeup.ikoba.utils.converters
-
-Converters for :mod:`zope.schema` based data.
-
-.. contents::
-
-
-.. :NOdoctest:
-.. :NOlayer: waeup.ikoba.testing.IkobaUnitTestLayer
-
-
-Constants
-=========
-
-.. attribute:: NONE_STRING_VALUE
-
-   This value determines, which string represents
-   'non-values'. Current setting is:
-
-      >>> from waeup.ikoba.utils.converters import NONE_STRING_VALUE
-      >>> NONE_STRING_VALUE
-      ''
-
-   If this value is found during conversion from strings, the value to
-   set will become ``None``.
-
-
-Adapters
-========
-
-The :mod:`waeup.ikoba.utils.converters` module is basically a collection of
-adapters for field types of :mod:`zope.schema`.
-
-:class:`Converter`
-------------------
-
-.. class:: Converter(field)
-
-   Base class for converter classes. You cannot create working
-   instances of this class. Instead derive from it and implement the
-   missing classes.
-
-   `field` should be an object implementing some interface from
-   :mod:`zope.schema` like :class:`zope.schema.TextLine`,
-   `zope.schema.Choice` or similar.
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: _convertValueFromString(string)
-
-       Raises :exc:`NotImplementedError`.
-
-       Called by :meth:`fromString()` to convert a given string to a
-       value appropriate for the applied field type. The string might
-       be ``None`` or ``missing_value`` from field definition.
-
-       Override this method in derived classes to get a working
-       converter.
-
-   .. method:: _convertValueToString(value)
-
-       Raises :exc:`NotImplementedError`.
-
-       Called by :meth:`toString()` to convert a given value to a
-       string suitable to be stored for instance in CSV files.
-
-       Override this method in derived classes to get a working
-       converter.
-
-   .. method:: fromString(string[, strict=True])
-
-       Compute a value for the context field from the given
-       `string`. Be aware that this might fail for several reasons,
-       for instance because the string cannot be converted to an
-       integer (if the applied field is an Integer field), etc.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-   .. method:: toString(value[, strict=True])
-
-       Generate a string from ``value``, suitable to be stored in CSV
-       files or similar. This method does preliminary checks, handles
-       defaults and ``missing_value`` stuff and then calls
-       :meth:`_convertValueToString(value)` to do the actual
-       conversion.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-       
-
-
-:class:`TextConverter`
-----------------------
-
-.. class:: TextConverter(field)
-
-   Create a converter for text fields.
-
-   `field` must be an object implementing
-   ``zope.schema.interfaces.IText``. This is true for instance for
-   :class:`zope.schema.TextLine` fields and most other, purely
-   text-related field types.
-
-   Instances of this class are available as adapters from :mod:`IText`
-   to :mod:`ISchemaTypeConverter`.
-
-   .. method:: adapts(zope.schema.IText)
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: fromString(string[, strict=True])
-
-       Compute a unicode string value from the given `string`. Be
-       aware that this might fail if no conversion of the delivered
-       data is possible.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-   .. method:: toString(value[, strict=True])
-
-       Return `value` as string or ``None``.
-
-       If `value` is ``None`` (or field's `missing_value`), ``None``
-       is returned.
-
-       If the value is not valid for the applied field an exception
-       might be raised.
-
-       Use `strict` to enable/disable validations of input data.
-
-
-:class:`BoolConverter`
-----------------------
-
-.. class:: BoolConverter(field)
-
-   Create a converter for boolean fields.
-
-   `field` must be an object implementing
-   ``zope.schema.interfaces.IBool``.
-
-   Instances of this class are available as adapters from :mod:`IBool`
-   to :mod:`ISchemaTypeConverter`.
-
-   .. method:: adapts(zope.schema.IBool)
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: fromString(string[, strict=True])
-
-       Compute an integer number value from the given `string`. In
-       general '1', 'true', and 'yes' are interpreted as ``True``,
-       ``None`` is interpreted as 'no value' and all other values are
-       interpreted as ``True``.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-   .. method:: toString(value[, strict=True])
-
-       Return `value` as string or ``None``.
-
-       If `value` is ``None`` (or field's `missing_value`), ``None``
-       is returned. Otherwise ``0`` or ``1`` are returned.
-
-       If the value is not valid for the applied field (i.e. not an
-       integer or `missing_value`) an exception
-       might be raised.
-
-       Use `strict` to enable/disable validations of input data.
-
-
-:class:`IntConverter`
----------------------
-
-.. class:: IntConverter(field)
-
-   Create a converter for integer fields.
-
-   `field` must be an object implementing
-   ``zope.schema.interfaces.IInt``.
-
-   Instances of this class are available as adapters from :mod:`IInt`
-   to :mod:`ISchemaTypeConverter`.
-
-   .. method:: adapts(zope.schema.IInt)
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: fromString(string[, strict=True])
-
-       Compute an integer number value from the given `string`. Be
-       aware that this might fail if no conversion of the delivered
-       data is possible.
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-   .. method:: toString(value[, strict=True])
-
-       Return `value` as string or ``None``.
-
-       If `value` is ``None`` (or field's `missing_value`), ``None``
-       is returned.
-
-       If the value is not valid for the applied field (i.e. not an
-       integer or `missing_value`) an exception
-       might be raised.
-
-       Use `strict` to enable/disable validations of input data.
-
-
-
-:class:`ChoiceConverter`
-------------------------
-
-.. class:: ChoiceConverter(field)
-
-   Create a converter for choice fields.
-
-   `field` must be an object implementing
-   ``zope.schema.interfaces.IChoice``.
-
-   Instances of this class are available as adapters from :mod:`IChoice`
-   to :mod:`ISchemaTypeConverter`.
-
-   .. method:: adapts(zope.schema.IChoice)
-
-   .. method:: provides(waeup.ikoba.interfaces.ISchemaTypeConverter)
-
-   .. method:: fromString(string[, strict=False])
-
-       Compute a value from the given `string`. Be aware that this
-       might fail if no conversion of the delivered data is possible.
-
-       As choices generally require a certain set of allowed values,
-       only these values be allowed, in their tokenized form. The
-       converter takes the given string as token and then tries to
-       generate the appropriate value out of it (see examples below).
-
-       Use `strict` to enable/disable validations of transformed
-       data.
-
-       As an invalid value will result in a `KeyError`, additional
-       validation checks are disabled with this converter. This speeds
-       up things as expensive lookups are avoided.
-
-   .. method:: toString(value[, strict=False])
-
-       Return `value` as string or ``None``.
-
-       If `value` is ``None`` (or field's `missing_value`), ``None``
-       is returned.
-
-       If the value is not valid for the applied field (i.e. not an
-       integer or `missing_value`) an exception
-       might be raised.
-
-       Use `strict` to enable/disable validations of input data.
-
-       As an invalid value will result in a `ValueError`, additional
-       validation checks are disabled with this converter. This speeds
-       up things as expensive lookups are avoided.
-
-
-:class:`DateConverter`
-----------------------
-
-.. autoclass:: waeup.ikoba.utils.converters.DateConverter
-   :members:
-   :inherited-members:
-
-
-Examples
-========
-
-Text field converters
----------------------
-
-We create a field that also implements ``IText``, a regular
-``TextLine`` field:
-
-    >>> from zope.schema import TextLine
-    >>> field = TextLine(title=u'Some Title',
-    ...                  default=u'Default Value',
-    ...                  required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`TextConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import TextConverter
-    >>> converter = TextConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.TextConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('blah')
-    u'blah'
-
-And back:
-
-    >>> converter.toString(u'blah')
-    'blah'
-
-Okay, not very surprising. But the field definitions can help also
-deliver values, if the given value is missing:
-
-    >>> converter.fromString(None)
-    u'Default Value'
-
-``None`` is not an acceptable value for fields which are required but
-provide no default:
-
-    >>> field = TextLine(title=u'Some Title',
-    ...                  required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-If we want to avoid this type of exception (and risk non-applicable
-data to be stored), we can use the ``strict`` parameter of
-``fromString()``:
-
-    >>> converter.fromString(None, strict=False) is None
-    True 
-
-
-If a field is not required, we will get the ``missing_value`` type in
-same case:
-
-    >>> field = TextLine(title=u'Some Title',
-    ...                  required=False)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None) is None
-    True
-
-We can also set a value that indicates missing data (``None`` by default):
-
-    >>> field = TextLine(title=u'Some Title',
-    ...                  required=False,
-    ...                  missing_value=u'<NO-VALUE>')
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    u'<NO-VALUE>'
-
-And put it back:
-
-    >>> converter.toString(u'<NO-VALUE>') is None
-    True
-
-Bool field converters
----------------------
-
-We create a field that also implements ``IBool``, a regular
-``Bool`` field:
-
-    >>> from zope.schema import Bool
-    >>> field = Bool(title=u'Some truth',
-    ...             default=True,
-    ...             required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`BoolConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import BoolConverter
-    >>> converter = BoolConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.BoolConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('yes')
-    True
-
-    >>> converter.fromString('Yes')
-    True
-
-    >>> converter.fromString('yEs')
-    True
-
-    >>> converter.fromString('1')
-    True
-
-    >>> converter.fromString('True')
-    True
-
-    >>> converter.fromString('0')
-    False
-
-    >>> converter.fromString('no')
-    False
-
-    >>> converter.fromString('false')
-    False
-
-
-And back:
-
-    >>> converter.toString(True)
-    '1'
-
-    >>> converter.toString(False)
-    '0'
-
-Okay, not very surprising. But the field definitions can help also
-deliver values, if the given value is missing:
-
-    >>> converter.fromString(None)
-    True
-
-``None`` is not an acceptable value for fields which are required but
-provide no default:
-
-    >>> field = Bool(title=u'Some Truth',
-    ...             required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-If we want to avoid this type of exception (and risk non-applicable
-data to be stored), we can use the ``strict`` parameter of
-``fromString()``:
-
-    >>> converter.fromString(None, strict=False) is None
-    True
-
-The same for the inverse operation:
-
-    >>> converter.toString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-    >>> converter.toString(None, strict=False) is None
-    True
-
-
-If a field is not required, we will get the ``missing_value`` type in
-same case:
-
-    >>> field = Bool(title=u'Some Title',
-    ...             required=False)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None) is None
-    True
-
-    >>> converter.toString(None) is None
-    True
-
-
-Int field converters
---------------------
-
-We create a field that also implements ``IInt``, a regular
-``Int`` field:
-
-    >>> from zope.schema import Int
-    >>> field = Int(title=u'Some number',
-    ...             default=12,
-    ...             required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`IntConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import IntConverter
-    >>> converter = IntConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.IntConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('666')
-    666
-
-And back:
-
-    >>> converter.toString(666)
-    '666'
-
-Okay, not very surprising. But the field definitions can help also
-deliver values, if the given value is missing:
-
-    >>> converter.fromString(None)
-    12
-
-``None`` is not an acceptable value for fields which are required but
-provide no default:
-
-    >>> field = Int(title=u'Some Title',
-    ...             required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-If we want to avoid this type of exception (and risk non-applicable
-data to be stored), we can use the ``strict`` parameter of
-``fromString()``:
-
-    >>> converter.fromString(None, strict=False) is None
-    True
-
-The same for the inverse operation:
-
-    >>> converter.toString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-    >>> converter.toString(None, strict=False) is None
-    True
-
-
-If a field is not required, we will get the ``missing_value`` type in
-same case:
-
-    >>> field = Int(title=u'Some Title',
-    ...             required=False)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None) is None
-    True
-
-    >>> converter.toString(None) is None
-    True
-
-
-Choice field converters
------------------------
-
-Before choices really work in unit tests, we have to make sure, that
-the following adapters are registered (this is normally done during
-startup automatically):
-
-    >>> from zc.sourcefactory.browser.source import FactoredTerms
-    >>> from zc.sourcefactory.browser.token import (
-    ...   fromString, fromUnicode, fromInteger, fromPersistent)
-    >>> from zope.component import getGlobalSiteManager
-    >>> gsm = getGlobalSiteManager()
-    >>> gsm.registerAdapter(FactoredTerms)
-    >>> gsm.registerAdapter(fromString)
-    >>> gsm.registerAdapter(fromUnicode)
-    >>> gsm.registerAdapter(fromInteger)
-    >>> gsm.registerAdapter(fromPersistent)
-
-We create a field that also implements ``IChoice``, a regular
-``Choice`` field:
-
-    >>> from zope.schema import Choice
-    >>> field = Choice(title=u'Some number',
-    ...                default=u'a',
-    ...                values=['a', 'b', 'c'],
-    ...                required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`ChoiceConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import ChoiceConverter
-    >>> converter = ChoiceConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.ChoiceConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('a')
-    'a'
-
-or values back to strings:
-
-    >>> converter.toString('a')
-    'a'
-
-Non-listed values will not be accepted:
-
-    >>> converter.fromString('nonsense')
-    Traceback (most recent call last):
-    ...
-    LookupError: nonsense
-
-Also non-string values will work:
-
-    >>> from zope.schema import Choice
-    >>> field = Choice(title=u'Some number',
-    ...                default=2,
-    ...                values=[1, 2, 3],
-    ...                required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString('1')
-    1
-
-If we define an own source, then it will cause no problems:
-
-    >>> from zc.sourcefactory.basic import BasicSourceFactory
-    >>> class CustomSource(BasicSourceFactory):
-    ...   def getValues(self):
-    ...     return [1,2,3]
-    ...    
-    ...   def getTitle(self, value):
-    ...     return "Number %s" % (value,)
-
-    >>> field = Choice(title=u'Some number',
-    ...                default=2,
-    ...                source=CustomSource(),
-    ...                required=True)
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString('1')
-    1
-
-Note, that when asking for conversion of Choice fields, you must
-deliver the token value of the source/vocabulary. This might be the
-same as the actual value as string, but it it might be different,
-depending of the kind of source/vocabulary used.
-
-Furthermore, when writing 'exporters', i.e. components that turn a
-value into a string for usage in CSV files, then you have to save the
-token value of a term.
-
-Using the :meth:`toString()` method will deliver the token for us:
-
-    >>> converter.toString(1)
-    '1'
-
-    >>> from waeup.ikoba.interfaces import SimpleIkobaVocabulary
-    >>> field = Choice(
-    ...   title = u'Favourite Dish',
-    ...   default = 0,
-    ...   vocabulary = SimpleIkobaVocabulary(
-    ...        ('N/A', 0), ('Pizza', 1),
-    ...        ('Cake', 2), ('Muffins', 3)),
-    ...   required = True,
-    ...   )
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString('0')
-    0
-
-    >>> converter.toString(1)
-    '1'
-
-A more complex (but still realistic example) for conversion of
-:mod:`zope.schema.Choice` fields follows. Here we have a special kind
-of object, the `Cave` class, and get their `name` attribute as token.
-
-    >>> class Cave(object):
-    ...   name = 'Home sweet home'
-
-    >>> def tokenizer(cave):
-    ...   return cave.name
-
-The tokenizer has to be registered as an Cave-to-IToken adapter to
-keep :mod:`zc.sourcefactory` happy:
-
-    >>> from zc.sourcefactory.interfaces import IToken
-    >>> from zope.component import getGlobalSiteManager
-    >>> gsm = getGlobalSiteManager()
-    >>> gsm.registerAdapter(tokenizer, required=(Cave,), provided=IToken)
-
-Now we finally can create two objects of `Cave` and make a field, that
-gives the choice between these two objects:
-
-    >>> obj1 = Cave()
-    >>> obj1.name = 'Wilma'
-
-    >>> obj2 = Cave()
-    >>> obj2.name = 'Fred'
-
-    >>> from zc.sourcefactory.basic import BasicSourceFactory
-    >>> class CustomSource(BasicSourceFactory):
-    ...   def getValues(self):
-    ...     return [obj1, obj2]
-    ...    
-    ...   def getTitle(self, value):
-    ...     return value.name
-
-    >>> field = Choice(title=u'Some cave',
-    ...                default=obj1,
-    ...                source=CustomSource(),
-    ...                required=True)
-
-We get a converter for this field in the usual way. The
-:meth:`waeup.ikoba.utils.converters.fromString()` method will return one of
-the objects if we feed it with ``'Wilma'`` or ``'Fred'``:
-
-    >>> converter = ISchemaTypeConverter(field)
-    >>> result = converter.fromString('Wilma')
-    >>> result
-    <Cave object at 0x...>
-
-    >>> result is obj1, result.name
-    (True, 'Wilma')
-
-If we use the converter the other way round, it will call the
-tokenizer above and deliver ``'Wilma'`` or ``'Fred'`` as they are the
-tokens of the objects in question:
-
-    >>> converter.toString(obj2)
-    'Fred'
-
-Date field converters
----------------------
-
-We create a field that also implements ``IDate``, a regular
-``Date`` field:
-
-    >>> from zope.schema import Date
-    >>> from datetime import date
-    >>> field = Date(title=u'Some date',
-    ...             default=date(2010, 4, 1),
-    ...             required=True)
-
-Now we can get a converter for this field by explicitly creating a
-:class:`IntConverter` instance:
-
-    >>> from waeup.ikoba.utils.converters import DateConverter
-    >>> converter = DateConverter(field)
-
-Or we can just grab a registered adapter:
-
-    >>> from waeup.ikoba.interfaces import ISchemaTypeConverter
-    >>> converter = ISchemaTypeConverter(field)
-
-This will select the correct converter for us automatically.
-
-    >>> converter
-    <waeup.ikoba.utils.converters.DateConverter object at 0x...>
-
-Now we can convert strings to this type:
-
-    >>> converter.fromString('2007-04-23')
-    datetime.date(2007, 4, 23)
-
-And back:
-
-    >>> converter.toString(date(2007,4,23))
-    '2007-04-23'
-
-As string input we also accept 'DD/MM/YYYY':
-
-    >>> converter.fromString('13/05/1945')
-    datetime.date(1945, 5, 13)
-
-and some more bizarre formats:
-
-    >>> converter.fromString('1945/05/13')
-    datetime.date(1945, 5, 13)
-
-    >>> converter.fromString('1945/05/13 12:12:34 GMT+1')
-    datetime.date(1945, 5, 13)
-
-    >>> converter.fromString('1945/5/6')
-    datetime.date(1945, 5, 6)
-
-    >>> converter.fromString('06.05.1945')
-    datetime.date(1945, 5, 6)
-
-    >>> converter.fromString('6.5.1945')
-    datetime.date(1945, 5, 6)
-
-while other date formats will mean trouble:
-
-    >>> converter.fromString('first of July 1960')
-    Traceback (most recent call last):
-    ...
-    ValueError: Cannot convert to date: first of July 1960. Use YYYY-MM-DD.
-
-Okay, not very surprising. But the field definitions can help also
-deliver values, if the given value is missing:
-
-    >>> converter.fromString(None)
-    datetime.date(2010, 4, 1)
-
-``None`` is not an acceptable value for fields which are required but
-provide no default:
-
-    >>> field = Date(title=u'Some Title',
-    ...             required=True)
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-
-If we want to avoid this type of exception (and risk non-applicable
-data to be stored), we can use the ``strict`` parameter of
-``fromString()``:
-
-    >>> converter.fromString(None, strict=False) is None
-    True
-
-The same for the inverse operation:
-
-    >>> converter.toString(None)
-    Traceback (most recent call last):
-    ...
-    RequiredMissing
-    >>> converter.toString(None, strict=False) is None
-    True
-
-
-If a field is not required, we will get the ``missing_value`` type in
-same case:
-
-    >>> field = Date(title=u'Some Title',
-    ...             required=False)
-    >>> converter = ISchemaTypeConverter(field)
-    >>> converter.fromString(None) is None
-    True
-
-    >>> converter.toString(None) is None
-    True
Index: in/waeup.ikoba/trunk/src/waeup/ikoba/utils/helpers.txt
===================================================================
--- /main/waeup.ikoba/trunk/src/waeup/ikoba/utils/helpers.txt	(revision 12989)
+++ 	(revision )
@@ -1,228 +1,0 @@
-:mod:`waeup.ikoba.utils.helpers` -- Helpers for Ikoba
-***************************************************
-
-.. module:: waeup.ikoba.utils.helpers
-
-Helper functions for Ikoba.
-
-.. :doctest:
-
-:func:`remove_file_or_directory`
-================================
-
-.. function:: remove_file_or_directory(path)
-
-   Removes a file or directory given by a path. We can remove files:
-
-     >>> import os
-     >>> import tempfile
-     >>> old_location = os.getcwd()
-     >>> new_location = tempfile.mkdtemp()
-     >>> os.chdir(new_location)
-
-     >>> from waeup.ikoba.utils.helpers import remove_file_or_directory
-     >>> open('blah', 'wb').write('nonsense')
-     >>> 'blah' in os.listdir('.')
-     True
-
-     >>> remove_file_or_directory('blah')
-     >>> 'blah' in os.listdir('.')
-     False
-
-   We can remove directories:
-
-     >>> os.mkdir('blah')
-     >>> 'blah' in os.listdir('.')
-     True
-
-     >>> remove_file_or_directory('blah')
-     >>> 'blah' in os.listdir('.')
-     False
-
-
-:func:`copy_filesystem_tree`
-============================
-
-.. function:: ccopy_filesystem_tree(src_path, dst_path[, overwrite=False[, del_old=False]])
-
-   Copies the contents of an (existing) directory to another
-   (existing) directory.
-
-   :param src_path: filesystem path to copy from
-   :type  src_path: string
-   :param dst_path: filesystem path to copy to
-   :type  dst_path: string
-   :keyword overwrite: Whether exiting files with same names should be
-                     overwritten.
-   :type  overwrite: bool
-   :keyword del_old: Whether old contents in destination path should be 
-                   removed.
-   :type  del_old: bool
-   :return: List of non-copied files
- 
-   Both directories must exist.
-
-   Unix hidden files and directories (starting with '.') are not
-   processed by this function.
-
-   Without any further parameters, we can copy complete file trees:
-
-     >>> os.mkdir('src')
-     >>> os.mkdir('dst')
-     >>> open(os.path.join('src', 'blah'), 'wb').write('nonsense')
-
-     >>> from waeup.ikoba.utils.helpers import copy_filesystem_tree
-     >>> result = copy_filesystem_tree('src', 'dst')
-
-   As a result we get a list of non-copied files:
-
-     >>> result
-     []
-
-   The created file was indeed copied:
-
-     >>> 'blah' in os.listdir('dst')
-     True
-
-   Hidden files (i.e. such starting with a dot) are not copied:
-
-     >>> open(os.path.join('src', '.blah'), 'wb').write('nonsense')
-     >>> result = copy_filesystem_tree('src', 'dst')
-     >>> '.blah' in os.listdir('dst')
-     False
-
-   This function supports some keyword parameters as explained below.
-
-Using ``overwrite``
--------------------
-
-Boolean. If set to ``True``, any existing and same named files and
-directories in the destination dir are overwritten with copies from
-the source. Default is `False`.
-
-Normally, existing same named files in the destination are not
-overwritten:
-
-    >>> open(os.path.join('src', 'blah'), 'wb').write('newnonsense')
-    >>> result = copy_filesystem_tree('src', 'dst')
-    >>> open(os.path.join('dst', 'blah'), 'rb').read()
-    'nonsense'
-
-Instead the filename is added to the result (a list of non-copied
-files):
-
-    >>> result
-    ['blah']
-
-If, however, we use `overwrite`, the existing file will be
-overwritten:
-
-    >>> result = copy_filesystem_tree('src', 'dst', overwrite=True)
-    >>> open(os.path.join('dst', 'blah'), 'rb').read()
-    'newnonsense'
-
-    >>> result
-    []
-
-This also works for complete directories:
-
-    >>> os.mkdir(os.path.join('src', 'mydir'))
-    >>> os.mkdir(os.path.join('dst', 'mydir'))
-    >>> open(os.path.join(
-    ...   'src', 'mydir', 'blah'), 'wb').write('srcblah')
-    >>> open(os.path.join(
-    ...   'dst', 'mydir', 'blah'), 'wb').write('dstblah')
-
-    >>> result = copy_filesystem_tree('src', 'dst', overwrite=True)
-    >>> open(os.path.join('dst', 'mydir', 'blah'), 'rb').read()
-    'srcblah'
-
-
-Using ``del_old``
------------------
-
-Boolean. If set to ``True``, any copied files and directories will be
-removed from the src dir. Default is `False`.
-
-    >>> result = copy_filesystem_tree('src', 'dst', overwrite=True,
-    ...                                           del_old=True)
-    >>> os.listdir('src')
-    ['.blah']
-
-All files and directories are removed from src, except the hidden file
-we created in the beginning.
-
-
-Clean up:
-
-    >>> import shutil
-    >>> shutil.rmtree(new_location)
-    >>> os.chdir(old_location)
-    
-
-
-:func:`get_inner_HTML_part()`
-=============================
-
-.. function:: get_inner_HTML_part(html_code)
-
-   Get the 'inner' part out of a piece of HTML code.
-
-   Helper function mainly to extract 'real content' from already
-   rendered forms.
-
-   The term 'inner part' here means the ``<form>`` part of an HTML
-   snippet. If this cannot be found, we look for a ``<body>`` part and
-   if this cannot be found as well, we simply return the whole input.
-
-   If a ``<form>`` part can be found in an HTML snippet, this is
-   returned with all preceeding/following stuff stripped:
-
-     >>> from waeup.ikoba.utils.helpers import get_inner_HTML_part
-     >>> print get_inner_HTML_part("""<html>
-     ... <head>
-     ... </head>
-     ... <BLANKLINE>
-     ... <body>
-     ... <form action="http://localhost/mycompany/faculties/TF/add"
-     ...       method="post" class="edit-form"
-     ...       enctype="multipart/form-data">
-     ...   <h1>Add a department</h1>
-     ... </form>
-     ... </body>
-     ... </html>
-     ... """)
-     <BLANKLINE>
-     <form action="http://localhost/mycompany/faculties/TF/add"
-           method="post" class="edit-form"
-           enctype="multipart/form-data">
-     <BLANKLINE>
-       <h1>Add a department</h1>
-     </form>
-     <BLANKLINE>
-     <BLANKLINE>
-
-   If there is no ``<form>`` part, try to find any ``<body>`` part:
-
-     >>> print get_inner_HTML_part("""<html>
-     ... <head>
-     ... </head>
-     ... <BLANKLINE>
-     ... <body>
-     ...  <div>Some content</div>
-     ... </body>
-     ... </html>
-     ... """)
-     <BLANKLINE>
-      <div>Some content</div>
-     <BLANKLINE>
-
-   If there is also no ``<body>`` tag, we return the input as-is:
-
-     >>> print get_inner_HTML_part("""<div>
-     ...  <div>Some content</div>
-     ... </div>
-     ... """)
-     <div>
-      <div>Some content</div>
-     </div>
