source: waeup/branches/ulif-rewrite/src/waeup/csvfile/README.txt @ 4288

:mod:`waeup.csvfile` -- generic support for handling CSV files
**************************************************************

:Test-Layer: unit

.. module:: waeup.csvfile
   :synopsis: generic support for handling CSV files.


.. note::

   This version of the :mod:`waeup.csvfile` module doesn't support
   Unicode input.  Also, there are currently some issues regarding
   ASCII NUL characters.  Accordingly, all input should be UTF-8 or
   printable ASCII to be safe. These restrictions will be removed in
   the future.


Module Contents
================

:class:`CSVFile`
----------------

.. class:: CSVFile(filepath)

   Wrapper around the path to a real CSV file. :class:`CSVFile` is an
   adapter that adapts basestring objects (aka regular and unicode
   strings).

   :class:`CSVFile` is designed as a base for derived, more
   specialized types of CSV files wrappers, although it can serve as a
   basic wrapper for simple, unspecified CSV files.

   .. method:: grok.context(basestring)
      :noindex:

      We bind to basestring objects.

   .. method:: grok.implements(ICSVFile)
      :noindex:

   .. attribute:: required_fields=[]

      A list of header fields (strings) required for this kind of
      CSVFile. Using the default constructor will fail with paths to
      files that do *not* provide those fields.

      The defaul value (empty list) means: no special fields required
      at all.

      Deriving classes can override this attribute to accept only
      files that provide the appropriate header fields. The default
      constructor already checks this.

   .. attribute:: path

      A string describing the path to the associated CSV file.

   .. method:: getData()

      Returns a generator delivering one data row of the denoted file
      at a time.

      Each data row is delivered as a dictionary mapping from header
      field names to the values. Therefore a source line like this::

        field1,field2
        ...
        data 1,data 2

      will become a dictionary like this::

        {'field1' : 'data 1', 
         'field2' : 'data 2'}

      The :meth:`getData` method does not evaluate file
      values for correct data types or so.

   .. method:: getHeaderFields()

      Get a sorted list of header fields in the wrapped CSV file.


:func:`getCSVFile`
----------------------

.. function:: getCSVFile(filepath)

   `filepath`
      String with a filepath to an existing CSV file.

   Get a CSV file wrapper for the given `filepath`. :func:`getCSVFile`
   knows about all registered :class:`CSVFile` wrappers registered and
   searches them for the most appropriate one.

   If none can be found ``None`` is returned.

   .. seealso:: 

      :ref:`getcsvfilewrapper`,

      :ref:`getcsvfiledecision`

Basic example
=============

To initialize the whole framework we have to grok the :mod:`waeup`
package first:

    >>> import grok
    >>> grok.testing.grok('waeup')

Create a file:

    >>> path = 'mycsvfile.csv'
    >>> open(path, 'wb').write(
    ... """col1,col2
    ... dataitem1,dataitem2
    ... item3,item4
    ... """)

A regular file path is difficult to handle in terms of a component
framework. Therefore we get a wrapper for it, that makes a CSV file
object out of a path string:

    >>> from waeup.csvfile.interfaces import ICSVFile
    >>> src = ICSVFile(path)
    >>> src
    <waeup.csvfile.csvfile.CSVFile object at 0x...>

Create a receiver:

    >>> from waeup.csvfile.interfaces import ICSVDataReceiver
    >>> class Receiver(object):
    ...   grok.implements(ICSVDataReceiver)
    ...   def receive(self, data):
    ...     print "RECEIVED: ", data

    >>> recv = Receiver()

Find a connector:

    >>> from zope.component import getMultiAdapter
    >>> from waeup.csvfile.interfaces import ICSVDataConnector
    >>> conn = getMultiAdapter((src, recv), ICSVDataConnector)
    Traceback (most recent call last):
    ...
    ComponentLookupError: ((<waeup.csvfile.csvfile.CSVFile object at 0x...>, 
                            <Receiver object at 0x...>),
        <InterfaceClass waeup.csvfile.interfaces.ICSVDataConnector>, u'')

Okay, create a connector:

    >>> class Connector1(grok.MultiAdapter):
    ...   grok.adapts(ICSVFile, ICSVDataReceiver)
    ...   grok.provides(ICSVDataConnector)
    ...   def __init__(self, source, receiver):
    ...     self.source = source
    ...     self.receiver = receiver
    ...   def doImport(self):
    ...     self.receiver.receive(
    ...        self.source.getData())

    >>> grok.testing.grok_component('Connector1', Connector1)
    True

Try again...

    >>> conn = getMultiAdapter((src, recv), ICSVDataConnector)
    >>> conn
    <Connector1 object at 0x...>

    >>> conn.doImport()
    RECEIVED: <generator object at 0x...>

Clean up:

    >>> import os
    >>> os.unlink(path)


CSV file wrappers
=================

CSV file wrappers can extract data from CSV files denoted by a path.

We create a CSV file:

    >>> path = 'mycsvfile.csv'
    >>> open(path, 'wb').write(
    ... """col1,col2
    ... dataitem1,dataitem2
    ... item3,item4
    ... """)

Now we get a CSV file wrapper for it. This is simply done by asking
for an adapter to the path string:

    >>> from waeup.csvfile.interfaces import ICSVFile
    >>> wrapper = ICSVFile(path)
    >>> wrapper
    <waeup.csvfile.csvfile.CSVFile object at 0x...>

This wrapper can return the CSV data as a sequence of dicts:

    >>> wrapper.getData()
    <generator object at 0x...>

As we see, the single dicts (each representing a row) are returned as
a generator. We can list them:

    >>> list(wrapper.getData())
    [{'col2': 'dataitem2', 'col1': 'dataitem1'}, 
     {'col2': 'item4', 'col1': 'item3'}]

We can get a list of headerfields found in the file:

    >>> wrapper.getHeaderFields()
    ['col1', 'col2']

.. _getcsvfilewrapper:

Getting a wrapper
=================

If we want to get a wrapper best suited for our purposes, we can also
use the :func:`getCSVFile` function:

    >>> from waeup.csvfile.csvfile import getCSVFile
    >>> wrapper = getCSVFile(path)
    >>> wrapper
    <waeup.csvfile.csvfile.CSVFile object at 0x...>

As we currently have only one type of wrapper, we get this. Let's
create another wrapper, that requires a column 'col1':

    >>> from waeup.csvfile.interfaces import ICSVFile
    >>> from waeup.csvfile import CSVFile
    >>> class ICSVFileWithCol1(ICSVFile):
    ...   """A CSV file that contains a 'col1' column.
    ...   """

    >>> class CSVFileWithCol1(CSVFile):
    ...   required_fields = ['col1']
    ...   grok.implements(ICSVFileWithCol1)
    ...   grok.provides(ICSVFileWithCol1)

We have to grok:

    >>> grok.testing.grok_component('CSVFileWithCol1', CSVFileWithCol1)
    True

Now we can ask for a wrapper again, but this time we will get a
CSVFileWithCol12 instance:

    >>> getCSVFile(path)
    <CSVFileWithCol1 object at 0x...>

If we cannot get a wrapper at all, ``None`` is returned:

    >>> getCSVFile('not-existent-file') is None
    True

.. _getcsvfiledecision:


How :func:`getCSVFile` decides which wrapper to use
---------------------------------------------------

Apparently, :func:`getCSVFile` performes some magic: given a certain
CSV file, it decides which one of all registered wrappers suits the
file best.

This decision is based on a score, which is computed as shown below
for each registered wrapper.

Before we can show this, we create some more CSV files.

One file that does not contain valid CSV data:

     >>> nocsvpath = 'nocsvfile.csv'
     >>> open(nocsvpath, 'wb').write(
     ... """blah blah blah.
     ... blubb blubb.
     ... """)

One file that contains a 'col1' and a 'col3' column:

    >>> path2 = 'mycsvfile2.csv'
    >>> open(path2, 'wb').write(
    ... """col1,col3
    ... dataitem1b,dataitem2b
    ... item3b,item4b
    ... """)

We create a wrapper that requires 'col1' and 'col2':

    >>> from waeup.csvfile.interfaces import ICSVFile
    >>> from waeup.csvfile import CSVFile
    >>> class ICSVFile12(ICSVFile):
    ...   """A CSV file that contains a 'special_col' column.
    ...   """

    >>> class CSVFile12(CSVFile):
    ...   required_fields = ['col1', 'col2']
    ...   grok.context(basestring)
    ...   grok.implements(ICSVFile12)
    ...   grok.provides(ICSVFile12)

    >>> grok.testing.grok_component('CSVFile12',  CSVFile12)
    True


* If no instance of a certain wrapper can be created from the given
  path (i.e. __init__ raises some kind of exception): score is -1:

    >>> from waeup.csvfile.csvfile import getScore
    >>> getScore('nonexistant', CSVFile)
    -1

* If a wrapper requires at least one header_field and the given file
  does not provide all of the required fields: score is -1:

    >>> getScore(path2, CSVFile12)
    -1

* If a wrapper requires no header fields at all
  (i.e. `required_fields` equals empty list): score is 0 (zero):

    >>> getScore(path, CSVFile)
    0

* If a wrapper requires at least one header_field and all header
  fields do also appear in the file: score is number of required
  fields.

    >>> getScore(path, CSVFileWithCol1)
    1

    >>> getScore(path, CSVFile12)
    2

If several wrappers get the same score for a certain file, the result
is not determined.


How to build custom CSV file wrappers
=====================================

A typical CSV file wrapper can be built like this:

    >>> import grok
    >>> from waeup.csvfile.interfaces import ICSVFile
    >>> from waeup.csvfile import CSVFile

    >>> class ICustomCSVFile(ICSVFile):
    ...   """A marker for custom CSV files."""

    >>> class CustomCSVFile(CSVFile):
    ...   required_fields = ['somecol', 'othercol']
    ...   grok.implements(ICustomCSVFile)
    ...   grok.provides(ICustomCSVFile)

    >>> grok.testing.grok_component('CustomCSVFile',  CustomCSVFile)
    True

The special things here are:

* Derive from :class:`CSVFile`

  :func:`getCSVFile` looks only for classes that are derived from
  :class:`CSVFile`. So if you want your wrapper to be found by this function,
  derive from :class:`CSVFile`.

  As :class:`CSVFile` is an adapter, also our custom wrapper will
  become one (adapting strings):

     >>> ICustomCSVFile(path)
     Traceback (most recent call last):
     ...
     TypeError: Missing columns in CSV file: ['somecol', 'othercol']

  If our input file provides the correct columns, it will work:

     >>> custompath = 'mycustom.csv'
     >>> open(custompath, 'wb').write(
     ... """somecol,othercol,thirdcol
     ... dataitem1,dataitem2,dataitem3
     ... """)

     >>> ICustomCSVFile(custompath)
     <CustomCSVFile object at 0x...>

* Provide and implement a custom interface

  A custom CSV file wrapper should provide and implement an own
  interface. Otherwise it could not be required by other components
  explicitly.

  We have to provide *and* implement the custom interface because
  otherwise instances would not implement the required interface
  (maybe due to a flaw in `grok`/`martian`).


Common Use Cases
================

Get a wrapper for a certain type of CSV file
--------------------------------------------

The type of a :class:`CSVFile` is determined by the interfaces it
provides.

If we want to get a wrapper that also guarantees to support certain
fields (or None), then we already know about the wanted type.

We create a file that does not have a 'special_col' field:

    >>> path = 'mycsvfile.csv'
    >>> open(path, 'wb').write(
    ... """col1,col2
    ... dataitem1,dataitem2
    ... item3,item4
    ... """)

Now we create a wrapper, that requires that field:

    >>> from waeup.csvfile.interfaces import ICSVFile
    >>> from waeup.csvfile import CSVFile
    >>> class ICSVFileWithSpecialCol(ICSVFile):
    ...   """A CSV file that contains a 'special_col' column.
    ...   """

    >>> class CSVFileWithSpecialCol(CSVFile):
    ...   required_fields = ['special_col']
    ...   grok.provides(ICSVFileWithSpecialCol)

    >>> grok.testing.grok_component('CSVFileWithSpecialCol', 
    ...                             CSVFileWithSpecialCol)
    True

If we want to get a wrapper for that kind of file:

    >>> ICSVFileWithSpecialCol(path)
    Traceback (most recent call last):
    ...
    TypeError: Missing columns in CSV file: ['special_col']

If the required col is available, however:

    >>> path2 = 'mycsvfile2.csv'
    >>> open(path2, 'wb').write(
    ... """col1,col2,special_col
    ... dataitem1,dataitem2,dataitem3
    ... item4,item5,item6
    ... """)
    >>> ICSVFileWithSpecialCol(path2)
    <CSVFileWithSpecialCol object at 0x...>

Build an importer framework
---------------------------

We can also build an importer framework with CSV file support using
the components described above.

To model this, we start with two files, we will import lateron:

    >>> path = 'mycsvfile.csv'
    >>> open(path, 'wb').write(
    ... """col1,col2
    ... dataitem1a,dataitem2a
    ... item3a,item4a
    ... """)

    >>> path2 = 'mycsvfile2.csv'
    >>> open(path2, 'wb').write(
    ... """col1,col3
    ... dataitem1b,dataitem2b
    ... item3b,item4b
    ... """)

Then we create two receivers for CSV file data:

    >>> from zope.interface import Interface
    >>> class IReceiver1(Interface):
    ...   """A CSV data receiver."""

    >>> class IReceiver2(Interface):
    ...   """Another CSV data receiver."""

    >>> class Receiver1(object):
    ...   grok.implements(IReceiver1)
    ...   def receive(self, data):
    ...     print "Receiver1 received: ", data

    >>> class Receiver2(object):
    ...   grok.implements(IReceiver2)
    ...   def receive(self, data):
    ...     print "Receiver2 received: ", data
    

If we want to be sure, that a wrapper requires these fields, we ask
for ICSVFile12:

    >>> wrapper1 = ICSVFile12(path)
    >>> wrapper1
    <CSVFile12 object at 0x...>

We could not use this interface (adapter) with the other CSV file:

    >>> wrapper2 = ICSVFile12(path2)
    Traceback (most recent call last):
    ...
    TypeError: Missing columns in CSV file: ['col2']

The last step is to build a bridge between the receivers and the
sources. We call it connector or importer here:

    >>> class IImporter(Interface):
    ...   """Import sources to receivers."""

    >>> class IImporter12(IImporter):
    ...   """Imports ICSVFile12 data into IReceiver1 objects."""

    >>> class Importer12(grok.MultiAdapter):
    ...   grok.adapts(ICSVFile12, IReceiver1)
    ...   grok.implements(IImporter12)
    ...   def __init__(self, csvfile, receiver):
    ...     self.csvfile = csvfile
    ...     self.receiver = receiver
    ...   def doImport(self):
    ...     self.receiver.receive(self.csvfile.getData())

    >>> grok.testing.grok_component('Importer12',  Importer12)
    True

We can create an importer if we know the type of CSV file:

    >>> myrecv = Receiver1()
    >>> myfile = ICSVFile12(path)
    >>> myfile
    <CSVFile12 object at 0x...>

    >>> ICSVFile12.providedBy(myfile)
    True

    >>> from zope.component import getMultiAdapter
    >>> myimporter = getMultiAdapter((myfile, myrecv), IImporter12)
    >>> myimporter
    <Importer12 object at 0x...>


We can also create an importer without knowing the type of CSV file
before, using the :func:`getCSVFile` function:

    >>> from waeup.csvfile import getCSVFile
    >>> myfile = getCSVFile(path)
    >>> ICSVFile12.providedBy(myfile)
    True

Apparently, the CSVFileWrapper getter knows, that a CSVFile12 suits
the contents of our file best. We did not specify what type of file
wrapper we want.

Getting an importer now is easy:

    >>> myimporter = getMultiAdapter((myfile, myrecv), IImporter12)
    >>> myimporter
    <Importer12 object at 0x...>


Clean up:

    >>> import os
    >>> os.unlink(path)
    >>> os.unlink(path2)
    >>> os.unlink(nocsvpath)
    >>> os.unlink(custompath)