source: main/waeup.sirp/trunk/src/waeup/sirp/utils/importexport.txt @ 4960

Importing and Exporting Things
******************************

The waeup package tries to support all special object types to be
exportable and importable.

:Test-Layer: unit

As imports and exports of data are a cruical point when doing updates
or to enable data backups, this support must be considered a serious
topic for the base functionality of the whole software.

Exports and imports are implemented using the Zope component framework
capabilities. This means we need some adapters and utilities, which we
register, by grokking the whole package::

    >>> import grok
    >>> grok.testing.grok('waeup')


Exporting XML
=============

One of the major formats we support for data export is XML.

When we want to export an object, we can choose the destination
file. If we do not give a file to write to, we get a StringIO object,
i.e. an in-memory file.

To show this basic functionality provided for all XML exportable
objects, we choose faculties.

We can export faculties. First we create a faculty::

    >>> from waeup.sirp.university.faculty import Faculty
    >>> myfaculty = Faculty()
    >>> myfaculty.name = 'My very faculty.'

Exporting plain XML
-------------------

Now let's export it to XML. To do this we first ask the framework for
an exporter::

    >>> from waeup.sirp.interfaces import IWAeUPXMLExporter
    >>> exporter = IWAeUPXMLExporter(myfaculty)
    >>> exporter
    <waeup.sirp.utils.importexport.XMLExporter object at 0x...>

All exporters provide an ``export(<obj>)`` method::

    >>> result = exporter.export()

The result is an object of type StringIO. We therefore use its
`read()` method to get the actual XML data::

    >>> print result.read()
    <?xml version="1.0" encoding="utf-8" ?>...


Generating XML files
--------------------

We can also let the results be written to a file::

    >>> result = exporter.export('myexport.xml')
    >>> print open('myexport.xml', 'rb').read()
    <?xml version="1.0" encoding="utf-8" ?>
    <pickle>
      <initialized_object>
    ...
    </pickle>

Clean up::

    >>> import os
    >>> os.unlink('myexport.xml')

Importing XML
=============

We can generate objects from XML.

Let's create a faculty instance, that we want to be restored
afterwards::

    >>> from waeup.sirp.university.faculty import Faculty
    >>> myfaculty = Faculty()
    >>> myfaculty.name = 'My very faculty.'

We create an XML dump of this object::

    >>> from waeup.sirp.interfaces import IWAeUPXMLExporter
    >>> exporter = IWAeUPXMLExporter(myfaculty)
    >>> result = exporter.export('myexport.xml')

We change the name of the faculty::

    >>> myfaculty.name = 'Another name'

Now we create an importer for that file::

    >>> from waeup.sirp.interfaces import IWAeUPXMLImporter
    >>> importer = IWAeUPXMLImporter(myfaculty)

Importing from filenames
------------------------

We can use a filepath to import from the denoted file::

    >>> new_obj = importer.doImport('myexport.xml')

The object created is indeed a faculty::

    >>> new_obj
    <waeup.sirp.university.faculty.Faculty object at 0x...>

    >>> new_obj.name
    'My very faculty.'

The new object is really a new object::

    >>> new_obj is myfaculty
    False

Importing from filelike objects
-------------------------------

We can also pass a file-like object instead of a filepath::

    >>> filelike_obj = open('myexport.xml', 'rb')
    >>> new_obj = importer.doImport(filelike_obj)

The object created is indeed a faculty::

    >>> new_obj
    <waeup.sirp.university.faculty.Faculty object at 0x...>

Clean up::

    >>> import os
    >>> os.unlink('myexport.xml')


Importing CSV data
==================

The WAeUP portal supports import of CSV data for several different
data types like faculties, students, and the like.

For our purposes we define CSV imports as tuples, containing a CSV
source and a CSV receiver:

  ``CSV Import := <CSVSource, CSVReceiver>``

CSV sources
-----------

CSV sources build a plugin framework inside the WAeUP portal (that
might be factored out one day). See `waeup.sirp.csvfile` to learn more
about that.

A `waeup.sirp.csvfile.CSVFile` plugin cares for a special kind of CSV file.

To make clear what that means, we start by creating a simple CSV
file.

CSVFile objects are abstractions of simple .csv-files. But as not
every CSV file contains the same kind of data, we can create different
flavours of it, for instance to import departments or faculties. As
each kind of import needs different kind of data, we have different
kinds of CSV sources.

CSV sources can check their associated files for consistency,
importability, etc.

We create a simple CSV file with cave data:

    >>> open('mycavedata.csv', 'wb').write(
    ... """size,owner
    ... 42,Manfred
    ... """)

We have a very simple CSVFile available, which is registered as an
adapter for strings. To make this work, the string should be a path to
a file:

    >>> from waeup.sirp.csvfile import getCSVFile
    >>> mysource = getCSVFile('mycavedata.csv')
    >>> mysource
    <waeup.sirp.csvfile.csvfile.CSVFile object at 0x...>

We define caves like this:

    >>> class Cave(object):
    ...   def __init__(self, size, owner):
    ...     self.size = size
    ...     self.owner = owner
    ...   def __repr__(self):
    ...     return '<Cave object [size: %s, owner: %s]>' % (
    ...       self.size, self.owner)

Let's assume, we have a container for caves:

    >>> from zope.interface import Interface
    >>> class ICaveContainer(Interface):
    ...   """A container for caves."""

    >>> class CaveContainer(object):
    ...   grok.implements(ICaveContainer)
    ...   caves = []
    ...   def addCave(self, cave):
    ...     self.caves.append(cave)

    >>> mycaves = CaveContainer()

Now, if we want to import caves from CSV files, we also need an
importer, that imports data to the container:

    >>> from waeup.sirp.csvfile.interfaces import ICSVFile
    >>> from waeup.sirp.utils.importexport import CSVImporter
    >>> from waeup.sirp.interfaces import IWAeUPCSVImporter
    >>> class CaveImporter(CSVImporter):
    ...   # Tell, what kinds of objects we connect...
    ...   grok.adapts(ICSVFile, ICaveContainer)
    ...   # Tell the world, that we are an importer...
    ...   grok.provides(IWAeUPCSVImporter)
    ...   # The datatype can be used by UI components and is required
    ...   # by the interface...
    ...   datatype = u'Cave Importer'
    ...   def doImport(self):
    ...     # CSVImporter instances have a `csvfile` and a `receiver`
    ...     # object defined which refer to the CSV file and the container.
    ...     for row in self.csvfile.getData():
    ...       cave = Cave(size = row['size'],
    ...                   owner = row['owner'])
    ...       self.receiver.addCave(cave) 


An CSV importer must be grokked before we can use it. In real life,
this is normally done on startup automatically:

    >>> grok.testing.grok_component('CaveImporter', CaveImporter)
    True

Importers are multi-adapters, so we need to ask for one:

    >>> from zope.component import getMultiAdapter
    >>> myimporter = getMultiAdapter((mysource, mycaves), IWAeUPCSVImporter)

Now we can finally do the import:

    >>> myimporter.doImport()

The import was really done:

    >>> mycaves.caves
    [<Cave object [size: 42, owner: Manfred]>]

**Important note**:

  This importer adapts ICSVFile and ICaveContainer, which means that
  the type of data receiver is specified correctly. For the CSV
  source, however, the importer cannot be sure to find a column
  ``size`` or ``owner`` in the file.

  This can be prevented, by defining a more special CSVFile type and
  adapting this instead of ICSVData.

  In the `waeup.sirp.csvfile` subpackage's README it is shown, how this can
  be accomplished.

  Summing up we would define an ICaveCSVFile type, provide an
  appropriate CSV file wrapper and then let our importer adapt::

    (ICaveCSVFile, ICaveContainer)

**Another note**:

  The importer we defined above does not support the ``clear_old_data``
  and ``overwrite`` keywords as would be required by the
  IWAeUPImporter interface.

  In real life this keywords should be supported.

Clean up:

   >>> import os
   >>> os.unlink('mycavedata.csv')