Changeset 7010 for main/waeup.sirp/branches/ulif-extimgstore

Timestamp:

6 Nov 2011, 23:36:21 (13 years ago)

Author:

uli

Message:

• Give an overview over file handling with the external file store.
• Minor fixes.

File:

• main/waeup.sirp/branches/ulif-extimgstore/src/waeup/sirp/imagestorage.py (modified) (4 diffs)

main/waeup.sirp/branches/ulif-extimgstore/src/waeup/sirp/imagestorage.py

@@ -21 +21 @@
 ##
 """A storage for image files.
+
+A few words about storing files with ``waeup.sirp``. The need for this
+feature arised initially from the need to store passport files for
+applicants and students. These files are dynamic (can be changed
+anytime), mean a lot of traffic and cost a lot of memory/disk space.
+
+**Design Basics**
+
+While one *can* store images and similar 'large binary objects' aka
+blobs in the ZODB, this approach quickly becomes cumbersome and
+difficult to understand. The worst approach here would be to store
+images as regular byte-stream objects. ZODB supports this but
+obviously access is slow (data must be looked up in the one
+``Data.fs`` file, each file has to be sent to the ZEO server and back,
+etc.).
+
+A bit less worse is the approach to store images in the ZODB but as
+Blobs. ZODB supports storing blobs in separate files in order to
+accelerate lookup/retrieval of these files. The files, however, have
+to be sent to the ZEO server (and back on lookups) which means a
+bottleneck and will easily result in an increased number of
+``ConflictErrors`` even on simple reads.
+
+The advantage of both ZODB-geared approaches is, of course, complete
+database consistency. ZODB will guarantee that your files are
+available under some object name and can be handled as any other
+Python object.
+
+Another approach is to leave the ZODB behind and to store images and
+other files in filesystem directly. This is faster (no ZEO contacts,
+etc.), reduces probability of `ConflictErrors`, keeps the ZODB
+smaller, and enables direct access (over filesystem) to the
+files. Furthermore steps might be better understandable for
+third-party developers. We opted for this last option.
+
+**External File Store**
+
+Our implementation for storing-files-API is defined in
+:class:`ExtFileStore`. An instance of this file storage (which is also
+able to store non-image files) is available at runtime as a global
+utility implementing :class:`waeup.sirp.interfaces.IExtFileStore`.
+
+The main task of this central component is to maintain a filesystem
+root path for all files to be stored. It also provides methods to
+store/get files under certain file ids which identify certain files
+locally.
+
+So, to store a file away, you can do something like this:
+
+  >>> from StringIO import StringIO
+  >>> from zope.component import getUtility
+  >>> from waeup.sirp.interfaces import IExtFileStore
+  >>> store = getUtility(IExtFileStore)
+  >>> store.createFile('myfile.txt', StringIO('some file content'))
+
+All you need is a filename and the file-like object containing the
+real file data.
+
+This will store the file somewhere (you shouldn't make too much
+assumptions about the real filesystem path here).
+
+Later, we can get the file back like this:
+
+  >>> store.getFile('myfile.txt')
+  <open file ...>
+
+What we get back is a file or file-like object already opened for
+reading:
+
+  >>> store.getFile('myfile.txt').read()
+  'some file content'
+
+**Handlers: Special Places for Special Files**
+
+The file store supports special handling for certain files. For
+example we want applicant images to be stored in a different directory
+than student images, etc. Because the file store cannot know all
+details about these special tratment of certain files, it looks up
+helpers (handlers) to provide the information it needs for really
+storing the files at the correct location.
+
+That a file stored in filestore needs special handling can be
+indicated by special filenames. These filenames start with a marker like
+this::
+
+  __<MARKER-STRING>__real-filename.jpg
+
+Please note the double underscores before and after the marker
+string. They indicate that all in between is a marker.
+
+If you store a file in file store with such a filename (we call this a
+`file_id` to distuingish it from real world filenames), the file store
+will look up a handler for ``<MARKER-STRING>`` and pass it the file to
+store. The handler then will return the internal path to store the
+file and possibly do additional things as well like validating the
+file or similar.
+
+Examples for such a file store handler can be found in the
+:mod:`waeup.sirp.applicants.applicant` module. Please see also the
+:class:`DefaultFileStoreHandler` class below for more details.
+
+The file store looks up handlers by utility lookups: it looks for a
+named utiliy providing
+:class:`waeup.sirp.interfaces.IFileStoreHandler` and named like the
+marker string (without leading/trailing underscores) in lower
+case. For example if the file id would be
+
+  ``__IMG_USER__manfred.jpg``
+
+then the looked up utility should be registered under name
+
+  ``img_user``
+
+and provide :class:`waeup.sirp.interfaces.IFileStoreHandler`. If no
+such utility can be found, a default handler is used instead
+(see :class:`DefaultFileStoreHandler`).
+
+**Context Adapters: Knowing Your Family**
+
+Often the internal filename or file id of a file depends on a
+context. For example when we store passport photographs of applicants,
+then each image belongs to a certain applicant instance. It is not
+difficult to maintain such a connection manually: Say every applicant
+had an id, then we could put this id into the filename as well and
+would build the filename to store/get the connected file by using that
+filename. You then would create filenames of a format like this::
+
+  __<MARKER-STRING>__applicant0001.jpg
+
+where ``applicant0001`` would tell exactly which applicant you can see
+on the photograph. You notice that the internal file id might have
+nothing to do with once uploaded filenames. The id above could have
+been uploaded with filename ``manfred.jpg`` but with the new file id
+we are able to find the file again later.
+
+Unfortunately it might soon get boring or cumbersome to retype this
+building of filenames for a certain type of context, especially if
+your filenames take more of the context into account than only a
+simple id.
+
+Therefore you can define filename building for a context as an adapter
+that then could be looked up by other components simply by doing
+something like:
+
+  >>> from waeup.sirp.interfaces import IFileStoreNameChooser
+  >>> file_id = IFileStoreNameChooser(my_context_obj)
+
+If you later want to change the way file ids are created from a
+certain context, you only have to change the adapter implementation
+accordingly.
+
+Note, that this is only a convenience component. You don't have to
+define context adapters but it makes things easier for others if you
+do, as you don't have to remember the exact file id creation method
+all the time and can change things quick and in only one location if
+you need to do so.
+
+Please see the :class:`FileStoreNameChooser` default implementation
+below for details.
+
 """
 import grok
@@ -180 +340 @@
 class ImageStorage(grok.Container):
     """A container for image files.
+
+    .. deprecated:: 0.2
+
+       Use :class:`waeup.sirp.ExtFileStore` instead.
     """
     def _del(self):
@@ -246 +410 @@
 
 
-class ExtFileStore(object): #grok.GlobalUtility):
+class ExtFileStore(object):
     """External file store.
 
@@ -408 +572 @@
     def createFile(self, store, root, filename, file_id, f):
         path = self.pathFromFileID(store, root, file_id)
-        return f, path, WAeUPImageFile(filename, file_id)
-        return path, HurryFile(filename, file_id)
+        return f, path, HurryFile(filename, file_id)