Changeset 7063 for main/waeup.sirp/trunk/src/waeup/sirp/imagestorage.py

Timestamp:

9 Nov 2011, 15:42:45 (13 years ago)

Author:

uli

Message:

Merge changes from branch ulif-extimgstore back into trunk.
Beside external image storage also waeupdocs should work again.

Location:

main/waeup.sirp/trunk

Files:

• . (modified) (1 prop)
• src/waeup/sirp/imagestorage.py (modified) (1 diff)

main/waeup.sirp/trunk/src/waeup/sirp/imagestorage.py

@@ -21 +21 @@
 ##
 """A storage for image files.
+
+A few words about storing files with ``waeup.sirp``. The need for this
+feature arised initially from the need to store passport files for
+applicants and students. These files are dynamic (can be changed
+anytime), mean a lot of traffic and cost a lot of memory/disk space.
+
+**Design Basics**
+
+While one *can* store images and similar 'large binary objects' aka
+blobs in the ZODB, this approach quickly becomes cumbersome and
+difficult to understand. The worst approach here would be to store
+images as regular byte-stream objects. ZODB supports this but
+obviously access is slow (data must be looked up in the one
+``Data.fs`` file, each file has to be sent to the ZEO server and back,
+etc.).
+
+A bit less worse is the approach to store images in the ZODB but as
+Blobs. ZODB supports storing blobs in separate files in order to
+accelerate lookup/retrieval of these files. The files, however, have
+to be sent to the ZEO server (and back on lookups) which means a
+bottleneck and will easily result in an increased number of
+``ConflictErrors`` even on simple reads.
+
+The advantage of both ZODB-geared approaches is, of course, complete
+database consistency. ZODB will guarantee that your files are
+available under some object name and can be handled as any other
+Python object.
+
+Another approach is to leave the ZODB behind and to store images and
+other files in filesystem directly. This is faster (no ZEO contacts,
+etc.), reduces probability of `ConflictErrors`, keeps the ZODB
+smaller, and enables direct access (over filesystem) to the
+files. Furthermore steps might be better understandable for
+third-party developers. We opted for this last option.
+
+**External File Store**
+
+Our implementation for storing-files-API is defined in
+:class:`ExtFileStore`. An instance of this file storage (which is also
+able to store non-image files) is available at runtime as a global
+utility implementing :class:`waeup.sirp.interfaces.IExtFileStore`.
+
+The main task of this central component is to maintain a filesystem
+root path for all files to be stored. It also provides methods to
+store/get files under certain file ids which identify certain files
+locally.
+
+So, to store a file away, you can do something like this:
+
+  >>> from StringIO import StringIO
+  >>> from zope.component import getUtility
+  >>> from waeup.sirp.interfaces import IExtFileStore
+  >>> store = getUtility(IExtFileStore)
+  >>> store.createFile('myfile.txt', StringIO('some file content'))
+
+All you need is a filename and the file-like object containing the
+real file data.
+
+This will store the file somewhere (you shouldn't make too much
+assumptions about the real filesystem path here).
+
+Later, we can get the file back like this:
+
+  >>> store.getFile('myfile.txt')
+  <open file ...>
+
+What we get back is a file or file-like object already opened for
+reading:
+
+  >>> store.getFile('myfile.txt').read()
+  'some file content'
+
+**Handlers: Special Places for Special Files**
+
+The file store supports special handling for certain files. For
+example we want applicant images to be stored in a different directory
+than student images, etc. Because the file store cannot know all
+details about these special tratment of certain files, it looks up
+helpers (handlers) to provide the information it needs for really
+storing the files at the correct location.
+
+That a file stored in filestore needs special handling can be
+indicated by special filenames. These filenames start with a marker like
+this::
+
+  __<MARKER-STRING>__real-filename.jpg
+
+Please note the double underscores before and after the marker
+string. They indicate that all in between is a marker.
+
+If you store a file in file store with such a filename (we call this a
+`file_id` to distuingish it from real world filenames), the file store
+will look up a handler for ``<MARKER-STRING>`` and pass it the file to
+store. The handler then will return the internal path to store the
+file and possibly do additional things as well like validating the
+file or similar.
+
+Examples for such a file store handler can be found in the
+:mod:`waeup.sirp.applicants.applicant` module. Please see also the
+:class:`DefaultFileStoreHandler` class below for more details.
+
+The file store looks up handlers by utility lookups: it looks for a
+named utiliy providing
+:class:`waeup.sirp.interfaces.IFileStoreHandler` and named like the
+marker string (without leading/trailing underscores) in lower
+case. For example if the file id would be
+
+  ``__IMG_USER__manfred.jpg``
+
+then the looked up utility should be registered under name
+
+  ``img_user``
+
+and provide :class:`waeup.sirp.interfaces.IFileStoreHandler`. If no
+such utility can be found, a default handler is used instead
+(see :class:`DefaultFileStoreHandler`).
+
+**Context Adapters: Knowing Your Family**
+
+Often the internal filename or file id of a file depends on a
+context. For example when we store passport photographs of applicants,
+then each image belongs to a certain applicant instance. It is not
+difficult to maintain such a connection manually: Say every applicant
+had an id, then we could put this id into the filename as well and
+would build the filename to store/get the connected file by using that
+filename. You then would create filenames of a format like this::
+
+  __<MARKER-STRING>__applicant0001.jpg
+
+where ``applicant0001`` would tell exactly which applicant you can see
+on the photograph. You notice that the internal file id might have
+nothing to do with once uploaded filenames. The id above could have
+been uploaded with filename ``manfred.jpg`` but with the new file id
+we are able to find the file again later.
+
+Unfortunately it might soon get boring or cumbersome to retype this
+building of filenames for a certain type of context, especially if
+your filenames take more of the context into account than only a
+simple id.
+
+Therefore you can define filename building for a context as an adapter
+that then could be looked up by other components simply by doing
+something like:
+
+  >>> from waeup.sirp.interfaces import IFileStoreNameChooser
+  >>> file_id = IFileStoreNameChooser(my_context_obj)
+
+If you later want to change the way file ids are created from a
+certain context, you only have to change the adapter implementation
+accordingly.
+
+Note, that this is only a convenience component. You don't have to
+define context adapters but it makes things easier for others if you
+do, as you don't have to remember the exact file id creation method
+all the time and can change things quick and in only one location if
+you need to do so.
+
+Please see the :class:`FileStoreNameChooser` default implementation
+below for details.
+
 """
 import grok
-import hashlib
 import os
-import transaction
-import warnings
-from StringIO import StringIO
-from ZODB.blob import Blob
-from persistent import Persistent
+import tempfile
+from hurry.file import HurryFile
 from hurry.file.interfaces import IFileRetrieval
-from waeup.sirp.image import WAeUPImageFile
-from waeup.sirp.utils.helpers import cmp_files
-
-def md5digest(fd):
-    """Get an MD5 hexdigest for the file stored in `fd`.
-
-    `fd`
-      a file object open for reading.
-
+from zope.component import queryUtility
+from zope.interface import Interface
+from waeup.sirp.interfaces import (
+    IFileStoreNameChooser, IExtFileStore, IFileStoreHandler,)
+
+class FileStoreNameChooser(grok.Adapter):
+    """Default file store name chooser.
+
+    File store name choosers pick a file id, a string, for a certain
+    context object. They are normally registered as adapters for a
+    certain content type and know how to build the file id for this
+    special type of context.
+
+    Provides the :class:`waeup.sirp.interfaces.IFileStoreNameChooser`
+    interface.
+
+    This default file name chosser accepts almost every name as long
+    as it is a string or unicode object.
     """
-    return hashlib.md5(fd.read()).hexdigest()
-
-class Basket(grok.Container):
-    """A basket holds a set of image files with same hash.
+    grok.context(Interface)
+    grok.implements(IFileStoreNameChooser)
+
+    def checkName(self, name):
+        """Check whether an object name is valid.
+
+        Raises a user error if the name is not valid.
+
+        For the default file store name chooser any name is valid.
+        """
+        if isinstance(name, basestring):
+            return True
+        return False
+
+    def chooseName(self, name):
+        """Choose a unique valid name for the object.
+
+        The given name and object may be taken into account when
+        choosing the name.
+
+        chooseName is expected to always choose a valid name (that
+        would pass the checkName test) and never raise an error.
+
+        For this default name chooser we return the given name if it
+        is valid or ``unknown_file`` else.
+        """
+        if self.checkName(name):
+            return name
+        return u'unknown_file'
+
+class ExtFileStore(object):
+    """External file store.
+
+    External file stores are meant to store files 'externally' of the
+    ZODB, i.e. in filesystem.
+
+    Most important attribute of the external file store is the `root`
+    path which gives the path to the location where files will be
+    stored within.
+
+    By default `root` is a ``'media/'`` directory in the root of the
+    datacenter root of a site.
+
+    The `root` attribute is 'read-only' because you normally don't
+    want to change this path -- it is dynamic. That means, if you call
+    the file store from 'within' a site, the root path will be located
+    inside this site (a :class:`waeup.sirp.University` instance). If
+    you call it from 'outside' a site some temporary dir (always the
+    same during lifetime of the file store instance) will be used. The
+    term 'temporary' tells what you can expect from this path
+    persistence-wise.
+
+    If you insist, you can pass a root path on initialization to the
+    constructor but when calling from within a site afterwards, the
+    site will override your setting for security measures. This way
+    you can safely use one file store for different sites in a Zope
+    instance simultanously and files from one site won't show up in
+    another.
+
+    An ExtFileStore instance is available as a global utility
+    implementing :class:`waeup.sirp.interfaces.IExtFileStore`.
+
+    To add and retrieve files from the storage, use the appropriate
+    methods below.
     """
 
-    def _del(self):
-        """Remove temporary files associated with local blobs.
-
-        A basket holds files as Blob objects. Unfortunately, if a
-        basket was not committed (put into ZODB), those blobs linger
-        around as real files in some temporary directory and won't be
-        removed.
-
-        This is a helper function to remove all those uncommitted
-        blobs that has to be called explicitly, for instance in tests.
-        """
-        key_list = list(self.keys())
-        for key in key_list:
-            item = self[key]
-            if getattr(item, '_p_oid', None):
-                # Don't mess around with blobs in ZODB
-                continue
-            fd = item.open('r')
-            name = getattr(fd, 'name', None)
-            fd.close()
-            if name is not None and os.path.exists(name):
-                os.unlink(name)
-            del self[key]
+    grok.implements(IExtFileStore)
+
+    _root = None
+
+    @property
+    def root(self):
+        """Root dir of this storage.
+
+        The root dir is a readonly value determined dynamically. It
+        holds media files for sites or other components.
+
+        If a site is available we return a ``media/`` dir in the
+        datacenter storage dir.
+
+        Otherwise we create a temporary dir which will be remembered
+        on next call.
+
+        If a site exists and has a datacenter, it has always
+        precedence over temporary dirs, also after a temporary
+        directory was created.
+
+        Please note that retrieving `root` is expensive. You might
+        want to store a copy once retrieved in order to minimize the
+        number of calls to `root`.
+
+        """
+        site = grok.getSite()
+        if site is not None:
+            root = os.path.join(site['datacenter'].storage, 'media')
+            return root
+        if self._root is None:
+            self._root = tempfile.mkdtemp()
+        return self._root
+
+    def __init__(self, root=None):
+        self._root = root
         return
 
-    def getInternalId(self, fd):
-        """Get the basket-internal id for the file stored in `fd`.
-
-        `fd` must be a file open for reading. If an (byte-wise) equal
-        file can be found in the basket, its internal id (basket id)
-        is returned, ``None`` otherwise.
-        """
-        fd.seek(0)
-        for key, val in self.items():
-            fd_stored = val.open('r')
-            file_len = os.stat(fd_stored.name)[6]
-            if file_len == 0:
-                # Nasty workaround. Blobs seem to suffer from being emptied
-                # accidentally.
-                site = grok.getSite()
-                if site is not None:
-                    site.logger.warn(
-                        'Empty Blob detected: %s' % fd_stored.name)
-                warnings.warn("EMPTY BLOB DETECTED: %s" % fd_stored.name)
-                fd_stored.close()
-                val.open('w').write(fd.read())
-                return key
-            fd_stored.seek(0)
-            if cmp_files(fd, fd_stored):
-                fd_stored.close()
-                return key
-            fd_stored.close()
-        return None
-
-    @property
-    def curr_id(self):
-        """The current basket id.
-
-        An integer number which is not yet in use. If there are
-        already `maxint` entries in the basket, a :exc:`ValueError` is
-        raised. The latter is _highly_ unlikely. It would mean to have
-        more than 2**32 hash collisions, i.e. so many files with the
-        same MD5 sum.
-        """
-        num = 1
-        while True:
-            if str(num) not in self.keys():
-                return str(num)
-            num += 1
-            if num <= 0:
-                name = getattr(self, '__name__', None)
-                raise ValueError('Basket full: %s' % name)
-
-    def storeFile(self, fd, filename):
-        """Store the file in `fd` into the basket.
-
-        The file will be stored in a Blob.
-        """
-        fd.seek(0)
-        internal_id = self.getInternalId(fd) # Moves file pointer!
-        if internal_id is None:
-            internal_id = self.curr_id
-            fd.seek(0)
-            self[internal_id] = Blob()
-            transaction.commit() # Urgently needed to make the Blob
-                                 # persistent. Took me ages to find
-                                 # out that solution, which makes some
-                                 # design flaw in ZODB Blobs likely.
-            self[internal_id].open('w').write(fd.read())
-            fd.seek(0)
-            self._p_changed = True
-        return internal_id
-
-    def retrieveFile(self, basket_id):
-        """Retrieve a file open for reading with basket id `basket_id`.
-
-        If there is no such id, ``None`` is returned. It is the
-        callers responsibility to close the open file.
-        """
-        if basket_id in self.keys():
-            return self[basket_id].open('r')
-        return None
-
-class ImageStorage(grok.Container):
-    """A container for image files.
+    def getFile(self, file_id):
+        """Get a file stored under file ID `file_id`.
+
+        Returns a file already opened for reading.
+
+        If the file cannot be found ``None`` is returned.
+
+        This methods takes into account registered handlers for any
+        marker put into the file_id.
+
+        .. seealso:: :class:`DefaultFileStoreHandler`
+        """
+        marker, filename, base, ext = self.extractMarker(file_id)
+        handler = queryUtility(IFileStoreHandler, name=marker,
+                               default=DefaultFileStoreHandler())
+        path = handler.pathFromFileID(self, self.root, file_id)
+        if not os.path.exists(path):
+            return None
+        fd = open(path, 'rb')
+        return fd
+
+    def getFileByContext(self, context):
+        """Get a file for given context.
+
+        Returns a file already opened for reading.
+
+        If the file cannot be found ``None`` is returned.
+
+        This method takes into account registered handlers and file
+        name choosers for context types.
+
+        This is a convenience method that internally calls
+        :meth:`getFile`.
+
+        .. seealso:: :class:`FileStoreNameChooser`,
+                     :class:`DefaultFileStoreHandler`.
+        """
+        file_id = IFileStoreNameChooser(context).chooseName()
+        return self.getFile(file_id)
+
+    def createFile(self, filename, f):
+        """Store a file.
+        """
+        file_id = filename
+        root = self.root # Calls to self.root are expensive
+        marker, filename, base, ext = self.extractMarker(file_id)
+        handler = queryUtility(IFileStoreHandler, name=marker,
+                               default=DefaultFileStoreHandler())
+        f, path, file_obj = handler.createFile(
+            self, root, file_id, filename, f)
+        dirname = os.path.dirname(path)
+        if not os.path.exists(dirname):
+            os.makedirs(dirname, 0755)
+        open(path, 'wb').write(f.read())
+        return file_obj
+
+    def extractMarker(self, file_id):
+        """split filename into marker, filename, basename, and extension.
+
+        A marker is a leading part of a string of form
+        ``__MARKERNAME__`` followed by the real filename. This way we
+        can put markers into a filename to request special processing.
+
+        Returns a quadruple
+
+          ``(marker, filename, basename, extension)``
+
+        where ``marker`` is the marker in lowercase, filename is the
+        complete trailing real filename, ``basename`` is the basename
+        of the filename and ``extension`` the filename extension of
+        the trailing filename. See examples below.
+
+        Example:
+
+           >>> extractMarker('__MaRkEr__sample.jpg')
+           ('marker', 'sample.jpg', 'sample', '.jpg')
+
+        If no marker is contained, we assume the whole string to be a
+        real filename:
+
+           >>> extractMarker('no-marker.txt')
+           ('', 'no-marker.txt', 'no-marker', '.txt')
+
+        Filenames without extension give an empty extension string:
+
+           >>> extractMarker('no-marker')
+           ('', 'no-marker', 'no-marker', '')
+
+        """
+        if not isinstance(file_id, basestring) or not file_id:
+            return ('', '', '', '')
+        parts = file_id.split('__', 2)
+        marker = ''
+        if len(parts) == 3 and parts[0] == '':
+            marker = parts[1].lower()
+            file_id = parts[2]
+        basename, ext = os.path.splitext(file_id)
+        return (marker, file_id, basename, ext)
+
+grok.global_utility(ExtFileStore, provides=IExtFileStore)
+
+class DefaultStorage(ExtFileStore):
+    """Default storage for files.
+
+    Registered globally as utility for
+    :class:`hurry.file.interfaces.IFileRetrieval`.
     """
-    def _del(self):
-        for basket in self.values():
-            try:
-                basket._del()
-            except:
-                pass
-
-    def storeFile(self, fd, filename):
-        fd.seek(0)
-        digest = md5digest(fd)
-        fd.seek(0)
-        if not digest in self.keys():
-            self[digest] = Basket()
-        basket_id = self[digest].storeFile(fd, filename)
-        full_id = "%s-%s" % (digest, basket_id)
-        return full_id
-
-    def retrieveFile(self, file_id):
-        if not '-' in file_id:
-            return None
-        full_id, basket_id = file_id.split('-', 1)
-        if not full_id in self.keys():
-            return None
-        return self[full_id].retrieveFile(basket_id)
-
-class ImageStorageFileRetrieval(Persistent):
-    grok.implements(IFileRetrieval)
-
-    def getImageStorage(self):
-        site = grok.getSite()
-        if site is None:
-            return None
-        return site.get('images', None)
-
-    def isImageStorageEnabled(self):
-        site = grok.getSite()
-        if site is None:
-            return False
-        if site.get('images', None) is None:
-            return False
-        return True
-
-    def getFile(self, data):
-        # ImageStorage is disabled, so give fall-back behaviour for
-        # testing without ImageStorage
-        if not self.isImageStorageEnabled():
-            return StringIO(data)
-        storage = self.getImageStorage()
-        if storage is None:
-            raise ValueError('Cannot find an image storage')
-        result = storage.retrieveFile(data)
-        if result is None:
-            return StringIO(data)
-        return storage.retrieveFile(data)
-
-    def createFile(self, filename, f):
-        if not self.isImageStorageEnabled():
-            return WAeUPImageFile(filename, f.read())
-        storage = self.getImageStorage()
-        if storage is None:
-            raise ValueError('Cannot find an image storage')
-        file_id = storage.storeFile(f, filename)
-        return WAeUPImageFile(filename, file_id)
+    grok.provides(IFileRetrieval)
+
+grok.global_utility(DefaultStorage, provides=IFileRetrieval)
+
+class DefaultFileStoreHandler(grok.GlobalUtility):
+    """A default handler for external file store.
+
+    This handler is the fallback called by external file stores when
+    there is no or an unknown marker in the file id.
+
+    Registered globally as utility for
+    :class:`waeup.sirp.interfaces.IFileStoreHandler`.
+    """
+    grok.implements(IFileStoreHandler)
+
+    def pathFromFileID(self, store, root, file_id):
+        """Return the root path of external file store appended by file id.
+        """
+        return os.path.join(root, file_id)
+
+    def createFile(self, store, root, filename, file_id, f):
+        """Infos about what to store exactly and where.
+
+        When a file should be handled by an external file storage, it
+        looks up any handlers (like this one), passes runtime infos
+        like the storage object, root path, filename, file_id, and the
+        raw file object itself.
+
+        The handler can then change the file, raise exceptions or
+        whatever and return the result.
+
+        This handler returns the input file as-is, a path returned by
+        :meth:`pathFromFileID` and an instance of
+        :class:`hurry.file.HurryFile` for further operations.
+
+        Please note: although a handler has enough infos to store the
+        file itself, it should leave that task to the calling file
+        store.
+        """
+        path = self.pathFromFileID(store, root, file_id)
+        return f, path, HurryFile(filename, file_id)