Changeset 12869 for main/waeup.kofa/trunk

Timestamp:

22 Apr 2015, 19:14:14 (10 years ago)

Author:

Henrik Bettermann

Message:

Start documenting batch processors.

Location:

main/waeup.kofa/trunk

Files:

• docs/source/userdocs/datacenter/import.rst (modified) (1 diff)
• src/waeup/kofa/applicants/batching.py (modified) (5 diffs)
• src/waeup/kofa/authentication.py (modified) (1 diff)
• src/waeup/kofa/students/batching.py (modified) (7 diffs)
• src/waeup/kofa/university/batching.py (modified) (9 diffs)
• src/waeup/kofa/university/export.py (modified) (1 diff)
• src/waeup/kofa/utils/batching.py (modified) (7 diffs)

main/waeup.kofa/trunk/docs/source/userdocs/datacenter/import.rst

@@ -53 +53 @@
 ================
 
-All batch processors inherit their methods from the :py:class:`waeup.kofa.utils.batching.BatchProcessor` base class. The core ``doImport`` method always remains unchanged.
+All batch processors inherit from the :py:class:`waeup.kofa.utils.batching.BatchProcessor` base class. The `doImport` method, described above, always remains unchanged. All processors have a property `available_fields` which defines the set of importable data. They correspond with the column titles of the import file. Available fields are usually composed of location fields, interface fields and additional fields. Overlaps are possible. Location fields define the minumum set of fields which are necessary to locate an existing object in order to update or remove it. Interface fields (schema fields) are the fields defined in the interface of the data entity. Additional fields are additionally needed for data processing. We further distinguish between required and optional fields or between schema and non-schema fields.
+
+In the following we list all available processors of the Kofa base package including some important methods which describe them best. We do not list available fields of each processor here. Available fields are shown in the browser user interface on the upload page of the portal.
+
+User Processor
+--------------
+
+.. autoclass:: waeup.kofa.authentication.UserProcessor()
+  :noindex:
+
+Faculty Processor
+-----------------
+
+.. autoclass:: waeup.kofa.university.batching.FacultyProcessor()
+  :noindex:
+
+Department Processor
+--------------------
+
+.. autoclass:: waeup.kofa.university.batching.DepartmentProcessor()
+  :noindex:
+
+Certificate Processor
+---------------------
+
+.. autoclass:: waeup.kofa.university.batching.CertificateProcessor()
+  :noindex:
+
+Course Processor
+----------------
+
+.. autoclass:: waeup.kofa.university.batching.CourseProcessor()
+  :noindex:
+
+
+Certificate Course Processor
+----------------------------
+
+.. autoclass:: waeup.kofa.university.batching.CertificateCourseProcessor()
+  :noindex:
+
+Applicants Container Processor
+------------------------------
+
+.. autoclass:: waeup.kofa.applicants.batching.ApplicantsContainerProcessor()
+  :noindex:
+
+Applicant Processor
+-------------------
+
+.. autoclass:: waeup.kofa.applicants.batching.ApplicantProcessor()
+  :noindex:

main/waeup.kofa/trunk/src/waeup/kofa/applicants/batching.py

@@ -35 +35 @@
 
 class ApplicantsContainerProcessor(BatchProcessor):
-    """A processor for applicants containers.
+    """The Applicants Container Processor imports containers for applicants.
+    It does not import their content. There is nothing special about this
+    processor.
     """
     grok.implements(IBatchProcessor)
@@ -76 +78 @@
 
 class ApplicantProcessor(BatchProcessor):
-    """A batch processor for IApplicant objects.
-
-    In create mode container_code is required. If application_number is given
-    an applicant with this number is created in the designated container.
-    If application_number is not given a random application_number is assigned.
-    applicant_id is being determined by the system and can't be imported.
-
-    In update or remove mode container_code and application_number columns
-    must not exist. The applicant object is solely searched by its applicant_id
-    or reg_number.
+    """The Applicant Processor imports application records (applicants).
+
+    In create mode `container_code` is required. If `application_number` is
+    given, an applicant with this number is created in the designated container.
+    If `application_number` is not given, a random `application_number` is
+    assigned. `applicant_id` is being determined by the system and can't be
+    imported.
+
+    In update or remove mode `container_code` and `application_number` columns
+    must not exist. The applicant object is solely localized by
+    `applicant_id` or `reg_number`.
     """
     grok.implements(IBatchProcessor)
@@ -95 +98 @@
     iface = IApplicant
     iface_byregnumber = IApplicantUpdateByRegNo
-    location_fields = ['']
     factory_name = 'waeup.Applicant'
 
@@ -303 +305 @@
         if cert is not None and (mode in ('create', 'update')):
             # course1 application category must match container's.
-            parent = self.getParent(row, self.site)
+            site = grok.getSite()
+            parent = self.getParent(row, site)
             if parent is None:
                 errs.append(('container', 'not found'))
@@ -332 +335 @@
             return 'Applicant is blocked.'
         return None
-
-    def doImport(self, *args, **kw):
-        # XXX: Not thread-safe.  Parallel applicant imports into
-        # different sites could mean a mess.  Luckily this is not a
-        # typical use-case. On the other hand it spares thousands of
-        # site lookups during large imports.
-        # XXX: Maybe this should go into Importer base.
-        self.site = grok.getSite() # needed by checkConversion()
-        return super(ApplicantProcessor, self).doImport(*args, **kw)

main/waeup.kofa/trunk/src/waeup/kofa/authentication.py

@@ -449 +449 @@
 
 class UserProcessor(BatchProcessor):
-    """A batch processor for IUserAccount objects.
+    """The User Processor processes user accounts, i.e. `Account` objects in the
+    ``users`` container.
+
+    The `roles` columns must contain Python list
+    expressions like ``['waeup.PortalManager', 'waeup.ImportManager']``.
+
+    The processor does not import local roles. These can be imported
+    by means of batch processors in the academic section.
     """
     grok.implements(IBatchProcessor)

main/waeup.kofa/trunk/src/waeup/kofa/students/batching.py

@@ -64 +64 @@
     iface_bymatricnumber = IStudentUpdateByMatricNo
 
-    location_fields = []
     factory_name = 'waeup.Student'
 
@@ -312 +311 @@
     additional_fields = []
 
-    #: header fields additionally required
-    additional_headers = []
+    # additional required fields (subset of additional_fields)
+    additional_fields_required = []
 
     @property
@@ -328 +327 @@
                 "Need at least columns student_id " +
                 "or reg_number or matric_number for import!")
-        for name in self.additional_headers:
+        for name in self.additional_fields_required:
             if not name in headerfields:
                 raise FatalCSVError(
@@ -410 +409 @@
     iface_transfer = IStudentStudyCourseTransfer
     factory_name = 'waeup.StudentStudyCourse'
-
-    location_fields = []
-    additional_fields = []
 
     def getParent(self, row, site):
@@ -510 +506 @@
     factory_name = 'waeup.StudentStudyLevel'
 
-    location_fields = []
-
     additional_fields = ['level']
-    additional_headers = ['level']
+    additional_fields_required = additional_fields
 
     @property
@@ -591 +585 @@
     factory_name = 'waeup.CourseTicket'
 
-    location_fields = []
     additional_fields = ['level', 'code']
-    additional_headers = ['level', 'code']
+    additional_fields_required = additional_fields
 
     @property
@@ -695 +688 @@
     factory_name = 'waeup.StudentOnlinePayment'
 
-    location_fields = []
     additional_fields = ['p_id']
-    additional_headers = []
 
     def checkHeaders(self, headerfields, mode='ignore'):

main/waeup.kofa/trunk/src/waeup/kofa/university/batching.py

@@ -44 +44 @@
 
 class FacultyProcessor(BatchProcessor):
-    """A batch processor for IFaculty objects.
+    """The Faculty Processor processes faculties in the `faculties` container.
+    The `FacultyProcessor` class also serves as a baseclass for all other 
+    batch processors in the academic section.
+
+    The processor makes some efforts to set local roles.
+    If new roles are provided, the `updateEntry` method first removes
+    all existing roles and then sets the new roles as given in the import
+    file. That means the entire set of local roles is replaced.
     """
     grok.implements(IBatchProcessor)
@@ -103 +110 @@
             role_map = IPrincipalRoleMap(obj)
             # Remove all existing local roles.
-            for local_role, user_name, setting in role_map.getPrincipalsAndRoles():
+            for local_role, user_name, setting in \
+                role_map.getPrincipalsAndRoles():
                 role_manager.unsetRoleForPrincipal(local_role, user_name)
                 notify(LocalRoleSetEvent(
@@ -114 +122 @@
                     local_role = rolemap['local_role']
                     role_manager.assignRoleToPrincipal(local_role, user)
-                    notify(LocalRoleSetEvent(obj, local_role, user, granted=True))
-                    items_changed += (
-                        '%s=%s, ' % ('local_roles', '%s|%s' % (user,local_role)))
+                    notify(LocalRoleSetEvent(
+                        obj, local_role, user, granted=True))
+                    items_changed += ('%s=%s, '
+                        % ('local_roles', '%s|%s' % (user,local_role)))
             row.pop('local_roles')
 
@@ -151 +160 @@
                 if not 'user_name' in rolemap.keys() or not \
                     'local_role' in rolemap.keys():
-                    errs.append(('local_roles','user_name or local_role missing'))
+                    errs.append((
+                        'local_roles','user_name or local_role missing'))
                     return errs, inv_errs, conv_dict
                 local_role = rolemap['local_role']
@@ -165 +175 @@
 
 class DepartmentProcessor(FacultyProcessor):
-    """A batch processor for IDepartment objects.
+    """The Department Processor works in the same way as the Faculty
+    Processor. Since department codes are not necessarily unique, it needs the
+    `faculty_code` to create and update objects.
     """
     grok.implements(IBatchProcessor)
@@ -213 +225 @@
 
 class CertificateProcessor(FacultyProcessor):
-    """A batch processor for ICertificate objects.
+    """The Certificate Processor gets the parent object (the
+    `certificates` attribute of the department container) in two ways.
+    If both faculty and department codes are provided, `getPartents` uses
+    these to locate the certificate. If department code or
+    faculty code are missing, it use the certificates catalog to find the
+    certificate.
     """
     grok.implements(IBatchProcessor)
@@ -270 +287 @@
             return dept.certificates
         # If department code or faculty code is missing,
-        # use catalog to get parent. Makes only sense in update mode but
-        # does also work in create mode.
+        # use catalog to get parent. 
         cat = queryUtility(ICatalog, name='certificates_catalog')
         results = list(
@@ -296 +312 @@
 
 class CourseProcessor(CertificateProcessor):
-    """A batch processor for ICourse objects.
+    """The Course Processor works exactly in the same way as the
+    Certificate Processor. It uses the courses catalog instead of the
+    certificates catalog.
     """
     grok.implements(IBatchProcessor)
@@ -343 +361 @@
 
 class CertificateCourseProcessor(FacultyProcessor):
-    """A batch processor for ICertificateCourse objects.
+    """The Certificate Course Processor needs more location fields.
+    Certificate courses are stored inside the certificate container.
+    Thus, `faculty_code`, `department_code` and the
+    `certificate_code` are necessary to find the parent container.
+    It furthermore needs the `course` and the `level` field to locate
+    existing objects as they are part of the object id (code).
     """
     grok.implements(IBatchProcessor)

main/waeup.kofa/trunk/src/waeup/kofa/university/export.py

@@ -26 +26 @@
 
 class FacultyExporter(grok.GlobalUtility, ExporterBase):
-    """The Faculty Exporter exports all faculties in the 'faculties'
+    """The Faculty Exporter exports all faculties in the `faculties`
     container. This is the only place where faculties are stored.
     """

main/waeup.kofa/trunk/src/waeup/kofa/utils/batching.py

@@ -56 +56 @@
 
     # Internal name...
-    util_name = 'baseprocessor'
+    util_name = ''
 
     # Items for this processor need an interface with zope.schema fields.
@@ -66 +66 @@
 
     # Headers needed to locate items...
-    location_fields = ['code', 'faculty_code']
+    location_fields = []
 
     # A factory with this name must be registered...
-    factory_name = 'waeup.Department'
+    factory_name = ''
 
     @property
@@ -128 +128 @@
     def applyMapping(self, row, mapping):
         """Apply mapping to a row of CSV data.
-
         """
         result = dict()
@@ -284 +283 @@
     def doImport(self, path, headerfields, mode='create', user='Unknown',
                  logger=None, ignore_empty=True):
-        """In contrast to most other methods, ``doImport`` is not supposed to
+        """In contrast to most other methods, `doImport` is not supposed to
         be customized, neither in custom packages nor in derived batch
         processor classes. Therefore, this is the only place where we
@@ -317 +316 @@
            record is stored in the pending data file.
 
-           Now ``doImport`` tries to add the new object with the data 
+           Now `doImport` tries to add the new object with the data
            from the conversion dictionary. In some cases this
-           may fail and a DuplicationError is raised. For example, a new
+           may fail and a `DuplicationError` is raised. For example, a new
            payment ticket is created but the same payment for same session
            has already been made. In this case the object id is unique, no
@@ -339 +338 @@
            transitions or states. 
 
-           Finally, ``doImport`` updates the existing object with the data
+           Finally, `doImport` updates the existing object with the data
            from the conversion dictionary.
 
@@ -348 +347 @@
            stored in the pending data file.
 
-           Finally, ``doImport`` removes the existing object.
+           Finally, `doImport` removes the existing object.
 
         """