Changeset 12869


Ignore:
Timestamp:
22 Apr 2015, 19:14:14 (10 years ago)
Author:
Henrik Bettermann
Message:

Start documenting batch processors.

Location:
main/waeup.kofa/trunk
Files:
7 edited

Legend:

Unmodified
Added
Removed

  • main/waeup.kofa/trunk/docs/source/userdocs/datacenter/import.rst

    r12868 r12869  
    5353================
    5454
    55 All batch processors inherit their methods from the :py:class:`waeup.kofa.utils.batching.BatchProcessor` base class. The core ``doImport`` method always remains unchanged.
     55All batch processors inherit from the :py:class:`waeup.kofa.utils.batching.BatchProcessor` base class. The `doImport` method, described above, always remains unchanged. All processors have a property `available_fields` which defines the set of importable data. They correspond with the column titles of the import file. Available fields are usually composed of location fields, interface fields and additional fields. Overlaps are possible. Location fields define the minumum set of fields which are necessary to locate an existing object in order to update or remove it. Interface fields (schema fields) are the fields defined in the interface of the data entity. Additional fields are additionally needed for data processing. We further distinguish between required and optional fields or between schema and non-schema fields.
     56
     57In the following we list all available processors of the Kofa base package including some important methods which describe them best. We do not list available fields of each processor here. Available fields are shown in the browser user interface on the upload page of the portal.
     58
     59User Processor
     60--------------
     61
     62.. autoclass:: waeup.kofa.authentication.UserProcessor()
     63  :noindex:
     64
     65Faculty Processor
     66-----------------
     67
     68.. autoclass:: waeup.kofa.university.batching.FacultyProcessor()
     69  :noindex:
     70
     71Department Processor
     72--------------------
     73
     74.. autoclass:: waeup.kofa.university.batching.DepartmentProcessor()
     75  :noindex:
     76
     77Certificate Processor
     78---------------------
     79
     80.. autoclass:: waeup.kofa.university.batching.CertificateProcessor()
     81  :noindex:
     82
     83Course Processor
     84----------------
     85
     86.. autoclass:: waeup.kofa.university.batching.CourseProcessor()
     87  :noindex:
     88
     89
     90Certificate Course Processor
     91----------------------------
     92
     93.. autoclass:: waeup.kofa.university.batching.CertificateCourseProcessor()
     94  :noindex:
     95
     96Applicants Container Processor
     97------------------------------
     98
     99.. autoclass:: waeup.kofa.applicants.batching.ApplicantsContainerProcessor()
     100  :noindex:
     101
     102Applicant Processor
     103-------------------
     104
     105.. autoclass:: waeup.kofa.applicants.batching.ApplicantProcessor()
     106  :noindex:

  • main/waeup.kofa/trunk/src/waeup/kofa/applicants/batching.py

    r11891 r12869  
    3535
    3636class ApplicantsContainerProcessor(BatchProcessor):
    37     """A processor for applicants containers.
     37    """The Applicants Container Processor imports containers for applicants.
     38    It does not import their content. There is nothing special about this
     39    processor.
    3840    """
    3941    grok.implements(IBatchProcessor)
    … …  
    7678
    7779class ApplicantProcessor(BatchProcessor):
    78     """A batch processor for IApplicant objects.
    79 
    80     In create mode container_code is required. If application_number is given
    81     an applicant with this number is created in the designated container.
    82     If application_number is not given a random application_number is assigned.
    83     applicant_id is being determined by the system and can't be imported.
    84 
    85     In update or remove mode container_code and application_number columns
    86     must not exist. The applicant object is solely searched by its applicant_id
    87     or reg_number.
     80    """The Applicant Processor imports application records (applicants).
     81
     82    In create mode `container_code` is required. If `application_number` is
     83    given, an applicant with this number is created in the designated container.
     84    If `application_number` is not given, a random `application_number` is
     85    assigned. `applicant_id` is being determined by the system and can't be
     86    imported.
     87
     88    In update or remove mode `container_code` and `application_number` columns
     89    must not exist. The applicant object is solely localized by
     90    `applicant_id` or `reg_number`.
    8891    """
    8992    grok.implements(IBatchProcessor)
    … …  
    9598    iface = IApplicant
    9699    iface_byregnumber = IApplicantUpdateByRegNo
    97     location_fields = ['']
    98100    factory_name = 'waeup.Applicant'
    99101
    … …  
    303305        if cert is not None and (mode in ('create', 'update')):
    304306            # course1 application category must match container's.
    305             parent = self.getParent(row, self.site)
     307            site = grok.getSite()
     308            parent = self.getParent(row, site)
    306309            if parent is None:
    307310                errs.append(('container', 'not found'))
    … …  
    332335            return 'Applicant is blocked.'
    333336        return None
    334 
    335     def doImport(self, *args, **kw):
    336         # XXX: Not thread-safe.  Parallel applicant imports into
    337         # different sites could mean a mess.  Luckily this is not a
    338         # typical use-case. On the other hand it spares thousands of
    339         # site lookups during large imports.
    340         # XXX: Maybe this should go into Importer base.
    341         self.site = grok.getSite() # needed by checkConversion()
    342         return super(ApplicantProcessor, self).doImport(*args, **kw)

  • main/waeup.kofa/trunk/src/waeup/kofa/authentication.py

    r12190 r12869  
    449449
    450450class UserProcessor(BatchProcessor):
    451     """A batch processor for IUserAccount objects.
     451    """The User Processor processes user accounts, i.e. `Account` objects in the
     452    ``users`` container.
     453
     454    The `roles` columns must contain Python list
     455    expressions like ``['waeup.PortalManager', 'waeup.ImportManager']``.
     456
     457    The processor does not import local roles. These can be imported
     458    by means of batch processors in the academic section.
    452459    """
    453460    grok.implements(IBatchProcessor)

  • main/waeup.kofa/trunk/src/waeup/kofa/students/batching.py

    r12623 r12869  
    6464    iface_bymatricnumber = IStudentUpdateByMatricNo
    6565
    66     location_fields = []
    6766    factory_name = 'waeup.Student'
    6867
    … …  
    312311    additional_fields = []
    313312
    314     #: header fields additionally required
    315     additional_headers = []
     313    # additional required fields (subset of additional_fields)
     314    additional_fields_required = []
    316315
    317316    @property
    … …  
    328327                "Need at least columns student_id " +
    329328                "or reg_number or matric_number for import!")
    330         for name in self.additional_headers:
     329        for name in self.additional_fields_required:
    331330            if not name in headerfields:
    332331                raise FatalCSVError(
    … …  
    410409    iface_transfer = IStudentStudyCourseTransfer
    411410    factory_name = 'waeup.StudentStudyCourse'
    412 
    413     location_fields = []
    414     additional_fields = []
    415411
    416412    def getParent(self, row, site):
    … …  
    510506    factory_name = 'waeup.StudentStudyLevel'
    511507
    512     location_fields = []
    513 
    514508    additional_fields = ['level']
    515     additional_headers = ['level']
     509    additional_fields_required = additional_fields
    516510
    517511    @property
    … …  
    591585    factory_name = 'waeup.CourseTicket'
    592586
    593     location_fields = []
    594587    additional_fields = ['level', 'code']
    595     additional_headers = ['level', 'code']
     588    additional_fields_required = additional_fields
    596589
    597590    @property
    … …  
    695688    factory_name = 'waeup.StudentOnlinePayment'
    696689
    697     location_fields = []
    698690    additional_fields = ['p_id']
    699     additional_headers = []
    700691
    701692    def checkHeaders(self, headerfields, mode='ignore'):

  • main/waeup.kofa/trunk/src/waeup/kofa/university/batching.py

    r11891 r12869  
    4444
    4545class FacultyProcessor(BatchProcessor):
    46     """A batch processor for IFaculty objects.
     46    """The Faculty Processor processes faculties in the `faculties` container.
     47    The `FacultyProcessor` class also serves as a baseclass for all other
     48    batch processors in the academic section.
     49
     50    The processor makes some efforts to set local roles.
     51    If new roles are provided, the `updateEntry` method first removes
     52    all existing roles and then sets the new roles as given in the import
     53    file. That means the entire set of local roles is replaced.
    4754    """
    4855    grok.implements(IBatchProcessor)
    … …  
    103110            role_map = IPrincipalRoleMap(obj)
    104111            # Remove all existing local roles.
    105             for local_role, user_name, setting in role_map.getPrincipalsAndRoles():
     112            for local_role, user_name, setting in \
     113                role_map.getPrincipalsAndRoles():
    106114                role_manager.unsetRoleForPrincipal(local_role, user_name)
    107115                notify(LocalRoleSetEvent(
    … …  
    114122                    local_role = rolemap['local_role']
    115123                    role_manager.assignRoleToPrincipal(local_role, user)
    116                     notify(LocalRoleSetEvent(obj, local_role, user, granted=True))
    117                     items_changed += (
    118                         '%s=%s, ' % ('local_roles', '%s|%s' % (user,local_role)))
     124                    notify(LocalRoleSetEvent(
     125                        obj, local_role, user, granted=True))
     126                    items_changed += ('%s=%s, '
     127                        % ('local_roles', '%s|%s' % (user,local_role)))
    119128            row.pop('local_roles')
    120129
    … …  
    151160                if not 'user_name' in rolemap.keys() or not \
    152161                    'local_role' in rolemap.keys():
    153                     errs.append(('local_roles','user_name or local_role missing'))
     162                    errs.append((
     163                        'local_roles','user_name or local_role missing'))
    154164                    return errs, inv_errs, conv_dict
    155165                local_role = rolemap['local_role']
    … …  
    165175
    166176class DepartmentProcessor(FacultyProcessor):
    167     """A batch processor for IDepartment objects.
     177    """The Department Processor works in the same way as the Faculty
     178    Processor. Since department codes are not necessarily unique, it needs the
     179    `faculty_code` to create and update objects.
    168180    """
    169181    grok.implements(IBatchProcessor)
    … …  
    213225
    214226class CertificateProcessor(FacultyProcessor):
    215     """A batch processor for ICertificate objects.
     227    """The Certificate Processor gets the parent object (the
     228    `certificates` attribute of the department container) in two ways.
     229    If both faculty and department codes are provided, `getPartents` uses
     230    these to locate the certificate. If department code or
     231    faculty code are missing, it use the certificates catalog to find the
     232    certificate.
    216233    """
    217234    grok.implements(IBatchProcessor)
    … …  
    270287            return dept.certificates
    271288        # If department code or faculty code is missing,
    272         # use catalog to get parent. Makes only sense in update mode but
    273         # does also work in create mode.
     289        # use catalog to get parent.
    274290        cat = queryUtility(ICatalog, name='certificates_catalog')
    275291        results = list(
    … …  
    296312
    297313class CourseProcessor(CertificateProcessor):
    298     """A batch processor for ICourse objects.
     314    """The Course Processor works exactly in the same way as the
     315    Certificate Processor. It uses the courses catalog instead of the
     316    certificates catalog.
    299317    """
    300318    grok.implements(IBatchProcessor)
    … …  
    343361
    344362class CertificateCourseProcessor(FacultyProcessor):
    345     """A batch processor for ICertificateCourse objects.
     363    """The Certificate Course Processor needs more location fields.
     364    Certificate courses are stored inside the certificate container.
     365    Thus, `faculty_code`, `department_code` and the
     366    `certificate_code` are necessary to find the parent container.
     367    It furthermore needs the `course` and the `level` field to locate
     368    existing objects as they are part of the object id (code).
    346369    """
    347370    grok.implements(IBatchProcessor)

  • main/waeup.kofa/trunk/src/waeup/kofa/university/export.py

    r12858 r12869  
    2626
    2727class FacultyExporter(grok.GlobalUtility, ExporterBase):
    28     """The Faculty Exporter exports all faculties in the 'faculties'
     28    """The Faculty Exporter exports all faculties in the `faculties`
    2929    container. This is the only place where faculties are stored.
    3030    """

  • main/waeup.kofa/trunk/src/waeup/kofa/utils/batching.py

    r12868 r12869  
    5656
    5757    # Internal name...
    58     util_name = 'baseprocessor'
     58    util_name = ''
    5959
    6060    # Items for this processor need an interface with zope.schema fields.
    … …  
    6666
    6767    # Headers needed to locate items...
    68     location_fields = ['code', 'faculty_code']
     68    location_fields = []
    6969
    7070    # A factory with this name must be registered...
    71     factory_name = 'waeup.Department'
     71    factory_name = ''
    7272
    7373    @property
    … …  
    128128    def applyMapping(self, row, mapping):
    129129        """Apply mapping to a row of CSV data.
    130 
    131130        """
    132131        result = dict()
    … …  
    284283    def doImport(self, path, headerfields, mode='create', user='Unknown',
    285284                 logger=None, ignore_empty=True):
    286         """In contrast to most other methods, ``doImport`` is not supposed to
     285        """In contrast to most other methods, `doImport` is not supposed to
    287286        be customized, neither in custom packages nor in derived batch
    288287        processor classes. Therefore, this is the only place where we
    … …  
    317316           record is stored in the pending data file.
    318317
    319            Now ``doImport`` tries to add the new object with the data
     318           Now `doImport` tries to add the new object with the data
    320319           from the conversion dictionary. In some cases this
    321            may fail and a DuplicationError is raised. For example, a new
     320           may fail and a `DuplicationError` is raised. For example, a new
    322321           payment ticket is created but the same payment for same session
    323322           has already been made. In this case the object id is unique, no
    … …  
    339338           transitions or states.
    340339
    341            Finally, ``doImport`` updates the existing object with the data
     340           Finally, `doImport` updates the existing object with the data
    342341           from the conversion dictionary.
    343342
    … …  
    348347           stored in the pending data file.
    349348
    350            Finally, ``doImport`` removes the existing object.
     349           Finally, `doImport` removes the existing object.
    351350
    352351        """
Note: See TracChangeset for help on using the changeset viewer.