Changeset 12868 for main/waeup.kofa/trunk

Timestamp:

21 Apr 2015, 20:56:58 (10 years ago)

Author:

Henrik Bettermann

Message:

More docs.

Location:

main/waeup.kofa/trunk

Files:

• docs/source/userdocs/datacenter/import.rst (modified) (3 diffs)
• src/waeup/kofa/browser/batchprocessing.txt (modified) (1 diff)
• src/waeup/kofa/students/tests/test_batching.py (modified) (3 diffs)
• src/waeup/kofa/university/tests/test_batching.py (modified) (4 diffs)
• src/waeup/kofa/utils/batching.py (modified) (4 diffs)
• src/waeup/kofa/utils/batching.txt (modified) (2 diffs)

main/waeup.kofa/trunk/docs/source/userdocs/datacenter/import.rst

@@ -4 +4 @@
 ***********
 
-The term 'Data Import' actually understates the range of functions importers really have. As already stated, many importers do not only restore data once backed up by exporters or, in other words, take values from CSV files and write them one-on-one into the database. The data undergo a complex staged data processing algorithm. Therefore, we prefer calling them 'batch processors' instead of importers. The staged import process is described in the following.
+Stages of Batch Processing
+==========================
 
-1. File Upload
-==============
+The term 'data import' actually understates the range of functions importers really have. As already stated, many importers do not only restore data once backed up by exporters or, in other words, take values from CSV files and write them one-on-one into the database. The data undergo a complex staged data processing algorithm. Therefore, we prefer calling them 'batch processors' instead of importers. The stages of the import process are as follows.
+
+Stage 1: File Upload
+--------------------
 
 Users with permission
@@ -17 +20 @@
 .. autoattribute:: waeup.kofa.browser.pages.DatacenterUploadPage.max_files
 
-When the upload succeeded the uploader sends an email to all import managers (users with role :py:class:`waeup.ImportManager<waeup.kofa.permissions.ImportManager>`) of the portal that a new file was uploaded.
+If the upload succeeded the uploader sends an email to all import managers (users with role :py:class:`waeup.ImportManager<waeup.kofa.permissions.ImportManager>`) of the portal that a new file was uploaded.
 
-The uploader changes the filename. An uploaded file foo.csv will be stored as foo_USERNAME.csv where username is the user id of the currently logged in user. Spaces in filename are replaced by underscores. Pending data filenames remain unchanged (see below)
+The uploader changes the filename. An uploaded file ``foo.csv`` will be stored as ``foo_USERNAME.csv`` where username is the user id of the currently logged in user. Spaces in filename are replaced by underscores. Pending data filenames remain unchanged (see below).
 
-After file upload the data center manager can click the 'Process data' button to open the page where files can be selected for import (import step 1). After selecting a file the data center manager can preview the header and the first three records of the uploaded file (import step 2). If the preview fails or the header contains duplicate column titles, an error message is raised. The user cannot proceed but is requested to replace the uploaded file. If the preview succeeds the user can proceed to the next step (import step 3) by selecting the appropriate processor and an import mode.
+After file upload the data center manager can click the 'Process data' button to open the page where files can be selected for import (**import step 1**). After selecting a file the data center manager can preview the header and the first three records of the uploaded file (**import step 2**). If the preview fails or the header contains duplicate column titles, an error message is raised. The user cannot proceed but is requested to replace the uploaded file. If the preview succeeds the user is able to proceed to the next step (**import step 3**) by selecting the appropriate processor and an import mode. In import mode ``create`` new objects are added to the database, `in `update`` mode existing objects are modified and in ``remove`` mode deleted.
 
-2. File Header Validation
-=========================
+Stage 2: File Header Validation
+-------------------------------
 
-Import step 3 is the stage where the file content is assessed for the first time and checked if the column titles correspond with the fields of the processor chosen. The page shows the header and the first record of the uploaded file. The page allows to change column fields or to ignore entire columns during import. It might have happened that one or more column titles are misspelled or that the person, who created the file, ignored the case-sensitivity of field names. Then the data center manager can easily fix this by selecting the correct title and click the 'Set headerfields' button. Setting the header fields is temporary, it does not change the file itself.
+Import step 3 is the stage where the file content is assessed for the first time and checked if the column titles correspond with the fields of the processor chosen. The page shows the header and the first record of the uploaded file. The page allows to change column titles or to ignore entire columns during import. It might have happened that one or more column titles are misspelled or that the person, who created the file, ignored the case-sensitivity of field names. Then the data import manager can easily fix this by selecting the correct title and click the 'Set headerfields' button. Setting the column titles is temporary, it does not modify the uploaded file. Consequently, it does not make sense to set new column titles if the file is not imported afterwards.
 
-The page also calls the `checkHeaders` method of the batch processor which checks for required fields. If a required column title is missing, a warning message is raised and the user can't proceed to the next step.
+The page also calls the `checkHeaders` method of the batch processor which checks for required fields. If a required column title is missing, a warning message is raised and the user can't proceed to the next step (**import step 4**).
 
-3. Data Validation and Import
-=============================
+Stage 3: Data Validation and Import
+-----------------------------------
+
+Import step 4 is the actual data import. The import is started by clicking the 'Perform import' button.
 
 Kofa does not validate the data in advance. It tries to import the data row-by-row while reading the CSV file. The reason is that import files very often  contain thousands or even tenthousands of records. It is not feasable for data managers to edit import files until they are error-free. Very often such an error is not really a mistake made by the person who compiled the file. Example: The import file contains course results although the student has not yet registered the courses. Then the import of this single record has to wait, i.e. it has to be marked pending, until the student has added the course ticket. Only then it can be edited by the batch processor.
@@ -40 +45 @@
    :noindex:
 
+Stage 4: Post-Processing
+------------------------
 
+The data import is finalized by calling :py:meth:`distProcessedFiles<waeup.kofa.datacenter.DataCenter.distProcessedFiles>`. This method moves the ``.pending`` and ``.finished`` files from their temporary to their final location in the storage path of the filesystem from where they can be accessed through browser user interface.
 
+Batch Processors
+================
+
+All batch processors inherit their methods from the :py:class:`waeup.kofa.utils.batching.BatchProcessor` base class. The core ``doImport`` method always remains unchanged.

main/waeup.kofa/trunk/src/waeup/kofa/browser/batchprocessing.txt

@@ -574 +574 @@
     >>> print open(pending_file).read()
     title_prefix,code,title,--ERRORS--
-    faculty,FAC1,Faculty 1,This object already exists. Skipping.
+    faculty,FAC1,Faculty 1,This object already exists.
     faculty,FAC 5,Faculty 5,code: Invalid input
 

main/waeup.kofa/trunk/src/waeup/kofa/students/tests/test_batching.py

@@ -548 +548 @@
             '2,Aaren,C123456,m,aa@aa.ng,1234,admitted,1990-01-04,Berson,mypw1,100000,matric_number: Invalid input\r\n'
             '1,Frank,F123456,m,aa@aa.ng,1234,,1990-01-06,Meyer,,100000,reg_number: Invalid input; matric_number: Invalid input\r\n'
-            '3,Uli,A123456,m,aa@aa.ng,1234,,1990-01-07,Schulz,,100002,This object already exists. Skipping.\r\n'
+            '3,Uli,A123456,m,aa@aa.ng,1234,,1990-01-07,Schulz,,100002,This object already exists.\r\n'
             )
         shutil.rmtree(os.path.dirname(fin_file))
@@ -891 +891 @@
         self.assertEqual(fail_file,
             'reg_number,code,mandatory,level,level_session,score,matric_number,--ERRORS--\r\n'
-            '1,COURSE1,,nonsense,,5,,Not all parents do exist yet. Skipping\r\n'
+            '1,COURSE1,,nonsense,,5,,Not all parents do exist yet.\r\n'
             '1,NONSENSE,,100,,5,,code: non-existent\r\n'
             '1,COURSE1,,200,2004,6,,level_session: does not match 2008\r\n'
@@ -1176 +1176 @@
             '1,942,online,BTECHBDT,2010/11/26 19:59:33.744 GMT+1,0,'
             '19500,schoolfee,19500,2015,paid,'
-            'Same payment has already been made. Skipping.'
+            'Same payment has already been made.'
             in content)
         shutil.rmtree(os.path.dirname(fin_file))

main/waeup.kofa/trunk/src/waeup/kofa/university/tests/test_batching.py

@@ -448 +448 @@
             'local_roles: user_name or local_role missing\r\n'
             'FAC11,DEP2,"[{\'user_name\':\'anne\',\'local_role\':\'waeup.local.DepartmentManager\'}]",'
-            'Not all parents do exist yet. Skipping\r\n'
+            'Not all parents do exist yet.\r\n'
             )
         # Anne got a local role in department DEP2.
@@ -537 +537 @@
             'local_roles: user_name or local_role missing\r\n'
             'FAC11,DEP2,CRS2,"[{\'user_name\':\'anne\',\'local_role\':\'waeup.local.Lecturer\'}]",'
-            'Not all parents do exist yet. Skipping\r\n'
+            'Not all parents do exist yet.\r\n'
             )
         # Anne got a local role in course CRS2.
@@ -636 +636 @@
             'local_roles: user_name or local_role missing\r\n'
             'FAC11,DEP2,CRT2,"[{\'user_name\':\'anne\',\'local_role\':\'waeup.local.CourseAdviser100\'}]",'
-            'Not all parents do exist yet. Skipping\r\n'
+            'Not all parents do exist yet.\r\n'
             )
         # Anne got a local role in certificate CRT2.
@@ -726 +726 @@
             'faculty_code,course,level,department_code,certificate_code,'
             '--ERRORS--\r\nFAC1,CRS1,100,DEP1,CRT1,'
-            'This object already exists. Skipping.\r\nFAC1,CRS1,100,DEP1,CRT2,'
-            'Not all parents do exist yet. Skipping\r\n'
+            'This object already exists.\r\nFAC1,CRS1,100,DEP1,CRT2,'
+            'Not all parents do exist yet.\r\n'
 
             )

main/waeup.kofa/trunk/src/waeup/kofa/utils/batching.py

@@ -298 +298 @@
         1. An empty row is skipped.
 
-        2. Empty strings are replaced by ignore-markers.
-
-        3. The `BatchProcessor.checkConversion` method validates all 
-           values in the row. If the validation fails
-
-        4.
-
-        5.
-
-        6.
+        2. Empty strings in the row are replaced by ignore-markers.
+
+        3. The `BatchProcessor.checkConversion` method validates and converts 
+           all values in the row. Conversion means the transformation of strings
+           into Python objects. For instance, number expressions have to be
+           transformed into integers, dates into datetime objects, phone number
+           expressions into phone number objects, etc. The converter returns a
+           dictionary with converted values or, if the validation of one of the
+           elements fails, an appropriate warning message. If the conversion
+           fails a pending record is created and stored in the pending data file
+           together with a warning message the converter has raised.
+
+        4. In **create mode** only:
+
+           The parent object must be found and a child
+           object with same object id must not exist. Otherwise the row
+           is skipped, a corresponding warning message is raised and a
+           record is stored in the pending data file.
+
+           Now ``doImport`` tries to add the new object with the data 
+           from the conversion dictionary. In some cases this
+           may fail and a DuplicationError is raised. For example, a new
+           payment ticket is created but the same payment for same session
+           has already been made. In this case the object id is unique, no
+           other object with same id exists, but making the 'same' payment
+           twice does not make sense. The import is skipped and a
+           record is stored in the pending data file.
+
+        5. In **update mode** only:
+
+           If the object can't be found, the row is skipped,
+           a ``no such entry`` warning message is raised and a record is
+           stored in the pending data file.
+
+           The `BatchProcessor.checkUpdateRequirements` method checks additional
+           requirements the object must fulfill before being updated. These
+           requirements are not imposed by the data type but the context
+           of the object. For example, post-graduate students have a different
+           registration workflow. With this method we do forbid certain workflow
+           transitions or states. 
+
+           Finally, ``doImport`` updates the existing object with the data
+           from the conversion dictionary.
+
+        6. In **remove mode** only:
+
+           If the object can't be found, the row is skipped,
+           a ``no such entry`` warning message is raised and a record is
+           stored in the pending data file.
+
+           Finally, ``doImport`` removes the existing object.
 
         """
@@ -364 +405 @@
                     self.writeFailedRow(
                         failed_writer, string_row,
-                        "Not all parents do exist yet. Skipping")
+                        "Not all parents do exist yet.")
                     continue
                 if self.entryExists(row, site):
@@ -370 +411 @@
                     self.writeFailedRow(
                         failed_writer, string_row,
-                        "This object already exists. Skipping.")
+                        "This object already exists.")
                     continue
                 obj = self.callFactory()
@@ -382 +423 @@
                     num_warns += 1
                     self.writeFailedRow(
-                        failed_writer, string_row,
-                        "%s Skipping." % error.message)
+                        failed_writer, string_row, error.message)
                     continue
                 except DuplicationError, error:
                     num_warns += 1
                     self.writeFailedRow(
-                        failed_writer, string_row,
-                        "%s Skipping." % error.msg)
+                        failed_writer, string_row, error.msg)
                     continue
             elif mode == 'remove':

main/waeup.kofa/trunk/src/waeup/kofa/utils/batching.txt

@@ -309 +309 @@
     >>> print open(result[3]).read()
     owner,name,taxpayer,dinoports,--ERRORS--
-    Barney,Barneys Home,1,2,This object already exists. Skipping.
-    Wilma,Wilmas Asylum,1,1,This object already exists. Skipping.
-    Fred,Freds Dinoburgers,0,10,This object already exists. Skipping.
-    Joey,Joeys Drive-in,0,110,This object already exists. Skipping.
+    Barney,Barneys Home,1,2,This object already exists.
+    Wilma,Wilmas Asylum,1,1,This object already exists.
+    Fred,Freds Dinoburgers,0,10,This object already exists.
+    Joey,Joeys Drive-in,0,110,This object already exists.
 
 This way we can correct the faulty entries and afterwards retry without
@@ -350 +350 @@
     >>> print open(result[3], 'rb').read()
     name,dinoports,--ERRORS--
-    Barneys Home,2,This object already exists. Skipping.
-    Wilmas Asylum,1,This object already exists. Skipping.
-    Freds Dinoburgers,10,This object already exists. Skipping.
-    Joeys Drive-in,110,This object already exists. Skipping.
+    Barneys Home,2,This object already exists.
+    Wilmas Asylum,1,This object already exists.
+    Freds Dinoburgers,10,This object already exists.
+    Joeys Drive-in,110,This object already exists.