Ignore:
Timestamp:
19 Apr 2015, 09:50:19 (10 years ago)
Author:
Henrik Bettermann
Message:

Build subsections. Reorganize content. Rename user documentation and developer documentation.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • main/waeup.kofa/trunk/docs/source/userdocs/datacenter.rst

    r12862 r12863  
    44***********
    55
    6 .. contents::
     6.. toctree::
     7   :maxdepth: 3
    78
    8 Introduction
    9 ============
    10 
    11 Most web portals store their data in a relational database like PostgreSQL, MySQL or Oracle. A relational database is organized in tables of rows and columns, with a unique key for each row. Each data entity gets its own table. Rows in tables can be linked to rows in other tables by storing the unique key of the row to which it should be linked. This sounds quite simple. Many computer users are familiar with this kind of data storage because they are used to spreadsheet programmes like Excel oder Calc which also organize data in tables.
    12 
    13 Kofa's persistent data are stored in a native object database designed for the Python programming language, the so-called ZODB_. An object database stores objects with attributes and not records as rows with columns in tables. These persistent objects can hold any kind of information in attributes and must not adhere to a specific schema like records in tables of a relational database.
    14 
    15 The ZODB_ also supports a hierarchical, treelike storage of objects. Objects can contain other objects if they are declared as containers. Objects are stored like folders and files in a filesystem. This makes the object handling very fast and transparent because we can access objects, or more precisely views of objects, by indicating their path in the database, i.e. by traversing the database tree to the object's location. Furthermore, we are accessing the views of objects through a web browser by entering a URL (Uniform Resource Locator). This publication path corresponds more or less to the traversal path of our objects. In Kofa the path always contains the object identifiers of all objects which are passed when traversing the database tree. Example:
    16 
    17 https://kofa-demo.waeup.org/students/K1000000/studycourse/100/DCO
    18 
    19 is the URL which requests a display view of a course ticket with id ``DCO``. This object is stored in a study level container object with id ``100``, stored in a study course container object with id ``studycourse``, stored in the student container object with id ``K1000000``, stored in the students root container, stored in the root container of the application, stored in the root of the database itself.
    20 
    21 This kind of storage requires that each object gets a unique object identifier (object id) within its container. The id string is visible in the browser address bar. Though it's technically possible for ids to contain spaces or slashes we do not allow these kinds of special characters in object ids to facilitate the readability of URLs.
    22 
    23 Administrators of web portals, which store their data in relational databases, are used to getting direct access to the portal's database. There are even tools to handle the administration of these databases over the Internet, like phpMyAdmin or phpPgAdmin to handle MySQL or PostgreSQL databases respectively. These user interfaces bypass the portals' user interfaces and give direct access to the database. They allow to easily import or export (dump) data tables or the entire database structure into CSV or SQL files. What at first sight appears to be very helpful and administration-friendly proves to be very dangerous on closer inspection. Data structures can be easily damaged or destroyed, or data can be easily manipulated by circumventing the portal's security machinery or logging system. Kofa does not provide any external user interface to access the ZODB directly, neither for viewing nor for editing data. This includes also the export and import of sets of data. Exports and imports are handled via the Kofa user interface itself. This is called batch processing which means either producing CSV files (comma-separated values) from portal data (export) or processing CSV files in order to add, update or remove portal data (import). Main premise of Kofa's batch processing technology is that the data stored in the ZODB can be specifically backed up and restored by exporting and importing data. But that's not all. Batch processors can do much more. They are an integral part of the student registration management.
    24 
    25 Data Export
    26 ===========
    27 
    28 Regular data exporters (1) collect objects from specific containers, (2) iterate over the collected objects, (3) extract and mangle information from each object, (4) write the information of each object into a row of a CSV file and (5) finally provide the file for download. The CSV file is neither stored in the database nor archived in the filesystem. (3) and (4) means a flattening of the hierarchical data structure, i.e. a mapping of objects to flat relational data to be stored in a CSV table. The extracted information must not necessarily be based only on static attributes of the collected object. The data, finally stored in the CSV file, can also be derived from parent or child objects, or dynamically computed by the object's methods and property attributes. These methods and properties can retrieve information from everywhere in the portal's database.
    29 
    30 In the following we list all exporter classes including two attributes and a method description. The `fields` attribute contain the column titles of the export file. These are not necessarily only attributes of the exported objects.
    31 
    32 .. note::
    33 
    34   The list of exported columns usually underlies heavy customizations. In the Kofa base package only very few columns are being exported. In some Kofa custom packages tons of data are being gathered from applicants and students and the number of columns increase accordingly.
    35 
    36 The `title` attribute unveils the name of the exporter under which this exporter will be displayed in the user interface. The `mangle_value()` method shows how some of fields are being dynamically computed.
    37 
    38 Regular Exporters
    39 -----------------
    40 
    41 .. autoclass:: waeup.kofa.userscontainer.UserExporter()
    42   :noindex:
    43 
    44   .. autoattribute:: waeup.kofa.userscontainer.UserExporter.fields
    45   .. autoattribute:: waeup.kofa.userscontainer.UserExporter.title
    46   .. automethod:: waeup.kofa.userscontainer.UserExporter.mangle_value()
    47 
    48 .. autoclass:: waeup.kofa.university.export.FacultyExporter()
    49   :noindex:
    50 
    51   .. autoattribute:: waeup.kofa.university.export.FacultyExporter.fields
    52   .. autoattribute:: waeup.kofa.university.export.FacultyExporter.title
    53   .. automethod:: waeup.kofa.university.export.FacultyExporter.mangle_value()
    54 
    55 
    56 .. autoclass:: waeup.kofa.university.export.DepartmentExporter()
    57   :noindex:
    58 
    59   .. autoattribute:: waeup.kofa.university.export.DepartmentExporter.fields
    60   .. autoattribute:: waeup.kofa.university.export.DepartmentExporter.title
    61   .. automethod:: waeup.kofa.university.export.DepartmentExporter.mangle_value()
    62 
    63 .. autoclass:: waeup.kofa.university.export.CourseExporter()
    64   :noindex:
    65 
    66   .. autoattribute:: waeup.kofa.university.export.CourseExporter.fields
    67   .. autoattribute:: waeup.kofa.university.export.CourseExporter.title
    68   .. automethod:: waeup.kofa.university.export.CourseExporter.mangle_value()
    69 
    70 .. autoclass:: waeup.kofa.university.export.CertificateExporter()
    71   :noindex:
    72 
    73   .. autoattribute:: waeup.kofa.university.export.CertificateExporter.fields
    74   .. autoattribute:: waeup.kofa.university.export.CertificateExporter.title
    75   .. automethod:: waeup.kofa.university.export.CertificateExporter.mangle_value()
    76 
    77 .. autoclass:: waeup.kofa.university.export.CertificateCourseExporter()
    78   :noindex:
    79 
    80   .. autoattribute:: waeup.kofa.university.export.CertificateCourseExporter.fields
    81   .. autoattribute:: waeup.kofa.university.export.CertificateCourseExporter.title
    82   .. automethod:: waeup.kofa.university.export.CertificateCourseExporter.mangle_value()
    83 
    84 
    85 .. autoclass:: waeup.kofa.accesscodes.export.AccessCodeBatchExporter()
    86   :noindex:
    87 
    88   .. autoattribute:: waeup.kofa.accesscodes.export.AccessCodeBatchExporter.fields
    89   .. autoattribute:: waeup.kofa.accesscodes.export.AccessCodeBatchExporter.title
    90 
    91 .. autoclass:: waeup.kofa.accesscodes.export.AccessCodeExporter()
    92   :noindex:
    93 
    94   .. autoattribute:: waeup.kofa.accesscodes.export.AccessCodeExporter.fields
    95   .. autoattribute:: waeup.kofa.accesscodes.export.AccessCodeExporter.title
    96   .. automethod:: waeup.kofa.accesscodes.export.AccessCodeExporter.mangle_value()
    97 
    98 .. autoclass:: waeup.kofa.hostels.export.HostelExporter()
    99   :noindex:
    100 
    101   .. autoattribute:: waeup.kofa.hostels.export.HostelExporter.fields
    102   .. autoattribute:: waeup.kofa.hostels.export.HostelExporter.title
    103 
    104 .. autoclass:: waeup.kofa.hostels.export.BedExporter()
    105   :noindex:
    106 
    107   .. autoattribute:: waeup.kofa.hostels.export.BedExporter.fields
    108   .. autoattribute:: waeup.kofa.hostels.export.BedExporter.title
    109 
    110 Application Data Exporters
    111 --------------------------
    112 
    113 .. autoclass:: waeup.kofa.applicants.export.ApplicantsContainerExporter()
    114   :noindex:
    115 
    116   .. autoattribute:: waeup.kofa.applicants.export.ApplicantsContainerExporter.fields
    117   .. autoattribute:: waeup.kofa.applicants.export.ApplicantsContainerExporter.title
    118 
    119 .. autoclass:: waeup.kofa.applicants.export.ApplicantExporter()
    120   :noindex:
    121 
    122   .. autoattribute:: waeup.kofa.applicants.export.ApplicantExporter.fields
    123   .. autoattribute:: waeup.kofa.applicants.export.ApplicantExporter.title
    124   .. automethod:: waeup.kofa.applicants.export.ApplicantExporter.mangle_value()
    125 
    126 Student Data Exporters
    127 ----------------------
    128 
    129 When starting a Student Data Exporter in the Data Center all student records will be taken into consideration, no matter what or where a student is studying. The exporter can also be started 'locally' at various levels in the academic section. Starting one of the exporters e.g. at faculty or department level means that only the data of students are exported who study in this faculty or department respectively. The exporter can also be started at certificate level. Then only the data of students, who are studying the named study course, will be taken into account. At course level the data of those students are being exported who have attended or taken this specific course.
    130 
    131 Student Data Exporter can be further configured through a configuration page. Search parameters like the student's current level, current session and current study mode  can be set to filter sets of students in order to decrease the size of the export file. The set of filter parameters varies and depends on the 'locatation' from where the exporter is called. A completely different set of filter parameters is provided for courses. In this case the session and level can be selected when the course was taken by the student.
    132 
    133 .. autoclass:: waeup.kofa.students.export.StudentExporter()
    134   :noindex:
    135 
    136   .. autoattribute:: waeup.kofa.students.export.StudentExporter.fields
    137   .. autoattribute:: waeup.kofa.students.export.StudentExporter.title
    138   .. automethod:: waeup.kofa.students.export.StudentExporter.mangle_value()
    139 
    140 .. autoclass:: waeup.kofa.students.export.StudentStudyCourseExporter()
    141   :noindex:
    142 
    143   .. autoattribute:: waeup.kofa.students.export.StudentStudyCourseExporter.fields
    144   .. autoattribute:: waeup.kofa.students.export.StudentStudyCourseExporter.title
    145   .. automethod:: waeup.kofa.students.export.StudentStudyCourseExporter.mangle_value()
    146 
    147 .. autoclass:: waeup.kofa.students.export.StudentStudyLevelExporter()
    148   :noindex:
    149 
    150   .. autoattribute:: waeup.kofa.students.export.StudentStudyLevelExporter.fields
    151   .. autoattribute:: waeup.kofa.students.export.StudentStudyLevelExporter.title
    152   .. automethod:: waeup.kofa.students.export.StudentStudyLevelExporter.mangle_value()
    153 
    154 .. autoclass:: waeup.kofa.students.export.CourseTicketExporter()
    155   :noindex:
    156 
    157   .. autoattribute:: waeup.kofa.students.export.CourseTicketExporter.fields
    158   .. autoattribute:: waeup.kofa.students.export.CourseTicketExporter.title
    159   .. automethod:: waeup.kofa.students.export.CourseTicketExporter.mangle_value()
    160 
    161 .. autoclass:: waeup.kofa.students.export.StudentPaymentsExporter()
    162   :noindex:
    163 
    164   .. autoattribute:: waeup.kofa.students.export.StudentPaymentsExporter.fields
    165   .. autoattribute:: waeup.kofa.students.export.StudentPaymentsExporter.title
    166   .. automethod:: waeup.kofa.students.export.StudentPaymentsExporter.mangle_value()
    167 
    168 .. autoclass:: waeup.kofa.students.export.BedTicketsExporter()
    169   :noindex:
    170 
    171   .. autoattribute:: waeup.kofa.students.export.BedTicketsExporter.fields
    172   .. autoattribute:: waeup.kofa.students.export.BedTicketsExporter.title
    173   .. automethod:: waeup.kofa.students.export.BedTicketsExporter.mangle_value()
    174 
    175 The above exporters refer to a specific content type (object class). They export all attributes of these objects and a few additional parameters derived from the parent objects. These exporters can be used for reimport, or more precisely for backing up and restoring data. The following 'special' exporters are made on request of some universities to collect and compose student data for analysis and postprocessing by the university.
    176 
    177 .. autoclass:: waeup.kofa.students.export.DataForBursaryExporter()
    178   :noindex:
    179 
    180   .. autoattribute:: waeup.kofa.students.export.DataForBursaryExporter.fields
    181   .. autoattribute:: waeup.kofa.students.export.DataForBursaryExporter.title
    182   .. automethod:: waeup.kofa.students.export.DataForBursaryExporter.mangle_value()
    183 
    184 .. autoclass:: waeup.kofa.students.export.StudentPaymentsOverviewExporter()
    185   :noindex:
    186 
    187   .. autoattribute:: waeup.kofa.students.export.StudentPaymentsOverviewExporter.fields
    188   .. autoattribute:: waeup.kofa.students.export.StudentPaymentsOverviewExporter.title
    189   .. autoattribute:: waeup.kofa.students.export.StudentPaymentsOverviewExporter.curr_year
    190   .. automethod:: waeup.kofa.students.export.StudentPaymentsOverviewExporter.mangle_value()
    191 
    192 .. autoclass:: waeup.kofa.students.export.StudentStudyLevelsOverviewExporter()
    193   :noindex:
    194 
    195   .. autoattribute:: waeup.kofa.students.export.StudentStudyLevelsOverviewExporter.fields
    196   .. autoattribute:: waeup.kofa.students.export.StudentStudyLevelsOverviewExporter.title
    197   .. automethod:: waeup.kofa.students.export.StudentStudyLevelsOverviewExporter.mangle_value()
    198 
    199 .. autoclass:: waeup.kofa.students.export.ComboCardDataExporter()
    200   :noindex:
    201 
    202   .. autoattribute:: waeup.kofa.students.export.ComboCardDataExporter.fields
    203   .. autoattribute:: waeup.kofa.students.export.ComboCardDataExporter.title
    204   .. automethod:: waeup.kofa.students.export.ComboCardDataExporter.mangle_value()
    205 
    206 Data Import
    207 ===========
    208 
    209 Logging
    210 =======
    211 
    212 
    213 
    214 
    215 
    216 
    217 
    218 .. _ZODB: http://www.zodb.org/
     9   datacenter/intro
     10   datacenter/export
     11   datacenter/import
     12   datacenter/logging
Note: See TracChangeset for help on using the changeset viewer.