source: main/waeup.kofa/trunk/docs/source/userdocs/datacenter.rst @ 12861

.. _datacenter:

Data Center
***********

.. contents::

Introduction
============

Most web portals store their data in a relational database like PostgreSQL, MySQL or Oracle. A relational database is organized in tables of rows and columns, with a unique key for each row. Each data entity gets its own table. Rows in tables can be linked to rows in other tables by storing the unique key of the row to which it should be linked. This sounds quite simple. Many computer users are familiar with this kind of data storage because they are used to spreadsheet programmes like Excel oder Calc which also organize data in tables.

Kofa's persistent data are stored in a native object database designed for the Python programming language, the so-called ZODB_. An object database stores objects with attributes and not records as rows with columns in tables. These persistent objects can hold any kind of information in attributes and must not adhere to a specific schema like records in tables of a relational database.

The ZODB_ also supports a hierarchical, treelike storage of objects. Objects can contain other objects if they are declared as containers. Objects are stored like folders and files in a filesystem. This makes the object handling very fast and transparent because we can access objects, or more precisely views of objects, by indicating their path in the database, i.e. by traversing the database tree to the object's location. Furthermore, we are accessing the views of objects through a web browser by entering a URL (Uniform Resource Locator). This publication path corresponds more or less to the traversal path of our objects. In Kofa the path always contains the object identifiers of all objects which are passed when traversing the database tree. Example:

https://kofa-demo.waeup.org/students/K1000000/studycourse/100/DCO

is the URL which requests a display view of a course ticket with id ``DCO``. This object is stored in a study level container object with id ``100``, stored in a study course container object with id ``studycourse``, stored in the student container object with id ``K1000000``, stored in the students root container, stored in the root container of the application, stored in the root of the database itself.

This kind of storage requires that each object gets a unique object identifier (object id) within its container. The id string is visible in the browser address bar. Though it's technically possible for ids to contain spaces or slashes we do not allow these kinds of special characters in object ids to facilitate the readability of URLs.

Administrators of web portals, which store their data in relational databases, are used to getting direct access to the portal's database. There are even tools to handle the administration of these databases over the Internet, like phpMyAdmin or phpPgAdmin to handle MySQL or PostgreSQL databases respectively. These user interfaces bypass the portals' user interfaces and give direct access to the database. They allow to easily import or export (dump) data tables or the entire database structure into CSV or SQL files. What at first sight appears to be very helpful and administration-friendly proves to be very dangerous on closer inspection. Data structures can be easily damaged or destroyed, or data can be easily manipulated by circumventing the portal's security machinery or logging system. Kofa does not provide any external user interface to access the ZODB directly, neither for viewing nor for editing data. This includes also the export and import of sets of data. Exports and imports are handled via the Kofa user interface itself. This is called batch processing which means either producing CSV files (comma-separated values) from portal data (export) or processing CSV files in order to add, update or remove portal data (import). Main premise of Kofa's batch processing technology is that the data stored in the ZODB can be specifically backed up and restored by exporting and importing data. But that's not all. Batch processors can do much more. They are an integral part of the student registration management.

Data Export
===========

Regular data exporters (1) collect objects from specific containers, (2) iterate over the collected objects, (3) extract and mangle information from each object, (4) write the information of each object into a row of a CSV file and (5) finally provide the file for download. The CSV file is neither stored in the database nor archived in the filesystem. (3) and (4) means a flattening of the hierarchical data structure, i.e. a mapping of objects to flat relational data to be stored in a CSV table. The extracted information must not necessarily be based only on static attributes of the collected object. The data, finally stored in the CSV file, can also be derived from parent or child objects, or dynamically computed by the object's methods and property attributes. These methods and properties can retrieve information from everywhere in the portal's database.

In the following we list all exporter classes including two attributes and a method description. The `fields` attribute contain the column titles of the export file. These are not necessarily only attributes of the exported objects.

.. note::

  The list of exported columns usually underlies heavy customizations. In the Kofa base package only very fiew columns are being exported. In some Kofa custom packages tons of data are being gathered from applicants and students and the number of columns increase accordingly.

The `title` attribute unveils the name of the exporter under which this exporter will be displayed in the user interface. The `mangle_value()` method shows how some of fields are being dynamically computed.

Regular Exporters
-----------------

.. autoclass:: waeup.kofa.userscontainer.UserExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.userscontainer.UserExporter.fields
  .. autoattribute:: waeup.kofa.userscontainer.UserExporter.title
  .. automethod:: waeup.kofa.userscontainer.UserExporter.mangle_value()

.. autoclass:: waeup.kofa.university.export.FacultyExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.university.export.FacultyExporter.fields
  .. autoattribute:: waeup.kofa.university.export.FacultyExporter.title
  .. automethod:: waeup.kofa.university.export.FacultyExporter.mangle_value()


.. autoclass:: waeup.kofa.university.export.DepartmentExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.university.export.DepartmentExporter.fields
  .. autoattribute:: waeup.kofa.university.export.DepartmentExporter.title
  .. automethod:: waeup.kofa.university.export.DepartmentExporter.mangle_value()

.. autoclass:: waeup.kofa.university.export.CourseExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.university.export.CourseExporter.fields
  .. autoattribute:: waeup.kofa.university.export.CourseExporter.title
  .. automethod:: waeup.kofa.university.export.CourseExporter.mangle_value()

.. autoclass:: waeup.kofa.university.export.CertificateExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.university.export.CertificateExporter.fields
  .. autoattribute:: waeup.kofa.university.export.CertificateExporter.title
  .. automethod:: waeup.kofa.university.export.CertificateExporter.mangle_value()

.. autoclass:: waeup.kofa.university.export.CertificateCourseExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.university.export.CertificateCourseExporter.fields
  .. autoattribute:: waeup.kofa.university.export.CertificateCourseExporter.title
  .. automethod:: waeup.kofa.university.export.CertificateCourseExporter.mangle_value()


.. autoclass:: waeup.kofa.accesscodes.export.AccessCodeBatchExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.accesscodes.export.AccessCodeBatchExporter.fields
  .. autoattribute:: waeup.kofa.accesscodes.export.AccessCodeBatchExporter.title

.. autoclass:: waeup.kofa.accesscodes.export.AccessCodeExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.accesscodes.export.AccessCodeExporter.fields
  .. autoattribute:: waeup.kofa.accesscodes.export.AccessCodeExporter.title
  .. automethod:: waeup.kofa.accesscodes.export.AccessCodeExporter.mangle_value()

.. autoclass:: waeup.kofa.hostels.export.HostelExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.hostels.export.HostelExporter.fields
  .. autoattribute:: waeup.kofa.hostels.export.HostelExporter.title

.. autoclass:: waeup.kofa.hostels.export.BedExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.hostels.export.BedExporter.fields
  .. autoattribute:: waeup.kofa.hostels.export.BedExporter.title

Application Data Exporters
--------------------------

.. autoclass:: waeup.kofa.applicants.export.ApplicantsContainerExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.applicants.export.ApplicantsContainerExporter.fields
  .. autoattribute:: waeup.kofa.applicants.export.ApplicantsContainerExporter.title

.. autoclass:: waeup.kofa.applicants.export.ApplicantExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.applicants.export.ApplicantExporter.fields
  .. autoattribute:: waeup.kofa.applicants.export.ApplicantExporter.title
  .. automethod:: waeup.kofa.applicants.export.ApplicantExporter.mangle_value()

Student Data Exporters
----------------------

When starting a Student Data Exporter in the Data Center all student records will be taken into consideration, no matter what or where a student is studying. The exporter can also be started 'locally' at various levels in the academic section. Starting one of the exporters e.g. at faculty or department level means that only the data of students are exported who study in this faculty or department respectively. The exporter can also be started at certificate level. Then only the data of students, who are studying the named study course, will be taken into account. At course level the data of those students are being exported who have attended or taken this specific course.

Student Data Exporter can be further configured through a configuration page. Search parameters like the student's current level, current session and current study mode  can be set to filter sets of students in order to decrease the size of the export file. The set of filter parameters varies and depends on the 'locatation' from where the exporter is called. A completely different set of filter parameters is provided for courses. In this case the session and level can be selected when the course was taken by the student.

.. autoclass:: waeup.kofa.students.export.StudentExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.students.export.StudentExporter.fields
  .. autoattribute:: waeup.kofa.students.export.StudentExporter.title
  .. automethod:: waeup.kofa.students.export.StudentExporter.mangle_value()

.. autoclass:: waeup.kofa.students.export.StudentStudyCourseExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.students.export.StudentStudyCourseExporter.fields
  .. autoattribute:: waeup.kofa.students.export.StudentStudyCourseExporter.title
  .. automethod:: waeup.kofa.students.export.StudentStudyCourseExporter.mangle_value()

.. autoclass:: waeup.kofa.students.export.StudentStudyLevelExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.students.export.StudentStudyLevelExporter.fields
  .. autoattribute:: waeup.kofa.students.export.StudentStudyLevelExporter.title
  .. automethod:: waeup.kofa.students.export.StudentStudyLevelExporter.mangle_value()

.. autoclass:: waeup.kofa.students.export.CourseTicketExporter()
  :noindex:

  .. autoattribute:: waeup.kofa.students.export.CourseTicketExporter.fields
  .. autoattribute:: waeup.kofa.students.export.CourseTicketExporter.title
  .. automethod:: waeup.kofa.students.export.CourseTicketExporter.mangle_value()

Data Import
===========

Logging
=======







.. _ZODB: http://www.zodb.org/