Changeset 12895 for main


Ignore:
Timestamp:
30 Apr 2015, 10:40:44 (10 years ago)
Author:
Henrik Bettermann
Message:

Wrap lines.

File:
1 edited

Legend:

Unmodified
Added
Removed
  • main/waeup.kofa/trunk/docs/source/userdocs/datacenter/logging.rst

    r12894 r12895  
    44********
    55
    6 Kofa is logging actions, which cause changes to the database or the filesystem, in separate human-redable event logfiles. Nearly all actions are being logged, except some actions committed by applicants and students during registration periods. During these data initialization periods, applicants and students are requested to edit many fields. Usually they press the 'Save' button quite often to backup the data entered so far. Logging all these changes would mean filling the logfiles with more or less useless or redundant information.
    7 
    8 The main purpose of Kofa's logging facilities is to trace changes effected by portal officers. Applicants and students can only access their own data. They don't have the permission to change anything else. Whereas portal officers bear responsibility for the integrity of the whole database. Depending on the roles they have, officers can add, edit or remove application and study data without leaving a trace in the object database itself. Kofa closes this security gap by recording nearly all actions of portal officers into various event logfiles.
     6Kofa logs actions, which cause changes to the database or the
     7filesystem, in separate human-redable event logfiles. Nearly all
     8actions are being logged, except some actions committed by
     9applicants and students during registration periods. During these
     10data initialization periods, applicants and students are requested
     11to edit many fields. Usually they press the 'Save' button quite
     12often to backup the data entered so far. Logging all these changes
     13would mean filling the logfiles with more or less useless or
     14redundant information.
     15
     16The main purpose of Kofa's logging facilities is to trace changes
     17effected by portal officers. Applicants and students can only access
     18their own data. They don't have the permission to change anything
     19else. Whereas portal officers bear responsibility for the integrity
     20of the whole database. Depending on the roles they have, officers
     21can add, edit or remove application and study data without leaving a
     22trace in the object database itself. Kofa closes this security gap
     23by recording nearly all actions of portal officers into various
     24event logfiles.
    925
    1026.. note::
    1127
    12  Kofa logs the changing but not the viewing of data (except downloading export files or reports). Also entering (login) end leaving (logout) the portal is **not** being recorded.
    13 
    14 Critics may claim that extensive logging may discourage officers from working with the portal. Persons in key functions often don't want to be controlled or supervised by anybody else. Indeed, Kofa does not respond to the desire for this kind of privacy. Datacenter managers can search all logfiles and can even see what their principles have done with the data. Goal behind is **transparency and security**, i.e. transparent data processing and reduction of the vulnerability to corruption combined with cutting-edge technologies to protect the data against attacks from outside.
    15 
    16 Logfiles can not be accessed directly. They can neither be downloaded nor viewed entirely online. But the files can be searched online for arbitrary `regular expressions`_. The simplest regular expression is a single word or a string. A technical description, how logfiles are being querried, can be found :py:meth:`here<waeup.kofa.datacenter.DataCenter.queryLogfiles>`.
    17 
    18 Each line in the logfile means a single log entry. It is composed of four parts: a datetime string, the log level (mainly 'INFO'), the id of the logged-in user, and a message part. Depending on the module, the message part is subdivided into further parts which are described below.
     28  Kofa logs the changing but not the viewing of data (except
     29  downloading export files or reports). Also entering (login) end
     30  leaving (logout) the portal is **not** being recorded.
     31
     32Critics may claim that extensive logging may discourage officers
     33from working with the portal. Persons in key functions often don't
     34want to be controlled or supervised by anybody else. Indeed, Kofa
     35does not respond to the desire for this kind of privacy. Datacenter
     36managers can search all logfiles and can even see what their
     37principles have done with the data. Goal behind is **transparency
     38and security**, i.e. transparent data processing and reduction of
     39the vulnerability to corruption combined with cutting-edge
     40technologies to protect the data against attacks from outside.
     41
     42Logfiles can not be accessed directly. They can neither be
     43downloaded nor viewed entirely online. But the files can be searched
     44online for arbitrary `regular expressions`_. The simplest regular
     45expression is a single word or a string. A technical description,
     46how logfiles are being querried, can be found
     47:py:meth:`here<waeup.kofa.datacenter.DataCenter.queryLogfiles>`.
     48
     49Each line in the logfile means a single log entry. It is composed of
     50four parts: a datetime string, the log level (mainly 'INFO'), the id
     51of the logged-in user, and a message part. Depending on the module,
     52the message part is subdivided into further parts which are
     53described below.
    1954
    2055
     
    2257========
    2358
    24 All actions in the academic section are logged in ``main.log``. Changes of faculties, departments, certificates, courses and certificate courses done via pages in the user interface (browser page) are recorded in the following form::
     59All actions in the academic section are logged in ``main.log``.
     60Changes of faculties, departments, certificates, courses and
     61certificate courses done via pages in the user interface (browser
     62page) are recorded in the following form::
    2563
    2664  2014-12-11 13:09:22,855 - INFO - admin - browser.pages.DepartmentAddFormPage - added: FAC1/DEP1
     
    2866  2014-12-12 13:19:43,255 - INFO - admin - browser.pages.FacultyManageFormPage - removed: DEP1
    2967
    30 The message part is composed of the browser page name and the action taken. In the example above a department ``DEP1`` was added on December 11, 2014 by user ``admin`` in faculty ``FAC1``, the title was changed 10 minutes later and, finally, the department was entirely removed one day later. In the same way user data and configuration data are logged in the main logfile. Examples::
    31 
     68The message part is composed of the browser page name and the action
     69taken. In the example above a department ``DEP1`` was added on
     70December 11, 2014 by user ``admin`` in faculty ``FAC1``, the title
     71was changed 10 minutes later and, finally, the department was
     72entirely removed one day later. In the same way user data and
     73configuration data are logged in the main logfile. Examples::
    3274
    3375  2015-04-01 08:33:47,474 - INFO - admin - browser.pages.UserManageFormPage - Test edited: roles
     
    5294``zope.anybody`` is the user id of anonnymous (non-logged-in) users.
    5395
    54 Some log entries do not contain information about a browser page. These entries were generated deeper in the system. The ``Document created`` entry, for example, was added by a workflow transition. The ``PasswordMandate used`` entry was added by :py:meth:`waeup.kofa.mandates.mandate.PasswordMandate.execute` and not directly by the browser page which calls this method (:py:meth:`waeup.kofa.mandates.browser.MandateView.update`).
    55 
    56 ``main.log`` is also the place where plugins store information about system upgrades. If catalogs have to be re-indexed or new attributes of objects have to
    57 be initialized, a corresponding message is stored in the main logfile.
     96Some log entries do not contain information about a browser page.
     97These entries were generated deeper in the system. The ``Document
     98created`` entry, for example, was added by a workflow transition.
     99The ``PasswordMandate used`` entry was added by
     100:py:meth:`waeup.kofa.mandates.mandate.PasswordMandate.execute` and
     101not directly by the browser page which calls this method
     102(:py:meth:`waeup.kofa.mandates.browser.MandateView.update`).
     103
     104``main.log`` is also the place where plugins store information about
     105system upgrades. If catalogs have to be re-indexed or new attributes
     106of objects have to be initialized, a corresponding message is stored
     107in the main logfile.
    58108
    59109
     
    61111==============
    62112
    63 All actions in the data center are logged in ``datacenter.log``. These are uploading import files, processing import files and downloading any kind of datacenter file::
     113All actions in the data center are logged in ``datacenter.log``.
     114These are uploading import files, processing import files and
     115downloading any kind of datacenter file::
    64116
    65117  2015-04-29 06:52:23,483 - INFO - admin - browser.pages.DatacenterUploadPage - uploaded: /kofa/trunk/var/datacenter/users_admin.csv
     
    69121  2015-04-29 06:54:51,748 - INFO - admin - processed: /kofa/trunk/var/datacenter/users2_admin.csv, create mode, 3 lines (3 successful/ 0 failed), 0.024 s (0.0079 s/item)
    70122
    71 In the example above a user data file ``users.csv`` was uploaded and processed in update mode. The import failed since the 3 users didn't exist. The file was then deleted. User ``admin`` uploaded the file a second time. The file name had to be changed because the uploader does not allow to upload files with same name twice. The file was finally successfully processed in create mode.
    72 
    73 Also the export of data (export, download and discard) is recorded in detail::
     123In the example above a user data file ``users.csv`` was uploaded and
     124processed in update mode. The import failed since the 3 users didn't
     125exist. The file was then deleted. User ``admin`` uploaded the file a
     126second time. The file name had to be changed because the uploader
     127does not allow to upload files with same name twice. The file was
     128finally successfully processed in create mode.
     129
     130Also the export of data (export, download and discard) is recorded
     131in detail::
    74132
    75133  2015-04-29 06:55:20,485 - INFO - admin - browser.pages.ExportCSVPage - exported: certificates, job_id=3036
     
    81139===============
    82140
    83 The creation of access code batches, disabling and re-enabling of single access codes and archiving and removal of entire batches is logged as follows::
     141The creation of access code batches, disabling and re-enabling of
     142single access codes and archiving and removal of entire batches is
     143logged as follows::
    84144
    85145  2015-04-29 08:12:26,091 - INFO - admin - accesscodes.browser.AddBatchPage - created: ABC-1-2015_04_29_06_12_24-admin.csv (1000, 2300.000000)
     
    89149  2015-04-29 08:14:14,152 - INFO - admin - accesscodes.browser.BatchContainerPage - deleted: ABC-1
    90150
    91 Not all access code workflow transitions are being logged. Each access code has a history attribute which contains a detailed list of all transitions. The history is shown on the `BatchContainerSearchPage`.
     151Not all access code workflow transitions are being logged. Each
     152access code has a history attribute which contains a detailed list
     153of all transitions. The history is shown on the
     154`BatchContainerSearchPage`.
    92155
    93156
     
    95158==============
    96159
    97 The creation, editing and removal of applicants containers as well as editing application records (applicants) is being logged. Also the approval of payment tickets and all other payment ticket transactions are being recorded in ``applicants.log``. Kofa also logs all workflow transitions into both the applicant's history attribute and the logfile. Okay, this is somehow redundant, but it has proved useful to get a complete overview over all application data transactions also in the logfile. In return, Kofa does not aditionally log actions of browser pages if a workflow transition is triggered at the same time. Let's see the example::
     160The creation, editing and removal of applicants containers as well
     161as editing application records (applicants) is being logged. Also
     162the approval of payment tickets and all other payment ticket
     163transactions are being recorded in ``applicants.log``. Kofa also
     164logs all workflow transitions into both the applicant's history
     165attribute and the logfile. Okay, this is somehow redundant, but it
     166has proved useful to get a complete overview over all application
     167data transactions also in the logfile. In return, Kofa does not
     168aditionally log actions of browser pages if a workflow transition is
     169triggered at the same time. Let's see the example::
    98170
    99171  2015-04-29 10:14:40,565 - INFO - admin - applicants.browser.ApplicantsContainerAddFormPage - added: app2015
     
    113185  2015-04-29 10:45:15,299 - INFO - admin - applicants.browser.ApplicantsRootManageFormPage - removed: app2015
    114186
    115 An applicants container was added first. The `startdate`,  `enddate` and the `application_fee` attributes were edited and a new application record (applicant) was added some seconds later. The `ApplicantManageFormPage` was opened, `reg_number`, `sex`, `course1` and `date_of_birth` was edited and the ``start`` transition was selected. This was done in the same transaction. The time difference between both log entries is only 0.001s. Furthermore, a payment ticket was created and the payment approved. Then the applicant was set to ``sumitted``, a `course_admitted` was selected and the applicant subsequently admitted. The form was automatically locked (see time difference). A student container was created and filled with the data from the application record. Finally, the entire applicants container including its content was removed in the same transaction, see time diffence between the last two log entries.
     187An applicants container was added first. The `startdate`, `enddate`
     188and the `application_fee` attributes were edited and a new
     189application record (applicant) was added some seconds later. The
     190`ApplicantManageFormPage` was opened, `reg_number`, `sex`, `course1`
     191and `date_of_birth` was edited and the ``start`` transition was
     192selected. This was done in the same transaction. The time difference
     193between both log entries is only 0.001s. Furthermore, a payment
     194ticket was created and the payment approved. Then the applicant was
     195set to ``sumitted``, a `course_admitted` was selected and the
     196applicant subsequently admitted. The form was automatically locked
     197(see time difference). A student container was created and filled
     198with the data from the application record. Finally, the entire
     199applicants container including its content was removed in the same
     200transaction, see time diffence between the last two log entries.
    116201
    117202
     
    119204============
    120205
    121 The following example shows a typical Nigerian logfile excerpt for a student from the very beginning (student record creation from application data) till the first registration of courses at level 100. Such an excerpt can be produced simply by searching for ``B1234567`` on the ``students.log`` search page::
     206The following example shows a typical Nigerian logfile excerpt for a
     207student from the very beginning (student record creation from
     208application data) till the first registration of courses at level
     209100. Such an excerpt can be produced simply by searching for
     210``B1234567`` on the ``students.log`` search page::
    122211
    123212  2014-12-11 10:21:21,930 - INFO - admin - B1234567 - Record created
     
    146235  2015-03-28 07:18:21,846 - INFO - admin - Student Processor - Matno_for_upload_25_03_2015-1_admin - B1234567 - updated: student_id=B1234567, matric_number=LSC123456
    147236
    148 The log entry format of transactions due to imports is slightly different::
     237The log entry format of transactions due to imports is slightly
     238different::
    149239
    150240  2015-04-30 08:40:51,088 - INFO - admin - Student Processor - studentcreate_admin - B3333333 - updated: state=returning, reg_number=BIA12326, firstname=FRIDAY, middlename=None, lastname=Olonko, sex=m, nationality=NG, matric_number=SSC4444444
    151241  2015-04-30 08:41:27,179 - INFO - admin - StudentStudyCourse Processor (update only) - studycourseaupdate_admin - B3333333 - updated: entry_mode=ug_pt, certificate=BIA, current_session=2002, entry_session=2002, current_level=100, current_verdict=Z
    152242
    153 Two files had been uploaded by admin (``studentcreate.csv`` and ``studycourseaupdate.csv``, see entries in ``datacenter.log``) and subsequently imported with the Student Processor and Student Study Course Processor respectively.
    154 
    155 As already stated above, not all actions committed by students are beeing logged. The student did not only upload a lot of files but also edited dozens of fields on the clearance form page and probably pressed the 'Save' button very often to backup the data entered so far. These backup transactions were skipped. All other transactions are being recorded. Some examples of further logfile records are listed below.
     243Two files had been uploaded by admin (``studentcreate.csv`` and
     244``studycourseaupdate.csv``, see entries in ``datacenter.log``) and
     245subsequently imported with the Student Processor and Student Study
     246Course Processor respectively.
     247
     248As already stated above, not all actions committed by students are
     249beeing logged. The student did not only upload a lot of files but
     250also edited dozens of fields on the clearance form page and probably
     251pressed the 'Save' button very often to backup the data entered so
     252far. These backup transactions were skipped. All other transactions
     253are being recorded. Some examples of further logfile records are
     254listed below.
    156255
    157256Setting a temporary password::
     
    164263  2014-11-17 10:19:05,735 - INFO - admin - students.browser.StudentActivatePage - W1111111 - account activated
    165264
    166 Rejection of clearance requests (an email is sent and the body of the email is stored as comment in the logfile)::
     265Rejection of clearance requests (an email is sent and the body of
     266the email is stored as comment in the logfile)::
    167267
    168268  2014-10-27 12:39:34,499 - INFO - clearanceofficer - students.browser.StudentRejectClearancePage - W2222222 - comment: no credit in english
    169269
    170 Approval of payment tickets (which does not automatically goes along with a student workflow transition)::
     270Approval of payment tickets (which does not automatically goes along
     271with a student workflow transition)::
    171272
    172273  2012-11-02 12:01:03,022 - INFO - admin - students.browser.OnlinePaymentApprovePage - W1000000 - schoolfee payment approved: p3517158705027
     
    185286============
    186287
    187 This file is hidden and can only be accessed directly in the filesystem. The payment logger is not needed in the Kofa base package. Only `OnlinePaymentApprovePage.update()` writes into the ``payments.log`` file. The logger is primarily intended for storing information about successfull financial transactions via external payment gateways (e.g. PayPal, Interswitch, eTranzact). No external payment gateway module is configured in the base package.
     288This file is hidden and can only be accessed directly in the
     289filesystem. The payment logger is not needed in the Kofa base
     290package. Only `OnlinePaymentApprovePage.update()` writes into the
     291``payments.log`` file. The logger is primarily intended for storing
     292information about successfull financial transactions via external
     293payment gateways (e.g. PayPal, Interswitch, eTranzact). No external
     294payment gateway module is configured in the base package.
    188295
    189296
     
    191298===========
    192299
    193 Whereas the reservation/allocation of bed spaces is being logged in ``students.log`` (see above), the management of hostels and beds is recorded in ``hostels.log``::
     300Whereas the reservation/allocation of bed spaces is being logged in
     301``students.log`` (see above), the management of hostels and beds is
     302recorded in ``hostels.log``::
    194303
    195304  2015-04-30 11:44:29,145 - INFO - admin - hostels.browser.HostelAddFormPage - hostels - added: Hall 1
     
    204313  2015-04-30 11:54:30,163 - INFO - admin - hostels.browser.HostelsContainerManagePage - hostels - deleted: hall-1
    205314
    206 In this example, the hostel ``Hall 1`` was added and configured, beds were updated, beds were reserved (switched), the hostel configuration was changed, beds were updated again, beds were unreserved, a single bed was released, a new student was allocated and, finally, the entire hostel was removed.
     315In this example, the hostel ``Hall 1`` was added and configured,
     316beds were updated, beds were reserved (switched), the hostel
     317configuration was changed, beds were updated again, beds were
     318unreserved, a single bed was released, a new student was allocated
     319and, finally, the entire hostel was removed.
    207320
    208321Also all transactions of the Hostel Processor are being logged::
Note: See TracChangeset for help on using the changeset viewer.