Changeset 12895
- Timestamp:
- 30 Apr 2015, 10:40:44 (10 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
main/waeup.kofa/trunk/docs/source/userdocs/datacenter/logging.rst
r12894 r12895 4 4 ******** 5 5 6 Kofa is logging actions, which cause changes to the database or the filesystem, in separate human-redable event logfiles. Nearly all actions are being logged, except some actions committed by applicants and students during registration periods. During these data initialization periods, applicants and students are requested to edit many fields. Usually they press the 'Save' button quite often to backup the data entered so far. Logging all these changes would mean filling the logfiles with more or less useless or redundant information. 7 8 The main purpose of Kofa's logging facilities is to trace changes effected by portal officers. Applicants and students can only access their own data. They don't have the permission to change anything else. Whereas portal officers bear responsibility for the integrity of the whole database. Depending on the roles they have, officers can add, edit or remove application and study data without leaving a trace in the object database itself. Kofa closes this security gap by recording nearly all actions of portal officers into various event logfiles. 6 Kofa logs actions, which cause changes to the database or the 7 filesystem, in separate human-redable event logfiles. Nearly all 8 actions are being logged, except some actions committed by 9 applicants and students during registration periods. During these 10 data initialization periods, applicants and students are requested 11 to edit many fields. Usually they press the 'Save' button quite 12 often to backup the data entered so far. Logging all these changes 13 would mean filling the logfiles with more or less useless or 14 redundant information. 15 16 The main purpose of Kofa's logging facilities is to trace changes 17 effected by portal officers. Applicants and students can only access 18 their own data. They don't have the permission to change anything 19 else. Whereas portal officers bear responsibility for the integrity 20 of the whole database. Depending on the roles they have, officers 21 can add, edit or remove application and study data without leaving a 22 trace in the object database itself. Kofa closes this security gap 23 by recording nearly all actions of portal officers into various 24 event logfiles. 9 25 10 26 .. note:: 11 27 12 Kofa logs the changing but not the viewing of data (except downloading export files or reports). Also entering (login) end leaving (logout) the portal is **not** being recorded. 13 14 Critics may claim that extensive logging may discourage officers from working with the portal. Persons in key functions often don't want to be controlled or supervised by anybody else. Indeed, Kofa does not respond to the desire for this kind of privacy. Datacenter managers can search all logfiles and can even see what their principles have done with the data. Goal behind is **transparency and security**, i.e. transparent data processing and reduction of the vulnerability to corruption combined with cutting-edge technologies to protect the data against attacks from outside. 15 16 Logfiles can not be accessed directly. They can neither be downloaded nor viewed entirely online. But the files can be searched online for arbitrary `regular expressions`_. The simplest regular expression is a single word or a string. A technical description, how logfiles are being querried, can be found :py:meth:`here<waeup.kofa.datacenter.DataCenter.queryLogfiles>`. 17 18 Each line in the logfile means a single log entry. It is composed of four parts: a datetime string, the log level (mainly 'INFO'), the id of the logged-in user, and a message part. Depending on the module, the message part is subdivided into further parts which are described below. 28 Kofa logs the changing but not the viewing of data (except 29 downloading export files or reports). Also entering (login) end 30 leaving (logout) the portal is **not** being recorded. 31 32 Critics may claim that extensive logging may discourage officers 33 from working with the portal. Persons in key functions often don't 34 want to be controlled or supervised by anybody else. Indeed, Kofa 35 does not respond to the desire for this kind of privacy. Datacenter 36 managers can search all logfiles and can even see what their 37 principles have done with the data. Goal behind is **transparency 38 and security**, i.e. transparent data processing and reduction of 39 the vulnerability to corruption combined with cutting-edge 40 technologies to protect the data against attacks from outside. 41 42 Logfiles can not be accessed directly. They can neither be 43 downloaded nor viewed entirely online. But the files can be searched 44 online for arbitrary `regular expressions`_. The simplest regular 45 expression is a single word or a string. A technical description, 46 how logfiles are being querried, can be found 47 :py:meth:`here<waeup.kofa.datacenter.DataCenter.queryLogfiles>`. 48 49 Each line in the logfile means a single log entry. It is composed of 50 four parts: a datetime string, the log level (mainly 'INFO'), the id 51 of the logged-in user, and a message part. Depending on the module, 52 the message part is subdivided into further parts which are 53 described below. 19 54 20 55 … … 22 57 ======== 23 58 24 All actions in the academic section are logged in ``main.log``. Changes of faculties, departments, certificates, courses and certificate courses done via pages in the user interface (browser page) are recorded in the following form:: 59 All actions in the academic section are logged in ``main.log``. 60 Changes of faculties, departments, certificates, courses and 61 certificate courses done via pages in the user interface (browser 62 page) are recorded in the following form:: 25 63 26 64 2014-12-11 13:09:22,855 - INFO - admin - browser.pages.DepartmentAddFormPage - added: FAC1/DEP1 … … 28 66 2014-12-12 13:19:43,255 - INFO - admin - browser.pages.FacultyManageFormPage - removed: DEP1 29 67 30 The message part is composed of the browser page name and the action taken. In the example above a department ``DEP1`` was added on December 11, 2014 by user ``admin`` in faculty ``FAC1``, the title was changed 10 minutes later and, finally, the department was entirely removed one day later. In the same way user data and configuration data are logged in the main logfile. Examples:: 31 68 The message part is composed of the browser page name and the action 69 taken. In the example above a department ``DEP1`` was added on 70 December 11, 2014 by user ``admin`` in faculty ``FAC1``, the title 71 was changed 10 minutes later and, finally, the department was 72 entirely removed one day later. In the same way user data and 73 configuration data are logged in the main logfile. Examples:: 32 74 33 75 2015-04-01 08:33:47,474 - INFO - admin - browser.pages.UserManageFormPage - Test edited: roles … … 52 94 ``zope.anybody`` is the user id of anonnymous (non-logged-in) users. 53 95 54 Some log entries do not contain information about a browser page. These entries were generated deeper in the system. The ``Document created`` entry, for example, was added by a workflow transition. The ``PasswordMandate used`` entry was added by :py:meth:`waeup.kofa.mandates.mandate.PasswordMandate.execute` and not directly by the browser page which calls this method (:py:meth:`waeup.kofa.mandates.browser.MandateView.update`). 55 56 ``main.log`` is also the place where plugins store information about system upgrades. If catalogs have to be re-indexed or new attributes of objects have to 57 be initialized, a corresponding message is stored in the main logfile. 96 Some log entries do not contain information about a browser page. 97 These entries were generated deeper in the system. The ``Document 98 created`` entry, for example, was added by a workflow transition. 99 The ``PasswordMandate used`` entry was added by 100 :py:meth:`waeup.kofa.mandates.mandate.PasswordMandate.execute` and 101 not directly by the browser page which calls this method 102 (:py:meth:`waeup.kofa.mandates.browser.MandateView.update`). 103 104 ``main.log`` is also the place where plugins store information about 105 system upgrades. If catalogs have to be re-indexed or new attributes 106 of objects have to be initialized, a corresponding message is stored 107 in the main logfile. 58 108 59 109 … … 61 111 ============== 62 112 63 All actions in the data center are logged in ``datacenter.log``. These are uploading import files, processing import files and downloading any kind of datacenter file:: 113 All actions in the data center are logged in ``datacenter.log``. 114 These are uploading import files, processing import files and 115 downloading any kind of datacenter file:: 64 116 65 117 2015-04-29 06:52:23,483 - INFO - admin - browser.pages.DatacenterUploadPage - uploaded: /kofa/trunk/var/datacenter/users_admin.csv … … 69 121 2015-04-29 06:54:51,748 - INFO - admin - processed: /kofa/trunk/var/datacenter/users2_admin.csv, create mode, 3 lines (3 successful/ 0 failed), 0.024 s (0.0079 s/item) 70 122 71 In the example above a user data file ``users.csv`` was uploaded and processed in update mode. The import failed since the 3 users didn't exist. The file was then deleted. User ``admin`` uploaded the file a second time. The file name had to be changed because the uploader does not allow to upload files with same name twice. The file was finally successfully processed in create mode. 72 73 Also the export of data (export, download and discard) is recorded in detail:: 123 In the example above a user data file ``users.csv`` was uploaded and 124 processed in update mode. The import failed since the 3 users didn't 125 exist. The file was then deleted. User ``admin`` uploaded the file a 126 second time. The file name had to be changed because the uploader 127 does not allow to upload files with same name twice. The file was 128 finally successfully processed in create mode. 129 130 Also the export of data (export, download and discard) is recorded 131 in detail:: 74 132 75 133 2015-04-29 06:55:20,485 - INFO - admin - browser.pages.ExportCSVPage - exported: certificates, job_id=3036 … … 81 139 =============== 82 140 83 The creation of access code batches, disabling and re-enabling of single access codes and archiving and removal of entire batches is logged as follows:: 141 The creation of access code batches, disabling and re-enabling of 142 single access codes and archiving and removal of entire batches is 143 logged as follows:: 84 144 85 145 2015-04-29 08:12:26,091 - INFO - admin - accesscodes.browser.AddBatchPage - created: ABC-1-2015_04_29_06_12_24-admin.csv (1000, 2300.000000) … … 89 149 2015-04-29 08:14:14,152 - INFO - admin - accesscodes.browser.BatchContainerPage - deleted: ABC-1 90 150 91 Not all access code workflow transitions are being logged. Each access code has a history attribute which contains a detailed list of all transitions. The history is shown on the `BatchContainerSearchPage`. 151 Not all access code workflow transitions are being logged. Each 152 access code has a history attribute which contains a detailed list 153 of all transitions. The history is shown on the 154 `BatchContainerSearchPage`. 92 155 93 156 … … 95 158 ============== 96 159 97 The creation, editing and removal of applicants containers as well as editing application records (applicants) is being logged. Also the approval of payment tickets and all other payment ticket transactions are being recorded in ``applicants.log``. Kofa also logs all workflow transitions into both the applicant's history attribute and the logfile. Okay, this is somehow redundant, but it has proved useful to get a complete overview over all application data transactions also in the logfile. In return, Kofa does not aditionally log actions of browser pages if a workflow transition is triggered at the same time. Let's see the example:: 160 The creation, editing and removal of applicants containers as well 161 as editing application records (applicants) is being logged. Also 162 the approval of payment tickets and all other payment ticket 163 transactions are being recorded in ``applicants.log``. Kofa also 164 logs all workflow transitions into both the applicant's history 165 attribute and the logfile. Okay, this is somehow redundant, but it 166 has proved useful to get a complete overview over all application 167 data transactions also in the logfile. In return, Kofa does not 168 aditionally log actions of browser pages if a workflow transition is 169 triggered at the same time. Let's see the example:: 98 170 99 171 2015-04-29 10:14:40,565 - INFO - admin - applicants.browser.ApplicantsContainerAddFormPage - added: app2015 … … 113 185 2015-04-29 10:45:15,299 - INFO - admin - applicants.browser.ApplicantsRootManageFormPage - removed: app2015 114 186 115 An applicants container was added first. The `startdate`, `enddate` and the `application_fee` attributes were edited and a new application record (applicant) was added some seconds later. The `ApplicantManageFormPage` was opened, `reg_number`, `sex`, `course1` and `date_of_birth` was edited and the ``start`` transition was selected. This was done in the same transaction. The time difference between both log entries is only 0.001s. Furthermore, a payment ticket was created and the payment approved. Then the applicant was set to ``sumitted``, a `course_admitted` was selected and the applicant subsequently admitted. The form was automatically locked (see time difference). A student container was created and filled with the data from the application record. Finally, the entire applicants container including its content was removed in the same transaction, see time diffence between the last two log entries. 187 An applicants container was added first. The `startdate`, `enddate` 188 and the `application_fee` attributes were edited and a new 189 application record (applicant) was added some seconds later. The 190 `ApplicantManageFormPage` was opened, `reg_number`, `sex`, `course1` 191 and `date_of_birth` was edited and the ``start`` transition was 192 selected. This was done in the same transaction. The time difference 193 between both log entries is only 0.001s. Furthermore, a payment 194 ticket was created and the payment approved. Then the applicant was 195 set to ``sumitted``, a `course_admitted` was selected and the 196 applicant subsequently admitted. The form was automatically locked 197 (see time difference). A student container was created and filled 198 with the data from the application record. Finally, the entire 199 applicants container including its content was removed in the same 200 transaction, see time diffence between the last two log entries. 116 201 117 202 … … 119 204 ============ 120 205 121 The following example shows a typical Nigerian logfile excerpt for a student from the very beginning (student record creation from application data) till the first registration of courses at level 100. Such an excerpt can be produced simply by searching for ``B1234567`` on the ``students.log`` search page:: 206 The following example shows a typical Nigerian logfile excerpt for a 207 student from the very beginning (student record creation from 208 application data) till the first registration of courses at level 209 100. Such an excerpt can be produced simply by searching for 210 ``B1234567`` on the ``students.log`` search page:: 122 211 123 212 2014-12-11 10:21:21,930 - INFO - admin - B1234567 - Record created … … 146 235 2015-03-28 07:18:21,846 - INFO - admin - Student Processor - Matno_for_upload_25_03_2015-1_admin - B1234567 - updated: student_id=B1234567, matric_number=LSC123456 147 236 148 The log entry format of transactions due to imports is slightly different:: 237 The log entry format of transactions due to imports is slightly 238 different:: 149 239 150 240 2015-04-30 08:40:51,088 - INFO - admin - Student Processor - studentcreate_admin - B3333333 - updated: state=returning, reg_number=BIA12326, firstname=FRIDAY, middlename=None, lastname=Olonko, sex=m, nationality=NG, matric_number=SSC4444444 151 241 2015-04-30 08:41:27,179 - INFO - admin - StudentStudyCourse Processor (update only) - studycourseaupdate_admin - B3333333 - updated: entry_mode=ug_pt, certificate=BIA, current_session=2002, entry_session=2002, current_level=100, current_verdict=Z 152 242 153 Two files had been uploaded by admin (``studentcreate.csv`` and ``studycourseaupdate.csv``, see entries in ``datacenter.log``) and subsequently imported with the Student Processor and Student Study Course Processor respectively. 154 155 As already stated above, not all actions committed by students are beeing logged. The student did not only upload a lot of files but also edited dozens of fields on the clearance form page and probably pressed the 'Save' button very often to backup the data entered so far. These backup transactions were skipped. All other transactions are being recorded. Some examples of further logfile records are listed below. 243 Two files had been uploaded by admin (``studentcreate.csv`` and 244 ``studycourseaupdate.csv``, see entries in ``datacenter.log``) and 245 subsequently imported with the Student Processor and Student Study 246 Course Processor respectively. 247 248 As already stated above, not all actions committed by students are 249 beeing logged. The student did not only upload a lot of files but 250 also edited dozens of fields on the clearance form page and probably 251 pressed the 'Save' button very often to backup the data entered so 252 far. These backup transactions were skipped. All other transactions 253 are being recorded. Some examples of further logfile records are 254 listed below. 156 255 157 256 Setting a temporary password:: … … 164 263 2014-11-17 10:19:05,735 - INFO - admin - students.browser.StudentActivatePage - W1111111 - account activated 165 264 166 Rejection of clearance requests (an email is sent and the body of the email is stored as comment in the logfile):: 265 Rejection of clearance requests (an email is sent and the body of 266 the email is stored as comment in the logfile):: 167 267 168 268 2014-10-27 12:39:34,499 - INFO - clearanceofficer - students.browser.StudentRejectClearancePage - W2222222 - comment: no credit in english 169 269 170 Approval of payment tickets (which does not automatically goes along with a student workflow transition):: 270 Approval of payment tickets (which does not automatically goes along 271 with a student workflow transition):: 171 272 172 273 2012-11-02 12:01:03,022 - INFO - admin - students.browser.OnlinePaymentApprovePage - W1000000 - schoolfee payment approved: p3517158705027 … … 185 286 ============ 186 287 187 This file is hidden and can only be accessed directly in the filesystem. The payment logger is not needed in the Kofa base package. Only `OnlinePaymentApprovePage.update()` writes into the ``payments.log`` file. The logger is primarily intended for storing information about successfull financial transactions via external payment gateways (e.g. PayPal, Interswitch, eTranzact). No external payment gateway module is configured in the base package. 288 This file is hidden and can only be accessed directly in the 289 filesystem. The payment logger is not needed in the Kofa base 290 package. Only `OnlinePaymentApprovePage.update()` writes into the 291 ``payments.log`` file. The logger is primarily intended for storing 292 information about successfull financial transactions via external 293 payment gateways (e.g. PayPal, Interswitch, eTranzact). No external 294 payment gateway module is configured in the base package. 188 295 189 296 … … 191 298 =========== 192 299 193 Whereas the reservation/allocation of bed spaces is being logged in ``students.log`` (see above), the management of hostels and beds is recorded in ``hostels.log``:: 300 Whereas the reservation/allocation of bed spaces is being logged in 301 ``students.log`` (see above), the management of hostels and beds is 302 recorded in ``hostels.log``:: 194 303 195 304 2015-04-30 11:44:29,145 - INFO - admin - hostels.browser.HostelAddFormPage - hostels - added: Hall 1 … … 204 313 2015-04-30 11:54:30,163 - INFO - admin - hostels.browser.HostelsContainerManagePage - hostels - deleted: hall-1 205 314 206 In this example, the hostel ``Hall 1`` was added and configured, beds were updated, beds were reserved (switched), the hostel configuration was changed, beds were updated again, beds were unreserved, a single bed was released, a new student was allocated and, finally, the entire hostel was removed. 315 In this example, the hostel ``Hall 1`` was added and configured, 316 beds were updated, beds were reserved (switched), the hostel 317 configuration was changed, beds were updated again, beds were 318 unreserved, a single bed was released, a new student was allocated 319 and, finally, the entire hostel was removed. 207 320 208 321 Also all transactions of the Hostel Processor are being logged::
Note: See TracChangeset for help on using the changeset viewer.
