Import of massdata into the portal

General procedure

A record is read from the input-source-file and the data is checked. If the record can be imported, it is appended to the file XXX_imported.csv, otherwise it is inserted into the file XXX_pending.csv, if no record with the exact same date already exists in the XXX_pending.csv. The check is done by comparing the md5-digests of the new record with the existing records.

When doing the import in pending-only-mode the XXX_pending.csv is taken as input-source-file and if a record can now be imported it is appended to the XXX_imported.csv file and removed from XXX_pending.csv.

Caveat:

if an record is inserted into XXX_pending, it will only be removed by importing in pending-only-mode. If a new input-file contains a record for the same object, with different data, that will not be detected at the moment.

The XXX_imported.csv and XXX_pending file has these columns added:

imported_by:the member who did the import
import_mode:edit, create, delete
import_date:the date and time of the import
imported_from:the filename of the input-source-file
error:the error message (empty in XXX_imported)