Cookbook: Interpreting MIK's log files
MIK writes out log files to record errors or warnings, and to document actions it takes while processing your content. The log files contain information that will help you track down problems, and also contain information that will let you decide if the various components of MIK, particularly its metadata manipulators, are working the way you expect.
MIK's log files are particularly useful when you are testing various configuration options, before you run MIK that one final time to generate your ingest packages for loading into Islandora. For example, the input_validators.log will point out missing or unexpected files in your input, a problem you will want to rectify before generating your packages for loading.
The path to this log is set in the [LOGGING] path_to_log .ini file option.
This is MIK's general log. All warnings and errors encountered by fetchers, filegetters, metadata parsers, and writers are logged here.
Each time MIK runs, it writes entries like the following to the log:
[2017-03-28 07:21:10] config.INFO: MIK Configuration {"config_id":"issue_156_test"} []
[2017-03-28 07:21:10] config.INFO: MIK Configuration {"last_updated_on":"2017-03-22"} []
[2017-03-28 07:21:10] config.INFO: MIK Configuration {"last_update_by":"Mark Jordan"} []
[2017-03-28 07:21:10] Info.INFO: MIK started running March 28, 2017, 7:21 am [] []
An entry is also written at the end of the log file indicating when MIK stopped:
[2017-03-28 07:21:14] Info.INFO: Completed in 0.06566778421402 minutes. [] []
This metadata about the MIK job is always written to the mik.log. The source of the metadata is the entries in the .ini file's [CONFIG] section. If you add additional entries there, they will be written to the mik.log.
If MIK encounters any errors or warnings, they are also written to this log. For example, here is a warning:
[2017-03-30 06:46:25] metadataparsers.WARNING: Mappings file contains a row that is not in source CSV row for this object. {"Record key":"image01","Missing field in metadata":"Related Item"} []
Errors are generally more verbose, and can be somewhat difficult to read. In this example, MIK detected a problem with the metadata parser, and because the problem was a serious exception, it dumped everything the metadata parser knew about the object being processed (in other words, the malformed MODS XML), plus information about which PHP file threw the exception and which line number the exception occurred on:
[2017-03-30 06:48:42] ErrorException.ERROR: ErrorException {"message":"Trying to get property of non-object","code":{"xmlString":"<mods xmlns=\"http://www.loc.gov/mods/v3\" xmlns:mods=\"http://www.loc.gov/mods/v3\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"><itleInfo><title>Small boats in Havana Harbour</title></titleInfo><name type=\"personal\"><namePart>Jordan, Mark</namePart><role><roleTerm type=\"text\">photographer</roleTerm></role></name><originInfo><dateCreated encoding=\"w3cdtf\" keyDate=\"yes\">2015-03-08</dateCreated></originInfo><subject><topic>Boats</topic></subject><subject><topic>water</topic></subject><identifier type=\"local\" displayLabel=\"Local identifier\">image01</identifier><note>Taken on vacation in Cuba.</note><genre authority=\"marcgt\">picture</genre><typeOfResource>still image</typeOfResource><physicalDescription><digitalOrigin>born digital</digitalOrigin></physicalDescription></mods>","xml":"[object] (DOMDocument: {})","childNodesElementSignatureStringArray":[]},"severity":8,"file":"/home/mark/Documents/hacking/mik/src/metadataparsers/mods/CsvToMods.php","line":176} []
This entry looks a bit overwhelming, especially when the log file contains a lot of them. Generally speaking, the bit of information just after the timestamp will contain the error message (in this example, "message":"Trying to get property of non-object"), and the text at the very end of the entry ("file":"/home/mark/Documents/hacking/mik/src/metadataparsers/mods/CsvToMods.php","line":176) will point out the file and line number where the error occurred.
If you see metadataparsers.WARNINGs like Mappings file contains a row that is not in source CSV row for this object., confirm that the values in the left-hand column in your mapping file match the CSV or CONTENTdm source field names. This warning indicates that MIK detects a mismatch in the field specified in the log entries.
This log is written to the same directory as the MIK log with the name problem_records.log.
Like the mik.log, the problem_records.log contains some metadata about the MIK job:
[2017-03-28 07:21:10] ProblemRecords.INFO: Initializing log {"configuration":{"config_id":"issue_156_test","last_updated_on":"2017-03-22","last_update_by":"Mark Jordan"}} []
If any errors occur that block or interfere with the creation of an ingest package, the problem records log will contain entries documenting the record keys of the problematic objects:
[2017-03-30 06:48:42] ProblemRecords.ERROR: image01 [] []
[2017-03-30 06:48:42] ProblemRecords.ERROR: image02 [] []
[2017-03-30 06:48:42] ProblemRecords.ERROR: image03 [] []
[2017-03-30 06:48:42] ProblemRecords.ERROR: image04 [] []
[2017-03-30 06:48:42] ProblemRecords.ERROR: image05 [] []
Essentially, the problem_records.log just gives you the record keys of objects that MIK couldn't generate ingest packages for, without the static that detailed error messages generates. Each entry in the problem_records.log should have a corresponding entry in mik.log (although mik.log may have entries that do not have a corresponding entry in problem_records.log), so you can get the detail there if needed.
MIK includes a utility scrip, located at extras/scripts/specificsetfromproblemrecords.php, that will generate a list of record keys from the problem_records.log that can be used with the SpecificSet fetcher manipulator. This allows your reprocess objects after you fix the errors identified in mik.log.
This log is written to the same directory as the MIK log with the name input_validator.log, but its location can be overridden in the [LOGGING] path_to_input_validator_log .ini file option.
Like the mik.log and problem_records.log, the input_validator.log contains some metadata about the MIK job:
[2017-03-29 16:30:40] input validator.INFO: Initializing log {"configuration":{"config_id":"cartoons","last_updated_on":"2017-03-01","last_update_by":"mj"}} []
If MIK's input validators detect any problems, they write log entries like this one to the input validation log:
[2017-03-30 06:53:58] input validator.ERROR: Input validation failed {"record ID":"image01","package path":"/home/mark/Downloads/mik_tutorial_data/IMG_1410.JPG","error":"File not found"} []
You should consult the input_validators.log while testing your MIK configuration, since it will tell you about issues you can probably fix easily, such as missing input files.
The path to this log is set in the [LOGGING] path_to_manipulator_log .ini file option.
The manipulator.log writes entries that document what various manipulators are doing. The most common type of entry in this log is generated by metadata manipulators. In this example, the SplitRepeatedValues metadata manipulator writes entries that show the "before" and "after" versions of the values it is changing during the creation of the MODS XML files:
[2017-03-30 06:48:42] config.INFO: SplitRepeatedValues {"Record key":"image01","Source field name":"Subjects","Source field value":"Boats; water","Output":"<subject><topic>Boats</topic></subject><subject><topic>water</topic></subject>"} []
[2017-03-30 06:48:42] config.INFO: SplitRepeatedValues {"Record key":"image05","Source field name":"Subjects","Source field value":"Alcatraz Federal Penitentiary; islands","Output":"<subject><topic>Alcatraz Federal Penitentiary</topic></subject>"} []
[2017-03-30 06:48:42] config.INFO: SplitRepeatedValues {"Record key":"image05","Source field name":"Subjects","Source field value":"Alcatraz Federal Penitentiary; islands","Output":"<subject><topic>Alcatraz Federal Penitentiary</topic></subject><subject><topic>islands</topic></subject>"} []
Specialized MIK components (e.g., specialized filegetters), post-write hook scripts and shutdown scripts can write their own log files. They works similar to the standard log files described above.
Content on the Move to Islandora Kit wiki is licensed under a Creative Commons Attribution 4.0 International License.
