DL1 Data Processing Tool

[kosack]

This is a "parent" issue to track some developments needed to implement proper DL1 data handling in ctapipe. This is part of a larger issue to make a standard chain (See #977)

Assumptions:

• DL1 data are in HDF5 format
• development should be synchronized to the protopipe development (the final Tool should replace the protopipe dl1 script, dl1-data-handler, and optionally later also the lstchain dl1 script, if that is useful)

Tasks:

• refactor basic image calibration code (see Major refactoring of calibration chain #1026 )
• Do gain-selection during or after calibration and charge extraction (so that we have only a single DL1 image), (See Major refactoring of calibration chain #1026, additional tasks)
• Decide on DL1 file format/layout: (discussion: DL1 data file layout discussion #1059 )
• Implement writing SubarrayDescription to an HDF5 file
• Implement creating a SubarrayDescription from an HDF5 file (written with prev step)
• Implement standard ctapipe DL0 -> DL1 Tool (based on protopipe, lstchain, dl1-dumper, etc)
  • Metadata tables:
    • simulation run configuration
    • observation configuration
  • Event tables:
    • monte-carlo events table
    • trigger table
    • standard calibrated image tables (easy, just needs calib)
    • standard image parameter tables (requires image processing chain and configuration thereof)
  • Monitoring tables (lower priority for now, but needed later)
    • pointing correction table
    • calibration monitoring (see implementation in LSTCamEventSource for example)
• Implement a DL1EventSource that reads output from the DL1 tool described in prev step so that a DL2 tool can also be created.
• Ensure the DL1 tool can be configured by a user (via config file and command-line) and that the input and output configuration is tracked and stored in the provenance metadata (see Improvements to Tool and Component functionality #1052)
• store full provenance metadata in output file

[kosack]

Just to note, I've made big progress on this and have a prototype working. During its development, i found a number of small bugs or missing features in ctapipe, so I will first start (in the next week or so) to create pull requests for those changes before committing the full tool.

[kosack]

I will close this, since it's all implemented (except for a DL1EventSource, but that can be a separate issue)