Update code to use more features of ctapipe

[maxnoe]

ctapipe 0.8 is not yet released, but we can collect necessary changes here we already know about.

Some of the points are more general ctapipe adaption and not related to 0.8.
These could be done already, so that the amount of work is less when ctapipe 0.8 comes out.

This is what I came up with, feel free to add stuff I forgot.

• Adapt the changes to the calibration code
• ctapipe.io.containers → ctapipe.containers
• CameraGeometry → CameraDescription.geometry
• Pointing container
• Gain Selection #197 -> Move gain selection from phe to counts (GainSelection should be applied to R0 waveform, not R1 ctapipe#1166)
• Warning: pulse time ouside window for NSB pixels #269 -> exclude negative from charge calculation (Improve extract_pulse_time_around_peak to ignore negative samples ctapipe#1143)
• Consider astropy 4.1 in order to use the new unix_tai format #324 -> Move to astropy 4 (consider Astropy 4 and Numpy 0.17 ctapipe#1211, Fixes for astropy 4.x ctapipe#1212, bump astropy version requirement to allow v4.0 ctapipe#1213)
• Update Hillas parameter naming, etc...

And stuff that is new in ctapipe v0.8, but will not be necessary for the functionality of lstchain when moving to v0.8

• Using TelescopeComponent where appropriate
• Use Path traitlet instead of Unicode where appropriate
• Use ctapipe Tool and configuration system for executables. (Could be done now already)
• Use the new cleaning components
• Clean up containers (e.g. no dl1-mother-of-all-containers) (DL1 columns #343, DL1 container cleanup #349 )
• Largest island
• Use new configurable muon analysis
• Remove lstchain/image/muon/muon_integrator.py, and use the (fixed in 0.8) ctapipe version
• Keep all events #237 -> option to choose whether DL1a/b output should include events which did not survive cleaning or not (the former is useful for keeping calibrated pedestal events for e.g. cleaning studies). This was not done until now because we would need to set defaults for all DL1b parameters (not worth considering the container is deprecated)
• Adapt DL1 data check to the new DL1 containers

[rlopezcoto]

What do you mean by Largest island?

[maxnoe]

There is now a function in ctapipe that gives you the largest island after cleaning., I think this got implemented here as well and this could be dropped for the ctapipe version.

[moralejo]

I would say it is better to start addressing this after the next lstchain release, right?
Otherwise the release will be delayed. We would like to have soon a new version of calibrated data to make progress e.g. with cleaning studies.

[rlopezcoto]

There is now a function in ctapipe that gives you the largest island after cleaning., I think this got implemented here as well and this could be dropped for the ctapipe version.

ok, right, this was implemented and we are currently using the main island for the analysis. In any case and after discussing with @jsitarek, for the lowest energies this may not be the best solution (using only the largest island), but that can be studied using Data/MC in the future

[rlopezcoto]

@moralejo I think this is just a proposal to keep track of these points, as discussed yesterday, as soon as we have a code that writes the information of interleaved events and the first datacheck scripts we can proceed with the 0.5 release.

[moralejo]

@moralejo I think this is just a proposal to keep track of these points, as discussed yesterday, as soon as we have a code that writes the information of interleaved events and the first datacheck scripts we can proceed with the 0.5 release.

Ok, I got nervous with the "could be done already". Fine if already means right after 0.5.0

[maxnoe]

I just wanted to note which changes are possible with the current release and which changes are releated to the new ctapipe release.

[moralejo]

• option to choose whether DL1a/b output should include events which did not survive cleaning or not (the former is useful for keeping calibrated pedestal events for e.g. cleaning studies). This was not done until now because we would need to set defaults for all DL1b parameters (not worth considering the container is deprecated)

• Adapt DL1 data check to the new DL1 containers

• Remove lstchain/image/muon/muon_integrator.py, and use the (fixed in 0.8) ctapipe version

(@rlopezcoto: already added them to the full list)

[FrancaCassol]

Hi to all,
to be sure not to have destructive interferences, should we better assign names to the different points in the above list ?
I am taking care of the calibration code and gain selection, but I can take care also of other points.
For example what to you mean for "point container"? The idea is to move it to ctapipe?

[maxnoe]

Since lstchain is now already using ctapipe 0.8 this issue is partly solved, but many improvements are not yet implemented.