Script to recompute DL1b parameters from a DL1 file

[vuillaut]

• Read a HDF5 DL1 file, recompute parameters based on calibrated images and pulse times and a config file
• write a new HDF5 file
• Updated parameters are : Hillas paramaters, wl, r, leakage, n_islands, intercept, time_gradient

This is not the best approach since it recodes the computation of the parameters.
A better approach would be to code an EventSource for DL1 (but this will come with ctapipe DL1 Tools) and use the same pipeline once we have calibrated images, no matter their origin.

[morcuended]

Maybe you should rename the script to lstchain_dl1ab.py since will work with MC and real data

[morcuended]

I have tested it with a dl1 file and it's working fine. I get new files with and without images. When I save the image table it takes quite long with respect to the no-image option though. However, the image is not recalculated, right? Shouldn't it be just the same? So I expected the run time to be more or less the same.

[vuillaut]

I have tested it with a dl1 file and it's working fine. I get new files with and without images. When I save the image table it takes quite long with respect to the no-image option though. However, the image is not recalculated, right? Shouldn't it be just the same? So I expected the run time to be more or less the same.

It should be more or less the same yes, the only difference is the copy. So depending on the size of the data, your specs, etc... time may vary but I don't expect such a huge difference.
Of what order of magnitude are we talking here?

[morcuended]

I ran the script in parallel using srun. These are the runtimes and memory usage I got with including image (first) and not including it (latter) for a file with 100 events.

    ReqMem     MaxRSS     AveRSS    Elapsed 
---------- ---------- ---------- ---------- 
    8000Mc    243808K    193928K   00:03:47 
    8000Mc    149600K    149600K   00:01:40

I would like to cross-check the runtimes but the IT container is down right now

[vuillaut]

I ran the script in parallel using srun. These are the runtimes and memory usage I got with including image (first) and not including it (latter) for a file with 100 events.

    ReqMem     MaxRSS     AveRSS    Elapsed 
---------- ---------- ---------- ---------- 
    8000Mc    243808K    193928K   00:03:47 
    8000Mc    149600K    149600K   00:01:40

I would like to cross-check the runtimes but the IT container is down right now

Weird, I don't have such difference on my laptop:
For a 676MB DL1 file containing ~11.000 events:
With images:

real    0m52.209s
user    1m22.858s
sys     0m9.207s

Without images:

real    0m48.386s
user    1m21.733s
sys     0m7.502s

[morcuended]

I'm probably doing something wrong then. Will check it again on my local machine

[morcuended]

I'm probably doing something wrong then. Will check it again on my local machine

My bad. I checked it again and the time values make sense now. These are for a whole subrun (53000 events):

Keeping images

real    2m16.617s
user    2m22.056s
sys     1m38.740s

Without images

real    2m9.230s
user    2m18.673s
sys     1m35.348s

The content of both files is OK as well. Let's merge it.

[rlopezcoto]

thanks @morcuended for the quick review and @vuillaut for the code!

[vuillaut]

Thanks for the tests @morcuended !