Different length of .../monitoring/flatfield in subruns blocks DL1 merging

[morcuended]

While merging DL1 files, OSA failed for one of them with the following error:

Traceback (most recent call last):
  File "/fefs/aswg/software/conda/envs/osa-v0.10/bin/lstchain_merge_hdf5_files", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/scripts/lstchain_merge_hdf5_files.py", line 117, in main
    auto_merge_h5files(
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/io/io.py", line 362, in auto_merge_h5files
    raise ValueError("Length of key {} from file {} different than in file {}".format(k, filename, file_list[0]))
ValueError: Length of key /dl1/event/telescope/monitoring/flatfield from file /fefs/aswg/data/real/DL1/20231211/v0.10/tailcut84/dl1_LST-1.Run15982.0052.h5 different than in file /fefs/aswg/data/real/DL1/20231211/v0.10/tailcut84/dl1_LST-1.Run15982.0000.h5

The merged DL1 file was halfway produced, though.

For approximately half of the files of run dl1_LST-1.Run15982.*.h5 the length of the flatfield table is 1, while for the other half is 2. For other runs is always 2.

Is this expected @FrancaCassol?

[SeiyaNozaki]

There were no calibration laser events at the beginning of run 15982, then shifters recovered the laser during the run. I think that is the reason.

[morcuended]

Thanks @SeiyaNozaki, that's it.

What should we do with the monitoring tables in these cases to be able to merge these files?

[maxnoe]

Why is this a problem at all? Why do we require that subruns have the same number of flatfield measurements?

[morcuended]

Why is this a problem at all? Why do we require that subruns have the same number of flatfield measurements?

It was decided like this in #1174. For keys to be copied, among which "/dl1/event/telescope/monitoring/flatfield" is, auto_merge_h5files checks that they have all the same length as the one in the first file.

[morcuended]

Found another run with the same merging problem (16129)

File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/scripts/lstchain_merge_hdf5_files.py", line 117, in main
    auto_merge_h5files(
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/io/io.py", line 362, in auto_merge_h5files
    raise ValueError("Length of key {} from file {} different than in file {}".format(k, filename, file_list[0]))
ValueError: Length of key /dl1/event/telescope/monitoring/calibration from file /fefs/aswg/data/real/DL1/20231215/v0.10/tailcut84/dl1_LST-1.Run16129.0117.h5 different than in file /fefs/aswg/data/real/DL1/20231215/v0.10/tailcut84/dl1_LST-1.Run16129.0000.h5

[morcuended]

Another failing run 16224

  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/scripts/lstchain_merge_hdf5_files.py", line 117, in main
    auto_merge_h5files(
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/io/io.py", line 362, in auto_merge_h5files
    raise ValueError("Length of key {} from file {} different than in file {}".format(k, filename, file_list[0]))
ValueError: Length of key /dl1/event/telescope/monitoring/pedestal from file /fefs/aswg/data/real/DL1/20231217/v0.10/tailcut84/dl1_LST-1.Run16224.0124.h5 different than in file /fefs/aswg/data/real/DL1/20231217/v0.10/tailcut84/dl1_LST-1.Run16224.0000.h5

[moralejo]

What should we do with the monitoring tables in these cases to be able to merge these files?

I do not understand the problem. If flatfield are missing, what happens if we just remove the check that currently fails?

[morcuended]

I do not understand the problem. If flatfield are missing, what happens if we just remove the check that currently fails?

We could do that. @FrancaCassol, if skip the check that all flatfield tables have the same length, could that be a problem for any other part of the analysis?

[FrancaCassol]

Hi, sorry for the delayed reply. This problem comes from the Cat-A calibration table which contains also the output of the interleaved processed by lstchain_data_r0_to_dl1 , this processing should be removed soon (since the interleaved must be processed by the Cat-B calibration script).
For what concerns the cat-B calibration table, it is read from the Cat-B calibration file at the beginning of lstchain_dl1ab , so per definition always the same per each subrun. Therefore, I think we can safely remove this check for the moment