Merging fails for some files

[maxnoe]

Doing some final tests for the school I found that when a sub run gives an error, an AssertionError is raised but, the tool doesn't fail (does not exit with error 1).

Below I attach part of the log of the merging to 10 sub dl1 files (only 6 are finally merged)

Metadata CTAPIPE_IO_LST_VERSION does not exist in file electron/training/dl1_proton_20deg_180deg_run1047___cta-prod5-lapalma_4LSTs_MAGIC_desert-2158m_mono.h5
electron/training/dl1_proton_20deg_180deg_run1047___cta-prod5-lapalma_4LSTs_MAGIC_desert-2158m_mono.h5 cannot be merged '¯\_(ツ)_/¯'
Traceback (most recent call last):
  File "/fefs/aswg/software/conda/envs/lstchain-v0.8.4/lib/python3.8/site-packages/lstchain/io/io.py", line 398, in merging_check
    check_mcheader(mcheader0, mcheader)
  File "/fefs/aswg/software/conda/envs/lstchain-v0.8.4/lib/python3.8/site-packages/lstchain/io/io.py", line 771, in check_mcheader
    assert mcheader1[k] == mcheader2[k]
AssertionError
  0%|                                                                                                                                      | 0/6 [00:00<?, ?it/s]/fefs/aswg/software/conda/envs/lstchain-v0.8.4/lib/python3.8/site-packages/tables/path.py:155: NaturalNameWarning: object name is not a valid Python identifier: 'geometry_LSTCam.__table_column_meta__'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  check_attribute_name(name)
/fefs/aswg/software/conda/envs/lstchain-v0.8.4/lib/python3.8/site-packages/tables/path.py:155: NaturalNameWarning: object name is not a valid Python identifier: 'geometry_MAGICCam.__table_column_meta__'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  check_attribute_name(name)
/fefs/aswg/software/conda/envs/lstchain-v0.8.4/lib/python3.8/site-packages/tables/path.py:155: NaturalNameWarning: object name is not a valid Python identifier: 'readout_LSTCam.__table_column_meta__'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  check_attribute_name(name)
/fefs/aswg/software/conda/envs/lstchain-v0.8.4/lib/python3.8/site-packages/tables/path.py:155: NaturalNameWarning: object name is not a valid Python identifier: 'layout.__table_column_meta__'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  check_attribute_name(name)
/fefs/aswg/software/conda/envs/lstchain-v0.8.4/lib/python3.8/site-packages/tables/path.py:155: NaturalNameWarning: object name is not a valid Python identifier: 'optics.__table_column_meta__'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  check_attribute_name(name)
/fefs/aswg/software/conda/envs/lstchain-v0.8.4/lib/python3.8/site-packages/tables/path.py:155: NaturalNameWarning: object name is not a valid Python identifier: 'readout_MAGICCam.__table_column_meta__'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  check_attribute_name(name)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:02<00:00,  2.65it/s]

Originally posted by @garciagenrique in #703 (comment)

[maxnoe]

@garciagenrique That's a completely independent issue.

The behavior you describe is definitely intentional. The automerge_h5_files ignores mc files that it deems "incompatible" with the first file and does not abort when a file does not match.

Did you try to e.g. merge proton and gamma files?

The assertion error in that case (we really shouldn't use assertions for that) is explicitly catched:

cta-lstchain/lstchain/io/io.py

Line 409 in 94f3170

except AssertionError:

[maxnoe]

@garciagenrique If you think that the files you try to merge should all be compatible with each other, we need to investigate why the check functions disagree.

You can try on this branch here: #868, it should give you an error message that tells you why the merge check failed

[garciagenrique]

Thanks for creating this @maxnoe. Will give soon feedback asap.

[vuillaut]

This error is linked to the MAGIC subarray issue. Some files show different subarrays, since the two MAGIC telescopes are different:

In [36]: sub0.optics_types
Out[36]: 
[OpticsDescription(name=MAGIC, equivalent_focal_length=16.97 m, num_mirrors=1, mirror_area=235.22 m2),
 OpticsDescription(name=LST, equivalent_focal_length=28.00 m, num_mirrors=1, mirror_area=386.73 m2)]

In [37]: sub.optics_types
Out[37]: 
[OpticsDescription(name=LST, equivalent_focal_length=28.00 m, num_mirrors=1, mirror_area=386.73 m2),
 OpticsDescription(name=MAGIC, equivalent_focal_length=16.97 m, num_mirrors=1, mirror_area=231.94 m2)]

The error should not appear with an updated version of lstchain (>0.8) and selecting the subarray [1 ,2 ,3, 4] but could still appear when working if MAGIC tels are included. It will however be solved entirely with ctapipe v0.12.

Thanks to @maxnoe for the inputs and fruitful exchanges.

[maxnoe]

After some discussion with @vuillaut the issue seems to be trying to auto_merge files produced with lstchain 0.7 / ctapipe 0.10.5 using lstchain 0.8 / ctapipe 0.11.

The 0.7 files include MAGIC in the subarray and there seems to be a non-determinism with respect to whether MAGIC-I or MAGIC-II are included in the subarray in the HDF due to the issue with storing optics descriptions using the string which does not differ for MAGIC but where the two MAGICS have different mirror areas.

So some of the files have MAGIC-I, some have MAGIC-II and due to auto_merge now checking the subarray equality, merging half of the files does not work.

For standard dl1 files produced with lstchain 0.8, this won't be a problem, since due to the change in ctapipe 0.11, the MAGIC telescopes won't be included in the hdf subarray when using allowed_telescopes={1, }.

For combined MAGIC / LST analysis this will still be an issue, that can only be fixed by going to ctapipe 0.12, which changes the data format to allow telescopes with the same name but different properties.

[garciagenrique]

Thanks a lot @vuillaut !