Merging of DL1 takes up to ~1 h

[morcuended]

With the current lstchain version, the merging of DL1b files run-wise can take quite long (~1 h). I do not know if it's something related to FEFS I/O saturation or something genuine of the merging script. With previous versions, the merging took a shorter time (about 10 min).

[moralejo]

Did you try (in parallel) with an earlier version? That should tell between the two possibilities.

[maxnoe]

Related to #1174 ?

[morcuended]

Related to #1174 ?

That was my impression

Did you try (in parallel) with an earlier version? That should tell between the two possibilities.

I will do it

[morcuended]

Tried to test the merging of DL1 v0.10 files using lstchain v0.10 and v0.9, but it's not possible to merge v0.10 data with v0.9

Traceback (most recent call last):
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/bin/lstchain_merge_hdf5_files", line 8, in <module>
    sys.exit(main())
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/lstchain/scripts/lstchain_merge_hdf5_files.py", line 88, in main
    auto_merge_h5files(
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/lstchain/io/io.py", line 317, in auto_merge_h5files
    file_list = merging_check(file_list)
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/lstchain/io/io.py", line 395, in merging_check
    subarray_info0 = SubarrayDescription.from_hdf(first_file)
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/ctapipe/instrument/subarray.py", line 512, in from_hdf
    geometry = CameraGeometry.from_table(
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/ctapipe/instrument/camera/geometry.py", line 589, in from_table
    return cls(
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/ctapipe/instrument/camera/geometry.py", line 133, in __init__
    raise TypeError(
TypeError: pix_type most be a PixelShape or the name of a PixelShape, got b'hexagon'

[moralejo]

Any idea of where is the time being spent? Can you make a simple test to find out? (e.g. disabling in the code some of the changes from #1174)

[morcuended]

Closing this after #1256. Now it is not as bad as before. We'll keep an eye on this anyway in the future.