Problems reprocessing 2021-02-13, since "case of broken pixels not contained in waveform is not implemented"

[marialainez]

All runs of night 2021-02-13 fail when trying to reprocess them with v0.10. This is the error traceback that we get:

ValueError: lstchain_data_r0_to_dl1 failed with output: 
 Traceback (most recent call last):
  File "/fefs/aswg/software/conda/envs/osa-v0.10/bin/lstchain_data_r0_to_dl1", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/scripts/lstchain_data_r0_to_dl1.py", line 227, in main
    r0_to_dl1.r0_to_dl1(
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/reco/r0_to_dl1.py", line 474, in r0_to_dl1
    for i, event in enumerate(source):
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/ctapipe/io/eventsource.py", line 323, in __iter__
    for event in self._generator():
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/ctapipe_io_lst/__init__.py", line 631, in _generator
    self.fill_r0r1_container(array_event, zfits_event)
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/ctapipe_io_lst/__init__.py", line 1079, in fill_r0r1_container
    r0, r1 = self.fill_r0r1_camera_container(zfits_event)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/ctapipe_io_lst/__init__.py", line 1037, in fill_r0r1_camera_container
    raise NotImplementedError(
NotImplementedError: Case of broken pixels not contained in waveform is not implemented.If you encounter this error, open an issue in ctapipe_io_lst noting the run for which this happened.
Closing remaining open files:/fefs/aswg/data/real/running_analysis/20210213/v0.10/interleaved_LST-1.Run03631.0030.h5...done

And here is an example of one of the commands that we launch:

lstchain_data_r0_to_dl1 --input-file=/fefs/aswg/data/real/R0/20210213/LST-1.1.Run03631.0030.fits.fz --output-dir=/fefs/aswg/data/real/running_analysis/20210213/v0.10 --pedestal-file=/fefs/aswg/data/real/monitoring/PixelCalibration/Cat-A/drs4_baseline/20210213/v0.10.4/drs4_pedestal.Run03630.0000.h5 --calibration-file=/fefs/aswg/data/real/monitoring/PixelCalibration/Cat-A/calibration/20210213/v0.10.4/calibration_filters_52.Run03640.0000.h5 --time-calibration-file=/fefs/aswg/data/real/monitoring/PixelCalibration/Cat-A/drs4_time_sampling_from_FF/20191124/v0.10.0/time_calibration.Run01625.0000.h5 --systematic-correction-file=/fefs/aswg/data/real/monitoring/PixelCalibration/Cat-A/ffactor_systematics/20201211/v0.10.3/calibration_scan_fit_20201211.0000.h5 --config=/fefs/aswg/lstosa/config_files_lstchain/lstchain_standard_v0.10_heuristic_ff.json --pointing-file=/fefs/onsite/monitoring/driveLST1/DrivePositioning/DrivePosition_log_20210213.txt --run-summary-path=/fefs/aswg/data/real/monitoring/RunSummary/RunSummary_20210213.ecsv

[maxnoe]

Hi @marialainez, I transferred the issue from lstchain to ctapipe_io_lst, since it fits better here.

I will have a look at the files, this error shouldn't happen (there should be no data that is like this), so I need to investigate why this error was triggered for the files you are analyzing.

[SeiyaNozaki]

Just FYI, there was a de-activated single module on that night, but it is the same case for other nights in February 2021...

[maxnoe]

Yes, thanks, I was aware of that and we have tests checking that this data is readable, at least the original files. I think here, we read gain selected or dvred files, I didn't check yet.

But I think we have an issue here with when we apply the offline DVR to this data. I need to check if it is a problem of the dvr or of the reader.

[marialainez]

Yes, for this reprocessing we are reading the files in R0V, since the other ones are not available in fefs anymore. But when we first processed this night with v0.9 from the original R0 files we didn't have any problem.

[maxnoe]

Checking the file on disk, there is something weird in the files itself:

In [1]: from protozfits import File

In [2]: f  = File("/fefs/aswg/data/real/R0V/20210213/LST-1.1.Run03631.0030.fits.fz")

In [3]: f
Out[3]: File({'CameraConfig': Table(1xR1.CameraConfiguration), 'Events': Table(13250xR1.CameraEvent)})

In [5]: c = f.CameraConfig[0]

In [6]: c.num_pixels
Out[6]: 1855

In [7]: c.lstcam.num_modules
Out[7]: 265

In [8]: e = f.Events[0]

In [9]: e.pixel_status
Out[9]: array([4, 4, 5, ..., 4, 4, 4], dtype=uint8)

In [10]: e.pixel_status.size
Out[10]: 1855

In [13]: from collections import Counter

In [14]: Counter(e.pixel_status)
Out[14]: Counter({4: 1694, 5: 154, 1: 7})

In [21]: e.waveform.size / c.num_samples
Out[21]: 161.0

The camera configuration says 1855 pixels, 265 modules, so the full camera, which should not be the case for the data with one module switched off. It should have 1848 pixels / 264 modules.

Then, looking into the pixel status, we have 1694 pixels that were HG readout but not stored by the DVR (4), 154 pixels which were HG readout and stored by the DVR (5) and 7 pixels that the DVR claims are "stored" but have nothing readout (1).

This shouldn't happen. The error is with the DVR tool on the data with the broken module, not here.