Update calibrator with pulse correction and gain selection #248

FrancaCassol · 2019-12-13T09:24:13Z

In this PR it has been added:

in calibrator.py:
-- the pulse time correction: if the calibrator find the time calibration file it applies the drs4 corrections otherwise it just applies the flat-field time corrections
-- the gain selection based on the threshold method: if the threshold is set to None, no selection is performed. I add the new notebook apply_calibration_with_gain_selection.ipynb to show an example of application
in onsite_create_calibration_file.py script: is added the calculation of the drs4 time corrections that are written in a hdf5 file in calibration directory of the onsite tree
in lstchain_data_create_calibration_file.py: I add new cuts to avoid magic lidar events

The configuration for all the calibration is in onsite_camera_calibration_param.json
Next step is to better coordinate all the configuration files

Do not use events without TIB data

Changed the time correction in order to put time arouns zero as for drs4 time correction

into update_calibrator

pulse time correction to the calibrator

review-notebook-app · 2019-12-13T09:24:19Z

Check out this pull request on

You'll be able to see Jupyter notebook diff and discuss changes. Powered by ReviewNB.

FrancaCassol · 2019-12-17T14:13:08Z

Hi @vuillaut, @rlopezcoto,

I think that at this point we should fast review this code

FrancaCassol · 2019-12-17T14:20:25Z

To make things more clear I will commit the correction of the reconstruction script in a different PR

rlopezcoto

I could not yet check the notebook, but the rest looks fine to me, except the small clarification I added.

rlopezcoto · 2019-12-17T14:47:25Z

lstchain/calib/camera/time_correction_calculate.py

-                      'from the peak_index (peak_index - shift)'
-                      ).tag(config=True)
+    charge_product = Unicode(
+        'LocalPeakWindowSum',


Shall this be hardcoded?

This extractor is not suppose to change often, but why do you want to have it hardcoded?

vuillaut · 2019-12-17T15:11:10Z

lstchain/calib/camera/calibrator.py

+
+        # perform the gain selection if the threshold is defined
+        if self.gain_threshold:
+            waveforms, gain_mask = self.gain_selector(event.r1.tel[telid].waveform)


shouldn't this be calculated on r0?
r1 are in p.e. at this point?

The r1 waveform is low-level corrected but still in ADC

Isn't this different from CTA data model?

Hi @vuillaut,
thinking gain to this point I agree that it is better to use the r0 waveform (the fact that it is not low level corrected should not affect the result). I am going to change it.
You are right, our r1 is different than in the ctpipe data model. This is due to the fact that we have a level more in the data (ctapipe r0 is supposed to be already low-level corrected), we should solve this issue in the next (but not in this PR ;-))

vuillaut · 2019-12-17T15:23:19Z

lstchain/calib/camera/calibrator.py

+            event.dl1.tel[telid].image = charge
+            event.dl1.tel[telid].pulse_time = pulse_corr_array


In this case, the image and pulse_time will be of shape (1855,2) instead of (1855,).
I don't think this is what we want (following algorithms expect images and pulse_time with 1 dimension).
Instead, I'd recommend a ManualGainSelection to select high gain by default.

And then the notebook will need to be modified to plot images with single dimension only.

Hi @vuillaut,
actually this is what I want: I would like to have the possibility to calibrate without performing any gain selection (mainly for debug purposes). The HG by default can be obtained just putting the default threshold to 4095 ;-)
By the way, here I fixed the gain selector to the Threshold one, should we give the possibility to choose other selectors?

I understand the advantage of having this for debugging but here the default option is to not select gains (since None is the default for gain_threshold). Maybe the default should be to have a gain selection to make sure that image and pulse_time have the correct dimension by default? And keeping both gains be a "hidden feature" for expert and debug?

To your question, yes I think it would be better to have the gain selector be the one passed in the config file.

I understand the advantage of having this for debugging but here the default option is to not select gains (since None is the default for gain_threshold). Maybe the default should be to have a gain selection to make sure that image and pulse_time have the correct dimension by default? And keeping both gains be a "hidden feature" for expert and debug?

No problem, I can make 4094 as default

To your question, yes I think it would be better to have the gain selector be the one passed in the config file.

Then I must take out the threshold trailet and pass the gain selector and one must not forget to pass the selector Config. @rlopezcoto what do you think?

And then the notebook will need to be modified to plot images with single dimension only.

which notebook?

The one in this PR

This is already working with images with single dimension

- Use the r0 waveform for the gain selection - Store the selected gain following the prescrition of K. Kosack's document "R1 & DL0 Telescope Event Interfaces and Prototype Evaluation"

FrancaCassol · 2019-12-18T10:09:18Z

Hi,
are there other comments on this PR?
Otherwise I think we can merge it.

vuillaut · 2019-12-18T13:15:04Z

lstchain/scripts/lstchain_data_muon_analysis.py

@@ -29,6 +21,7 @@
 --output_file Data_table.fits --pedestal_file pedestal_file_run446_0000.fits 
 --calibration_file calibration.hdf5
 --max_events 1000
+--tel_id 0


shouldn't this be 1 by default?
ok it is just an example...

vuillaut · 2019-12-18T13:17:11Z

lstchain/scripts/lstchain_data_muon_analysis.py

@@ -74,7 +72,7 @@ def main():
    print("max events: {}".format(args.max_events))

    # Camera geometry
-    geom = CameraGeometry.from_name("LSTCam-002")
+    geom = CameraGeometry.from_name("LSTCam-003")


Is LSTCam-003 already part of current environment?
ctapipe v0.7 comes with what? ctapipe_resources v0.2.18 ?

LSTCam-003 was recently included in ctapipe_io_lst (see New time and geometry #19)

… subtracted

rlopezcoto · 2019-12-20T12:35:29Z

@FrancaCassol it seems that the latest commits made the code break on the calibration part, could you please check it out? Thanks

rlopezcoto · 2020-01-09T07:46:39Z

Hi @FrancaCassol, did you have time to have a look at this? It would be great if we could move on with this one to get #241 also merged

FrancaCassol · 2020-01-09T13:43:05Z

Hi @rlopezcoto,
yes I will have a look

rlopezcoto · 2020-01-09T15:54:21Z

Thanks @FrancaCassol!
Regarding the discussion we had in #252, and just for the record, now for the gain selection, the threshold=3500 and is set on R1 waveforms, right?

FrancaCassol · 2020-01-09T17:17:34Z

Hi @ruben,
yes, the threshold in lstchian_standard_config.json is 3500 and the selection is done on the R1 waveform :

waveforms, gain_mask = self.gain_selector(event.r1.tel[telid].waveform)

FrancaCassol added 13 commits December 9, 2019 15:38

Add cut on low gain charge std to elimunate Magic lidar events

76e4221

Do not use events without TIB data

Updated the trailets for selecting calibraiton events

c646d56

Changed the time correction in order to put time arouns zero as for drs4 time correction

Changed charge cut values valid for filters 5-2

7a378fc

Correct function declaration

efe6510

Add drs4 time correction

4ff2c9a

dd the the gain selection if the threshold is not None

18521a1

Add a try condition

f0d239d

Pass as trailet the charge integrator

68c7afe

Added the TimeCorrectionCalculate configuration

036e06c

Add the production of the time calibration file

27f377f

Merge branch 'master' of https://github.com/cta-observatory/cta-lstchain

4beb7b8

into update_calibrator

change write more

ff07186

Notebook to show the introduction of the gain selection and of the

e0c5720

pulse time correction to the calibrator

FrancaCassol changed the title ~~Update calibrator with pulse coorction and gain selection~~ Update calibrator with pulse correction and gain selection Dec 13, 2019

FrancaCassol mentioned this pull request Dec 17, 2019

Wrong initialisation values for gain selection and charge integration for data #252

Closed

rlopezcoto reviewed Dec 17, 2019

View reviewed changes

vuillaut reviewed Dec 17, 2019

View reviewed changes

morcuended mentioned this pull request Dec 17, 2019

Not writing DL1b if cleaning levels are set to 0 #249

Merged

- Change the default thresold value from None to 4095

09b36d1

- Use the r0 waveform for the gain selection - Store the selected gain following the prescrition of K. Kosack's document "R1 & DL0 Telescope Event Interfaces and Prototype Evaluation"

rlopezcoto approved these changes Dec 18, 2019

View reviewed changes

vuillaut reviewed Dec 18, 2019

View reviewed changes

FrancaCassol added 4 commits December 19, 2019 12:23

Change the default values to the correct one

aedb810

Consider the r1 waveform for defining the threshold, this is baseline…

50da5c2

… subtracted

use the localpeakintegrator and add the charge configuration

351724b

Add the time calibration and move the gain selection to the calibrator

da8b26a

FrancaCassol added 2 commits December 19, 2019 12:31

Add the time calibration file as input for the dl1 production

946dec1

Remove assert for type of charge integrator

138b269

Remove comments

0386bbf

rlopezcoto merged commit a9de4c3 into cta-observatory:master Jan 10, 2020

rlopezcoto mentioned this pull request Jan 15, 2020

Time calibration file not being produced by onsite calibration scripts #260

Closed

FrancaCassol mentioned this pull request Jan 16, 2020

Add time calibration script #261

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update calibrator with pulse correction and gain selection #248

Update calibrator with pulse correction and gain selection #248

FrancaCassol commented Dec 13, 2019

review-notebook-app bot commented Dec 13, 2019

FrancaCassol commented Dec 17, 2019

FrancaCassol commented Dec 17, 2019

rlopezcoto left a comment

rlopezcoto Dec 17, 2019

FrancaCassol Dec 17, 2019

vuillaut Dec 17, 2019

FrancaCassol Dec 17, 2019

vuillaut Dec 17, 2019

FrancaCassol Dec 18, 2019

vuillaut Dec 17, 2019 •

edited

Loading

vuillaut Dec 17, 2019

FrancaCassol Dec 17, 2019

vuillaut Dec 17, 2019

FrancaCassol Dec 17, 2019

FrancaCassol Dec 17, 2019

vuillaut Dec 17, 2019

FrancaCassol Dec 17, 2019 •

edited

Loading

FrancaCassol commented Dec 18, 2019

vuillaut Dec 18, 2019

vuillaut Dec 18, 2019

morcuended Dec 19, 2019

rlopezcoto commented Dec 20, 2019

rlopezcoto commented Jan 9, 2020

FrancaCassol commented Jan 9, 2020

rlopezcoto commented Jan 9, 2020 •

edited

Loading

FrancaCassol commented Jan 9, 2020

		event.dl1.tel[telid].image = charge
		event.dl1.tel[telid].pulse_time = pulse_corr_array

Update calibrator with pulse correction and gain selection #248

Update calibrator with pulse correction and gain selection #248

Conversation

FrancaCassol commented Dec 13, 2019

review-notebook-app bot commented Dec 13, 2019

FrancaCassol commented Dec 17, 2019

FrancaCassol commented Dec 17, 2019

rlopezcoto left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vuillaut Dec 17, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrancaCassol Dec 17, 2019 • edited Loading

Choose a reason for hiding this comment

FrancaCassol commented Dec 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rlopezcoto commented Dec 20, 2019

rlopezcoto commented Jan 9, 2020

FrancaCassol commented Jan 9, 2020

rlopezcoto commented Jan 9, 2020 • edited Loading

FrancaCassol commented Jan 9, 2020

vuillaut Dec 17, 2019 •

edited

Loading

FrancaCassol Dec 17, 2019 •

edited

Loading

rlopezcoto commented Jan 9, 2020 •

edited

Loading