Proposals for charge calibration flow

[watsonjj]

I have prepared some proposals for the flow of charge calibration. I am mainly concerned with the charge calibration at the DL1 level.

Certain aspects of the charge calibration are common in all of these proposals, and I think they are mostly agreed upon:

• The Calculators attach calibration coefficients to the MON container after they are calculated
• The CameraCalibrator takes the calibration coefficients from the MON container and applies them to the extracted charge
• Events are labelled with a trigger type, which dictates if it is used in a calibration calculator
• The three points above allow an on-the-fly calibration to occur in an intuitive way
• A Tool can be created that iterates over a file, and stores the resulting MON container to a file
• A MON source can be setup to read MON information into the container (therefore providing either an initial calibration, or an alternative to simultaneously running a Calculator in the event loop for on-the-fly calibration)

Connected previous discussions: #1037 #1041 #1026

I will post the proposals I currently have below, so they can be discussed.

[watsonjj]

[watsonjj]

[watsonjj]

[watsonjj]

[watsonjj]

At the moment my preferred option is Approach 3

[watsonjj]

EDIT for approach 3:

[kosack]

Let's separate calibration calculation from calibration application.

Let's start with a simple 2-pass method:

• Pass 0:

  • process DC/PE data and generate coefficients per pixel (special run). This should include the overall calib_scale for MC data.

• pass 1 : calculate calibration: loop through data (perhaps only calib events), output a DL0/Monitoring files with:

  • pedestal/pixel (every second or so)
  • flatfield correction /pixel

• pass 2: process data (stage 1) and apply the calibration monitoring info. The monitoring data should be loaded, and a scipy.interpolation function generated, so that we have a single function that gives the pedestal, flatfield, etc .for a single event (based on it's timestamp). For MC, this function can just return the same constant value when called.

Where is data?

• Monitoring: calibration coefficients generated from many events, etc (e.g a table of time, values)
• Event: the calibration coefficients interpolated to exactly the value for the current event

[FrancaCassol]

Hi @watsonjj

At the moment my preferred option is Approach 3

if I understand correctly this is what is do proposed now for the calibration at R1 level.
Why do you say that some steps are performed multiple times? Could you give an example?

[FrancaCassol]

Hi @kosack,

Let's start with a simple 2-pass method:

* Pass 0:
  
  * process DC/PE data and generate coefficients per pixel (special run).  This should include the overall calib_scale for MC data.

yes, let call it the R0 calibration level, this is what we are already doing in LST

* pass 1 : **calculate calibration**:  loop through data (perhaps only calib events), output a DL0/Monitoring files with:
  
  * pedestal/pixel (every second or so)
  * flatfield correction /pixel

In the requirements, the calibration frequency is 100 Hz, so in order to have enough statistics we will have calibration events each 10-100 s (depending also on how fast the values are supposed to change)

* pass 2: process data (stage 1) and apply the calibration monitoring info.  The monitoring data should be loaded, and a `scipy.interpolation`  function generated, so that we have a single function that gives the pedestal, flat-field, etc .for a single event (based on it's timestamp).   For MC, this function can just return the same constant value when called.

As a first approach, one can probably simply take the calibration event closer in time to the shower event. It will be interesting to test if the interpolation makes a big difference statistically speaking (it will depend on how smooth are the changes in the calibration coefficients)

Where is data?

* Monitoring: calibration coefficients generated from many events, etc (e.g a table of time, values)

* Event: the calibration coefficients interpolated to exactly the value for the current event

In the first approach (without interpolation), the event values are just pointing to the corresponding monitoring values

[watsonjj]

yes, let call it the R0 calibration level, this is what we are already doing in LST

No, this thread is specifically concerned with the DL1 calibration. Yes I can assume the camera do something in their R1 calibration that puts the waveform in some units proportional to p.e. (or photons) per sample. This is necessary for the online analysis. But the calibration I am concerned with is the DL1 calibration into absolute p.e. or photons. This calibration is dependant on the charge extraction used.

Similar processes will occur for the calibration of the charge in the R1 and DL1 calibrations, and they can potentially use similar methods. But R1 is intended to be an approximate calibration. An additional calibration of the charge is required for the DL1 level. We need to be clear and careful about what we are discussing here....

[watsonjj]

R1 calibration is already provided to the waveforms from simtelarray (stored in the simtel file and applied in the SimtelEventSource). What ctapipe is missing at the moment is the DL1 calibration. In #1160 I added the functionality to apply the charge calibration in the DL1 calibration step. The calibration coefficients are retrieved from event.calibration.tel[telid].dl1. In a future PR I will create the calculators that fill the event.calibration.tel[telid].dl1.