protopipe

A pipeline prototype for the Cherenkov Telescope Array.

Installation

These instructions are for users only, i.e. you want to use the latest version of the code but do not intend to make any change to it. If your wish is to develop the package then follow the instruction below entitled "Instructions for developpers".

Get the source code and create the required basic conda environment:

      git clone https://github.com/cta-observatory/protopipe.git
      cd protopipe
      conda env create -f protopipe_environment.yml
      conda activate protopipe

In case you have already an environment 'myenv' set up for that kind of development, use:

      conda env update -n myenv -f protopipe_environment.yml

This environment contains the bare minimum in order to run the scripts and build the documentation.

It doesn't take into account any additional tool you could use later on (it is suggested to install ipython, jupyter and vitables, especially if you want to contribute to the code).

Next you need to install protopipe itself (protopipe is the folder where the code was expanded. The name differs if you downloaded a released tagged version, it can be protopipe-0.2 for instance) :

      cd protopipe
      python setup.py develop

This will let you make changes to your local git repository without the need to update your environment every time. In the abscence of a conda or pip installation made avalaible, to install a released version (no further development), use :

      python setup.py install

Remember that the environment needs to be activated in order for protopipe to work. This procedure has been successfully tested on macOS (10.10.5 & 10.14.6) and on Scientific Linux 7.

Building the documentation

From the main folder go down to the documentation repository and create the documentation :

      cd docs
      make html

You will probably get some harmless warnings. The initial page is stored in _ build/html/index.html, which you can open using your favorite internet browser.

Test if it works

Before starting to use protopipe, be sure to be inside the relevant environment (e.g. conda activate protopipe). Get a simtel Monte Carlo file obtained with prod3b with only LSTCam and NectarCam cameras. Any other kind of cameras in the file would lead to a crash, see issues #23. Note that some generic La Palma files can contain FlashCam cameras. The write_dl1.py script will prodcue a DL1 output from the simtel data.

      cd protopipe/scripts
      python write_dl1.py -h

Analysis chain general description

The following instructions refer to local use of protopipe, which doesn't involve the use of the DIRAC computing grid (you can ignore the grid.yaml configuration file). Typical analysis steps are the following :

1. create an analysis parent folder with the auxiliary script create_dir_structure.py
2. prepare the configuration files
   1. copy the example YAML configuration files in the relative subfolders
   2. edit them for the particular needs of your analysis
3. build a model for energy estimation
   1. Create tables for gamma-rays using write_dl1.py and analysis.yaml
   2. Merge them with the auxiliary script merge.sh
   3. create the model with build_model.py and regressor.yaml
   4. check it's performance with model_diagnostic.py and regressor.yaml
4. build a model for gamma/hadron separation
   1. set 'estimate_energy' to True when using write_dl1.py so the reconstructed energy can be estimated and further used as a discriminant parameter.
   2. same process as for the energy estimation but using classifier.yaml
5. produce DL2-level data
   1. create tables for gamma-rays, protons and electrons using write_dl2.py and analysis.yaml
   2. merge them with the auxiliary script merge.sh
6. Estimate the final performance (IRFs) with make_performance.py and performance.yaml

Note: DL1/DL2 scripts take as input only 1 file at the time.

Instructions for developers:

1. Fork the master protopipe remote repository as explained here
2. Follow the installation instructions above, but using your remote repository (we'll call origin yours and upstream the official one)
3. if you want to develop something new:
   1. update local master branch (git pull upstream master)
   2. create a new branch from your local master branch
   3. develop inside it
   4. push it to origin
   5. continue to develop and push until you feel ready
4. start a pull request from origin/your_branch to upstream/master
   1. wait for an outcome
   2. if necessary, you can update or fix things in your branch because now everything is traced (local/your_branch --> origin/your_branch --> pull request)

Note: if your developments take a relatively long time,

1. update periodically your local master branch,
2. if updates have been made, go to your local development branch (git checkout your_branch)
3. if there are no conflicts, move the beginning of your branch at the end of the updated master (git rebase master)
4. push your branch to your remote