Merge pull request #157 from HealthyPear/feature-add_protopipe_BENCHMARK

Add new script for benchmarking

docs/index.rst

@@ -10,11 +10,11 @@ Welcome to protopipe's documentation!
 (CTA) based on the `ctapipe <https://cta-observatory.github.io/ctapipe>`__ library.
 The package has been developed and tested in the department of astrophysics at
 `CEA-Saclay/IRFU <http://irfu.cea.fr/dap/en/index.php>`__,
-but since Sep 23, 2019 is also open for development by other members of the CTA consortium.
+but since September 2019 has been open for development to all members of the CTA consortium.
 
 The source code is currently hosted on a `GitHub repository <https://github.com/cta-observatory/protopipe>`__ 
 to which this documentation is linked.  
-It will soon migrate to the CTAO-GitLab installation under the ASWG group
+It is planned to migrate it to the CTAO-GitLab installation under the ASWG group
 repository (`link <https://gitlab.cta-observatory.org/cta-consortium/aswg>`__).
 
 Current performance is stored internally at `this RedMine page <https://forge.in2p3.fr/projects/benchmarks-reference-analysis/wiki/Protopipe_performance_data>`__.

docs/install/development.rst

@@ -4,7 +4,7 @@ Development version
 ===================
 
   1. `fork <https://help.github.com/en/articles/fork-a-repo>`__ the `repository <https://github.com/cta-observatory/protopipe>`_
-  2. create a virtual environment if necessary (Anaconda users can use the ``environment_development.yml`` file)
+  2. if necessary, create a virtual environment (Anaconda users can use the ``environment_development.yml`` file found at the root of the repository)
   3. ``pip install -e '.[all]'``
 
   The ``all`` keyword will install all extra requirements,

docs/install/grid.rst

@@ -28,42 +28,14 @@ are the following,
   Even if *protopipe* itself allows for Python 3.7, the minimum version required to use
   Dirac is 3.8.
 
-For convenience, a ready-to-use conda environment recipe is reported below.
-You can also find it in the `root directory of the interface <https://github.com/HealthyPear/protopipe-grid-interface>`_.
+An example base conda environment recipe can be found
+`here <https://github.com/HealthyPear/protopipe-grid-interface/blob/master/environment_development.yaml>`_.
 You can create the corresponding environment with a command like this,
 
-``conda env create -f my_env_recipe.yml -n my_env``
+``conda env create -f environment_development.yaml``
 
-.. code-block:: yaml
-
-  name: protopipe-CTADIRAC
-  channels:
-    - conda-forge
-  dependencies:
-    - python>=3.8
-    - pip
-    - dirac-grid
-    - voms
-    - pytables
-    - pyyaml
-    - eventio # required for protopipe with ctapipe <0.12.0
-    - gammapy=0.18 # required for protopipe with ctapipe <0.12.0
-    - pip:
-        - CTADIRAC
-
-In this environment you can then install first protopipe and then the interface.
-
-If you want to install them in their released versions, integrate the ``pip`` section
-with,
-
-- ``protopipe``/``protopipe=x.y.z`` for a latest/specific released version of protopipe,
-- ``git+https://github.com/HealthyPear/[email protected]`` for a released version of its interface.
-
-An overview of the versioning between the two sofware is described below (:ref:`versioning`)
-
-If you are a developer, or you want to work with the development version,
-please refer to the development installation instructions for protopipe (:ref:`install-development`)
-and its interface (:ref:`install-grid-dev`).
+In this environment you can then install first *protopipe* and then its interface
+following the respective instructions.
 
 DIRAC GRID certificate
 ----------------------
@@ -81,8 +53,9 @@ The interface
 Getting a released version
 --------------------------
 
-For versions >=0.4.0 you can install it as a Python3-based package in your environment as explained
-above.
+For versions >=0.4.0 you can install it as a Python3-based package in your environment,
+
+``pip install git+https://github.com/HealthyPear/[email protected]``
 
 The following table refers to all versions and their compatibility with _protopipe_.
 
@@ -102,29 +75,30 @@ The following table refers to all versions and their compatibility with _protopi
     * - v0.2.X
       - v0.2.X
 
-The latest released version of the interface is also compatible with
+The latest released version of the interface is always compatible with
 the development version of *protopipe*.
 
 .. warning::
 
   After the Python3 upgrade of DIRAC and CTADIRAC,
   the interface installation and usage have changed considerably,
-  while its relation with _protopipe_ has only improved.
+  while its relation with *protopipe* has only improved.
   It is very unlikely that you will ever need to work with a version older than v0.4.0,
   but if this were to happen, please check older versions of this documentation
-  either from readthedocs or from the repository of _protopipe_.
+  either from readthedocs or from the repository of *protopipe*.
 
 .. _install-grid-dev:
 
 Getting the development version
 -------------------------------
 
 This version is:
+
 - always compatible with the development version of *protopipe*,
 - possibly compatible with the latest release of *protopipe*,
 
 The procedure to install with this version is similar to the same one
-for _protopipe_:
+for *protopipe*:
 
 - ``git clone https://github.com/HealthyPear/protopipe-grid-interface.git``
 - ``cd protopipe-grid-interface``
@@ -151,8 +125,8 @@ You will be asked to generate your proxy and then to choose the ``Setup`` and th
 You need to choose the default values.
 
 .. warning::
-  The defaults right now are lacking redundance in the configuration system.
-  It is suggested to edit the configuration file that you can find inside your conda enviroment
+  The configuration system defaults could lack redundance depending on the version of CTADIRAC.
+  It is suggested to check or edit the configuration file that you can find inside your conda enviroment
   under ``etc/dirac.cfg`` like the following,
 
   .. code-block::

docs/install/index.rst

@@ -90,26 +90,37 @@ Some use-cases could be the following,
 Docker
 ++++++
 
-The container used by the interface requires the `installation of Docker <https://docs.docker.com/get-docker/>`_.
+The `installation of Docker <https://docs.docker.com/get-docker/>`_ is a requirement.
 
-To enter the container (and the first time downloading the image),
+Two docker containers are provided for development and released content respectively.
 
-``docker run --rm -v $HOME/.globus:/home/dirac/.globus -v $PWD/shared_folder:/home/dirac/shared_folder -v [...]/protopipe:/home/dirac/protopipe -v [...]/protopipe-grid-interface:/home/dirac/protopipe-grid-interface -it ctadirac/client``
+To enter the *development* container,
+
+``docker run --rm -v $HOME/.globus:/home/dirac/.globus -v [...]/protopipe:/home/cta/protopipe -v [...]/protopipe-grid-interface:/home/cta/protopipe-grid-interface -it mperesano/protopipe-ctadirac-dev``
+
+where,
+
+- ``[...]`` is the path of your source code on the host,
+- the ``--rm`` flag will erase the container at exit if you don't want to persist it in order to save disk space (any data produced will not disappear if you use a bind-mount).
+
+To enter the *release* container,
+
+``docker run --rm -v $HOME/.globus:/home/dirac/.globus -it mperesano/protopipe-ctadirac:tag``
+
+where ``tag`` if omitted will always refer to the latest release, otherwise you will need to specify it.
 
-where ``[...]`` is the path of your source code on the host and the ``--rm`` flag will erase the container at exit
-to save disk space (the data stored in the ``shared_folder`` won't disappear).
 Please, refer to the Docker documentation for other use cases.
 
-**WARNING**
-If you are using *macos* you could encounter some disk space issues.
-Please check `this <https://docs.docker.com/docker-for-mac/space/>`_ and `also this <https://djs55.github.io/jekyll/update/2017/11/27/docker-for-mac-disk-space.html>`_ on how to manage disk space.
+.. warning::
+  If you are using *macos* you could encounter some disk space issues.
+  Please check `this <https://docs.docker.com/docker-for-mac/space/>`_ and `also this <https://djs55.github.io/jekyll/update/2017/11/27/docker-for-mac-disk-space.html>`_ on how to manage disk space.
 
 Vagrant
 +++++++
 
-**NOTE**
-Only required for users that want to use a *Singularity*
-container on a *macos* and *Microsoft Windows* machine.
+.. note::
+  Only required for users that want to use a *Singularity*
+  container on a *macos* and *Microsoft Windows* machine.
 
 All users, regardless of their operative systems, can use protopipe and its interface via
 `Vagrant <https://www.vagrantup.com/>`_. 
@@ -126,9 +137,9 @@ used by the interface to setup the analysis.
 Singularity
 +++++++++++
 
-**WARNING**
-Support for *Singularity* has been dropped by the mantainers of *CTADIRAC*.
-The following solutions have not been tested in all possible cases.
+.. warning::
+  Support for *Singularity* has been dropped by the mantainers of *CTADIRAC*.
+  The following solutions have not been tested in all possible cases.
 
 - **macos / Microsoft Windows**
 
@@ -183,9 +194,9 @@ but the recipe for the container has been saved here.
 In this case you won't need to do ``. /home/dirac/dirac_env.sh``: the 
 commands will be already stored in your ``$PATH``.
 
-**WARNING**
-The recipe ``CTADIRAC_singularity`` is maintained by the author; if any bug arises,
-reverting to the methods described above (if possible) will provide you with a working environment.
+.. warning::
+  The recipe ``CTADIRAC_singularity`` is not currently maintained; if any bug arises,
+  reverting to the methods described above (if possible) will provide you with a working environment.
 
 If you have root privileges you can just build your own image with,
 

docs/install/release.rst

@@ -10,7 +10,8 @@ If you prefer to work from an Anaconda virtual environment you can create it wit
 
 ``conda env create -f environment_latest_release.yml``
 
-or any other compatible virtual environment, depending on your needs.
+The recipe can be found in the root folder of the project repository,
+but any other compatible virtual environment can be used, depending on your needs.
 
 For previous releases,
 

docs/mva/index.rst

@@ -6,27 +6,30 @@ Multivariate analysis (``protopipe.mva``)
 Introduction
 ------------
 
-`protopipe.mva` contains utilities to build models for regression or
-classification problems. It is based on machine learning methods available in
+`protopipe.mva` contains utilities to build models for regression and
+classification. It is based on machine learning methods available in
 scikit-learn_. Internally, the tables are dealt with the Pandas_ Python module.
 
 For each type of camera a regressor/classifier should be trained.
-For both type of models an average of the image estimates is later computed to
+
+For both type of models an average of the image estimates is computed
+during the :ref:`data_training` and/or :ref:`DL2` steps to
 determine a global output for the event (energy or score/gammaness).
 
 Details
 -------
 
 Data is split in train and test subsamples by single telescope images.
 
-The class ```TrainModel``` uses a training sample composed of gamma-rays for a
-regression model. In addition of a gamma-ray sample, a sample of
-protons is also used to build a classifier.
+The class ```TrainModel``` uses a training sample composed of 
+
+- signal for a regression model,
+- signal and background for a classifier.
 
 The training of a model can be done also via the GridSearchCV_ algorithm which 
 allows to find the best hyper-parameters of the models.
 
-Supported models:
+Currently tested models:
 
 - ``sklearn.ensemble.RandomForestClassifier``
 - ``sklearn.ensemble.RandomForestRegressor``

docs/perf/index.rst

@@ -6,34 +6,31 @@ Performance generation (``protopipe.perf``)
 Introduction
 ------------
 
-The DL2-to-DL3 step of the *protopipe* pipeline is now based on the 
+The DL2-to-DL3 step of the *protopipe* pipeline is based on the 
 `pyirf <https://cta-observatory.github.io/pyirf/>`_ library.
 
 The ``protopipe.perf`` module contains utility tools used to interface with this
 library, mainly to translate the DL2 data in the internal nomenclature used by
 *pyirf* to produce the DL3 file.
 
-In general this step takes care of
+In general this step takes care of,
 
 * handling the determination of the best-cutoffs to separate gamma and
   the background (protons + electrons),
 * estimating the sensitivity from the optimized cuts,
 * producing the instrument response functions (IRFs).
 
-
-Provided that a performance script makes used of 
+Provided that a performance script makes use of 
 ``protopipe.perf.utils.read_DL2_pyirf``, *protopipe* supports the addition and 
 testing of multiple scripts based on pyirf.
 
 .. note::
   Some functions are discontinued or will be with the next release
-  and they will be removed.
+  and should be removed.
 
-The current approach is based on the EventDisplay historical pipeline for which
+The current approach is based on the *EventDisplay* historical pipeline for which
 the following main points apply,
 
-- minimum telescope multiplicity cut set to 3 (for 30' and 100'' exposures)
-  and 4 (for 5h and 50h exposures),
 - maximum signal efficiency set to 80%,
 - optimisation cuts are performed following these steps,
   - applying a global cut of 40% signal efficiency,

docs/pipeline/index.rst

@@ -6,7 +6,7 @@ Event reconstruction (``protopipe.pipeline``)
 Introduction
 ------------
 
-``protopipe.pipeline`` contains classes that are used in scripts to produce
+``protopipe.pipeline`` contains classes that are used in scripts to produce,
 
 - tables with images information (DL1), typically for g/h classifier and energy regressor,
 - tables with event information, typically used for performance estimation (DL2).
@@ -45,7 +45,7 @@ Management of events, triggers and images
 In case ``LST_stereo`` is set to ``True``, events with,
 
 - 1 LST trigger and less than 2 triggers from any other telescope type will not
-  be processed after DL1b
+  be processed after DL1b,
 - 1 LST trigger and at least 2 triggers from any other telescope type will get processed
   at DL2a **without** the single-LST image and provided the remaining
   images are still at least 2 after the image quality cuts have been taken into