Merge pull request #166 from cta-observatory/update-notebooks-and-res…

…ults

Update the entire benchmark suite

docs/benchmarks.rst

@@ -0,0 +1,30 @@
+.. _benchmarks:
+
+Benchmarks (``protopipe.benchmarks``)
+=====================================
+
+The ``protopipe.benchmarks`` module is composed of 3 parts:
+
+- ``notebooks``, a folder containing benchmarking Jupyter notebooks divided by analysis stage,
+- ``operations``, a sub-module containing functions to perform many different operations on data related to benchmarking
+- ``plot``, a sub-module containing plotting functions
+- ``utils``, a sub-module containing utility functions for the notebooks
+- ``book_template``, a folder containing the Jupyter Book template for a CTA analysis
+
+.. note::
+    Much of what is contained in the sub-modules is the product of a long refactoring process
+    of old material from the notebooks.  
+    Many things can be improved or imported by *ctaplot*/*cta-benchmarks* and *ctapipe* 
+    as the refactoring of the pipeline takes progress.  
+    Also, not all notebooks are exatcly the same in terms of global options,
+    a notebook template will be added.
+
+.. note::
+    All benchmarks will be launched by means of a new ``protopipe-BENCHMARK`` script.
+    This will become the recommended method, as it will integrates with the rest of the analysis interface.
+
+API reference
+-------------
+
+.. automodapi:: protopipe.benchmarks
+    :skip: accuracy_score, auc, binned_statistic, curve_fit, roc_curve, Column, LogNorm, Table, Path, cone_solid_angle, RectBivariateSpline, read_table

docs/contribute/beforepushing.rst

@@ -109,6 +109,15 @@ and it is analysed using the same workflow as in a standard full-scale analysis.
 Benchmarks
 ----------
 
+.. note::
+
+  Benchmarks will soon disappear from this documentation as hosting more than 1 analysis
+  started to be non-ideal.
+  A new repository to host the results has been created under the CTAO Gitlab instance
+  (`link <https://gitlab.cta-observatory.org/mperesano/protopipe-results>`__).
+  It will be soon filled with the currently available analyses and with the
+  publication of protopipe v0.5.0, contribute  will be even easier than before.
+
 .. toctree::
    :hidden:
 
@@ -130,10 +139,10 @@ Benchmarks
 
 This documentation hosts a series of notebooks used for benchmarking.
 
-Their contents follow the development triggered by the ongoing
+Their contents follow the development triggered by the 
 comparison between protopipe and CTA-MARS (see
 `this issue <https://github.com/cta-observatory/protopipe/issues/24>`__ and
-references therein for a summary).
+references therein for a summary), but also with EventDisplay.
 
 The following tables summarize the analyses currently performed and planned with
 *protopipe* to evaluate its performance and that of the observatory.
@@ -164,12 +173,6 @@ pipelines, which is used throughout the notebooks for comparison and performance
      - 180°
      - `link <https://forge.in2p3.fr/attachments/download/63177/CTA-N_from_South.zip>`__
      - `link <https://forge.in2p3.fr/projects/step-by-step-reference-mars-analysis/wiki>`__
-   * - South (Paranal desert)
-     - baseline
-     - 20°
-     - 180°
-     - ...
-     - ...
 
 .. list-table:: **PROD 5**
   :widths: 25 25 25 25 25 25
@@ -189,7 +192,7 @@ pipelines, which is used throughout the notebooks for comparison and performance
     - ...
   * - South (Paranal desert)
     - baseline
-    - 20°
+    - 40°
     - 180°
     - ...
     - ...

docs/index.rst

@@ -80,6 +80,7 @@ Please, check the development version of the README for up-to-date links.
    mva/index
    perf/index
    scripts/index
+   benchmarks
 
 Indices and tables
 ==================

protopipe/aux/example_config_files/protopipe_CTAMARS_1stPass_DL1a.json

@@ -0,0 +1,37 @@
+{
+  "ProcessorTool": {
+      "progress_bar": true,
+      "log_level": 30
+  },
+  "DataWriter": {
+        "Contact": {
+            "name": "YOUR-NAME-HERE",
+            "email": "[email protected]",
+            "organization": "YOUR-ORGANIZATION"
+        },
+        "output_path": "events_protopipe_CTAMARS_calibration_1stPass.dl1.h5",
+        "overwrite": false,
+        "write_images": true,
+        "write_parameters": false,
+        "write_stereo_shower": false,
+        "write_mono_shower": false,
+        "transform_image": true,
+        "transform_peak_time": true
+  },
+  "SimTelEventSource": {
+      "calib_scale": 0.897
+  },
+  "CameraCalibrator": {
+      "apply_peak_time_shift": false,
+      "apply_waveform_time_shift": false,
+      "image_extractor_type": "TwoPassWindowSum"
+  },
+  "TwoPassWindowSum": {
+  "apply_integration_correction": false,
+      "core_threshold" : [
+        ["type", "LST_LST_LSTCam", 6.0],
+        ["type", "MST_MST_NectarCam", 8.0]
+      ],
+      "disable_second_pass": true
+  }
+}

protopipe/aux/example_config_files/protopipe_CTAMARS_2ndPass_Dl1a.json

@@ -0,0 +1,37 @@
+{
+  "ProcessorTool": {
+      "progress_bar": true,
+      "log_level": 30
+  },
+  "DataWriter": {
+        "Contact": {
+            "name": "YOUR-NAME-HERE",
+            "email": "[email protected]",
+            "organization": "YOUR-ORGANIZATION"
+        },
+        "output_path": "events_protopipe_CTAMARS_calibration_2ndPass.dl1.h5",
+        "overwrite": false,
+        "write_images": true,
+        "write_parameters": false,
+        "write_stereo_shower": false,
+        "write_mono_shower": false,
+        "transform_image": true,
+        "transform_peak_time": true
+  },
+  "SimTelEventSource": {
+      "calib_scale": 0.897
+  },
+  "CameraCalibrator": {
+      "apply_peak_time_shift": false,
+      "apply_waveform_time_shift": false,
+      "image_extractor_type": "TwoPassWindowSum"
+  },
+  "TwoPassWindowSum": {
+  "apply_integration_correction": false,
+      "core_threshold" : [
+        ["type", "LST_LST_LSTCam", 6.0],
+        ["type", "MST_MST_NectarCam", 8.0]
+      ],
+      "disable_second_pass": false
+  }
+}

protopipe/benchmarks/__init__.py

@@ -0,0 +1,3 @@
+from .operations import *
+from .plot import *
+from .utils import *