Merge pull request #116 from HealthyPear/maintenance-integration_test…

…_pipeline_up_to_models

Setup of pipeline integration testing up to modeling

conftest.py

@@ -0,0 +1,6 @@
+import pytest
+
+
+@pytest.fixture(scope="session")
+def pipeline_testdir(tmp_path_factory):
+    return tmp_path_factory.mktemp("test_pipeline")

docs/contribute/beforepushing.rst

@@ -35,18 +35,16 @@ You will have to fix any warning that appears during documentation building,
 because the documentation also runs on `readthedocs <https://readthedocs.org/>`__
 with an option to treat warnings as errors.
 
-Unit and integration tests
---------------------------
+Unit tests
+----------
 
 .. note::
   This is a maintenance activity which has being long overdue and we need
   manpower for it, so if you have experience on this or you want to contribute
   please feel free to do so.
 
-  For more information on how to contribute to this effort check these issues,
-
-  - `unit-tests <https://github.com/cta-observatory/protopipe/issues/69>`__,
-  - `integration tests <https://github.com/cta-observatory/protopipe/issues/70>`__.
+  For more information on how to contribute to this effort check
+  `this issue <https://github.com/cta-observatory/protopipe/issues/69>`__.
 
 Being *protopipe* based on *ctapipe*, all the tools imported from the latter
 have been already tested and approved (*protopipe* uses always a version of
@@ -64,18 +62,28 @@ Same for *pyirf*.
   order to code a new feature, this has to be pull-requested to *ctapipe* and at
   the same time hardcoded in *protopipe*, until the new version of *ctapipe* is released.
 
-For the moment there is only **one integration test**.
+Integration tests
+^^^^^^^^^^^^^^^^^
+
+.. note::
+  For more information on how to contribute to this effort check
+  `this issue <https://github.com/cta-observatory/protopipe/issues/70>`__.
+
+The integration tests are defined in the dedicated module ``pipeline/scripts/tests/test_pipeline.py``
+and start from test simtel files stored on a CC-IN2P3 dataserver.  
+
+The test data is diffuse data from the Prod3b baseline simulations of both 
+CTAN and CTAS produced with the following Corsika settings,
 
-This test is in charge to detect if changes in the script
-``protopipe.scripts.data_training`` and any code used by it, produce
-any fatal behaviour or crash.
+- gammas, ``NSHOW=10 ESLOPE=-2.0 EMIN=10 EMAX=20 NSCAT=1 CSCAT=200 VIEWCONE=3``
+- protons, ``NSHOW=10 ESLOPE=-2.0 EMIN=100 EMAX=200 NSCAT=1 CSCAT=200 VIEWCONE=3``
+- electrons, ``NSHOW=10 ESLOPE=-2.0 EMIN=10 EMAX=20 NSCAT=1 CSCAT=200 VIEWCONE=3``
 
-For the moment it uses an old test file from *ctapipe* (a Prod2 CTA South array composed of
-LSTCam, FlashCam and ASTRICam with about ~100 simulated showers).
-Each test is expected to produce a non-empty HDF5 file.
+in the same proportions as a standard full-scale analysis.
 
-The test can be executed directly from the main folder of *protopipe* by launching
-``pytest``. It is also automatically triggered by the CI every time a new
+The pipeline integration testing can be executed directly from the main folder 
+of *protopipe* by launching ``pytest``.
+It is also automatically triggered by the CI every time a new
 pull-request is pushed to the repository, and its correct
 execution is a mandatory condition for merging.
 

protopipe/scripts/tests/test_classifier.yaml

@@ -0,0 +1,51 @@
+General:
+ model_type: 'classifier'
+ # [...] = your analysis local full path OUTSIDE the Vagrant box
+ data_dir: '[...]/shared_folder/analyses/v0.4.0_dev1/data/TRAINING/for_particle_classification/'
+ data_sig_file: 'TRAINING_classification_{}_gamma_merged.h5'
+ data_bkg_file: 'TRAINING_classification_{}_proton_merged.h5'
+ cam_id_list: ['LSTCam', 'NectarCam']
+ table_name_template: '' # leave empty (TO BE REMOVED)
+ outdir: '[...]/shared_folder/analyses/v0.4.0_dev1/estimators/gamma_hadron_classifier'
+
+Split:
+ train_fraction: 0.8
+ use_same_number_of_sig_and_bkg_for_training: False  # Lowest statistics will drive the split
+
+Method:
+ name: 'RandomForestClassifier'  # AdaBoostClassifier or RandomForestClassifier
+ target_name: 'label'
+ tuned_parameters: # these are lists of values used by the GridSearchCV algorithm
+  n_estimators: [200]
+  max_depth: [10]  # null for None
+  max_features: [3] # possible choices are “auto”, “sqrt”, “log2”, int or float
+  min_samples_split: [10]
+  min_samples_leaf: [10]
+ scoring: 'roc_auc' # possible choices are 'roc_auc', 'explained_variance'
+ cv: 2
+ use_proba: True  # If not output is score
+ calibrate_output: False  # If true calibrate probability
+
+FeatureList:
+ # - 'log10_reco_energy'
+ # - 'log10_reco_energy_tel'
+ - 'log10_hillas_intensity'
+ - 'hillas_width'
+ - 'hillas_length'
+ - 'h_max'
+ - 'impact_dist'
+
+SigFiducialCuts:
+ - 'good_image == 1'
+ - 'is_valid == True'
+
+BkgFiducialCuts:
+ - 'good_image == 1'
+ - 'is_valid == True'
+
+Diagnostic:
+ # Energy binning (used for reco and true energy)
+ energy:
+  nbins: 4
+  min: 0.02
+  max: 200

protopipe/scripts/tests/test_config_analysis_south.yaml

@@ -9,7 +9,7 @@ General:
  # WARNING: for simulations containing multiple copies of the telescopes,
  # only 'full_array' or custom list are supported options!
  array: full_array
- cam_id_list : ['LSTCam', 'NectarCam'] # Selected cameras (disabled option)
+ cam_id_list : ['LSTCam', 'FlashCam', 'CHEC'] # Selected cameras (disabled option)
 
 # Cleaning for reconstruction
 ImageCleaning: