pytest failing on ctapipe tools in CI

[jlenain]

Describe the bug
pytest runs in CI currently fails when testing ctapipe tools being implemented in nectarchain. We encountered this while working on PR #115 with @tibaldo .

To Reproduce
Steps to reproduce the behavior:

1. Run pytest as implemented in CI:

pytest --cov=nectarchain --cov-report=xml

2. pytest fails when the initialize method is called. It behaves as if pytest inherited from options from the tested tool:

usage: pytest [-h] [-q] [--quiet] [-v] [--verbose] [--overwrite] [--progress]
              [--no-progress]
              [-c PedestalNectarCAMCalibrationTool.config_files]
              [--config PedestalNectarCAMCalibrationTool.config_files]
              [--log-level PedestalNectarCAMCalibrationTool.log_level]
              [-l PedestalNectarCAMCalibrationTool.log_file]
              [--log-file PedestalNectarCAMCalibrationTool.log_file]
              [--log-file-level PedestalNectarCAMCalibrationTool.log_file_level]
              [--provenance-log PedestalNectarCAMCalibrationTool.provenance_log]
              [-i PedestalNectarCAMCalibrationTool.run_file]
              [--input PedestalNectarCAMCalibrationTool.run_file]
              [-r PedestalNectarCAMCalibrationTool.run_number]
              [--run-number PedestalNectarCAMCalibrationTool.run_number]
              [-m PedestalNectarCAMCalibrationTool.max_events]
              [--max-events PedestalNectarCAMCalibrationTool.max_events]
              [-o PedestalNectarCAMCalibrationTool.output_path]
              [--output PedestalNectarCAMCalibrationTool.output_path]
              [--events-per-slice PedestalNectarCAMCalibrationTool.events_per_slice]
              [--charge_sigma_high_thr PedestalEstimationComponent.charge_sigma_high_thr]
              [--charge_sigma_low_thr PedestalEstimationComponent.charge_sigma_low_thr]
              [--extractor_kwargs PedestalEstimationComponent.extractor_kwargs]
              [--filter_method PedestalEstimationComponent.filter_method]
              [--method PedestalEstimationComponent.method]
              [--ucts_tmax PedestalEstimationComponent.ucts_tmax]
              [--ucts_tmin PedestalEstimationComponent.ucts_tmin]
              [--wfs_std_threshold PedestalEstimationComponent.wfs_std_threshold]
              [extra_args ...]
pytest: error: unrecognized arguments: --cov-report=xml

and does not recognize the --cov-report option any more.

Dissecting the CI workflow of ctapipe, it appears that when installing the pytest-xdist addon for pytest and running with:

pytest -n auto --dist loadscope --cov=nectarchain --cov-report=xml

pytest itself is running smoothly.

pytest-xdist should thus be part of the test installation configuration and enabled in CI.

Expected behavior
pytest itself should not fail in CI.

Supporting information
See the attached log.

[jlenain]

I do not understand what black magic is at work so that pytest runs smoothly when pytest-xdist is present, and fails when it is not.

@maxnoe , @kosack , have you already encountered this on the ctapipe side ?

[kosack]

Do you maybe not have the pytest-cov package installed?

Perhaps take a look a the official CTA python template on GitLab that @maxnoe created. It solves a lot of problems.

Though having pytest inherit options from the tool seems strange and is probably the true origin of the problem... however, the pytest executable itself is just a wrapper that essentially runs the equivalent of python -mpytest <yourscript>

[jlenain]

Thanks @kosack !
Yes, pytest-cov is definitely installed and used on our side.
I regularly look through the python template project that @maxnoe has set up to follow high quality rules, but I may have failed on this specific point.