Introduce pre-commit hooks

.github/workflows/ci.yml

@@ -14,7 +14,7 @@ env:
   PYTEST_ADDOPTS: --color=yes
 
 jobs:
-  flake8-lint:
+  lint:
     runs-on: ubuntu-latest
     name: Lint
     steps:
@@ -23,19 +23,23 @@ jobs:
       - name: Set up Python environment
         uses: actions/setup-python@v4
         with:
-          python-version: "3.8"
-      - name: flake8 Lint
-        uses: py-actions/flake8@v2 
+          python-version: "3.9"
+      - name: Runs pre-commit hooks
+        uses: pre-commit/[email protected]
+        with:
+          extra_args: --files $(git diff origin/main --name-only)
 
   tests:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.8, 3.9, "3.10", 3.11]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
         ctapipe-version: [v0.19.1]
+        install-method: [ "mamba", "pip" ]
 
     defaults:
       run:
+        # We need login shells (-l) for micromamba to work.
         shell: bash -leo pipefail {0}
 
     steps:
@@ -49,15 +53,22 @@ jobs:
         run: |
           sed -i -e "s/- python=.*/- python=$PYTHON_VERSION/g" environment.yml
 
-      - name: Create and activate env
+      - name: Mamba setup
+        if: matrix.install-method == 'mamba'
         uses: mamba-org/provision-with-micromamba@v15
         with:
           environment-name: "ci"
           environment-file: environment.yml
 
+      - name: Python setup
+        if: matrix.install-method == 'pip'
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+          check-latest: true
+
       - name: Install dependencies
         env:
-          PYTHON_VERSION: ${{ matrix.python-version }}
           CTAPIPE_VERSION: ${{ matrix.ctapipe-version }}
 
         run: |

.gitignore

@@ -63,7 +63,8 @@ distribute-*.tar.gz
 # Rope refactoring data:
 .ropeproject
 
-# ipython notebook checkpoints
+# ipython/jupyter notebooks and checkpoints
+*.ipynb
 .ipynb_checkpoints
 .cache*
 .cache

.pre-commit-config.yaml

@@ -0,0 +1,22 @@
+repos:
+  # https://pycqa.github.io/isort/docs/configuration/black_compatibility.html#integration-with-pre-commit
+  - repo: https://github.com/pycqa/isort
+    rev: 5.12.0
+    hooks:
+      - id: isort
+        args: ["--profile", "black", "--filter-files"]
+  - repo: https://github.com/psf/black
+    rev: 23.3.0
+    hooks:
+      - id: black-jupyter
+  # https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html?highlight=other%20tools#flake8
+  - repo: https://github.com/PyCQA/flake8
+    rev: 6.0.0
+    hooks:
+      - id: flake8
+        args: [--max-line-length=88, "--extend-ignore=E203,E712"]
+  - repo: https://github.com/mwouts/jupytext
+    rev: v1.14.6
+    hooks:
+    - id: jupytext
+      args: [--from, ipynb, --to, "py:percent"]

README.md

@@ -35,11 +35,11 @@ apptainer shell oras://ghcr.io/cta-observatory/nectarchain:latest
 
 The `nectarchain` code is then available under `/opt/cta/nectarchain`.
 
-[DIRAC support](#optinal-dirac-support) is fully available and configured within such a container.
+[DIRAC support](#optional-dirac-support) is fully available and configured within such a container.
 
-#### Note to Mac OS users
+#### Note to macOS users
 
-Mac OS users may experience errors when trying to initialize a proxy to DIRAC when the [DIRAC support is enabled](#optional-dirac-support), especially with recent hardware equipped with M1 or M2 Apple CPU chips. The container alternative can then help having an environment with CTADIRAC fully configured. However, [Apptainer](https://apptainer.org/) is [not readily available on Mac OS](https://apptainer.org/docs/admin/main/installation.html#mac), but there is a workaround using [`lima` virtualization technology](https://lima-vm.io/) on a Mac.
+macOS users may experience errors when trying to initialize a proxy to DIRAC when the [DIRAC support is enabled](#optional-dirac-support), especially with recent hardware equipped with M1 or M2 Apple CPU chips. The container alternative can then help having an environment with CTADIRAC fully configured. However, [Apptainer](https://apptainer.org/) is [not readily available on macOS](https://apptainer.org/docs/admin/main/installation.html#mac), but there is a workaround using [`lima` virtualization technology](https://lima-vm.io/) on a Mac.
 
 **TL;DR**
 
@@ -69,6 +69,12 @@ mamba activate nectarchain
 pip install -e .
 ```
 
+Enable [pre-commit hooks](https://pre-commit.com/), which enforces adherence to PEP8 coding style:
+
+```shell
+pre-commit install
+```
+
 Please follow the [same conventions as `ctapipe`](https://cta-observatory.github.io/ctapipe/getting_started/index.html#developing-a-new-feature-or-code-change) regarding settings of Git remotes for pull requests.
 
 ### Optional DIRAC support
@@ -96,7 +102,7 @@ one may try:
 ```shell
 mamba install dirac-grid "voms=2.1.0rc2=h7a71a8a_7"
 ```
-or the [container alternative](#note-to-mac-os-users) as explained above.
+or the [container alternative](#note-to-macos-users) as explained above.
 
 `nectarchain` is currently pinned to `ctapipe` version 0.12.
 

environment.yml

@@ -7,6 +7,8 @@ dependencies:
   - ctapipe=0.19.1
   - ctapipe-io-nectarcam
   - jupyterlab
+  - jupytext
+  - pre-commit
   - pytest-runner
   - pip
   - pandas

notebooks/Access NectarCAM data using DIRAC API.py

@@ -0,0 +1,86 @@
+# ---
+# jupyter:
+#   jupytext:
+#     text_representation:
+#       extension: .py
+#       format_name: percent
+#       format_version: '1.3'
+#       jupytext_version: 1.14.5
+#   kernelspec:
+#     display_name: Python (nectar-dev)
+#     language: python
+#     name: nectar-dev
+# ---
+
+# %% [markdown]
+# # How to access NectarCAM data from the EGI grid directly using the DIRAC API ?
+#
+# In this short notebook, we will see how to access data stored on the grid using the
+# DIRAC API, without the hassle of first manually downloading them locally.
+#
+# In order to achieve this, you will obviously need a `conda` environment in which all
+# relevant code is installed, such as `ctapipe`, `nectarchain`, but also `CTADIRAC`
+# itself. Please refer to the `nectarchain` installation procedure at
+# <https://github.com/cta-observatory/nectarchain> and follow the instructions to enable
+# DIRAC support.
+#
+# You will also need to have an active proxy for EGI, initialized e.g. with:
+#
+# ```
+# dirac-proxy-init -U -M -g cta_nectarcam
+# ```
+#
+# You can also check whether you currently have an active proxy with the command
+# `dirac-proxy-info`.
+
+# %%
+# %matplotlib inline
+import os
+from glob import glob
+
+from ctapipe.coordinates import EngineeringCameraFrame
+from ctapipe.instrument import CameraGeometry
+from ctapipe.io import EventSeeker, EventSource
+from ctapipe.visualization import CameraDisplay
+from DIRAC.Interfaces.API.Dirac import Dirac
+
+# %%
+dirac = Dirac()
+
+# %%
+# ?dirac.getFile
+
+# %%
+lfns = [
+    "/vo.cta.in2p3.fr/nectarcam/2022/20220411/NectarCAM.Run3169.0000.fits.fz",
+    "/vo.cta.in2p3.fr/nectarcam/2022/20220411/NectarCAM.Run3169.0001.fits.fz",
+]
+
+# %%
+tmpdir = f"/tmp/{os.environ['USER']}/scratch"
+if not os.path.isdir(tmpdir):
+    print(f"{tmpdir} does not exist yet, I will create it for you")
+    os.makedirs(tmpdir)
+
+# %%
+dirac.getFile(lfn=lfns, destDir=tmpdir, printOutput=True)
+
+# %% [markdown]
+# **You are now ready to work with `ctapipe` as usual!**
+
+# %%
+path = glob(f"{tmpdir}/NectarCAM.*.fits.fz")
+path.sort()
+reader = EventSource(input_url=path[0])
+seeker = EventSeeker(reader)
+
+# Get some event, and display camera charges for the high gain channel
+# (no time window optimization)
+evt = seeker.get_event_index(10)
+image = evt.r0.tel[0].waveform.sum(axis=2)
+camera = CameraGeometry.from_name("NectarCam-003").transform_to(
+    EngineeringCameraFrame()
+)
+disp = CameraDisplay(geometry=camera, image=image[0])
+
+# %%