Disable collecting test coverage by default

[bouweandela]

Description

Collecting code coverage is time consuming. Disable it by default and enable it on request by running pytest --cov. This will save some time when developing locally and also reduces the runtime of the GitHub Actions tests by about 1 (Linux) to 2 (OSX) minutes.

Link to documentation: https://esmvaltool--2456.org.readthedocs.build/projects/ESMValCore/en/2456/contributing.html#test-coverage

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Changes are backward compatible
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.83%. Comparing base (4b0dd41) to head (9fe2d82).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2456   +/-   ##
=======================================
  Coverage   94.83%   94.83%           
=======================================
  Files         251      251           
  Lines       14191    14191           
=======================================
  Hits        13458    13458           
  Misses        733      733           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[valeriupredoi]

OK, I like the idea of buying us time via not cov-outputting at CI level, but how much time is it saved at user-level? I find it very useful to have the coverage report immediately available after all tests have run, and I bet you my boots if one is not a serious developer, always interested in the coverage report, then they don't even bother running the tests. So I'd say remove it from the CI via --no-cov flag, but leave it for the user - what say you, bud?

[bouweandela]

I usually only look at the coverage report provided by Codecov because it shows me the relevant coverage changes as opposed to everything. I also run the tests way more often to check if I did not mess things up, e.g. before committing, then to check if I covered every changed line of code with tests. But I admit this is just a personal preference.

Let's ask the others: @ESMValGroup/technical-lead-development-team What is your experience with code coverage? Would you prefer to have it on or off on the command line by default? i.e. should the default be pytest --cov or pytest --no-cov? Running the tests with coverage takes 38 seconds on my laptop, while running them without coverage takes 24 seconds.

[valeriupredoi]

we may want to use --testmon and have both jam and peanut butter, it looks like the previous issues with --cov and --testmon have now been solved tarpas/pytest-testmon#86 NOTE: testmon allows coverage production/updating only for the bits of code that changed

[ehogan]

Running the tests with coverage takes 38 seconds on my laptop, while running them without coverage takes 24 seconds.

Wow, it's more like ~3 minutes and ~2 minutes on my machine! 😮

My preference would be to have coverage run via a GitHub workflow, but not locally (since I like to test locally frequently, and not for coverage reasons, but I would like to make sure my coverage is good when creating a PR) 👍

[bouweandela]

@ehogan Are you running the tests in parallel? Of your machine has multiple cpu cores you might be able to run the tests faster withpytest -n auto

[ehogan]

Thanks @bouweandela, pytest -n auto used 4 workers and reduced the timings to ~2 minutes and ~1 minute 👍 (I'm guessing my Linux machine isn't as good as yours!) 😝

[valeriupredoi]

nobody make comments about Linux boxes, my Ubuntu 18.04 is flying through those tests 🤣