Drop the use of .coveragerc

[frenck]

A proposal to change how we monitor and report the test coverage of our codebase.

Current State

Currently, our project's code test coverage is reported to be around 97%. However, this high percentage is artificially maintained by excluding files that do not meet our coverage threshold using the .coveragerc file.

Objective

The primary goal of this proposal is to obtain a more accurate insight into our actual test coverage and identify areas for improvement.

Proposal

I propose to stop excluding files from test coverage reports for all integrations. By doing so, coverage tools like CodeCov will provide a more accurate representation of our coverage gaps. This change will:

• Enhance the value of coverage tools by providing better visibility into coverage gaps.
• Eliminate the need to maintain coverage exclusions.
• Simplify our process by removing the need to explain the reasons behind these exclusions constantly.

Impact

Accepting this proposal will result in a significant drop in reported test coverage. However, CodeCov will continue to ensure 100% coverage for essential files (e.g., configuration flows), as it currently does.

For reference the current configuration at the time of writing:

coverage:
  status:
    project:
      default:
        target: 90
        threshold: 0.09
      required:
        target: auto
        threshold: 1
        paths:
          - homeassistant/components/*/config_flow.py
          - homeassistant/components/*/device_action.py
          - homeassistant/components/*/device_condition.py
          - homeassistant/components/*/device_trigger.py
          - homeassistant/components/*/diagnostics.py
          - homeassistant/components/*/group.py
          - homeassistant/components/*/intent.py
          - homeassistant/components/*/logbook.py
          - homeassistant/components/*/media_source.py
          - homeassistant/components/*/recorder.py
          - homeassistant/components/*/scene.py

Implementation plan

Implementing the suggested changes requires two changes:

1. Remove .coveragerc file (get rid of all exclusions)
2. Move coverage "general" settings into pyproject.toml
3. Adjust the codecov.yml file, removing the default project target (and letting it automatically handle it based on a threshold) and adding a target for the required threshold.
4. Remove coveragerc references from the developer documentation.

The Codecov default project threshold might need to be re-evaluated afterward.

Optional future enhancements

• Instate the python-coverage-comment-action in our CI pipelines.
  https://github.com/py-cov-action/python-coverage-comment-action
  Example: [Internal PR] Improve code py-cov-action/python-coverage-comment-action-v3-example#1 (comment)
• "Coverage goals"
  https://nedbatchelder.com/blog/202111/coverage_goals.html

Changelog

4 July, 2024, 15:24 CEST: Adjusted the proposal to remove all coverage exclusions (instead of only integrations/components).

[MartinHjelmare]

How can we avoid the review annotations from codecov on non covered lines for integrations that don't aim to add tests? Those are very annoying during review when a module is completely uncovered.

[frenck]

By default, CodeCov will only annotate the changes in a PR (patch coverage). However, I agree that it might be annoying in the situation you are pointing out.

I see a few options for solving this.

1. Learn to ignore them (you can toggle them in the diff view on GitHub). Example:
   
2. Disable annotations for CodeCov completely (which will still enforce).

The second one, could be combined with enabling commenting again (so CodeCov will post a summery report in the Pull Request).

[joostlek]

Tested it out, only a 19% decrease is quite reasonable (I expected way more)

[Jc2k]

That's pretty good! Was expecting it to drop muuuuch more too.

[timmo001]

Seems reasonable, a lot of tests are missed when they are added to .coveragerc and are generally forgotten over time.

---

This may be for a separate ADR, but there is the potential of splitting components from the rest of the codebase:

• https://about.codecov.io/blog/codecov-components-breaking-down-coverage-by-filters/
• https://docs.codecov.com/docs/components
• https://docs.codecov.com/docs/flags

This would allow target(s) could be set higher for internal / core code and lower for the community for example. Moving components/integrations to a different file structure could potentially allow a more fine tuned coverage report, such as moving core components under their own directory etc.

• https://about.codecov.io/blog/the-case-against-100-code-coverage/
• https://testing.googleblog.com/2020/08/code-coverage-best-practices.html

[frenck]

Yeah that might fit, e.g., a general categorization of framework vs built-in components vs platinum integrations vs gold integrations vs others...

I think that might be a good next step.

I've looked at doing it per integration, but that will quickly become a hot mess.

[sdb9696]

This seems like a v.good idea to me, I’ve never really understood why integrations could be excluded from coverage and I think it drives a misunderstanding of whether the tests should be exercising the HA code or the code of the dependency library. The ring camera entity is a good example of something that seems to be excluded for no good reason afaict (and I plan to fix). Better to report coverage as coverage.

[frenck]

PR: home-assistant/core#121286