Multi-package coverage report #8609

NorfairKing · 2022-11-19T14:36:42Z

I'd like to have a project with two packages, one of which has a test suite that tests the other package, run cabal test all --coverage and get a unified code coverage report.

The text was updated successfully, but these errors were encountered:

NorfairKing · 2022-11-27T17:32:39Z

Multi-package coverage reports will hopefully become easy with https://github.com/NorfairKing/dekking

This commits re-enables per-component builds when coverage checking is enabled. This restriction was previously added in haskell#5004 to fix haskell#4798. - haskell#4798 was subsequently fixed "again" with the fix for haskell#5213, in haskell#7493 by fixing the paths of the testsuite `.mix` files to the same location as that of the main library component. Therefore the restriction to treat testsuites per-package (legacy-fallback) is no longer needed. We went further and fixed coverage for internal sublibraries, packages with backpack (but without generating coverage information for indefinite and instantiated units -- it is not clear what it would mean for HPC to support this), and coverage for multi-package projects. 1. We allow hpc in per-component builds 2. To generate hpc files in the appropriate component directories in the distribution tree, we remove the hack from haskell#7493 and instead determine the `.mix` directories that are included in the call to `hpc markup` by passing the list of libraries in the project from the cabal-install invocation of test. We also drop an unnecessary directory in the hpc file hierarchy. 3. To account for internal (non-backpack) libraries, we include the mix dirs and modules of all (non-indefinite and non-instantiations) libraries in the project Indefinite libraries and instantiations are ignored as it is not obvious what it means for HPC to support backpack, e.g. covering a library function that two different instantiations 4. We now only reject coverage if there are no libraries at all in the project, rather than if there are no libraries in the package. This allows us to drop the coverage masking logic in cabal.project.coverage while still having coverage of cabal-install (i.e. cabal test --enable-coverage cabal-install now works without the workaround) Even though we allow multi-package project coverage, we still cover each package independently -- the tix files resulting from all packages are not combined for the time being. Includes tests for haskell#6440, haskell#6397, haskell#8609, and haskell#4798 (the test for haskell#5213 already exists) Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage) , haskell#8609 (multi-package coverage report) and fixes in a new way the previously fixed haskell#4798, haskell#5213.

This commits re-enables per-component builds when coverage checking is enabled. This restriction was previously added in haskell#5004 to fix haskell#4798. - haskell#4798 was subsequently fixed "again" with the fix for haskell#5213, in haskell#7493 by fixing the paths of the testsuite `.mix` files to the same location as that of the main library component. Therefore the restriction to treat testsuites per-package (legacy-fallback) is no longer needed. We went further and fixed coverage for internal sublibraries, packages with backpack (but without generating coverage information for indefinite and instantiated units -- it is not clear what it would mean for HPC to support this), and coverage for multi-package projects. 1. We allow hpc in per-component builds 2. To generate hpc files in the appropriate component directories in the distribution tree, we remove the hack from haskell#7493 and instead determine the `.mix` directories that are included in the call to `hpc markup` by passing the list of libraries in the project from the cabal-install invocation of test. We also drop an unnecessary directory in the hpc file hierarchy. 3. To account for internal (non-backpack) libraries, we include the mix dirs and modules of all (non-indefinite and non-instantiations) libraries in the project Indefinite libraries and instantiations are ignored as it is not obvious what it means for HPC to support backpack, e.g. covering a library function that two different instantiations 4. We now only reject coverage if there are no libraries at all in the project, rather than if there are no libraries in the package. This allows us to drop the coverage masking logic in cabal.project.coverage while still having coverage of cabal-install (i.e. cabal test --enable-coverage cabal-install now works without the workaround) Even though we allow multi-package project coverage, we still cover each package independently -- the tix files resulting from all packages are not combined for the time being. TODO: Multi-package project coverage is fixed in Cabal, however, the paths to the source files listed in the `.mix` files will be incorrect because package sources will no longer be in the root of the project tree, but rather under the subdir with the package... Includes tests for haskell#6440, haskell#6397, haskell#8609, and haskell#4798 (the test for haskell#5213 already exists) Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage) , haskell#8609 (multi-package coverage report) and fixes in a new way the previously fixed haskell#4798, haskell#5213.

This commits re-enables per-component builds when coverage checking is enabled. This restriction was previously added in haskell#5004 to fix haskell#4798. - haskell#4798 was subsequently fixed "again" with the fix for haskell#5213, in haskell#7493 by fixing the paths of the testsuite `.mix` files to the same location as that of the main library component. Therefore the restriction to treat testsuites per-package (legacy-fallback) is no longer needed. We went further and fixed coverage for internal sublibraries, packages with backpack (but without generating coverage information for indefinite and instantiated units -- it is not clear what it would mean for HPC to support this), and coverage for multi-package projects. 1. We allow hpc in per-component builds 2. To generate hpc files in the appropriate component directories in the distribution tree, we remove the hack from haskell#7493 and instead determine the `.mix` directories that are included in the call to `hpc markup` by passing the list of components in the project from the cabal-install invocation of test. We also drop an unnecessary directory in the hpc file hierarchy. 3. To account for internal (non-backpack) libraries, we include the mix dirs and modules of all (non-indefinite and non-instantiations) libraries in the project Indefinite libraries and instantiations are ignored as it is not obvious what it means for HPC to support backpack, e.g. covering a library function that two different instantiations 4. We now only reject coverage if there are no libraries at all in the project, rather than if there are no libraries in the package. This allows us to drop the coverage masking logic in cabal.project.coverage while still having coverage of cabal-install (i.e. cabal test --enable-coverage cabal-install now works without the workaround) Even though we allow multi-package project coverage, we still cover each package independently -- the tix files resulting from all packages are not combined for the time being. TODO: Multi-package project coverage is fixed in Cabal, however, the paths to the source files listed in the `.mix` files will be incorrect because package sources will no longer be in the root of the project tree, but rather under the subdir with the package... Includes tests for haskell#6440, haskell#6397, haskell#8609, and haskell#4798 (the test for haskell#5213 already exists) Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage) , haskell#8609 (multi-package coverage report) and fixes in a new way the previously fixed haskell#4798, haskell#5213.

This commits re-enables per-component builds when coverage checking is enabled. This restriction was previously added in haskell#5004 to fix haskell#4798. - haskell#4798 was subsequently fixed "again" with the fix for haskell#5213, in haskell#7493 by fixing the paths of the testsuite `.mix` files to the same location as that of the main library component. Therefore the restriction to treat testsuites per-package (legacy-fallback) is no longer needed. We went further and fixed coverage for internal sublibraries, packages with backpack (but without generating coverage information for indefinite and instantiated units -- it is not clear what it would mean for HPC to support this), and coverage for multi-package projects. 1. We allow hpc in per-component builds 2. To generate hpc files in the appropriate component directories in the distribution tree, we remove the hack from haskell#7493 and instead determine the `.mix` directories that are included in the call to `hpc markup` by passing the list of components in the project from the cabal-install invocation of test. We also drop an unnecessary directory in the hpc file hierarchy. 3. To account for internal (non-backpack) libraries, we include the mix dirs and modules of all (non-indefinite and non-instantiations) libraries in the project Indefinite libraries and instantiations are ignored as it is not obvious what it means for HPC to support backpack, e.g. covering a library function that two different instantiations 4. We now only reject coverage if there are no libraries at all in the project, rather than if there are no libraries in the package. This allows us to drop the coverage masking logic in cabal.project.coverage while still having coverage of cabal-install (i.e. cabal test --enable-coverage cabal-install now works without the workaround) Even though we allow multi-package project coverage, we still cover each package independently -- the tix files resulting from all packages are not combined for the time being. Multi-package project coverage is fixed in Cabal, however, the paths to the source files listed in the `.mix` files will be incorrect because package sources will no longer be in the root of the project tree, but rather under the subdir with the package. We add an error for multi-package projects when coverage is enabled, and track lifting this error in haskell#9493. Includes tests for haskell#6440, haskell#6397, haskell#8609, and haskell#4798 (the test for haskell#5213 already exists) Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage) , doesn't yet fix haskell#8609 (multi-package coverage report) and fixes in a new way the previously fixed haskell#4798, haskell#5213.

This commit re-designs the mechanism by which we make the .mix files of libraries available to produce the Haskell Program Coverage report after running testsuites. The idea, for the Cabal library, is: * Cabal builds libraries with -fhpc, and store the hpc artifacts in build </> `extraCompilationArtifacts` * At Cabal install time, `extraCompilationArtifacts` is copied into the package database * At Cabal configure time, we both - receive as --coverage-for flags unit-ids of library components from the same package (ultimately, when haskell#9493 is resolved, we will receive unit ids of libraries in other packages in the same project too), - and, when configuring a whole package instead of just a testsuite component, we determine the unit-ids of libraries in the package these unit-ids are written into `configCoverageFor` in `ConfigFlags` * At Cabal test time, for each library to cover (stored in `configCoverageFor`), we look in the package database for the hpc dirs, which we eventually pass along to the `hpc markup` call as `--hpcdir` flags As for cabal-install: * After a plan has been elaborated, we select the packages which can be covered and pass them to Cabal's ./Setup configure as --coverage-for=<unit-id> flags. - Notably, valid libraries are non-indefinite and non-instantiations, since HPC does not support backpack. - Furthermore, we only include libraries in the same package as the component being configured, despite possibly there being more library components in other packages of the same project. When haskell#9493 is resolved, we could lift this restriction and pass all libraries local to the package as --coverage-for. See `determineCoverageFor` and `shouldCoverPkg` in Distribution.Client.ProjectPlanning. Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage), doesn't yet fix haskell#8609 (multi-package coverage report) which is tracked in haskell#9493, and fixes in a new way the previously fixed haskell#4798, haskell#5213.

This commit re-designs the mechanism by which we make the .mix files of libraries available to produce the Haskell Program Coverage report after running testsuites. The idea, for the Cabal library, is: * Cabal builds libraries with -fhpc, and store the hpc artifacts in build </> `extraCompilationArtifacts` * At Cabal install time, `extraCompilationArtifacts` is copied into the package database * At Cabal configure time, we both - receive as --coverage-for flags unit-ids of library components from the same package (ultimately, when haskell#9493 is resolved, we will receive unit ids of libraries in other packages in the same project too), - and, when configuring a whole package instead of just a testsuite component, we determine the unit-ids of libraries in the package these unit-ids are written into `configCoverageFor` in `ConfigFlags` * At Cabal test time, for each library to cover (stored in `configCoverageFor`), we look in the package database for the hpc dirs, which we eventually pass along to the `hpc markup` call as `--hpcdir` flags As for cabal-install: * After a plan has been elaborated, we select the packages which can be covered and pass them to Cabal's ./Setup configure as --coverage-for=<unit-id> flags. - Notably, valid libraries are non-indefinite and non-instantiations, since HPC does not support backpack. - Furthermore, we only include libraries in the same package as the component being configured, despite possibly there being more library components in other packages of the same project. When haskell#9493 is resolved, we could lift this restriction and pass all libraries local to the package as --coverage-for. See `determineCoverageFor` and `shouldCoverPkg` in Distribution.Client.ProjectPlanning. Detail: We no longer pass the path to the testsuite's mix dirs to `hpc markup` because we only ever include modules in libraries, which means they were previously unused. Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage), doesn't yet fix haskell#8609 (multi-package coverage report) which is tracked in haskell#9493, and fixes in a new way the previously fixed haskell#4798, haskell#5213.

This commits re-enables per-component builds when coverage checking is enabled. This restriction was previously added in haskell#5004 to fix haskell#4798. - haskell#4798 was subsequently fixed "again" with the fix for haskell#5213, in haskell#7493 by fixing the paths of the testsuite `.mix` files to the same location as that of the main library component. Therefore the restriction to treat testsuites per-package (legacy-fallback) is no longer needed. We went further and fixed coverage for internal sublibraries, packages with backpack (but without generating coverage information for indefinite and instantiated units -- it is not clear what it would mean for HPC to support this), and coverage for multi-package projects. 1. We allow hpc in per-component builds 2. To generate hpc files in the appropriate component directories in the distribution tree, we remove the hack from haskell#7493 and instead determine the `.mix` directories that are included in the call to `hpc markup` by passing the list of components in the project from the cabal-install invocation of test. We also drop an unnecessary directory in the hpc file hierarchy. 3. To account for internal (non-backpack) libraries, we include the mix dirs and modules of all (non-indefinite and non-instantiations) libraries in the project Indefinite libraries and instantiations are ignored as it is not obvious what it means for HPC to support backpack, e.g. covering a library function that two different instantiations 4. We now only reject coverage if there are no libraries at all in the project, rather than if there are no libraries in the package. This allows us to drop the coverage masking logic in cabal.project.coverage while still having coverage of cabal-install (i.e. cabal test --enable-coverage cabal-install now works without the workaround) Even though we allow multi-package project coverage, we still cover each package independently -- the tix files resulting from all packages are not combined for the time being. Multi-package project coverage is fixed in Cabal, however, the paths to the source files listed in the `.mix` files will be incorrect because package sources will no longer be in the root of the project tree, but rather under the subdir with the package. We add an error for multi-package projects when coverage is enabled, and track lifting this error in haskell#9493. Includes tests for haskell#6440, haskell#6397, haskell#8609, and haskell#4798 (the test for haskell#5213 already exists) Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage) , doesn't yet fix haskell#8609 (multi-package coverage report) and fixes in a new way the previously fixed haskell#4798, haskell#5213.

This commit re-designs the mechanism by which we make the .mix files of libraries available to produce the Haskell Program Coverage report after running testsuites. The idea, for the Cabal library, is: * Cabal builds libraries with -fhpc, and store the hpc artifacts in build </> `extraCompilationArtifacts` * At Cabal install time, `extraCompilationArtifacts` is copied into the package database * At Cabal configure time, we both - receive as --coverage-for flags unit-ids of library components from the same package (ultimately, when haskell#9493 is resolved, we will receive unit ids of libraries in other packages in the same project too), - and, when configuring a whole package instead of just a testsuite component, we determine the unit-ids of libraries in the package these unit-ids are written into `configCoverageFor` in `ConfigFlags` * At Cabal test time, for each library to cover (stored in `configCoverageFor`), we look in the package database for the hpc dirs, which we eventually pass along to the `hpc markup` call as `--hpcdir` flags As for cabal-install: * After a plan has been elaborated, we select the packages which can be covered and pass them to Cabal's ./Setup configure as --coverage-for=<unit-id> flags. - Notably, valid libraries are non-indefinite and non-instantiations, since HPC does not support backpack. - Furthermore, we only include libraries in the same package as the component being configured, despite possibly there being more library components in other packages of the same project. When haskell#9493 is resolved, we could lift this restriction and pass all libraries local to the package as --coverage-for. See `determineCoverageFor` and `shouldCoverPkg` in Distribution.Client.ProjectPlanning. Detail: We no longer pass the path to the testsuite's mix dirs to `hpc markup` because we only ever include modules in libraries, which means they were previously unused. Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage), doesn't yet fix haskell#8609 (multi-package coverage report) which is tracked in haskell#9493, and fixes in a new way the previously fixed haskell#4798, haskell#5213.

This commits re-enables per-component builds when coverage checking is enabled. This restriction was previously added in haskell#5004 to fix haskell#4798. - haskell#4798 was subsequently fixed "again" with the fix for haskell#5213, in haskell#7493 by fixing the paths of the testsuite `.mix` files to the same location as that of the main library component. Therefore the restriction to treat testsuites per-package (legacy-fallback) is no longer needed. We went further and fixed coverage for internal sublibraries, packages with backpack (but without generating coverage information for indefinite and instantiated units -- it is not clear what it would mean for HPC to support this), and coverage for multi-package projects. 1. We allow hpc in per-component builds 2. To generate hpc files in the appropriate component directories in the distribution tree, we remove the hack from haskell#7493 and instead determine the `.mix` directories that are included in the call to `hpc markup` by passing the list of components in the project from the cabal-install invocation of test. We also drop an unnecessary directory in the hpc file hierarchy. 3. To account for internal (non-backpack) libraries, we include the mix dirs and modules of all (non-indefinite and non-instantiations) libraries in the project Indefinite libraries and instantiations are ignored as it is not obvious what it means for HPC to support backpack, e.g. covering a library function that two different instantiations 4. We now only reject coverage if there are no libraries at all in the project, rather than if there are no libraries in the package. This allows us to drop the coverage masking logic in cabal.project.coverage while still having coverage of cabal-install (i.e. cabal test --enable-coverage cabal-install now works without the workaround) Even though we allow multi-package project coverage, we still cover each package independently -- the tix files resulting from all packages are not combined for the time being. Multi-package project coverage is fixed in Cabal, however, the paths to the source files listed in the `.mix` files will be incorrect because package sources will no longer be in the root of the project tree, but rather under the subdir with the package. We add an error for multi-package projects when coverage is enabled, and track lifting this error in haskell#9493. Includes tests for haskell#6440, haskell#6397, haskell#8609, and haskell#4798 (the test for haskell#5213 already exists) Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage) , doesn't yet fix haskell#8609 (multi-package coverage report) and fixes in a new way the previously fixed haskell#4798, haskell#5213.

This commit re-designs the mechanism by which we make the .mix files of libraries available to produce the Haskell Program Coverage report after running testsuites. The idea, for the Cabal library, is: * Cabal builds libraries with -fhpc, and store the hpc artifacts in build </> `extraCompilationArtifacts` * At Cabal install time, `extraCompilationArtifacts` is copied into the package database * At Cabal configure time, we both - receive as --coverage-for flags unit-ids of library components from the same package (ultimately, when haskell#9493 is resolved, we will receive unit ids of libraries in other packages in the same project too), - and, when configuring a whole package instead of just a testsuite component, we determine the unit-ids of libraries in the package these unit-ids are written into `configCoverageFor` in `ConfigFlags` * At Cabal test time, for each library to cover (stored in `configCoverageFor`), we look in the package database for the hpc dirs, which we eventually pass along to the `hpc markup` call as `--hpcdir` flags As for cabal-install: * After a plan has been elaborated, we select the packages which can be covered and pass them to Cabal's ./Setup configure as --coverage-for=<unit-id> flags. - Notably, valid libraries are non-indefinite and non-instantiations, since HPC does not support backpack. - Furthermore, we only include libraries in the same package as the component being configured, despite possibly there being more library components in other packages of the same project. When haskell#9493 is resolved, we could lift this restriction and pass all libraries local to the package as --coverage-for. See `determineCoverageFor` and `shouldCoverPkg` in Distribution.Client.ProjectPlanning. Detail: We no longer pass the path to the testsuite's mix dirs to `hpc markup` because we only ever include modules in libraries, which means they were previously unused. Fixes haskell#6440 (internal libs coverage), haskell#6397 (backpack breaks coverage), doesn't yet fix haskell#8609 (multi-package coverage report) which is tracked in haskell#9493, and fixes in a new way the previously fixed haskell#4798, haskell#5213.

Mikolaj added type: enhancement Cabal: code-coverage labels Dec 29, 2022

alt-romes mentioned this issue Nov 30, 2023

Allow per-component builds with coverage enabled #9464

Merged

5 tasks

mergify bot closed this as completed in 073ccc8 Dec 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-package coverage report #8609

Multi-package coverage report #8609

NorfairKing commented Nov 19, 2022

NorfairKing commented Nov 27, 2022

Multi-package coverage report #8609

Multi-package coverage report #8609

Comments

NorfairKing commented Nov 19, 2022

NorfairKing commented Nov 27, 2022