Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclude non-executable lines #892

Merged
merged 10 commits into from
Feb 16, 2022
15 changes: 14 additions & 1 deletion src/CodeCoverage.php
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,8 @@ public function append(RawCodeCoverageData $rawData, $id = null, bool $append =

$this->applyFilter($rawData);

$this->applyExecutableLinesFilter($rawData);

if ($this->useAnnotationsForIgnoringCode) {
$this->applyIgnoredLinesFilter($rawData);
}
Expand Down Expand Up @@ -466,7 +468,8 @@ private function applyCoversAnnotationFilter(RawCodeCoverageData $rawData, $line

if (is_array($linesToBeCovered)) {
foreach ($linesToBeCovered as $fileToBeCovered => $includedLines) {
$rawData->keepCoverageDataOnlyForLines($fileToBeCovered, $includedLines);
$rawData->keepLineCoverageDataOnlyForLines($fileToBeCovered, $includedLines);
$rawData->keepFunctionCoverageDataOnlyForLines($fileToBeCovered, $includedLines);
}
}
}
Expand All @@ -484,6 +487,16 @@ private function applyFilter(RawCodeCoverageData $data): void
}
}

private function applyExecutableLinesFilter(RawCodeCoverageData $data): void
{
foreach (array_keys($data->lineCoverage()) as $filename) {
$data->keepLineCoverageDataOnlyForLines(
$filename,
$this->uncoveredFileAnalyser()->executableLinesIn($filename)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be moved out of uncoveredFileAnalyser if it's going to be called for covered files as well

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there will also a performance impact from calling executableLinesIn so much that it might be a good idea to try and mitigate - if someone has caching enabled, then that will read from file each time but it might be a good idea to add a memory cache somewhere? Reads on Windows particularly are notoriously slow, and I think this would do an additional read per source file per test if my code-skim is correct?

Copy link
Contributor

@dvdoug dvdoug Feb 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For non-cached version:

Current

    public function enterNode(Node $node): void
    {
        if (!$this->isExecutable($node)) {
            return;
        }

        $this->executableLines[] = $node->getStartLine();
    }

    /**
     * @psalm-return list<int>
     */
    public function executableLines(): array
    {
        $executableLines = array_unique($this->executableLines);

        sort($executableLines);

        return $executableLines;
    }

Faster?



    public function enterNode(Node $node): void
    {
        if (!$this->isExecutable($node)) {
            return;
        }

        $this->executableLines[$node->getStartLine()] = $node->getStartLine();
    }

    /**
     * @psalm-return list<int>
     */
    public function executableLines(): array
    {
        return $executableLines;
    }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be moved out of uncoveredFileAnalyser if it's going to be called for covered files as well

Sure, I'll leave refactoring after the main solution has landed with all tests passing

I think there will also a performance impact from calling executableLinesIn so much

I have no experience on Windows, but on Linux real cases give me negligible impact:

Files: 906, Methods: 4402, Lines: 33708
Tests: 2989, Assertions: 19314, Skipped: 22

3 runs on Paratest, Standard SATA disk for cache

Before

Time: 02:02.831, Memory: 200.54 MB
Generating code coverage report ... done [00:06.284]

Time: 02:03.884, Memory: 198.84 MB
Generating code coverage report ... done [00:05.837]

Time: 02:02.206, Memory: 200.58 MB
Generating code coverage report ... done [00:05.936]

After

Time: 02:04.346, Memory: 176.52 MB
Generating code coverage report ... done [00:05.902]

Time: 02:06.492, Memory: 176.50 MB
Generating code coverage report ... done [00:05.635]

Time: 02:05.663, Memory: 176.52 MB
Generating code coverage report ... done [00:05.731]

$this->executableLines[$node->getStartLine()] = $node->getStartLine();

I thought this too: be aware that we need to version the cache, otherwise new release will break user reports if run onto old caches

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an interesting post by an MS engineer at microsoft/WSL#873 (comment) in the context of WSL that compares/contrasts performance of the filesystem in Linux vs Windows. Suffice to say it's real

Windows 11, PHP 8.1.2, XDebug 3.1.3, running tests for this repo

9.2 branch

Time: 00:19.726, Memory: 360.00 MB

Time: 00:21.713, Memory: 360.00 MB

Time: 00:23.309, Memory: 360.00 MB

This PR

Time: 00:53.541, Memory: 374.00 MB

Time: 00:44.610, Memory: 374.00 MB

Time: 00:48.638, Memory: 374.00 MB

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the pointer, I'll have a look (hopefully tomorrow morning, otherwise over the weekend).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

I think the uncovered file analyser not having a memory cache is OK because it's normally only called once at the end of the entire test suite run just before generating the report, unlike the analyser for covered files that's called after every test. It's just that here, the relevant function is now being called from a very different context...

Copy link
Owner

@sebastianbergmann sebastianbergmann Feb 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed 3a3285f just now. It should not cause any harm, apart from producing a merge conflict for this PR.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought about this some more: the idea to separate CoveredFileAnalyzer and UncoveredFileAnalyzer from each other was because the former was required after each test and the latter only once at the end. Now we also need the data from UncoveredFileAnalyzer after each test (right?). If that is really the case, then we should merge the two into FileAnalyzer.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reverted 3a3285f (so that this PR no longer has conflicts). I think we should separate caching refactorings/improvements from the topic of this PR.

);
}
}

private function applyIgnoredLinesFilter(RawCodeCoverageData $data): void
{
foreach (array_keys($data->lineCoverage()) as $filename) {
Expand Down
28 changes: 18 additions & 10 deletions src/RawCodeCoverageData.php
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ public function removeCoverageDataForFile(string $filename): void
/**
* @param int[] $lines
*/
public function keepCoverageDataOnlyForLines(string $filename, array $lines): void
public function keepLineCoverageDataOnlyForLines(string $filename, array $lines): void
{
if (!isset($this->lineCoverage[$filename])) {
return;
Expand All @@ -136,17 +136,25 @@ public function keepCoverageDataOnlyForLines(string $filename, array $lines): vo
$this->lineCoverage[$filename],
array_flip($lines)
);
}

if (isset($this->functionCoverage[$filename])) {
foreach ($this->functionCoverage[$filename] as $functionName => $functionData) {
foreach ($functionData['branches'] as $branchId => $branch) {
if (count(array_diff(range($branch['line_start'], $branch['line_end']), $lines)) > 0) {
unset($this->functionCoverage[$filename][$functionName]['branches'][$branchId]);
/**
* @param int[] $lines
*/
public function keepFunctionCoverageDataOnlyForLines(string $filename, array $lines): void
{
if (!isset($this->functionCoverage[$filename])) {
return;
}

foreach ($functionData['paths'] as $pathId => $path) {
if (in_array($branchId, $path['path'], true)) {
unset($this->functionCoverage[$filename][$functionName]['paths'][$pathId]);
}
foreach ($this->functionCoverage[$filename] as $functionName => $functionData) {
foreach ($functionData['branches'] as $branchId => $branch) {
if (count(array_diff(range($branch['line_start'], $branch['line_end']), $lines)) > 0) {
unset($this->functionCoverage[$filename][$functionName]['branches'][$branchId]);

foreach ($functionData['paths'] as $pathId => $path) {
if (in_array($branchId, $path['path'], true)) {
unset($this->functionCoverage[$filename][$functionName]['paths'][$pathId]);
}
}
}
Expand Down
8 changes: 5 additions & 3 deletions src/StaticAnalysis/CachingUncoveredFileAnalyser.php
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,15 @@ public function __construct(string $directory, UncoveredFileAnalyser $uncoveredF

public function executableLinesIn(string $filename): array
{
if ($this->has($filename, __METHOD__)) {
return $this->read($filename, __METHOD__);
$cacheKey = 'v2' . __METHOD__;

if ($this->has($filename, $cacheKey)) {
return $this->read($filename, $cacheKey);
}

$data = $this->uncoveredFileAnalyser->executableLinesIn($filename);

$this->write($filename, __METHOD__, $data);
$this->write($filename, $cacheKey, $data);

return $data;
}
Expand Down
60 changes: 51 additions & 9 deletions src/StaticAnalysis/ExecutableLinesFindingVisitor.php
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@
*/
namespace SebastianBergmann\CodeCoverage\StaticAnalysis;

use function array_unique;
use function sort;
use PhpParser\Node;
use PhpParser\Node\Expr\BinaryOp;
use PhpParser\Node\Expr\CallLike;
use PhpParser\Node\Scalar;
use PhpParser\Node\Stmt\Break_;
use PhpParser\Node\Stmt\Case_;
use PhpParser\Node\Stmt\Catch_;
Expand All @@ -26,6 +27,7 @@
use PhpParser\Node\Stmt\Foreach_;
use PhpParser\Node\Stmt\Goto_;
use PhpParser\Node\Stmt\If_;
use PhpParser\Node\Stmt\Property;
use PhpParser\Node\Stmt\Return_;
use PhpParser\Node\Stmt\Switch_;
use PhpParser\Node\Stmt\Throw_;
Expand All @@ -40,34 +42,69 @@
final class ExecutableLinesFindingVisitor extends NodeVisitorAbstract
{
/**
* @psalm-var list<int>
* @psalm-var array<int, int>
*/
private $executableLines = [];

/**
* @psalm-var array<int, int>
*/
private $propertyLines = [];

public function enterNode(Node $node): void
{
$this->savePropertyLines($node);

if (!$this->isExecutable($node)) {
return;
}

$this->executableLines[] = $node->getStartLine();
$line = $this->getLine($node);

if (isset($this->propertyLines[$line])) {
return;
}

$this->executableLines[$line] = $line;
}

/**
* @psalm-return list<int>
* @psalm-return array<int, int>
*/
public function executableLines(): array
{
$executableLines = array_unique($this->executableLines);
return $this->executableLines;
}

private function savePropertyLines(Node $node): void
{
if (!$node instanceof Property && !$node instanceof Node\Stmt\ClassConst) {
return;
}

sort($executableLines);
foreach (range($node->getStartLine(), $node->getEndLine()) as $index) {
$this->propertyLines[$index] = $index;
}
}

private function getLine(Node $node): int
{
if (
$node instanceof Node\Expr\PropertyFetch ||
$node instanceof Node\Expr\NullsafePropertyFetch ||
$node instanceof Node\Expr\StaticPropertyFetch
) {
return $node->getEndLine();
}

return $executableLines;
return $node->getStartLine();
}

private function isExecutable(Node $node): bool
{
return $node instanceof Break_ ||
return $node instanceof BinaryOp ||
$node instanceof Break_ ||
$node instanceof CallLike ||
$node instanceof Case_ ||
$node instanceof Catch_ ||
$node instanceof Continue_ ||
Expand All @@ -82,10 +119,15 @@ private function isExecutable(Node $node): bool
$node instanceof Goto_ ||
$node instanceof If_ ||
$node instanceof Return_ ||
$node instanceof Scalar ||
$node instanceof Switch_ ||
$node instanceof Throw_ ||
$node instanceof TryCatch ||
$node instanceof Unset_ ||
$node instanceof Node\Expr\Assign ||
$node instanceof Node\Expr\PropertyFetch ||
$node instanceof Node\Expr\NullsafePropertyFetch ||
$node instanceof Node\Expr\StaticPropertyFetch ||
$node instanceof While_;
}
}
10 changes: 4 additions & 6 deletions tests/TestCase.php
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ protected function getLineCoverageXdebugDataForBankAccount()
29 => -1,
31 => -1,
32 => -2,
33 => -2,
35 => 1,
],
]),
RawCodeCoverageData::fromXdebugWithoutPathCoverage([
Expand All @@ -59,6 +61,7 @@ protected function getLineCoverageXdebugDataForBankAccount()
29 => 1,
31 => -1,
32 => -2,
33 => -2,
],
]),
RawCodeCoverageData::fromXdebugWithoutPathCoverage([
Expand Down Expand Up @@ -90,6 +93,7 @@ protected function getLineCoverageXdebugDataForBankAccount()
29 => 1,
31 => 1,
32 => -2,
33 => -2,
],
]),
];
Expand Down Expand Up @@ -1346,20 +1350,17 @@ protected function getExpectedLineCoverageDataArrayForBankAccount(): array
0 => 'BankAccountTest::testBalanceIsInitiallyZero',
1 => 'BankAccountTest::testDepositWithdrawMoney',
],
9 => null,
13 => [],
14 => [],
15 => [],
16 => [],
18 => [],
22 => [
0 => 'BankAccountTest::testBalanceCannotBecomeNegative2',
1 => 'BankAccountTest::testDepositWithdrawMoney',
],
24 => [
0 => 'BankAccountTest::testDepositWithdrawMoney',
],
25 => null,
29 => [
0 => 'BankAccountTest::testBalanceCannotBecomeNegative',
1 => 'BankAccountTest::testDepositWithdrawMoney',
Expand All @@ -1380,20 +1381,17 @@ protected function getExpectedLineCoverageDataArrayForBankAccountInReverseOrder(
0 => 'BankAccountTest::testDepositWithdrawMoney',
1 => 'BankAccountTest::testBalanceIsInitiallyZero',
],
9 => null,
13 => [],
14 => [],
15 => [],
16 => [],
18 => [],
22 => [
0 => 'BankAccountTest::testBalanceCannotBecomeNegative2',
1 => 'BankAccountTest::testDepositWithdrawMoney',
],
24 => [
0 => 'BankAccountTest::testDepositWithdrawMoney',
],
25 => null,
29 => [
0 => 'BankAccountTest::testDepositWithdrawMoney',
1 => 'BankAccountTest::testBalanceCannotBecomeNegative',
Expand Down
7 changes: 3 additions & 4 deletions tests/_files/BankAccount-clover-line.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<project timestamp="%i" name="BankAccount">
<file name="%s%eBankAccount.php">
<class name="BankAccount" namespace="global">
<metrics complexity="5" methods="4" coveredmethods="3" conditionals="0" coveredconditionals="0" statements="10" coveredstatements="5" elements="14" coveredelements="8"/>
<metrics complexity="5" methods="4" coveredmethods="3" conditionals="0" coveredconditionals="0" statements="9" coveredstatements="5" elements="13" coveredelements="8"/>
</class>
<line num="6" type="method" name="getBalance" visibility="public" complexity="1" crap="1" count="2"/>
<line num="8" type="stmt" count="2"/>
Expand All @@ -12,15 +12,14 @@
<line num="14" type="stmt" count="0"/>
<line num="15" type="stmt" count="0"/>
<line num="16" type="stmt" count="0"/>
<line num="18" type="stmt" count="0"/>
<line num="20" type="method" name="depositMoney" visibility="public" complexity="1" crap="1" count="2"/>
<line num="22" type="stmt" count="2"/>
<line num="24" type="stmt" count="1"/>
<line num="27" type="method" name="withdrawMoney" visibility="public" complexity="1" crap="1" count="2"/>
<line num="29" type="stmt" count="2"/>
<line num="31" type="stmt" count="1"/>
<metrics loc="33" ncloc="33" classes="1" methods="4" coveredmethods="3" conditionals="0" coveredconditionals="0" statements="10" coveredstatements="5" elements="14" coveredelements="8"/>
<metrics loc="34" ncloc="34" classes="1" methods="4" coveredmethods="3" conditionals="0" coveredconditionals="0" statements="9" coveredstatements="5" elements="13" coveredelements="8"/>
</file>
<metrics files="1" loc="33" ncloc="33" classes="1" methods="4" coveredmethods="3" conditionals="0" coveredconditionals="0" statements="10" coveredstatements="5" elements="14" coveredelements="8"/>
<metrics files="1" loc="34" ncloc="34" classes="1" methods="4" coveredmethods="3" conditionals="0" coveredconditionals="0" statements="9" coveredstatements="5" elements="13" coveredelements="8"/>
</project>
</coverage>
7 changes: 3 additions & 4 deletions tests/_files/BankAccount-clover-path.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<project timestamp="%i" name="BankAccount">
<file name="%s%eBankAccount.php">
<class name="BankAccount" namespace="global">
<metrics complexity="5" methods="4" coveredmethods="3" conditionals="7" coveredconditionals="3" statements="10" coveredstatements="5" elements="21" coveredelements="11"/>
<metrics complexity="5" methods="4" coveredmethods="3" conditionals="7" coveredconditionals="3" statements="9" coveredstatements="5" elements="20" coveredelements="11"/>
</class>
<line num="6" type="method" name="getBalance" visibility="public" complexity="1" crap="1" count="2"/>
<line num="8" type="stmt" count="2"/>
Expand All @@ -12,15 +12,14 @@
<line num="14" type="stmt" count="0"/>
<line num="15" type="stmt" count="0"/>
<line num="16" type="stmt" count="0"/>
<line num="18" type="stmt" count="0"/>
<line num="20" type="method" name="depositMoney" visibility="public" complexity="1" crap="1" count="2"/>
<line num="22" type="stmt" count="2"/>
<line num="24" type="stmt" count="1"/>
<line num="27" type="method" name="withdrawMoney" visibility="public" complexity="1" crap="1" count="2"/>
<line num="29" type="stmt" count="2"/>
<line num="31" type="stmt" count="1"/>
<metrics loc="33" ncloc="33" classes="1" methods="4" coveredmethods="3" conditionals="7" coveredconditionals="3" statements="10" coveredstatements="5" elements="21" coveredelements="11"/>
<metrics loc="34" ncloc="34" classes="1" methods="4" coveredmethods="3" conditionals="7" coveredconditionals="3" statements="9" coveredstatements="5" elements="20" coveredelements="11"/>
</file>
<metrics files="1" loc="33" ncloc="33" classes="1" methods="4" coveredmethods="3" conditionals="7" coveredconditionals="3" statements="10" coveredstatements="5" elements="21" coveredelements="11"/>
<metrics files="1" loc="34" ncloc="34" classes="1" methods="4" coveredmethods="3" conditionals="7" coveredconditionals="3" statements="9" coveredstatements="5" elements="20" coveredelements="11"/>
</project>
</coverage>
8 changes: 3 additions & 5 deletions tests/_files/BankAccount-cobertura-line.xml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage line-rate="0.5" branch-rate="0" lines-covered="5" lines-valid="10" branches-covered="0" branches-valid="0" complexity="5" version="0.4" timestamp="%i">
<coverage line-rate="0.55555555555556" branch-rate="0" lines-covered="5" lines-valid="9" branches-covered="0" branches-valid="0" complexity="5" version="0.4" timestamp="%i">
<sources>
<source>%s</source>
</sources>
<packages>
<package name="BankAccount" line-rate="0.5" branch-rate="0" complexity="5">
<package name="BankAccount" line-rate="0.55555555555556" branch-rate="0" complexity="5">
<classes>
<class name="BankAccount" filename="BankAccount.php" line-rate="0.5" branch-rate="0" complexity="5">
<class name="BankAccount" filename="BankAccount.php" line-rate="0.55555555555556" branch-rate="0" complexity="5">
<methods>
<method name="getBalance" signature="" line-rate="1" branch-rate="0" complexity="1">
<lines>
Expand All @@ -20,7 +20,6 @@
<line number="14" hits="0"/>
<line number="15" hits="0"/>
<line number="16" hits="0"/>
<line number="18" hits="0"/>
</lines>
</method>
<method name="depositMoney" signature="$balance" line-rate="1" branch-rate="0" complexity="1">
Expand All @@ -42,7 +41,6 @@
<line number="14" hits="0"/>
<line number="15" hits="0"/>
<line number="16" hits="0"/>
<line number="18" hits="0"/>
<line number="22" hits="2"/>
<line number="24" hits="1"/>
<line number="29" hits="2"/>
Expand Down
8 changes: 3 additions & 5 deletions tests/_files/BankAccount-cobertura-path.xml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage line-rate="0.5" branch-rate="0.42857142857143" lines-covered="5" lines-valid="10" branches-covered="3" branches-valid="7" complexity="5" version="0.4" timestamp="%i">
<coverage line-rate="0.55555555555556" branch-rate="0.42857142857143" lines-covered="5" lines-valid="9" branches-covered="3" branches-valid="7" complexity="5" version="0.4" timestamp="%i">
<sources>
<source>%s</source>
</sources>
<packages>
<package name="BankAccount" line-rate="0.5" branch-rate="0.42857142857143" complexity="5">
<package name="BankAccount" line-rate="0.55555555555556" branch-rate="0.42857142857143" complexity="5">
<classes>
<class name="BankAccount" filename="BankAccount.php" line-rate="0.5" branch-rate="0.42857142857143" complexity="5">
<class name="BankAccount" filename="BankAccount.php" line-rate="0.55555555555556" branch-rate="0.42857142857143" complexity="5">
<methods>
<method name="getBalance" signature="" line-rate="1" branch-rate="1" complexity="1">
<lines>
Expand All @@ -20,7 +20,6 @@
<line number="14" hits="0"/>
<line number="15" hits="0"/>
<line number="16" hits="0"/>
<line number="18" hits="0"/>
</lines>
</method>
<method name="depositMoney" signature="$balance" line-rate="1" branch-rate="1" complexity="1">
Expand All @@ -42,7 +41,6 @@
<line number="14" hits="0"/>
<line number="15" hits="0"/>
<line number="16" hits="0"/>
<line number="18" hits="0"/>
<line number="22" hits="2"/>
<line number="24" hits="1"/>
<line number="29" hits="2"/>
Expand Down
Loading