SF#334 Test coverage is incomplete

[perldl-bot]

From http://sourceforge.net/p/pdl/bugs/334

[duffee]

I started having a look at this one by running Devel::Cover across PDL with

HARNESS_PERL_SWITCHES=-MDevel::Cover make test
cover

(it took 23 hours on my machine) It reports 4% coverage for blib-lib-PDL-IO-FITS-pm.html, hitting only the BEGIN statements. The tests in IO/FITS/t/fits.t exercise a lot more than that, so something is preventing it from running correctly. I can't figure out how to get Devel::Cover to Do What I Mean, not helped by PDL's archaic test layout.
I've been to the coveralls site and still can't pick out any hints on how to run coverage on my local machine.

Is trying to generate a coverage report doomed to failure or does someone have a suggestion?

[mohawk2]

Thank you for wanting to contribute to this important issue! Ironically, I've been focusing on it myself recently, with getting Test::PDL ready to incorporate into main-PDL.

We run coverage tests in our CI on every commit (it's the Coveralls tick or cross). It isn't very quick, but where CI of a fairly complete build (with many or all of the optional builds present) takes about 6 mins (less when ccache is helping), with coverage it's about 8 (less with ccache). I don't know why it would take 23 hours. The canonical way to run coverage tests is cover -test, though your incantation above should work similarly. Take a look at .github/workflows/ci.yml and https://github.com/PDLPorters/devops/blob/master/github-actions/ci-dist/action.yml (the "with coverage" step). It's doing $MYPERL -S cover -test -relative_only -gcov_chdir -report Coveralls.

You allude to an "archaic" layout of tests. Before I changed it, nearly all the tests were in the top t directory, and needed various complicated guard mechanisms to make themselves not run if their component (being optional) wasn't built. That adds considerably to the difficulty of splitting up the unwieldy blob that PDL still is, but soon won't be. It also added to the burden of maintaining the complicated guard mechanisms. Moving the tests next to their relevant code is more modular.

[duffee]

Upgrading Devel::Cover from old system version 1.36 to new cpan version 1.44 (and using your CI step) made the cover run in a decent time. (23 hours was a sign that I was doing something wrong, but didn't know what)

It reports 4% which is still low if my eyeballs can estimate how much fits.t exercises, but it will have to wait until next week for me to look at it.

I think this is now the 28th anniversary of PDL and I assumed that the tests buried in subdirectories were an artifact of it growing organically before CPAN practices converged. If I have any decent ideas on how to handle the optional components, I'll bring it up for discussion.

[mohawk2]

Test::PDL is now incorporated into main PDL, under Basic, which will form the new breakaway "PDL::Basic" as discussed. It's quite nice writing tests that use it, which will mean tapprox can be consigned to the dustbin of history!

[duffee]

I'm starting to think that a blog post using Test::PDL to test whether I truely understand how a $matrix * $vector operation works. (Advent calendar warning)

[mohawk2]

I can tell you upfront that a $m * $v will just broadcast each row of the $m, doing a scalar multiply over each element of each. Is that your expectation? I feel like the general understanding of matrix x vector stuff would be inner-product related, which the scalar multiply would not do.

[mohawk2]

As a starter for an Advent posting, the newly-incorporated Test::PDL, which I'm using to replace all tapprox and nearly all instances of these constructs in the codebase:

• approx
• ok all($x == $y)
• is_deeply [$p->unpdl], [...]
• ok sum(abs($out-$in)) < 1e-4
• is $out == $in, 1
• ok tapprox(...)
• ok all approx(...)
• ok all($x == $y)
• is( int(at(sum($c-$ans))), 0, "conv2d()" )
• ok +($ans[0] == 50) && ($ans[1] == 1) && ($ans[2] == 2)

... is really good. It checks all types (unless you say not to) and dims are identical, whether NaNs/badvalues/infinities are all the same, and when something fails, you get a message like:

#   Failed test 'inplace sorting with bad values'
#   at Basic/t/ufunc.t line 56.
#     1/10 values do not match
#          got: Double   D [10]       (PB   ) [0 1 2 4 5 6 7 8 BAD BAD]
#     expected: Double   D [10]       (PB   ) [0 1 2 4 5 6 7 8 9 BAD]
# First <=5 values differ at: 
# [
#  [8]
# ]
# Those 'got' values: [BAD]
# Those 'expected' values: [9]

That one was from finding a bug in qsort that had lain hidden, because all(pdl('5 3 BAD') == pdl('5 BAD 2')) returns 1; the BAD does not make all return false because there are any true values.

[mohawk2]

More PDL Advent fodder: make a dataflowing (using the newish flowing idiom) version of the scoring formula in https://blogs.perl.org/users/enkidu/2020/03/enter-the-matrix-with-pdl.html together with an explanation of broadcasting (no longer "threading") being adding from:

C_ij = 1/(x_i - y_j)

to being:

C_ij = 1/(x_ij - y_ij)

with the additional indices being non-bolded, and "dummy". Article 1, dummy dims; article 2, dataflow.

EDIT I commented on one of Enkidu's PDL posts, asking them if they'd like to be involved.