test: basic execution test for EPSS #4484 #4510
Conversation
Add a test to the cli tests to check the EPSS functionality: It first tests if the update of EPSS source runs without errors (regression test for intel#4473). Then checks for an example SBOM if EPSS values are written to csv report.
weichslgartner
force-pushed
the
basic_execution_test_for_EPSS
branch
from
31c09d7
to
e35ae71
Compare
There was a problem hiding this comment.
Looks like the test is failing:
=========================== short test summary info ============================
FAILED test/test_cli.py::TestCLI::test_basic_epss - ValueError: could not convert string to float: ''
============= 1 failed, 22 passed, 11 skipped in 68.76s (0:01:08) ==============
|
@terriko I branched from main when the other fix was not yet merged so this was expected. Yet it didn't fail during the update (as was skipped because of the cache), but only in the step when it checked if the EPSS value from the csv file. I will update with the main branch, then this test should pass. |
|
Ah, my bad -- for some reason I had it in my head that this had been updated but of course it hadn't. (that's what I get for multitasking too much this morning while scanning PRs.) Letting things re-run now so I can see what's up. |
There was a problem hiding this comment.
Still getting the same error:
=========================== short test summary info ============================
FAILED test/test_cli.py::TestCLI::test_basic_epss - ValueError: could not convert string to float: ''
============= 1 failed, 22 passed, 11 skipped in 68.77s (0:01:08) ==============
I haven't dug into it, but I do see that error often when we forget to trim whitespace on a string, so there may be a wayward " " somewhere? Or as you say, maybe the log level isn't giving you the correct messages and you'll need to swap it?
|
@terriko It seems it didn't update the EPSS data because the cache seems to be still old (before the EPSS fix). The Value error comes from parsing values out of the generated csv files. The two last values should be epss_percentile and epss_probability (it also asserts the header and they didn't trigger). If EPSS failed there are no values there hence the conversion fails. (I will create a better error message here). I reproduced the this fail locally with an old cache with not epss data. Only if I do an "-u now" update it downloads the data correctly. So having a way to force epss update without have to download all the cve data would be good. |
|
Hm, the cache was broken but I fixed it earlier this week and I did update last night. Let me go pull latest and then try re-running this. |
|
(I'm wondering if there's something else up with caching that I'm not seeing, though, because I'm still having some weirdness on jobs.) |
|
Okay, cache has updated. I'm going to try re-running the failing tests again and see if that unstuck it or if there's more to do. |
Adds test described in #4484.
Add a test to the cli tests to check the EPSS functionality: It first tests if the update of EPSS source runs without errors (regression test for #4473).
Then checks for an example SBOM if EPSS values are written to csv report.
I tried to disable the other data sources, but needed NVD. Any suggestions to minimize the test are welcome.
Should fail on current main, should pass after #4475 is merged.
Tested it locally with https://github.com/weichslgartner/cve-bin-tool/tree/basic_execution_test_for_EPSS