Add performance tests

[terriko]

As per #97, we could probably use some performance tests so that travis will warn us if there's a huge performance regression we should look at. What the threshold for "failing" should be and how they work can be a topic of discussion here before an implementation is written.

[rossburton]

If you want to scan a representative rootfs as part of the CI then maybe our release tarballs are a candidate? For example core-image-minimal-dev is a rootfs that is small (35M download), trivial to unpack (as it's a tarball) but actually exercises the code:

$ time cve-bin-tool -xv .
Connecting to NVD database and extracting the CVE list ... Please hold on.. This will take few minutes...
Last Update: 2019-03-11
Local database has been updated in the past 24h.
New data not downloaded.  Remove old files to force the update.
usr/lib/libssl.so.1.1 is openssl 1.1.1
Known CVEs in version 1.1.1
CVE-2018-12433, CVE-2018-12438, CVE-2018-12437, CVE-2019-0190
usr/lib/libexpat.so.1.6.8 is expat 2.2.6
Known CVEs in version 2.2.6
CVE-2012-6702, CVE-2016-5300, CVE-2016-0718
usr/lib/libcrypto.so.1.1 is openssl 1.1.1
Known CVEs in version 1.1.1
CVE-2018-12433, CVE-2018-12438, CVE-2018-12437, CVE-2019-0190
usr/lib/libxml2.so.2.9.8 is xml2 2.9.8
Known CVEs in version 2.9.8
CVE-2018-14404, CVE-2018-14567, CVE-2017-5130, CVE-2016-4614, CVE-2016-4616, CVE-2017-15412, CVE-2018-9251, CVE-2016-9596, CVE-2016-4615, CVE-2017-7376, CVE-2016-4483, CVE-2017-18258, CVE-2015-8806, CVE-2016-9598
lib/libz.so.1.2.11 is zlib 1.2.11

Overall CVE summary:
There are 4 files with known CVEs detected
Known CVEs in zlib 1.2.11, xml2 2.9.8, openssl 1.1.1, expat 2.2.6:
xml2,2.9.8,CVE-2018-14404,HIGH
xml2,2.9.8,CVE-2018-14567,MEDIUM
xml2,2.9.8,CVE-2017-5130,HIGH
xml2,2.9.8,CVE-2016-4614,CRITICAL
xml2,2.9.8,CVE-2016-4616,CRITICAL
xml2,2.9.8,CVE-2017-15412,HIGH
xml2,2.9.8,CVE-2018-9251,MEDIUM
xml2,2.9.8,CVE-2016-9596,MEDIUM
xml2,2.9.8,CVE-2016-4615,CRITICAL
xml2,2.9.8,CVE-2017-7376,CRITICAL
xml2,2.9.8,CVE-2016-4483,HIGH
xml2,2.9.8,CVE-2017-18258,MEDIUM
xml2,2.9.8,CVE-2015-8806,HIGH
xml2,2.9.8,CVE-2016-9598,MEDIUM
openssl,1.1.1,CVE-2018-12433,MEDIUM
openssl,1.1.1,CVE-2018-12438,MEDIUM
openssl,1.1.1,CVE-2018-12437,MEDIUM
openssl,1.1.1,CVE-2019-0190,HIGH
expat,2.2.6,CVE-2012-6702,MEDIUM
expat,2.2.6,CVE-2016-5300,HIGH
expat,2.2.6,CVE-2016-0718,CRITICAL

real	0m37.567s
user	0m25.125s
sys	0m15.380s

That URL won't be changing as it's a release artefact, so you can download it as part of the CI job.

[rossburton]

(that time is with 0.2.0 obviously)

[terriko]

This sounds like a perfect real-world test setup. I'm going to flag this with the 'gsoc' tag because this might be a reasonable addition to a gsoc project proposal, although people are of course welcome to work on it outside of that scope.

[terriko]

write tests using the tarballs! I'm working on a real-file test over in #99 (although it has a bug) and you can look at the existing ones for the extractors. You'd want to hide it behind a LONG_TEST flag. Once the test works at all, it'd be nice to figure out some logic for caching results so that you can compare vs last release or some previous commit.

[terriko]

We don't have quite the performance tests we envisioned, but because of the timeouts and regular monitoring of CI I think we've got something good enough that this can be closed. If anyone's got any brilliant visions for new tests, they can open an new issue.