NVD is not working.

[imsahil007]

Every time I try to run tests for test/test_ci.py. The tests fail for curl with this assertion error.
I tried creating pulling a fresh copy of the repo. But the tests still failed.

I noticed that the tests are failing for codecov as well.

[terriko]

0 != 0, eh?
I'm heading out shortly for an appointment, but that's an interesting one. I'll try to get to it this afternoon.

[terriko]

I am indeed getting some tests failures locally, but more than just this one. It looks like I'm getting some nvd issues that are probably masking what's going on.

[imsahil007]

I ran the tests again. Nothing seems to work right now. I tried scanning some binaries. You can see the output here

[imsahil007]

[anthonyharrison]

@terriko @imsahil007 I have got to the bottom of this after a bit of digging around the cvedb.py module which has revealed that the problem is with the way the NVD page is being scrapped off the web. I don't know if the format of the URLs has changed recently but the feeds are now relative URLs (i.e. /feeds/....) rather than absolute URLs (https://nvd.nist.gov/feeds/...). Unfortunately the processing of the URLs assumes that they are absolute URLs so when the URLs are passed to the rountines which download the CVEs, as there are no URLs which have matched the regular expression, no data is retrieved.

A simple hack is to make the following change to the nist_scrape routine to make the meta_url absolute

async def nist_scrape(self, session):
    async with session.get(self.feed) as response:
        page = await response.text()
        json_meta_links = self.META_REGEX.findall(page)
        return dict(
            await asyncio.gather(
                *[self.getmeta(session, "https://nvd.nist.gov"+meta_url) for meta_url in json_meta_links]
            )
        )

The META_REGEX needs to also change to

META_REGEX = re.compile(r"/./json/.-[0-9].[0-9]-[0-9]*.meta")

[imsahil007]

@terriko @imsahil007 I have got to the bottom of this after a bit of digging around the cvedb.py module which has revealed that the problem is with the way the NVD page is being scrapped off the web. I don't know if the format of the URLs has changed recently but the feeds are now relative URLs (i.e. /feeds/....) rather than absolute URLs (https://nvd.nist.gov/feeds/...). Unfortunately the processing of the URLs assumes that they are absolute URLs so when the URLs are passed to the rountines which download the CVEs, as there are no URLs which have matched the regular expression, no data is retrieved.

A simple hack is to make the following change to the nist_scrape routine to make the meta_url absolute

async def nist_scrape(self, session):
    async with session.get(self.feed) as response:
        page = await response.text()
        json_meta_links = self.META_REGEX.findall(page)
        return dict(
            await asyncio.gather(
                *[self.getmeta(session, "https://nvd.nist.gov"+meta_url) for meta_url in json_meta_links]
            )
        )

The META_REGEX needs to also change to

META_REGEX = re.compile(r"/./json/.-[0-9].[0-9]-[0-9]*.meta")

Yeah, I noticed that as well. Thanks for your input @anthonyharrison

[terriko]

Thank you! I got as far as seeing that nvd wasn't working at all, but hadn't had a chance to debug why. This is incredibly helpful.