NIST CVE SYNC JOB CREATED #91

bminnix · 2022-05-12T21:53:02Z

Created job that looks up current software in the DLC, finds/pulls/creates/associates CVEs for the software and updates existing ones if necessary.

nautobot_device_lifecycle_mgmt/forms.py

development/nautobot_config.py

progala

Thanks for putting work into this PR @bminnix , this is a really useful feature!

I'm including some general notes below. Questions and comments around code are inlined with the relevant lines of code.

Try to avoid using \ when wrapping long lines. PEP8 recommends using line continuation inside parethneses, brackets and braces. I also find implied line continuation easier to read.
I don't think linters have been applied to the code you wrote. invoke tests will run linters along with tests and should flag up anything needing changes. invoke -l will show you available individual linting tasks if you'r prefer to run them separately.

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

progala · 2022-06-17T17:41:44Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+
+    def get_cve_info(self, cpe_software_search_url: str, software_id=None) -> dict:
+        """Search NIST for software and related CVEs"""
+        cpe_info = json.loads(requests.get(cpe_software_search_url).text)


Will this always succeed or is it possible that we get non-json payload?

As long as the req completes, even if no match in the db a json response is returned.

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

progala · 2022-06-17T19:59:19Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+        """Create the list of items that will need to be inserted to DLC CVEs"""
+        dlc_cves = CVELCM.objects.all()
+        for cve, info in cpe_cves.items():
+            if cve not in dlc_cves:


Is cve an instance of CVELCM, i.e. will this conditional ever be True?

At this point, cve may or may not be a current object in CVELCM. cpe_cves are items being passed in from a call to the NIST CVE DB for the currently scoped CPE so this logic is looking to see if it already exists before trying to create it like you've mentioned in next comment, but doing it pre- get_or_create, so that it should only be creating, or associating.

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

bminnix · 2022-07-15T11:23:15Z

I think the only thing that is left are pylint issues.
Running into crashing with:
TypeError: __new__() missing 1 required positional argument: 'scope_type'

Per pylint-dev/pylint-django#358
Upgrading to pylint-django 2.5.3 seemed to resolve all but one traceback/crashing issue:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pylint/utils/ast_walker.py", line 78, in walk
    callback(astroid)
  File "/usr/local/lib/python3.7/site-packages/pylint_django/augmentations/__init__.py", line 739, in wrap_func
    return with_method(orig_method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pylint_django/augmentations/__init__.py", line 317, in ignore_import_warnings_for_related_fields
    consumer._atomic = ScopeConsumer(new_things, consumer.consumed, consumer.scope_type)  # pylint: disable=W0212
TypeError: __new__() missing 1 required positional argument: 'scope_type'

The develop branch this was built from passes all tests. More investigation to come.

itdependsnetworks · 2022-07-16T02:01:36Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+        """Performs search of NIST CVE DB and prepares data for insertion to DLC Management Plugin."""
+        cve_name = url.split("/")[-1]
+        cve_search_url = f"{url}"
+        result = json.loads(requests.get(cve_search_url).text)


Suggested change

result = json.loads(requests.get(cve_search_url).text)

result = requests.get(cve_search_url).json()

itdependsnetworks · 2022-07-16T02:10:22Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+        all_cve_info = {}
+        if len(cpe_info["result"]["cpes"]) > 0:
+            cve_list = cpe_info["result"]["cpes"][0].get("vulnerabilities", [])
+            base_url = "https://services.nvd.nist.gov/rest/json/cve/1.0/"
+
+            dlc_cves = [cve["name"] for cve in CVELCM.objects.values("name")]
+
+            for cve in cve_list:
+                if cve not in dlc_cves and cve.startswith("CVE"):
+                    all_cve_info[cve] = self.prep_cve_for_dlc(base_url + cve)
+                    sleep(0.25)
+                else:
+                    existing_cve = CVELCM.objects.get(name=cve)
+                    self.associate_software_to_cve(software_id, existing_cve.id)
+


Suggested change

all_cve_info = {}

if len(cpe_info["result"]["cpes"]) > 0:

cve_list = cpe_info["result"]["cpes"][0].get("vulnerabilities", [])

base_url = "https://services.nvd.nist.gov/rest/json/cve/1.0/"

dlc_cves = [cve["name"] for cve in CVELCM.objects.values("name")]

for cve in cve_list:

if cve not in dlc_cves and cve.startswith("CVE"):

all_cve_info[cve] = self.prep_cve_for_dlc(base_url + cve)

sleep(0.25)

else:

existing_cve = CVELCM.objects.get(name=cve)

self.associate_software_to_cve(software_id, existing_cve.id)

if len(cpe_info["result"]["cpes"]) <= 0:

return {}

all_cve_info = {}

cve_list = cpe_info["result"]["cpes"][0].get("vulnerabilities", [])

base_url = "https://services.nvd.nist.gov/rest/json/cve/1.0/"

dlc_cves = [cve["name"] for cve in CVELCM.objects.values("name")]

for cve in cve_list:

if cve not in dlc_cves and cve.startswith("CVE"):

all_cve_info[cve] = self.prep_cve_for_dlc(base_url + cve)

sleep(0.25)

else:

existing_cve = CVELCM.objects.get(name=cve)

self.associate_software_to_cve(software_id, existing_cve.id)

itdependsnetworks · 2022-07-16T02:10:55Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+
+    def get_cve_info(self, cpe_software_search_url: str, software_id=None) -> dict:
+        """Search NIST for software and related CVEs."""
+        cpe_info = json.loads(requests.get(cpe_software_search_url).text)


Suggested change

cpe_info = json.loads(requests.get(cpe_software_search_url).text)

cpe_info = requests.get(cpe_software_search_url).json()

itdependsnetworks · 2022-07-16T02:17:55Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+                self.log_info(message=f"""Created {cve}""")
+                self.associate_software_to_cve(software_id, dlc_cve.id)
+
+            else:
+                cve = CVELCM.objects.get(name=cve)
+                self.associate_software_to_cve(software_id, cve)


Suggested change

self.log_info(message=f"""Created {cve}""")

self.associate_software_to_cve(software_id, dlc_cve.id)

else:

cve = CVELCM.objects.get(name=cve)

self.associate_software_to_cve(software_id, cve)

self.log_info(message=f"Updated {cve}")

else:

dlc_cve = CVELCM.objects.get(name=cve)

self.associate_software_to_cve(software_id, dlc_cve)

^^ something like that. cve is recasted in one instance, and pk/id in the other. I am not sure what dlc_cve is, so perhaps that is not the best for the generic obj that could be.

itdependsnetworks · 2022-07-16T02:25:18Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+        if len(reference_data) > 0:
+            cve_url = reference_data[0].get("url", f"https://www.cvedetails.com/cve/{cve_name}/")
+        else:
+            cve_url = f"https://www.cvedetails.com/cve/{cve_name}/"


Suggested change

if len(reference_data) > 0:

cve_url = reference_data[0].get("url", f"https://www.cvedetails.com/cve/{cve_name}/")

else:

cve_url = f"https://www.cvedetails.com/cve/{cve_name}/"

cve_url = f"https://www.cvedetails.com/cve/{cve_name}/"

if reference_data and reference_data[0].get('url'):

cve_url = reference_data[0]["url"])

In general, I suspect you can use this pattern:

if my_list: vs if len(my_list) > 0

There may be edge cases I am not considering, but for the most part, if my_list = [], that returns falsey on a conditional check.

itdependsnetworks · 2022-07-16T02:27:57Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+        platform = platform.split(" ", 1)[1].lower()
+        platform = platform.replace(" ", "_")


Suggested change

platform = platform.split(" ", 1)[1].lower()

platform = platform.replace(" ", "_")

platform = platform.split(" ", 1)[1].lower().replace(" ", "_")

itdependsnetworks · 2022-07-16T02:28:35Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+        all_software = SoftwareLCM.objects.all()
+        cve_counter = 0
+
+        for software in all_software:


Suggested change

all_software = SoftwareLCM.objects.all()

cve_counter = 0

for software in all_software:

cve_counter = 0

for software in SoftwareLCM.objects.all():

itdependsnetworks · 2022-07-16T02:31:27Z

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py

+        dlc_cves = CVELCM.objects.all()
+
+        for cve in dlc_cves:


Suggested change

dlc_cves = CVELCM.objects.all()

for cve in dlc_cves:

for cve in CVELCM.objects.all():

itdependsnetworks · 2022-07-16T02:33:05Z

Looks pretty good! I went a little nitpicky to introduce some patterns. None of it is tested, so take with grain of salt.

glennake · 2022-07-28T16:57:35Z

Hey @bminnix, thanks for your work on this, I will definitely be using this feature when it is merged in.

One thing I've noticed when looking through the code is that it takes the manufacturer, platform and software version to query NVD. In some cases, such as Juniper, they have patch releases within a version. Such as:

12.3x48-d10
12.3x48-d25
12.3x48-d80
etc

The last part is often in the "update" field on the CPE name string, for example:

12.3x48-d10 = cpe:2.3:o:juniper:junos:12.3x48:d10 (99 CVE's)
12.3x48-d25 = cpe:2.3:o:juniper:junos:12.3x48:d25 (72 CVE's)
12.3x48-d80 = cpe:2.3:o:juniper:junos:12.3x48:d80 (22 CVE's)

These different "updates" have different CVE's that apply to them, so without the update field the returned CVE's will be inaccurate where this field is used to differentiate a patch release. I'm not fully sure how this can be resolved without an additional "patch" or "update" field under the software objects. Or a standard naming convention for software names in Nautobot where the string can be split to get the value for the update field in the CPE match string.

Hope I have understood the code correctly and this all makes sense? I suppose an accurate and reliable way to resolve could be some sort of child objects under the software objects for patch/update releases. Or a standard software naming string as above where the patch/update release identifier can be pulled out.

glennake · 2022-07-29T12:29:06Z

Possible solution to my previous comment, the NVD API does not appear to require the complete CPE match string. Any fields that are not supplied are evaluated as wildcards.

For example you could just send the CPE match string:
cpe:2.3:*:juniper:junos:12.3x48

Instead of:
cpe:2.3:*:juniper:junos:12.3x48:*:*:*:*:*:*:*

Which means you could probably just change the following line to not add all the additional fields as wildcards:
https://github.com/bminnix/nautobot-plugin-device-lifecycle-mgmt/blob/nist-cve-sync/nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py#L140
extended_url = f"{manufacturer}:{platform}:{version}"

Then when creating the software versions in Nautobot device lifecycle they could be named like this:

12.3x48:d10
12.3x48:d25
12.3x48:d80

Which should automatically pull through in the "version" variable of your job to create the match strings as below:

cpe:2.3:*:juniper:junos:12.3x48:d10
cpe:2.3:*:juniper:junos:12.3x48:d25
cpe:2.3:*:juniper:junos:12.3x48:d80

Which should then return the CVE's that are appropriate to the specific patch release.

There may be an issue with this if software version information is imported from the network, as JunOS for example formats these versions with a "-" between the software version and patch/update version number. e.g.

SRX> show version | grep Release
JUNOS Software Release [12.1X47-D20.7]

It doesn't help that NVD has inconsistencies in CPE string formatting too...

cpe:2.3:*:juniper:junos:15.1x49-d140
cpe:2.3:*:juniper:junos:15.1x49:d140

Created job that looks up current software in the DLC, finds/pulls/creates/associates CVEs for the software and updates existing ones if necessary. Ugraded to minor revision 1.1.0 Validated against 1.1.6 and 1.3.3 Overall functionality is resolved. Conversions for specific vendors may be necessary, or a required format of software

bminnix · 2022-10-28T18:00:03Z

@glennake sorry for the lengthy delay for work on this, another project was stealing my time.

Yes, these are a couple of issues I have also been trying to figure out the best way to make it vendor agnostic and have highlighted the exact parts that makes this piece a bit more difficult (naming conventions vary by vendor, NIST isn't necessarily keeping them in a standardized format).

My original thought was to require a certain format if using this Plugin with Job, but it was stated that trying to post requirements on software version entry would likely cause more user problems.

I'm curious if I need to make some kind of vendor conversion. Have a default way of handling standard stuff, but if there is a vendor that has an odd way of doing it, write a conversion for that vendor.

…h 50 requests in 30 second rolling window API

bminnix · 2023-05-08T16:21:51Z

This feature is still in progress. A netutils utility is in the progress of being PR'd that will be used here to address the Juniper JunOS version inconsistencies that we've seen.
networktocode/netutils#220

gsnider2195 · 2023-08-28T16:55:33Z

development/dev.env

+NAUTOBOT_CELERY_TASK_SOFT_TIME_LIMIT=21600
+NAUTOBOT_CELERY_TASK_TIME_LIMIT=43200


Are these only required for dev environments or should these time limits be added to the job classes that were failing due to the default time limits?

bradh11 · 2024-01-09T15:32:38Z

@bminnix Any updates to share on this PR?

bminnix · 2024-01-23T05:00:33Z

I'm making time to go through the PR for netutils which is required here. Currently down to 2 comments to resolve before I can hopefully get that accepted and move back to this.

My apologies for the lack of timely movement.

bminnix · 2024-03-21T16:25:47Z

This PR is being closed. This feature will not be supported on Nautobot 1.6.

A new PR will be opened to add this feature for use with later versions.

bminnix · 2024-03-21T16:47:42Z

The branch for supporting new versions can be found here:
https://github.com/nautobot/nautobot-app-device-lifecycle-mgmt/tree/app-nist-cve-sync

bminnix requested review from jvanderaa and progala as code owners May 12, 2022 21:53

bminnix force-pushed the nist-cve-sync branch 2 times, most recently from bd3b7fc to b943147 Compare May 23, 2022 19:09

bminnix commented May 26, 2022

View reviewed changes

nautobot_device_lifecycle_mgmt/forms.py Show resolved Hide resolved

bminnix commented May 26, 2022

View reviewed changes

development/nautobot_config.py Outdated Show resolved Hide resolved

bminnix force-pushed the nist-cve-sync branch 2 times, most recently from 31c7584 to a78836b Compare May 26, 2022 18:03

bminnix requested a review from glennmatthews May 26, 2022 18:18

progala requested changes Jun 17, 2022

View reviewed changes

itdependsnetworks reviewed Jul 15, 2022

View reviewed changes

nautobot_device_lifecycle_mgmt/jobs/cve_tracking.py Outdated Show resolved Hide resolved

itdependsnetworks reviewed Jul 16, 2022

View reviewed changes

bminnix force-pushed the nist-cve-sync branch 3 times, most recently from a30975c to 33e6c98 Compare October 28, 2022 17:19

bminnix force-pushed the nist-cve-sync branch from 33e6c98 to 4be2c47 Compare October 28, 2022 17:33

Bumped the sleep timer for CVE queries from .5s to .75s to comply wit…

2db31ad

…h 50 requests in 30 second rolling window API

bminnix force-pushed the nist-cve-sync branch from d5096e5 to 2db31ad Compare October 29, 2022 01:16

Additional Cleanup

a2fdc7b

bradh11 assigned bminnix Apr 13, 2023

gsnider2195 reviewed Aug 28, 2023

View reviewed changes

bradh11 changed the base branch from develop to ltm-1.6 October 17, 2023 14:15

bminnix closed this Mar 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIST CVE SYNC JOB CREATED #91

NIST CVE SYNC JOB CREATED #91

bminnix commented May 12, 2022

progala left a comment

progala Jun 17, 2022

bminnix Jul 12, 2022

progala Jun 17, 2022

bminnix Jul 12, 2022

bminnix commented Jul 15, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks Jul 16, 2022

itdependsnetworks commented Jul 16, 2022

glennake commented Jul 28, 2022 •

edited

Loading

glennake commented Jul 29, 2022 •

edited

Loading

bminnix commented Oct 28, 2022

bminnix commented May 8, 2023

gsnider2195 Aug 28, 2023

bradh11 commented Jan 9, 2024

bminnix commented Jan 23, 2024

bminnix commented Mar 21, 2024

bminnix commented Mar 21, 2024

	result = json.loads(requests.get(cve_search_url).text)
	result = requests.get(cve_search_url).json()

	cpe_info = json.loads(requests.get(cpe_software_search_url).text)
	cpe_info = requests.get(cpe_software_search_url).json()

		platform = platform.split(" ", 1)[1].lower()
		platform = platform.replace(" ", "_")

		NAUTOBOT_CELERY_TASK_SOFT_TIME_LIMIT=21600
		NAUTOBOT_CELERY_TASK_TIME_LIMIT=43200

NIST CVE SYNC JOB CREATED #91

NIST CVE SYNC JOB CREATED #91

Conversation

bminnix commented May 12, 2022

progala left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bminnix commented Jul 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itdependsnetworks commented Jul 16, 2022

glennake commented Jul 28, 2022 • edited Loading

glennake commented Jul 29, 2022 • edited Loading

bminnix commented Oct 28, 2022

bminnix commented May 8, 2023

Choose a reason for hiding this comment

bradh11 commented Jan 9, 2024

bminnix commented Jan 23, 2024

bminnix commented Mar 21, 2024

bminnix commented Mar 21, 2024

glennake commented Jul 28, 2022 •

edited

Loading

glennake commented Jul 29, 2022 •

edited

Loading