Nosey Parker v0.16.0

[bradlarsen]

Additions

• The scan command now supports a new --copy-blobs={all,matching,none} parameter. When specified as matching, a copy of each encountered blob that has matches will be saved to the datastore's blobs directory. When specified as all, a copy of each encountered blob will be saved. The default value is none. This mechanism exists to aid in ad-hoc downstream investigation. Copied blobs are not used elsewhere in Nosey Parker at this point.

• A new advanced global command-line parameter has been exposed:

  • --sqlite-cache-size=SIZE to control the pragma cache_size value used in sqlite database connections

• The datastore now contains two additional tables for to represent freeform comments and accept/reject status associated with findings. These additional tables are not currently populated in the open-source version of Nosey Parker. The report command now emits finding status and comment if populated. Note: the datastore format is not settled and is subject to change.

• A new "ruleset" mechanism has been added. A ruleset is a named collection of rules that can be selected as a group. The new --ruleset=NAME parameter to scan can be used to enable alternative rulesets. Three built-in rulesets are provided (default, np.assets and np.hashes); the special ruleset name all enables all known rules. See the built-in rulesets at crates/noseyparker/data/default/builtin/rulesets for an example for writing your own.

• The default collection of rules has been pruned down to further emphasize signal-to-noise. Only rules that detect secret things are included in the default collection. Rules that detect other things, such as cloud assets, application IDs, or public keys, are not included in this set. Instead, those are in the np.assets ruleset, which is not enabled by default. No rules have been removed from Nosey Parker; rather, the defaults have been adjusted to support the most common use case (secrets detection).

• Additional checks have been added to the rules check command:

  • Each regex rule must have at least one capture group
  • Each ruleset must have a globally-unique ID
  • A ruleset's included rules must resolve to actual rules
  • A ruleset should not include duplicate rules

• A new rules list command is available, which lists available rules and rulesets. This command can emit its output in human-oriented format or in JSON format.

• New rules have been added:

  • Dependency-Track API Key (Thank you @tpat13!)
  • Password Hash (sha256crypt)
  • Password Hash (sha512crypt)
  • Password Hash (Cisco IOS PBKDF2 with SHA256)
  • React App Username
  • React App Password

• A new global --quiet / -q option has been added, which suppresses non-error feedback messages and disables progress bars.

Fixes

• Command-line parameters that can meaningfully accept negative numbers can now be specified without having to use --PARAMETER=NEGATIVE_VALUE syntax; a space can now separate the paraemter and the value.

• Fixed three rules that were missing capture groups:

  • Age Recipient (X25519 public key)
  • Age Identity (X22519 secret key)
  • crates.io API Key

  Due to nuanced details of how scanning is performed, rules without capture groups will never produce reported matches. An additional check was added to the rules check command and a couple assertions were added that should help prevent this type of error in the future.

• Fixed several rules:

  • Amazon MWS Auth Token: the capture group was smaller than it should have been
  • Microsoft Teams Webhook: changed 3 capture groups to 1; full URL is now included
  • Slack Webhook: full URL is now included

• The LICENSE, README.md, and CHANGELOG.md files are now included in prebuilt binary releases.

• ANSI formatting sequences are now no longer included by default by the report command when the output is redirected to a file using the -o/--outfile parameter (#55).

• The scan command should no longer emit warnings like Failed to decode entry in tree. These warnings were due to a bug in the Git object parsing code in the gix dependency, which was fixed upstream.

Changes

• The rules check command invocation now behaves differently. It now no longer requires input paths to be specified. It will check the built-in rules for problems, and if additional paths are specified, will check those rules as well. This change was made so that the scan, rules check, and rules list invocations have consistent interfaces.

• The default path-based ignore rules in Nosey Parker now ignore packed-refs files from Git repositories.

• Several rules have been changed:

  • The Slack rule (id np.slack.1) has been removed, as it was redundant with Slack Token.
  • Slack Token has been split into Slack Bot Token, Slack Legacy Bot Token, Slack User Token, and Slack App Token.
  • CodeClimate was enhanced to detect additional cases and was renamed to CodeClimate Reporter ID.
  • md5crypt Hash (id np.md5.1) has been renamed to Password Hash (md5crypt) and re-identified as np.pwhash.1.
  • bcrypt Hash (id np.bcrypt.1) has been renamed to Password Hash (bcrypt) and re-identified as np.pwhash.2.

• Log messages are written to stderr instead of stdout.

---

This discussion was created from the release Nosey Parker v0.16.0.