How to ignore some strings ?

[yilas]

Hi,

Thank you for your tool which works very well. However, I have a question about the best way to ignore certain strings.

For example, I would like to be able to ignore some AWS keys :

AKIA111111111EXAMPLE
AKIA111111112EXAMPLE

So I've created a yaml file with this pattern (~/tmp/nosey/aws.yml) :

rules:
- name: AWS API Key Key
  pattern: '\b(?:A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16})\b'
  references:
  - https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html
  - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
  - https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html
  - https://docs.aws.amazon.com/accounts/latest/reference/credentials-access-keys-best-practices.html

  examples:
  - 'A3T0ABCDEFGHIJKLMNOP'
  - AKIADEADBEEFDEADBEEF'

  negative_examples:
  - 'AKIA111111111EXAMPLE'
  - 'AKIA222222222EXAMPLE'
  - 'AKIAI44QH8DHBEXAMPLE'
  - '======================'
  - '//////////////////////'
  - '++++++++++++++++++++++'

Then :

noseyparker scan -r ~/tmp/nosey/ --datastore ~/gitlab-dump.db ~/gitlab-dump/

But these keys are still detected by the tool.
I guess I'm doing something wrong but I don't see how to solve my issue.

Thanks in advance for your help !

[bradlarsen]

Hi @yilas . Good question!

In the current Nosey Parker rules language there is no mechanism to express deny-lists or ignore particular strings.

The negative_examples field in the rules language is used not for ignoring particular strings, but instead for expressing test cases: inputs that the rule should not match. Both the examples and negative_examples test cases are checked by the noseyparker rules check command.

I presume you are writing the rule that you shared in the issue here primarily to play around with custom Nosey Parker rules? There is already an AWS API Key rule that uses a more restrictive pattern than what you have written above.

[Coruscant11]

Maybe by creating a .noseyparkerignore ?
I think we could take inspiration from gitleaks : https://github.com/gitleaks/gitleaks#gitleaksignore

[bradlarsen]

To be clear: this is not functionality that exists currently in Nosey Parker. But it might be a useful addition.

[munntjlx]

I did put in a POST processing JSON file you could use (see open PR's). Just use 'jsonl' and process and skip the ones labeled 'skip' in the JSON provided in the PR.

[munntjlx]

Post processing might be the best option. Have a file like thus:

{
    "AWS API Key": 
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "Age Recipient (X25519 public key)":
        {
            "Priority": "Low",
            "Skip": "True",
            "Resolution": "NAN"
        },
    "Age Identity (X22519 secret key)":
        {
            "Priority": "Low",
            "Skip": "True",
            "Resolution": "NAN"
        },
    "Artifactory API Key":
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "Azure Connection String":
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution":"Reset the account or ID"
        },
    "Azure App Configuration Connection String":
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution":"Reset the account or ID"
        },
    "CodeClimate":
        {
            "Priority": "Low",
            "Skip": "True",
            "Resolution": "NAN"
        },
    "crates.io API Key":
        {
            "Priority": "Low",
            "Skip": "True",
            "Resolution": "NAN"
        },
    "DigitalOcean Application Access Token":
        {
            "Priority": "Low",
            "Skip": "True",
            "Resolution": "NAN"
        },
    "DigitalOcean Refresh Token":
        {
            "Priority": "Low",
            "Skip": "True",
            "Resolution": "NAN"
        },
    "Dynatrace Token":
        {
            "Priority": "Medium",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "Facebook Secret Key":
        {
            "Priority": "Low",
            "Skip": "True",
            "Resolution": "NAN"
        }, 
    "Figma Personal Access Token":
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "Generic Secret":
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "Generic API Key":
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "Generic Username and Password (quoted)":
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "Generic Username and Password (unquoted)":
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "Generic Password (single quoted)":
        {
            "Priority": "Medium",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "GitHub Personal Access Token":
        {
            "Priority": "High",
            "Skip": "False",
            "Resolution": "Reset the account or ID"
        },
    "GitHub OAuth Access Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "GitHub App Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "GitHub Refresh Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "GitHub Client ID":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "GitHub Secret Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "GitHub Personal Access Token (fine-grained permissions)":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "GitLab Runner Registration Token":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "GitLab Personal Access Token":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "GitLab Pipeline Trigger Token":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "Google Client ID":
    {
        "Priority": "Medium",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Google OAuth Access Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Google API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Google Cloud Storage Bucket (subdomain style)":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "Google Cloud Storage Bucket (path style)":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "Hardcoded Gradle Credentials":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Grafana API Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Grafana Service Account Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "md5crypt Hash":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "bcrypt Hash":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Heroku API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "HuggingFace User Access Token":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "Jenkins Token or Crumb":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "JSON Web Token (base64url-encoded)":
    {
        "Priority": "High",
        "Skip": "True",
        "Resolution": "Reset the account or ID"
    },
    "LinkedIn Client ID":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "LinkedIn Secret Key":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "MailChimp API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Mapbox Public Access Token":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "Microsoft Teams Webhook":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "netrc Credentials":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "New Relic License Key":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "New Relic License Key (non-suffixed)":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "New Relic API Service Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "New Relic Admin API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "New Relic Insights Insert Key":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "New Relic Insights Query Key":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "New Relic REST API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "New Relic Pixie API Key":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "New Relic Pixie Deploy Key":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "NPM Access Token (fine-grained)":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "NuGet API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Credentials in ODBC Connection String":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Okta API Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "OpenAI API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    
    "PEM-Encoded Private Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID or cert"
    },
    
    "Postman API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    
    "Credentials in PsExec":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    
    "PyPI Upload Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    
    "Salesforce Access Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    
    "Square Access Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    
    "StackHawk API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    
    "Sauce Token":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "Segment Public API Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "SendGrid API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Shopify Domain":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "Shopify App Secret":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Shopify Access Token (Public App)":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Shopify Access Token (Custom App)":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Slack":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Slack Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Slack Webhook":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "SonarQube Token":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Square OAuth Secret":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Telegram Bot Token":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "Reset the account or ID"
    },
    "Twilio API Key":
    {
        "Priority": "High",
        "Skip": "False",
        "Resolution": "Reset the account or ID"
    },
    "Twitter Client ID":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    },
    "Twitter Secret Key":
    {
        "Priority": "Low",
        "Skip": "True",
        "Resolution": "NAN"
    }
    
}

[bradlarsen]

Another thought: SARIF format has some built-in mechanisms for suppressing particular findings.

I believe that format is used by GitHub itself for communicating code scanning diagnostics and automated feedback like you see bots placing on PRs sometimes.

SARIF format is supported somewhat in Nosey Parker (#33, #34). It's not heavily tested, but that might be another plausible path for suppressing particular findings, even across multiple scan runs. A number of other tools support that format.