Adding baseline functionality for benchmark script

[domanchi]

Example Usage

$ python scripts/benchmark.py > baseline.json
$ python scripts/benchmark.py --baseline baseline.json
---------------------------------------------------------
plugin                            time          change
---------------------------------------------------------
All Plugins                     1.38231s        ▲  0.02
AWSKeyDetector                  0.63596s           -
ArtifactoryDetector             0.66454s        ▲  0.01
Base64HighEntropyString         0.76522s        ▼ -0.02
BasicAuthDetector                0.6484s        ▼ -0.03
HexHighEntropyString            0.75845s        ▼ -0.04
KeywordDetector                 0.68238s        ▼ -0.03
PrivateKeyDetector              0.67607s        ▼ -0.04
SlackDetector                   0.66106s        ▼ -0.03
StripeDetector                  0.64006s        ▲  0.07
---------------------------------------------------------
$ cat baseline.json | jq
{
    "filenames": [
        "/mnt/d/playground/detect-secrets"
    ],
    "timings": {
        "AWSKeyDetector": 0.63614,
        "ArtifactoryDetector": 0.67135,
        "Base64HighEntropyString": 0.74892,
        "BasicAuthDetector": 0.62003,
        "HexHighEntropyString": 0.71833,
        "KeywordDetector": 0.64856,
        "PrivateKeyDetector": 0.63874,
        "SlackDetector": 0.62901,
        "StripeDetector": 0.71252,
        "all-plugins": 1.40302
    }
}

This allows us to do:

$ python scripts/run_performance_tests.py -m LONG_FILE > baseline.json
Generating content...
Running checks...
$ python scripts/run_performance_tests.py -m LONG_FILE --baseline baseline.json
Generating content...
Running checks...
---------------------------------------------------------
plugin                            time          change
---------------------------------------------------------
All Plugins                    15.75589s        ▲  0.35
AWSKeyDetector                  1.35012s        ▲  0.06
ArtifactoryDetector             1.88477s        ▲  0.01
Base64HighEntropyString         2.87111s        ▲  0.02
BasicAuthDetector               1.33566s        ▲  0.02
HexHighEntropyString            2.90824s        ▼ -0.04
KeywordDetector                 2.43115s        ▲  0.03
PrivateKeyDetector              1.86511s        ▼ -0.01
SlackDetector                   1.48878s        ▲  0.01
StripeDetector                  1.38918s        ▲  0.02
---------------------------------------------------------
$ cat baseline.json | jq
{
    "config": {
        "length": null,
        "mode": "LONG_FILE"
    },
    "filenames": [
        "/tmp/tmpxmbg9rgl"
    ],
    "timings": {
        "AWSKeyDetector": 1.40741,
        "ArtifactoryDetector": 1.89773,
        "Base64HighEntropyString": 2.89092,
        "BasicAuthDetector": 1.35538,
        "HexHighEntropyString": 2.8728,
        "KeywordDetector": 2.45626,
        "PrivateKeyDetector": 1.85466,
        "SlackDetector": 1.50198,
        "StripeDetector": 1.40959,
        "all-plugins": 16.10408
    }
}

[KevinHock]

🚢 🐑

[domanchi]

@killuazhu , with regards to #136, can you see what changes this code needs to undergo in order to reproduce your performance hit? Even with my sample of 250K lines, it only took ~16s.

[killuazhu]

@domanchi The performance hit was resolved with the short term fix in #144 and #147. If we want to reproduce, we will need to revert those 2 PRs.

[domanchi]

Thanks for the convenient links!

After reverting those changes locally, I get these results:

$ scripts/run_performance_tests.py -m LONG_FILE --baseline baseline.json
Generating content...
Running checks...
------------------------------------------------------------
plugin                          time                change
------------------------------------------------------------
All Plugins               Timeout exceeded!        ▲  13.9
AWSKeyDetector                 1.38776s            ▲  0.02
ArtifactoryDetector            1.87043s            ▲  0.03
Base64HighEntropyString   Timeout exceeded!       ▲  27.11
BasicAuthDetector              1.37341s            ▼ -0.02
HexHighEntropyString      Timeout exceeded!       ▲  27.13
KeywordDetector                2.47949s            ▼ -0.02
PrivateKeyDetector             1.85911s               -
SlackDetector                  1.5571s             ▼ -0.06
StripeDetector                 1.40493s               -
------------------------------------------------------------

Good to know our changes were effective, and that this script would help us identify any further issues in the future!