[custom_ti] Add support for basic authentication #11202

chemamartinez · 2024-09-20T18:11:11Z

Proposed commit message

This pull request adds:

Support for basic authentication to the Custom Threat Intelligence integration.
Fingerprint to avoid duplicated indicators to be ingested.

I recommend reviewing the PR commit by commit, and notice that ecfafdf only adds formatting changes (replacing tabs by spaces).

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Closes [custom_ti] Add Basic auth to integration #11192

Screenshots

nicpenning · 2024-09-20T19:04:52Z

I went down the road of trying to add this and test but got this error:

elastic-package test
2024/09/20 13:24:31  INFO New version is available - v0.104.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.104.0
Run test suite for the package
Run asset tests for the package
2024/09/20 13:24:31  INFO License text found in "/home/napsta/integrations/LICENSE.txt" will be included in package
Error: error running package asset tests: could not complete test run: can't install the package: can't install the package: could not zip-install package; API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"Encountered 1 errors creating saved objects: [{\"type\":\"dashboard\",\"id\":\"ti_custom-e336dd7a-d5cb-4b7f-a6cd-85c45d0bd1ac\",\"error\":{\"isBoom\":true,\"isServer\":false,\"data\":null,\"output\":{\"statusCode\":400,\"payload\":{\"message\":\"[attributes.controlGroupInput.panelsJSON]: expected value of type [string] but got [Object]: Bad Request\",\"statusCode\":400,\"error\":\"Bad Request\"},\"headers\":{}},\"type\":\"unknown\"}}]"}

Did you see this in your testing?

nicpenning · 2024-09-21T00:41:24Z

👀

chemamartinez · 2024-10-02T11:17:11Z

Hi @nicpenning,

I have just run the same command successfully:

% elastic-package test
2024/10/02 13:14:17  INFO New version is available - v0.104.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.104.0
Run asset tests for the package
2024/10/02 13:14:17  INFO License text found in "/Users/chemamartinez/devel/integrations/LICENSE.txt" will be included in package
--- Test results for package: ti_custom - START ---
╭───────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                                                          │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ ti_custom │             │ asset     │ dashboard ti_custom-e336dd7a-d5cb-4b7f-a6cd-85c45d0bd1ac is loaded │ PASS   │       1.25µs │
│ ti_custom │             │ asset     │ search ti_custom-a06e63dc-01d3-4005-b24f-2fd46c3962b1 is loaded    │ PASS   │        417ns │
│ ti_custom │ indicator   │ asset     │ index_template logs-ti_custom.indicator is loaded                  │ PASS   │        334ns │
│ ti_custom │ indicator   │ asset     │ ingest_pipeline logs-ti_custom.indicator-0.2.0 is loaded           │ PASS   │        250ns │
╰───────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_custom - END   ---
Done
Run pipeline tests for the package
--- Test results for package: ti_custom - START ---
╭───────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                                                             │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-asn-ndjson.log)              │ PASS   │  214.81975ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-domain-name-ndjson.log)      │ PASS   │ 201.276292ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-email-ndjson.log)            │ PASS   │  209.85875ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-file-ndjson.log)             │ PASS   │ 203.720208ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-ip-ndjson.log)               │ PASS   │ 216.358709ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-unknown-ndjson.log)          │ PASS   │ 212.144917ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-url-ndjson.log)              │ PASS   │ 216.560917ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-windows-registry-ndjson.log) │ PASS   │ 214.379583ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-x509-ndjson.log)             │ PASS   │   200.5165ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-asn-ndjson.log                                         │ PASS   │    85.3065ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-domain-name-ndjson.log                                 │ PASS   │  85.485333ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-email-ndjson.log                                       │ PASS   │  78.396541ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-file-ndjson.log                                        │ PASS   │  91.986958ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-ip-ndjson.log                                          │ PASS   │   77.96225ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-unknown-ndjson.log                                     │ PASS   │  65.644709ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-url-ndjson.log                                         │ PASS   │  64.541625ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-windows-registry-ndjson.log                            │ PASS   │  69.884042ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-x509-ndjson.log                                        │ PASS   │ 111.277333ms │
╰───────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_custom - END   ---
Done
Run policy tests for the package
--- Test results for package: ti_custom - START ---
No test results
--- Test results for package: ti_custom - END   ---
Done
Run static tests for the package
--- Test results for package: ti_custom - START ---
╭───────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ ti_custom │ indicator   │ static    │ Verify sample_event.json │ PASS   │  76.768125ms │
╰───────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_custom - END   ---
Done
Run system tests for the package
2024/10/02 13:08:17  INFO License text found in "/Users/chemamartinez/devel/integrations/LICENSE.txt" will be included in package
2024/10/02 13:09:06  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/stix-taxii-1727867346767679000.log
2024/10/02 13:09:14  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/elastic-agent-1727867354872666000.log
2024/10/02 13:10:01  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/stix-taxii-1727867401965130000.log
2024/10/02 13:10:10  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/elastic-agent-1727867410076043000.log
2024/10/02 13:11:04  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/stix-taxii-1727867464120480000.log
2024/10/02 13:11:12  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/elastic-agent-1727867472202292000.log
2024/10/02 13:11:56  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/stix-filestream-1727867516252147000.log
2024/10/02 13:12:04  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/elastic-agent-1727867524453067000.log
--- Test results for package: ti_custom - START ---
╭───────────┬─────────────┬───────────┬──────────────────┬────────┬───────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME        │ RESULT │  TIME ELAPSED │
├───────────┼─────────────┼───────────┼──────────────────┼────────┼───────────────┤
│ ti_custom │ indicator   │ system    │ filestream       │ PASS   │ 34.117025291s │
│ ti_custom │ indicator   │ system    │ stix             │ PASS   │ 41.620459791s │
│ ti_custom │ indicator   │ system    │ taxii            │ PASS   │    35.843603s │
│ ti_custom │ indicator   │ system    │ taxii-basic-auth │ PASS   │ 37.779323916s │
╰───────────┴─────────────┴───────────┴──────────────────┴────────┴───────────────╯
--- Test results for package: ti_custom - END   ---
Done

Maybe there is something broken in your elastic-package installation? you can try upgrading it to the latest version.

elastic-vault-github-plugin-prod · 2024-10-02T11:20:07Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-10-02T11:22:34Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

nicpenning · 2024-10-02T12:21:14Z

Could have been, looks like yours is coming along nicely, thanks for adding this!

packages/ti_custom/data_stream/indicator/agent/stream/cel.yml.hbs

elasticmachine · 2024-10-03T08:17:29Z

💚 Build Succeeded

Buildkite Build
Commit: 4a0e8c0

History

💚 Build #16709 succeeded ecfafdf
💔 Build #16245 failed 60b13d14c1dfc9076609aff212e46f478157c941

cc @chemamartinez

elastic-sonarqube · 2024-10-03T08:17:33Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
95.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2024-10-03T08:59:24Z

Package ti_custom - 0.2.0 containing this change is available at https://epr.elastic.co/search?package=ti_custom

chemamartinez added enhancement New feature or request Integration:ti_custom Custom Threat Intelligence labels Sep 20, 2024

chemamartinez self-assigned this Sep 20, 2024

chemamartinez added 4 commits October 2, 2024 09:07

Add fingerprint to STIX events

a3aafd1

Add support for basic auth

72f1143

Update changelog

57b0965

Replace tabs by spaces in CEL code

ecfafdf

chemamartinez force-pushed the 11192-custom_ti-basic-auth branch from 60b13d1 to ecfafdf Compare October 2, 2024 11:01

chemamartinez marked this pull request as ready for review October 2, 2024 11:22

chemamartinez requested a review from a team as a code owner October 2, 2024 11:22

chemamartinez added the Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] label Oct 2, 2024

efd6 reviewed Oct 2, 2024

View reviewed changes

packages/ti_custom/data_stream/indicator/agent/stream/cel.yml.hbs Outdated Show resolved Hide resolved

packages/ti_custom/data_stream/indicator/agent/stream/cel.yml.hbs Outdated Show resolved Hide resolved

chemamartinez added 2 commits October 3, 2024 09:48

Merge branch 'main' into 11192-custom_ti-basic-auth

0131597

Simplify CEL conditions

4a0e8c0

chemamartinez requested a review from efd6 October 3, 2024 07:58

efd6 approved these changes Oct 3, 2024

View reviewed changes

chemamartinez merged commit 07a9442 into elastic:main Oct 3, 2024
5 checks passed

nicpenning mentioned this pull request Oct 3, 2024

[ti_custom]: 406 Not Acceptable - Integration won't return any indicators #11324

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[custom_ti] Add support for basic authentication #11202

[custom_ti] Add support for basic authentication #11202

chemamartinez commented Sep 20, 2024 •

edited

Loading

nicpenning commented Sep 20, 2024

nicpenning commented Sep 21, 2024

chemamartinez commented Oct 2, 2024

elastic-vault-github-plugin-prod bot commented Oct 2, 2024

elasticmachine commented Oct 2, 2024

nicpenning commented Oct 2, 2024

elasticmachine commented Oct 3, 2024

elastic-sonarqube bot commented Oct 3, 2024

elastic-vault-github-plugin-prod bot commented Oct 3, 2024

[custom_ti] Add support for basic authentication #11202

[custom_ti] Add support for basic authentication #11202

Conversation

chemamartinez commented Sep 20, 2024 • edited Loading

Proposed commit message

Checklist

Related issues

Screenshots

nicpenning commented Sep 20, 2024

nicpenning commented Sep 21, 2024

chemamartinez commented Oct 2, 2024

elastic-vault-github-plugin-prod bot commented Oct 2, 2024

🚀 Benchmarks report

elasticmachine commented Oct 2, 2024

nicpenning commented Oct 2, 2024

elasticmachine commented Oct 3, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Oct 3, 2024

Quality Gate passed

elastic-vault-github-plugin-prod bot commented Oct 3, 2024

chemamartinez commented Sep 20, 2024 •

edited

Loading