Osquerybeat: Add action responses data stream #39143

aleksmaus · 2024-04-22T17:49:39Z

Proposed commit message

Add action responses data stream: logs-osquery_manager.action.responses-default

This allows osquerybeat to post the actions responses directly to elasticsearch and fix the issues with the current the transform job based approach, where the actions results could be lost at scale, and presently there is no better solution to address this at the elasticsearch stack level.

For more details check this ticket:
https://github.com/elastic/security-team/issues/8893

This change is also sets up things to be able to handle client-side processors correctly per stream as needed, the issue ticket https://github.com/elastic/security-team/issues/9041. Kibana side configuration UI needs to be updated in order to support this feature.

This change is backwards compatible and works with the policy before the osquery_manager package update elastic/integrations#9661
The action response document will be sent to logs-osquery_manager.action.responses-default only when this new integration package is installed, and the osquery_manager.action.responses stream becomes available in the policy.

Since the results are now posted into the proper logs-osquery_manager.action.responses-default datastream, Kibana would need to be adjusted to use it instead of the currently used index .logs-osquery_manager.action.responses-default

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Full regression tests need to be performed:

Test new osquerybeat with the stack and older integration package, with packs and no packs, live queries, custom osquery configuration.
Test new osquerybeat with the stack and the new integration package, with packs and no packs, live queries, custom osquery configuration.

The new action responses will be created in the new datastream

GET logs-osquery_manager.action.responses-default/_search

Related issues

Screenshots

The new logs-osquery_manager.action.responses-default document example:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-logs-osquery_manager.action.responses-default-2024.04.22-000001",
        "_id": "mmpnBo8BWSVhX2NCTq2S",
        "_score": 1,
        "_source": {
          "agent": {
            "name": "mi6",
            "id": "33a86aa3-4932-4846-8125-6ef996d8b2e7",
            "type": "osquerybeat",
            "ephemeral_id": "670ae42f-cbcb-49c2-8214-4d9bd401f55a",
            "version": "8.13.1"
          },
          "agent_id": "33a86aa3-4932-4846-8125-6ef996d8b2e7",
          "elastic_agent": {
            "id": "33a86aa3-4932-4846-8125-6ef996d8b2e7",
            "version": "8.13.1",
            "snapshot": false
          },
          "action_input_type": "osquery",
          "action_data": {
            "query": "select * from osquery_info",
            "id": "c98b474d-1635-45ce-8fd9-9d814ee1dfcc"
          },
          "completed_at": "2024-04-22T15:22:54.306025Z",
          "action_response": {
            "osquery": {
              "count": 1
            }
          },
          "@timestamp": "2024-04-22T15:22:54.306Z",
          "ecs": {
            "version": "8.0.0"
          },
          "action_id": "281f35b4-fffd-4c36-a9f1-bfed002b9f9a",
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "osquery_manager.action.responses"
          },
          "host": {
            "hostname": "mi6",
            "os": {
              "build": "23E224",
              "kernel": "23.4.0",
              "name": "macOS",
              "family": "darwin",
              "type": "macos",
              "version": "14.4.1",
              "platform": "darwin"
            },
            "ip": [
              "fe80::f4d4:88ff:fe6c:4520",
              "fe80::88:cf6:575c:32d5",
              "192.168.50.246",
              "fe80::10c9:d6ff:febc:c723",
              "fe80::10c9:d6ff:febc:c723",
              "fe80::1b7d:a2a5:2d5d:3265",
              "fe80::64a0:4808:f967:c032",
              "fe80::7166:a04c:47bb:2b29",
              "fe80::ce81:b1c:bd2c:69e"
            ],
            "name": "mi6",
            "id": "70A8C8D7-AF7E-5BCB-BF12-E2B381A5B1AC",
            "mac": [
              "12-C9-D6-BC-C7-23",
              "36-35-D5-EB-20-80",
              "36-35-D5-EB-20-84",
              "36-35-D5-EB-20-88",
              "CA-B1-D3-08-D0-0F",
              "CA-B1-D3-08-D0-10",
              "CA-B1-D3-08-D0-11",
              "CA-B1-D3-08-D0-2F",
              "CA-B1-D3-08-D0-30",
              "CA-B1-D3-08-D0-31",
              "F4-D4-88-6C-45-20",
              "F6-D4-88-6C-45-20"
            ],
            "architecture": "arm64"
          },
          "started_at": "2024-04-22T15:22:54.100066Z",
          "event": {
            "agent_id_status": "verified",
            "ingested": "2024-04-22T15:23:04Z",
            "dataset": "osquery_manager.action.responses"
          }
        }
      }
    ]
  }
}

mergify · 2024-04-22T17:50:19Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @aleksmaus? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-04-22T17:52:39Z

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

elasticmachine · 2024-04-22T18:01:51Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 58 min 5 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

pkoutsovasilis

I am not an osquery beat expert but in terms of go code this lgtm. I would vote for a Changelog entry in CHANGELOG.next.asciidoc

mjwolf · 2024-05-14T18:04:08Z

x-pack/osquerybeat/internal/pub/publisher_test.go

+		},
+	}
+
+	for _, tc := range tests {


If you use T.Run() here, it will have some advantages, each test case would be run and return results separately.

Ooops, sorry, saw the approval, merged, but missed these comments. Will open a follow up PR to address this.

mjwolf · 2024-05-14T18:05:28Z

x-pack/osquerybeat/internal/pub/publisher_test.go

+				"count": 1,
+				"started_at": "2024-04-18T19:39:39.532125Z"
+			} `),
+			// "agent_id": "bf3d6036-2260-4bbf-94a3-5ccce0d75d9e",


I think you could remove agent_id if it's not needed

tomsonpl

Thanks @aleksmaus 🥇
I looked through the code, and seems logical to me :) However, you know that I do not know go :) Would you like me to test this within kibana, or you already tested everything?

aleksmaus · 2024-05-15T12:30:02Z

@tomsonpl @szwarckonrad Please test this with corresponding integration change, that is not merged yet.
In order to fix the issue completely you would need to make some Kibana side changes in order to to query logs-osquery_manager.action.responses-default datastream for the results if the new integration is installed.

I will be holding off on integration PR (elastic/integrations#9661) merge for now until you adjust Kibana and confirm that everything works for you as expected. Please reach out on Slack if you have questions.

…-actions * upstream/main: (313 commits) github-action: delete opentelemetry workflow (elastic#39559) updatecli: move to the .github folder and support for signed commits (elastic#39472) Osquerybeat: Add action responses data stream (elastic#39143) [winlogbeat] performance improvment; avoid rendering event message twice (elastic#39544) Fix the AWS SDK dependencies issue causing the "not found, ResolveEndpointV2" error (elastic#39454) x-pack/filebeat/input/cel: add http metrics collection (elastic#39503) build(deps): bump github.com/elastic/elastic-agent-libs from 0.9.4 to 0.9.7 (elastic#39424) Remove unused env vars from pipelines (elastic#39534) [BK] - Remove osx steps from branch execution (elastic#39552) [BK] - Remove certain steps from running for Branches (elastic#39533) Allow dependabot report BK status checks (elastic#39540) Remove hardcoded module definitions in CI (elastic#39506) Explicitly set DOCKER_PULL, RACE_DETECTOR and TEST_COVERAGE for pipelines (elastic#39510) Fixed pipelines formatting (elastic#39513) Update filebeat pipeline to match Jenkins steps (elastic#39261) Add error check to groupToEvents so we don't blindly add error values (elastic#39404) Remove fields not needed for session view in add_session_view processor (elastic#39500) `aws-s3` input: Split S3 poller and SQS reader into explicit input objects (elastic#39353) ci(jenkins): remove post-build notifications (elastic#39483) [DOCS] Add the `read_pipeline` cluster privilege for winlogbeat and the `auto_configure` index privilege to beats documentation (elastic#38534) ...

Add action responses data stream

e377ef9

aleksmaus added the enhancement label Apr 22, 2024

aleksmaus requested review from tomsonpl and szwarckonrad April 22, 2024 17:49

aleksmaus requested a review from a team as a code owner April 22, 2024 17:49

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 22, 2024

mergify bot assigned aleksmaus Apr 22, 2024

aleksmaus added backport-skip Skip notification from the automated backport with mergify Team:Security-Deployment and Devices Deployment and Devices Team in Security Solution labels Apr 22, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 22, 2024

Add missing copyright header

c426cb0

aleksmaus added 7 commits April 22, 2024 14:05

Make linter happy

6706c22

Linting

e193afd

Merge branch 'main' into feature/osquery_action_responses

252db5c

Merge branch 'main' into feature/osquery_action_responses

58567c5

Fix the compatibility with older stack

177ec88

Merge branch 'main' into feature/osquery_action_responses

520c712

Merge branch 'main' into feature/osquery_action_responses

72762aa

pkoutsovasilis approved these changes May 13, 2024

View reviewed changes

mjwolf approved these changes May 14, 2024

View reviewed changes

tomsonpl approved these changes May 14, 2024

View reviewed changes

aleksmaus changed the title ~~Add action responses data stream~~ Osquerybeat: Add action responses data stream May 15, 2024

aleksmaus merged commit 86ef638 into elastic:main May 15, 2024
23 checks passed

aleksmaus mentioned this pull request May 15, 2024

Osquerybeat: Follow up with changelog and unit test tweak for "Add action responses data stream PR" #39577

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Osquerybeat: Add action responses data stream #39143

Osquerybeat: Add action responses data stream #39143

aleksmaus commented Apr 22, 2024 •

edited

Loading

mergify bot commented Apr 22, 2024

elasticmachine commented Apr 22, 2024

elasticmachine commented Apr 22, 2024 •

edited

Loading

Build stats

pkoutsovasilis left a comment •

edited

Loading

mjwolf May 14, 2024

aleksmaus May 15, 2024

mjwolf May 14, 2024

tomsonpl left a comment

aleksmaus commented May 15, 2024 •

edited

Loading

Osquerybeat: Add action responses data stream #39143

Osquerybeat: Add action responses data stream #39143

Conversation

aleksmaus commented Apr 22, 2024 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

mergify bot commented Apr 22, 2024

elasticmachine commented Apr 22, 2024

elasticmachine commented Apr 22, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

pkoutsovasilis left a comment • edited Loading

Choose a reason for hiding this comment

mjwolf May 14, 2024

Choose a reason for hiding this comment

aleksmaus May 15, 2024

Choose a reason for hiding this comment

mjwolf May 14, 2024

Choose a reason for hiding this comment

tomsonpl left a comment

Choose a reason for hiding this comment

aleksmaus commented May 15, 2024 • edited Loading

aleksmaus commented Apr 22, 2024 •

edited

Loading

elasticmachine commented Apr 22, 2024 •

edited

Loading

pkoutsovasilis left a comment •

edited

Loading

aleksmaus commented May 15, 2024 •

edited

Loading