[CM] Event Stream service #153211

vadimkibana · 2023-03-14T18:35:25Z

Summary

This PR implements the Event Stream service for Content Management.

For high-level overview see:

Event Stream technical summary (a bit old, but still good as general overview read)

Implementation details in this PR:

This PR introduces the EventStreamService high-level class, which is the public interface to the Event Stream, holds any necessary state, and follows plugin life-cycle methods.
On a lower level the actual event storage is defined in the EventStreamClient interface.
There are two EventStreamClient implementations:
- EsEventStreamClient is the production implementation, which stores events to Elasticsearch.
- MemoryEventStreamClient is used for testing and could be used for demo purposes.
The same test suite testEventStreamClient is reused for EsEventStreamClient and MemoryEventStreamClient, which should help with verifying that both implements work correctly and the same. For EsEventStreamClient it is executed as Kibana integration test, but for MemoryEventStreamClient it is executed as a Jest test.
In EventStreamService events are buffered for 250ms or up to 100 events before they are flushed to the storage.
Events are stored in the .kibana-event-stream data stream.
The data stream and index template are create during plugin initialization "start" life-cycle, similar to how it is done in the Event Log and in the Reporting index.
The mappings define a meta field, which is currently unused, but will allow to add more fields in the future without needing to change the schema of the data stream.
The mappings define a transaction ID txId field, which can be used to correlate multiple related events together or to store the transaction ID.
Events are written to Elasticsearch using the _bulk request API.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2023-03-15T07:45:16Z

Pinging @elastic/appex-sharedux (Team:SharedUX)

src/plugins/content_management/server/core/core.ts

src/plugins/content_management/server/event_stream/event_stream_service.ts

src/plugins/content_management/server/event_stream/memory/memory_event_stream_client.test.ts

src/plugins/content_management/server/event_stream/types.ts

sebelga

Amazing work @vadimkibana ! Overall looks great, I left a few questions but no blockers. Great test coverage 👍

src/plugins/content_management/server/core/core.ts

src/plugins/content_management/server/event_stream/memory/util.ts

src/plugins/content_management/server/core/core.ts

src/plugins/content_management/server/event_stream/es/init/mappings.ts

src/plugins/content_management/server/event_stream/es/es_event_stream_client.ts

...s/content_management/server/event_stream/es/integration_tests/es_event_stream_client.test.ts

src/plugins/content_management/server/event_stream/tests/util.ts

src/plugins/content_management/server/event_stream/memory/memory_event_stream_client.test.ts

src/plugins/content_management/server/event_stream/event_stream_service.ts

vadimkibana · 2023-03-20T11:11:10Z

@elasticmachine merge upstream

lukasolson

DataDiscovery changes LGTM (code review only), suggested tests.

packages/kbn-es-query/src/kuery/node_types/node_builder.ts

vadimkibana · 2023-03-21T13:32:36Z

Maybe take a look at ES client retry logic, maybe disable. Maybe idempotent id generate on client.

vadimkibana · 2023-03-24T17:19:34Z

@elasticmachine merge upstream

…-ref HEAD~1..HEAD --fix'

vadimkibana · 2023-03-30T11:53:50Z

@elasticmachine merge upstream

pmuellr

LGTM, thanks for the changes!

azasypkin

LGTM

Dosant · 2023-04-03T09:39:08Z

@elasticmachine merge upstream

…vent-stream-6

Dosant · 2023-04-03T14:41:19Z

@elasticmachine merge upstream

Dosant · 2023-04-04T09:12:34Z

@elasticmachine merge upstream

Dosant · 2023-04-04T15:52:24Z

@elasticmachine merge upstream

kibana-ci · 2023-04-05T09:28:41Z

💚 Build Succeeded

Buildkite Build
Commit: e2b14b4

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/es-query`	193	197	+4
`bfetch`	74	75	+1
total			+5

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`bfetch`	6.7KB	6.9KB	+128.0B
`kbnUiSharedDeps-srcJs`	2.2MB	2.2MB	+136.0B
total			+264.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/es-query`	251	255	+4
`bfetch`	89	91	+2
total			+6

ESLint disabled line counts

id	before	after	diff
`securitySolution`	432	435	+3

Total ESLint disabled count

id	before	after	diff
`securitySolution`	512	515	+3

History

💔 Build #117746 failed f6f8575
💚 Build #117618 succeeded ca22ed4
💚 Build #117421 succeeded af53ac9
💔 Build #117302 failed 8aafe33
💔 Build #116715 failed 8b907d0

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @Dosant @vadimkibana

## Summary This PR implements the Event Stream service for Content Management. For high-level overview see: - [Event Stream technical summary](https://docs.google.com/document/d/1nyMhb0p4gNV43OVF6cLJkxhMf2V4V1BIjPsnACxe_t0/edit#heading=h.typ7x7sxmeye) (a bit old, but still good as general overview read) Implementation details in this PR: - This PR introduces the `EventStreamService` high-level class, which is the public interface to the Event Stream, holds any necessary state, and follows plugin life-cycle methods. - On a lower level the actual event storage is defined in the `EventStreamClient` interface. - There are two `EventStreamClient` implementations: - `EsEventStreamClient` is the production implementation, which stores events to Elasticsearch. - `MemoryEventStreamClient` is used for testing and could be used for demo purposes. - The same test suite `testEventStreamClient` is reused for `EsEventStreamClient` and `MemoryEventStreamClient`, which should help with verifying that both implements work correctly and the same. For `EsEventStreamClient` it is executed as Kibana integration test, but for `MemoryEventStreamClient` it is executed as a Jest test. - In `EventStreamService` events are buffered for 250ms or up to 100 events before they are flushed to the storage. - Events are stored in the `.kibana-event-stream` data stream. - The data stream and index template are create during plugin initialization "start" life-cycle, similar to how it is done in the Event Log and in the Reporting index. - The mappings define a `meta` field, which is currently unused, but will allow to add more fields in the future without needing to change the schema of the data stream. - The mappings define a transaction ID `txId` field, which can be used to correlate multiple related events together or to store the transaction ID. - Events are written to Elasticsearch using the `_bulk` request API. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Aleh Zasypkin <[email protected]> Co-authored-by: Anton Dosov <[email protected]>

vadimkibana changed the title ~~Event stream 6~~ [CM] Event Stream service Mar 15, 2023

streamich force-pushed the event-stream-6 branch from 686d8bf to 97eb918 Compare March 15, 2023 07:40

vadimkibana added review release_note:skip Skip the PR/issue when compiling release notes Team:SharedUX Team label for AppEx-SharedUX (formerly Global Experience) Feature:Content Management User generated content (saved objects) management v8.8.0 labels Mar 15, 2023

vadimkibana self-assigned this Mar 15, 2023

vadimkibana marked this pull request as ready for review March 15, 2023 07:45

vadimkibana requested review from a team as code owners March 15, 2023 07:45

vadimkibana requested review from sebelga and Dosant March 15, 2023 07:45

vadimkibana commented Mar 15, 2023

View reviewed changes

sebelga approved these changes Mar 15, 2023

View reviewed changes

streamich force-pushed the event-stream-6 branch from 596fc9a to 7424307 Compare March 17, 2023 11:09

vadimkibana requested a review from a team March 17, 2023 11:10

event stream

f254c6c

streamich force-pushed the event-stream-6 branch from 0055e78 to f254c6c Compare March 20, 2023 09:03

Merge branch 'main' into event-stream-6

3434201

lukasolson approved these changes Mar 20, 2023

View reviewed changes

packages/kbn-es-query/src/kuery/node_types/node_builder.ts Show resolved Hide resolved

kibanamachine and others added 5 commits March 24, 2023 13:19

Merge branch 'main' into event-stream-6

91caf62

add .range tests

fc9a791

flush event stream buffer on plugin stop

e5f101a

disable ES retries

91ae129

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

f013ad7

…-ref HEAD~1..HEAD --fix'

vadimkibana added 4 commits March 30, 2023 13:07

do not track total hits

dc544fc

fix pagination in in-memory client implementation

feff975

check data stream returns consistent error

a81947f

remove test todos

ebb2469

Merge branch 'main' into event-stream-6

8b907d0

vadimkibana requested review from azasypkin and pmuellr March 30, 2023 11:54

Dosant self-assigned this Mar 30, 2023

pmuellr approved these changes Mar 30, 2023

View reviewed changes

azasypkin approved these changes Mar 31, 2023

View reviewed changes

fix lint

57c1c68

kibanamachine and others added 5 commits April 3, 2023 05:39

Merge branch 'main' into event-stream-6

8aafe33

don’t crash if no events

b05e2f1

Merge branch 'event-stream-6' of github.com:vadimkibana/kibana into e…

8b51d0a

…vent-stream-6

Merge branch 'main' of github.com:elastic/kibana into event-stream-6

33b94bf

handle stop error

eb3dd05

Merge branch 'main' into event-stream-6

af53ac9

Merge branch 'main' into event-stream-6

ca22ed4

kibanamachine and others added 3 commits April 4, 2023 11:52

Merge branch 'main' into event-stream-6

f6f8575

Merge branch 'main' of github.com:elastic/kibana into event-stream-6

53bd2a7

fix ts

e2b14b4

Dosant merged commit f442a0d into elastic:main Apr 5, 2023

kibanamachine added the backport:skip This commit does not require backporting label Apr 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CM] Event Stream service #153211

[CM] Event Stream service #153211

vadimkibana commented Mar 14, 2023 •

edited

Loading

elasticmachine commented Mar 15, 2023

sebelga left a comment

vadimkibana commented Mar 20, 2023

lukasolson left a comment

vadimkibana commented Mar 21, 2023 •

edited

Loading

vadimkibana commented Mar 24, 2023

vadimkibana commented Mar 30, 2023

pmuellr left a comment

azasypkin left a comment

Dosant commented Apr 3, 2023

Dosant commented Apr 3, 2023

Dosant commented Apr 4, 2023

Dosant commented Apr 4, 2023

kibana-ci commented Apr 5, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

[CM] Event Stream service #153211

[CM] Event Stream service #153211

Conversation

vadimkibana commented Mar 14, 2023 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Mar 15, 2023

sebelga left a comment

Choose a reason for hiding this comment

vadimkibana commented Mar 20, 2023

lukasolson left a comment

Choose a reason for hiding this comment

vadimkibana commented Mar 21, 2023 • edited Loading

vadimkibana commented Mar 24, 2023

vadimkibana commented Mar 30, 2023

pmuellr left a comment

Choose a reason for hiding this comment

azasypkin left a comment

Choose a reason for hiding this comment

Dosant commented Apr 3, 2023

Dosant commented Apr 3, 2023

Dosant commented Apr 4, 2023

Dosant commented Apr 4, 2023

kibana-ci commented Apr 5, 2023

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

vadimkibana commented Mar 14, 2023 •

edited

Loading

vadimkibana commented Mar 21, 2023 •

edited

Loading