[MD]Enable data source audit log to file #2215

kristenTian · 2022-08-29T20:00:21Z

Description

Enable data source audit log to file

Default log file name:

/tmp/opensearch-dashboards-data-source-audit.log

The example log message in file:

[2022-08-29T19:46:49.882Z][INFO ][�[35mplugins.dataSource.audit�[39m]{"type":"opensearch.dataSourceClient.call.internalUser"} 127.0.0.1 accessing through 127.0.0.1 on data source: 6d9a33e0-1f3a-11ed-b2d8-a74b67b7c8dc

User could also update appender through config yaml like so:

data_source.audit.appender.kind: 'file'
data_source.audit.appender.layout.kind: 'pattern'
data_source.audit.appender.path: <path>

Tested when disabled, will not generate log file
data_source.audit.eanbled = false

Issues Resolved

#1986

Check List

Commits are signed per the DCO using --signoff

src/plugins/data_source/server/audit/scopted_audit_client.ts

src/plugins/data_source/config.ts

src/plugins/data_source/server/audit/scopted_audit_client.ts

src/plugins/data_source/server/plugin.ts

src/plugins/data_source/server/audit/logging_auditor.ts

src/plugins/data_source/server/data_source_service.ts

zengyan-amazon · 2022-08-30T07:17:31Z

src/plugins/data_source/server/plugin.ts

+              const auditor = auditTrailPromise.then((auditTrail) => auditTrail.asScoped(req));
+
              return dataSourceService.getDataSourceClient(
                dataSourceId,
                context.core.savedObjects.client,
-                cryptographyClient
+                cryptographyClient,
+                auditor


is auditTrailPromise.then block and guarantee auditor is initialized/created when calling dataSourceService.getDataSourceClient on line 136? Or we will need to use await?

meanwhile, shall we declare getClient as async ?

And how about just log it here in this getClient function instead of in dataSourceService ?

no, it will not block unless we use await;

not need to, it already returns promise

make sense to me so request can be private with current implementation.-- future maybe use observable.

I also think auditor should be pass down to the configure_client. Because we'll log more than datasourceID, we'll probably log username, query(emitted by client), error, all of those are retrieved from deeper level.

As u mentioned, this should be separate PR/task. -- As audit query still need further investigation, l'd just keep the current position and iterate in future. subtask: #2228

zhongnansu

Left some comments.
I see there are still things missing. For example, client emitter, how to log query, error, etc. It doesn't have to be done all in one PR, but could you add/create an issue/task-break down.

src/plugins/data_source/server/plugin.ts

zhongnansu · 2022-08-30T19:28:09Z

src/plugins/data_source/server/audit/logging_auditor.ts

+
+  public withAuditScope(name: string) {}
+
+  public add(event: AuditableEvent) {


I think we need more than type and message in the log line. For example, username or credential id?

for now let's leave that. need security plugin's support to fully enable the log for authorization related stuff.

zhongnansu · 2022-08-30T19:37:09Z

src/plugins/data_source/server/plugin.ts

+              const auditor = auditTrailPromise.then((auditTrail) => auditTrail.asScoped(req));
+
              return dataSourceService.getDataSourceClient(
                dataSourceId,
                context.core.savedObjects.client,
-                cryptographyClient
+                cryptographyClient,
+                auditor


I also think auditor should be pass down to the configure_client. Because we'll log more than datasourceID, we'll probably log username, query(emitted by client), error, all of those are retrieved from deeper level.

src/plugins/data_source/config.ts

zhongnansu · 2022-08-30T19:51:19Z

src/plugins/data_source/server/plugin.ts

+
+    auditor.add({
+      message: auditMessage,
+      type: 'opensearch.dataSourceClient.call.internalUser',


this doesn't make sense, it's always current user, not internal user. Type here should by dynamic, possible values for example creation, access, change, deletion. reference: https://www.elastic.co/guide/en/kibana/7.11/xpack-security-audit-logging.html#field-event-type

Or come up with some better candidates

good catch, going to update the string , but limiting the scope for this pr to be some static string for now. --e.g. opensearch.dataSourceClient.fetchClient

zhongnansu · 2022-08-30T19:52:56Z

src/plugins/data_source/server/plugin.ts

+  private getAuditMessage(request: OpenSearchDashboardsRequest, dataSourceId: string) {
+    const rawRequest = ensureRawRequest(request);
+    const remoteAddress = rawRequest?.info?.remoteAddress;
+    const xForwardFor = request.headers['x-forwarded-for'];


why is remoteAddress and x-forwarded-for needed ?

to log the requester's info, later will be utilized by security plugin

Signed-off-by: Kristen Tian <[email protected]>

src/plugins/data_source/server/plugin.ts

Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]>

* Instantiate credential management plugin code structure (#1996) Signed-off-by: Kristen Tian <[email protected]> * Data source inside stack management setup (#2017) (#2030) Signed-off-by: Kristen Tian <[email protected]> * enable CI for feature branch (#2010) Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add empty data source plugin (#2052) Adds empty data source plugin. Signed-off-by: Kristen Tian <[email protected]> * [MD] Add initial credential management CRUD pages (#2040) * Add credential management CRUD pages 1. List all credentials 2. Create your saved credential 3. Edit your credential 4. Delete credentials Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Register Data source to savedObject & Update license header (#2037) Signed-off-by: Kristen Tian <[email protected]> * Move credential saved object to data source plugin (#2062) Signed-off-by: Louis Chu <[email protected]> Move credential saved object to data source plugin Resolve follow up comments on UI Signed-off-by: Kristen Tian <[email protected]> * breadcrumbfix for datasource management (#2066) * breadcrumbfix for datasource management Signed-off-by: mpabba3003 <[email protected]> * breadcrumbfix for datasource management - refactoring code Signed-off-by: mpabba3003 <[email protected]> * using services to update breadcrumb on data sources management page Signed-off-by: mpabba3003 <[email protected]> * Changing the license header on breadcrumbs.ts datasource management Signed-off-by: Kristen Tian <[email protected]> * Fix breadcrumb on listing page and update saved object mapping (#2069) Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add data source step into IndexPattern with Mock switch (#2064) (#2086) Signed-off-by: Kristen Tian <[email protected]> * Add delete button for credential detailed page (#2067) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Link datasource to indexpattern (#2118) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add encrypt/decrypt module on data source plugin (#2120) Signed-off-by: Louis Chu <[email protected]> 1. Add encrypt/decrypt module with UT 2. Add client factory wrapper for encrypt credential 3. Add encryption config support 4. Bugfix on Jest interpret Buffer Signed-off-by: Kristen Tian <[email protected]> * Integrate index pattern with new data client (#2146) Signed-off-by: Kristen Tian <[email protected]> * Add noAuth to dataSource attributes (#2154) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource Management - creation & listing - UI only (#2128) * data source management - creation & Listing UI only * data source management - creation & Listing UI only * Create/edit data source feature * toggling default value * refactoring code as per review comments * toggling server flag to false Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Apply dataSource plugin as dependent for cm and dsm plugins (#2150) Signed-off-by: Louis Chu <[email protected]> Apply dataSource plugin as dependent for cm and dsm plugins (#2150) Signed-off-by: Kristen Tian <[email protected]> * Leverage datasource enablement in index pattern management Signed-off-by: Kristen Tian <[email protected]> * [MD] Add client management module and register `getClient()` to route handler context (#2121) * Add client management module and register `getClient()` interface to route handler context Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Make step info in index pattern creation dynamic (#2164) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Refactor for credential listing page & add loading effect (#2142) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Integration sequal - replace data client placeholders (#2167) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Integrate with crypto module to decrpt password (#2170) Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] noAuth integration, credential & endpoint validation (#2165) * noAuth integration, credential & endpoint validation Signed-off-by: mpabba3003 <[email protected]> * Refactoring validation message Signed-off-by: mpabba3003 <[email protected]> * Adding back accidentally deleted file home/tutorials/haproxy_metrics/index.ts Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Refactor search strategy to conditionally user datasource client (#2171) Signed-off-by: Kristen Tian <[email protected]> * adding relation between credential selection and no auth checkbox (#2175) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update getIndicesViaSearch with datasource (#2176) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add null header to child client spawn (#2188) Signed-off-by: Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD][IP]Update button position & Add UT & Add skip option (#2195) * Add UT - index pattern related Signed-off-by: Kristen Tian <[email protected]> * Update button position Signed-off-by: Kristen Tian <[email protected]> * Add skip option to allow use default os data source Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD]Refactor layout and validate input fields for listing and create pages (#2202) Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Enable datasource link in saveObjectManagement (#2209) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update configure data source per UX input (#2235) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Enable data source audit log to file (#2215) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Refactor of credential editing page layout & refactor backend field validation method (#2222) * Refactor of credential editing page layout & refactor backend field validation method * Resolved the comments & fix the multiple call for one operation Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Tweak fetch data back to original (#2238) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Revamped UX for data source management (#2239) * revamped UX for data source management Signed-off-by: mpabba3003 <[email protected]> * refactored datasource screens as per PR comments Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Credentials security redesign (#2253) 1. Data model changes for data source saved object 2. Server side changes for data source saved object a. Implement data_source_saved_objects_client_wrapper to integrate with CryptographyClient for password encryption / decryption. b. Change data_source_service to fetch credentials directly from data source (still decrypt via CryptographyClient) c. Fix unit tests accordingly Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add step data source UI test (#2264) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD]Improve datasource server side error handling (#2236) Signed-off-by: Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource management new API change integrations (#2282) * changing datasource management design to integrate with new API changes Signed-off-by: mpabba3003 <[email protected]> * Update edit_data_source_form.tsx moving masked password to constants Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update stream test to bypass CI env generate domain attribute Signed-off-by: Kristen Tian <[email protected]> * Delete credential management Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource Management - Create data source - Unit tests (#2341) * Unit test cases for data source management - create Signed-off-by: mpabba3003 <[email protected]> * adding tests to utils.ts & changing it to test Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Fix update data source & block update endpint (#2364) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * MD datasource management-datasource table-UTs (#2350) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * reafctor based on PR 2334 comments to merge to main (#2375) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * removing invalid urls as CI fails on link checker for inavlid urls in git (#2376) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * Fix DS snapshot test Signed-off-by: Kristen Tian <[email protected]> * Add https://test.com/ to lychee exclude Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * Remove unnessacry check Signed-off-by: Kristen Tian <[email protected]> * Remove not needed check Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Su <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Zhongnan Su <[email protected]> Co-authored-by: Yan Zeng <[email protected]> Co-authored-by: Manideep Pabba <[email protected]> Co-authored-by: Yibo Wang <[email protected]>

* Instantiate credential management plugin code structure (opensearch-project#1996) Signed-off-by: Kristen Tian <[email protected]> * Data source inside stack management setup (opensearch-project#2017) (opensearch-project#2030) Signed-off-by: Kristen Tian <[email protected]> * enable CI for feature branch (opensearch-project#2010) Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add empty data source plugin (opensearch-project#2052) Adds empty data source plugin. Signed-off-by: Kristen Tian <[email protected]> * [MD] Add initial credential management CRUD pages (opensearch-project#2040) * Add credential management CRUD pages 1. List all credentials 2. Create your saved credential 3. Edit your credential 4. Delete credentials Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Register Data source to savedObject & Update license header (opensearch-project#2037) Signed-off-by: Kristen Tian <[email protected]> * Move credential saved object to data source plugin (opensearch-project#2062) Signed-off-by: Louis Chu <[email protected]> Move credential saved object to data source plugin Resolve follow up comments on UI Signed-off-by: Kristen Tian <[email protected]> * breadcrumbfix for datasource management (opensearch-project#2066) * breadcrumbfix for datasource management Signed-off-by: mpabba3003 <[email protected]> * breadcrumbfix for datasource management - refactoring code Signed-off-by: mpabba3003 <[email protected]> * using services to update breadcrumb on data sources management page Signed-off-by: mpabba3003 <[email protected]> * Changing the license header on breadcrumbs.ts datasource management Signed-off-by: Kristen Tian <[email protected]> * Fix breadcrumb on listing page and update saved object mapping (opensearch-project#2069) Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add data source step into IndexPattern with Mock switch (opensearch-project#2064) (opensearch-project#2086) Signed-off-by: Kristen Tian <[email protected]> * Add delete button for credential detailed page (opensearch-project#2067) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Link datasource to indexpattern (opensearch-project#2118) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add encrypt/decrypt module on data source plugin (opensearch-project#2120) Signed-off-by: Louis Chu <[email protected]> 1. Add encrypt/decrypt module with UT 2. Add client factory wrapper for encrypt credential 3. Add encryption config support 4. Bugfix on Jest interpret Buffer Signed-off-by: Kristen Tian <[email protected]> * Integrate index pattern with new data client (opensearch-project#2146) Signed-off-by: Kristen Tian <[email protected]> * Add noAuth to dataSource attributes (opensearch-project#2154) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource Management - creation & listing - UI only (opensearch-project#2128) * data source management - creation & Listing UI only * data source management - creation & Listing UI only * Create/edit data source feature * toggling default value * refactoring code as per review comments * toggling server flag to false Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Louis Chu <[email protected]> Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Kristen Tian <[email protected]> * Leverage datasource enablement in index pattern management Signed-off-by: Kristen Tian <[email protected]> * [MD] Add client management module and register `getClient()` to route handler context (opensearch-project#2121) * Add client management module and register `getClient()` interface to route handler context Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Make step info in index pattern creation dynamic (opensearch-project#2164) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Refactor for credential listing page & add loading effect (opensearch-project#2142) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Integration sequal - replace data client placeholders (opensearch-project#2167) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Integrate with crypto module to decrpt password (opensearch-project#2170) Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] noAuth integration, credential & endpoint validation (opensearch-project#2165) * noAuth integration, credential & endpoint validation Signed-off-by: mpabba3003 <[email protected]> * Refactoring validation message Signed-off-by: mpabba3003 <[email protected]> * Adding back accidentally deleted file home/tutorials/haproxy_metrics/index.ts Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Refactor search strategy to conditionally user datasource client (opensearch-project#2171) Signed-off-by: Kristen Tian <[email protected]> * adding relation between credential selection and no auth checkbox (opensearch-project#2175) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update getIndicesViaSearch with datasource (opensearch-project#2176) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add null header to child client spawn (opensearch-project#2188) Signed-off-by: Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD][IP]Update button position & Add UT & Add skip option (opensearch-project#2195) * Add UT - index pattern related Signed-off-by: Kristen Tian <[email protected]> * Update button position Signed-off-by: Kristen Tian <[email protected]> * Add skip option to allow use default os data source Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD]Refactor layout and validate input fields for listing and create pages (opensearch-project#2202) Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Enable datasource link in saveObjectManagement (opensearch-project#2209) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update configure data source per UX input (opensearch-project#2235) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Enable data source audit log to file (opensearch-project#2215) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Refactor of credential editing page layout & refactor backend field validation method (opensearch-project#2222) * Refactor of credential editing page layout & refactor backend field validation method * Resolved the comments & fix the multiple call for one operation Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Tweak fetch data back to original (opensearch-project#2238) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Revamped UX for data source management (opensearch-project#2239) * revamped UX for data source management Signed-off-by: mpabba3003 <[email protected]> * refactored datasource screens as per PR comments Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Credentials security redesign (opensearch-project#2253) 1. Data model changes for data source saved object 2. Server side changes for data source saved object a. Implement data_source_saved_objects_client_wrapper to integrate with CryptographyClient for password encryption / decryption. b. Change data_source_service to fetch credentials directly from data source (still decrypt via CryptographyClient) c. Fix unit tests accordingly Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add step data source UI test (opensearch-project#2264) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD]Improve datasource server side error handling (opensearch-project#2236) Signed-off-by: Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource management new API change integrations (opensearch-project#2282) * changing datasource management design to integrate with new API changes Signed-off-by: mpabba3003 <[email protected]> * Update edit_data_source_form.tsx moving masked password to constants Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update stream test to bypass CI env generate domain attribute Signed-off-by: Kristen Tian <[email protected]> * Delete credential management Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource Management - Create data source - Unit tests (opensearch-project#2341) * Unit test cases for data source management - create Signed-off-by: mpabba3003 <[email protected]> * adding tests to utils.ts & changing it to test Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Fix update data source & block update endpint (opensearch-project#2364) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * MD datasource management-datasource table-UTs (opensearch-project#2350) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * reafctor based on PR 2334 comments to merge to main (opensearch-project#2375) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * removing invalid urls as CI fails on link checker for inavlid urls in git (opensearch-project#2376) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * Fix DS snapshot test Signed-off-by: Kristen Tian <[email protected]> * Add https://test.com/ to lychee exclude Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * Remove unnessacry check Signed-off-by: Kristen Tian <[email protected]> * Remove not needed check Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Su <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Zhongnan Su <[email protected]> Co-authored-by: Yan Zeng <[email protected]> Co-authored-by: Manideep Pabba <[email protected]> Co-authored-by: Yibo Wang <[email protected]>

* Instantiate credential management plugin code structure (opensearch-project#1996) Signed-off-by: Kristen Tian <[email protected]> * Data source inside stack management setup (opensearch-project#2017) (opensearch-project#2030) Signed-off-by: Kristen Tian <[email protected]> * enable CI for feature branch (opensearch-project#2010) Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add empty data source plugin (opensearch-project#2052) Adds empty data source plugin. Signed-off-by: Kristen Tian <[email protected]> * [MD] Add initial credential management CRUD pages (opensearch-project#2040) * Add credential management CRUD pages 1. List all credentials 2. Create your saved credential 3. Edit your credential 4. Delete credentials Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Register Data source to savedObject & Update license header (opensearch-project#2037) Signed-off-by: Kristen Tian <[email protected]> * Move credential saved object to data source plugin (opensearch-project#2062) Signed-off-by: Louis Chu <[email protected]> Move credential saved object to data source plugin Resolve follow up comments on UI Signed-off-by: Kristen Tian <[email protected]> * breadcrumbfix for datasource management (opensearch-project#2066) * breadcrumbfix for datasource management Signed-off-by: mpabba3003 <[email protected]> * breadcrumbfix for datasource management - refactoring code Signed-off-by: mpabba3003 <[email protected]> * using services to update breadcrumb on data sources management page Signed-off-by: mpabba3003 <[email protected]> * Changing the license header on breadcrumbs.ts datasource management Signed-off-by: Kristen Tian <[email protected]> * Fix breadcrumb on listing page and update saved object mapping (opensearch-project#2069) Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add data source step into IndexPattern with Mock switch (opensearch-project#2064) (opensearch-project#2086) Signed-off-by: Kristen Tian <[email protected]> * Add delete button for credential detailed page (opensearch-project#2067) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Link datasource to indexpattern (opensearch-project#2118) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add encrypt/decrypt module on data source plugin (opensearch-project#2120) Signed-off-by: Louis Chu <[email protected]> 1. Add encrypt/decrypt module with UT 2. Add client factory wrapper for encrypt credential 3. Add encryption config support 4. Bugfix on Jest interpret Buffer Signed-off-by: Kristen Tian <[email protected]> * Integrate index pattern with new data client (opensearch-project#2146) Signed-off-by: Kristen Tian <[email protected]> * Add noAuth to dataSource attributes (opensearch-project#2154) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource Management - creation & listing - UI only (opensearch-project#2128) * data source management - creation & Listing UI only * data source management - creation & Listing UI only * Create/edit data source feature * toggling default value * refactoring code as per review comments * toggling server flag to false Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Louis Chu <[email protected]> Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Kristen Tian <[email protected]> * Leverage datasource enablement in index pattern management Signed-off-by: Kristen Tian <[email protected]> * [MD] Add client management module and register `getClient()` to route handler context (opensearch-project#2121) * Add client management module and register `getClient()` interface to route handler context Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Make step info in index pattern creation dynamic (opensearch-project#2164) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Refactor for credential listing page & add loading effect (opensearch-project#2142) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Integration sequal - replace data client placeholders (opensearch-project#2167) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Integrate with crypto module to decrpt password (opensearch-project#2170) Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] noAuth integration, credential & endpoint validation (opensearch-project#2165) * noAuth integration, credential & endpoint validation Signed-off-by: mpabba3003 <[email protected]> * Refactoring validation message Signed-off-by: mpabba3003 <[email protected]> * Adding back accidentally deleted file home/tutorials/haproxy_metrics/index.ts Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Refactor search strategy to conditionally user datasource client (opensearch-project#2171) Signed-off-by: Kristen Tian <[email protected]> * adding relation between credential selection and no auth checkbox (opensearch-project#2175) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update getIndicesViaSearch with datasource (opensearch-project#2176) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add null header to child client spawn (opensearch-project#2188) Signed-off-by: Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD][IP]Update button position & Add UT & Add skip option (opensearch-project#2195) * Add UT - index pattern related Signed-off-by: Kristen Tian <[email protected]> * Update button position Signed-off-by: Kristen Tian <[email protected]> * Add skip option to allow use default os data source Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD]Refactor layout and validate input fields for listing and create pages (opensearch-project#2202) Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Enable datasource link in saveObjectManagement (opensearch-project#2209) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update configure data source per UX input (opensearch-project#2235) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Enable data source audit log to file (opensearch-project#2215) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Refactor of credential editing page layout & refactor backend field validation method (opensearch-project#2222) * Refactor of credential editing page layout & refactor backend field validation method * Resolved the comments & fix the multiple call for one operation Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Tweak fetch data back to original (opensearch-project#2238) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Revamped UX for data source management (opensearch-project#2239) * revamped UX for data source management Signed-off-by: mpabba3003 <[email protected]> * refactored datasource screens as per PR comments Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Credentials security redesign (opensearch-project#2253) 1. Data model changes for data source saved object 2. Server side changes for data source saved object a. Implement data_source_saved_objects_client_wrapper to integrate with CryptographyClient for password encryption / decryption. b. Change data_source_service to fetch credentials directly from data source (still decrypt via CryptographyClient) c. Fix unit tests accordingly Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add step data source UI test (opensearch-project#2264) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD]Improve datasource server side error handling (opensearch-project#2236) Signed-off-by: Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource management new API change integrations (opensearch-project#2282) * changing datasource management design to integrate with new API changes Signed-off-by: mpabba3003 <[email protected]> * Update edit_data_source_form.tsx moving masked password to constants Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update stream test to bypass CI env generate domain attribute Signed-off-by: Kristen Tian <[email protected]> * Delete credential management Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource Management - Create data source - Unit tests (opensearch-project#2341) * Unit test cases for data source management - create Signed-off-by: mpabba3003 <[email protected]> * adding tests to utils.ts & changing it to test Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Fix update data source & block update endpint (opensearch-project#2364) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * MD datasource management-datasource table-UTs (opensearch-project#2350) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * reafctor based on PR 2334 comments to merge to main (opensearch-project#2375) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * removing invalid urls as CI fails on link checker for inavlid urls in git (opensearch-project#2376) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * Fix DS snapshot test Signed-off-by: Kristen Tian <[email protected]> * Add https://test.com/ to lychee exclude Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * Remove unnessacry check Signed-off-by: Kristen Tian <[email protected]> * Remove not needed check Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Su <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Zhongnan Su <[email protected]> Co-authored-by: Yan Zeng <[email protected]> Co-authored-by: Manideep Pabba <[email protected]> Co-authored-by: Yibo Wang <[email protected]> Signed-off-by: Sergey V. Osipov <[email protected]>

kristenTian added the multiple datasource multiple datasource project label Aug 29, 2022

kristenTian changed the title ~~Enable data source audit log to file~~ [MD]Enable data source audit log to file Aug 29, 2022

zengyan-amazon requested changes Aug 29, 2022

View reviewed changes

kristenTian force-pushed the OSD5 branch 2 times, most recently from a3ae14c to 3f04e07 Compare August 30, 2022 01:45

zengyan-amazon reviewed Aug 30, 2022

View reviewed changes

kristenTian force-pushed the OSD5 branch from 3f04e07 to ee51b69 Compare August 30, 2022 18:04

zhongnansu reviewed Aug 30, 2022

View reviewed changes

Enable data source audit log to file

cc0bf56

Signed-off-by: Kristen Tian <[email protected]>

kristenTian force-pushed the OSD5 branch from ee51b69 to cc0bf56 Compare August 31, 2022 00:17

zengyan-amazon marked this pull request as ready for review August 31, 2022 00:35

zengyan-amazon requested a review from a team as a code owner August 31, 2022 00:35

zengyan-amazon approved these changes Aug 31, 2022

View reviewed changes

src/plugins/data_source/server/plugin.ts Show resolved Hide resolved

zhongnansu approved these changes Aug 31, 2022

View reviewed changes

zhongnansu mentioned this pull request Aug 31, 2022

[MD] Logging and Auditing #1986

Open

3 tasks

kristenTian merged commit bda8df6 into opensearch-project:feature/multi-datasource Aug 31, 2022

kristenTian added a commit to kristenTian/OpenSearch-Dashboards that referenced this pull request Sep 12, 2022

Enable data source audit log to file (opensearch-project#2215)

01e3c85

Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]>

kristenTian added a commit to kristenTian/OpenSearch-Dashboards that referenced this pull request Sep 14, 2022

Enable data source audit log to file (opensearch-project#2215)

9a40b46

Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]>

kristenTian added a commit to kristenTian/OpenSearch-Dashboards that referenced this pull request Sep 14, 2022

Enable data source audit log to file (opensearch-project#2215)

0aabd8f

Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]>

kristenTian added a commit to kristenTian/OpenSearch-Dashboards that referenced this pull request Sep 15, 2022

Enable data source audit log to file (opensearch-project#2215)

144649d

Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MD]Enable data source audit log to file #2215

[MD]Enable data source audit log to file #2215

kristenTian commented Aug 29, 2022 •

edited by zhongnansu

Loading

zengyan-amazon Aug 30, 2022

zengyan-amazon Aug 30, 2022

zengyan-amazon Aug 30, 2022

kristenTian Aug 30, 2022 •

edited

Loading

zhongnansu Aug 30, 2022

kristenTian Aug 30, 2022 •

edited

Loading

zhongnansu left a comment

zhongnansu Aug 30, 2022

kristenTian Aug 30, 2022 •

edited

Loading

zhongnansu Aug 30, 2022

zhongnansu Aug 30, 2022

kristenTian Aug 30, 2022 •

edited

Loading

zhongnansu Aug 30, 2022

kristenTian Aug 30, 2022 •

edited

Loading


		public withAuditScope(name: string) {}

		public add(event: AuditableEvent) {

[MD]Enable data source audit log to file #2215

[MD]Enable data source audit log to file #2215

Conversation

kristenTian commented Aug 29, 2022 • edited by zhongnansu Loading

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kristenTian Aug 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kristenTian Aug 30, 2022 • edited Loading

Choose a reason for hiding this comment

zhongnansu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kristenTian Aug 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kristenTian Aug 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kristenTian Aug 30, 2022 • edited Loading

Choose a reason for hiding this comment

kristenTian commented Aug 29, 2022 •

edited by zhongnansu

Loading

kristenTian Aug 30, 2022 •

edited

Loading

kristenTian Aug 30, 2022 •

edited

Loading

kristenTian Aug 30, 2022 •

edited

Loading

kristenTian Aug 30, 2022 •

edited

Loading

kristenTian Aug 30, 2022 •

edited

Loading