MultiDataSource feature merge (#2334)

* Instantiate credential management plugin code structure (#1996) Signed-off-by: Kristen Tian <[email protected]> * Data source inside stack management setup (#2017) (#2030) Signed-off-by: Kristen Tian <[email protected]> * enable CI for feature branch (#2010) Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add empty data source plugin (#2052) Adds empty data source plugin. Signed-off-by: Kristen Tian <[email protected]> * [MD] Add initial credential management CRUD pages (#2040) * Add credential management CRUD pages 1. List all credentials 2. Create your saved credential 3. Edit your credential 4. Delete credentials Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Register Data source to savedObject & Update license header (#2037) Signed-off-by: Kristen Tian <[email protected]> * Move credential saved object to data source plugin (#2062) Signed-off-by: Louis Chu <[email protected]> Move credential saved object to data source plugin Resolve follow up comments on UI Signed-off-by: Kristen Tian <[email protected]> * breadcrumbfix for datasource management (#2066) * breadcrumbfix for datasource management Signed-off-by: mpabba3003 <[email protected]> * breadcrumbfix for datasource management - refactoring code Signed-off-by: mpabba3003 <[email protected]> * using services to update breadcrumb on data sources management page Signed-off-by: mpabba3003 <[email protected]> * Changing the license header on breadcrumbs.ts datasource management Signed-off-by: Kristen Tian <[email protected]> * Fix breadcrumb on listing page and update saved object mapping (#2069) Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add data source step into IndexPattern with Mock switch (#2064) (#2086) Signed-off-by: Kristen Tian <[email protected]> * Add delete button for credential detailed page (#2067) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Link datasource to indexpattern (#2118) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add encrypt/decrypt module on data source plugin (#2120) Signed-off-by: Louis Chu <[email protected]> 1. Add encrypt/decrypt module with UT 2. Add client factory wrapper for encrypt credential 3. Add encryption config support 4. Bugfix on Jest interpret Buffer Signed-off-by: Kristen Tian <[email protected]> * Integrate index pattern with new data client (#2146) Signed-off-by: Kristen Tian <[email protected]> * Add noAuth to dataSource attributes (#2154) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource Management - creation & listing - UI only (#2128) * data source management - creation & Listing UI only * data source management - creation & Listing UI only * Create/edit data source feature * toggling default value * refactoring code as per review comments * toggling server flag to false Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Apply dataSource plugin as dependent for cm and dsm plugins (#2150) Signed-off-by: Louis Chu <[email protected]> Apply dataSource plugin as dependent for cm and dsm plugins (#2150) Signed-off-by: Kristen Tian <[email protected]> * Leverage datasource enablement in index pattern management Signed-off-by: Kristen Tian <[email protected]> * [MD] Add client management module and register `getClient()` to route handler context (#2121) * Add client management module and register `getClient()` interface to route handler context Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Make step info in index pattern creation dynamic (#2164) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Refactor for credential listing page & add loading effect (#2142) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Integration sequal - replace data client placeholders (#2167) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Integrate with crypto module to decrpt password (#2170) Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] noAuth integration, credential & endpoint validation (#2165) * noAuth integration, credential & endpoint validation Signed-off-by: mpabba3003 <[email protected]> * Refactoring validation message Signed-off-by: mpabba3003 <[email protected]> * Adding back accidentally deleted file home/tutorials/haproxy_metrics/index.ts Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Refactor search strategy to conditionally user datasource client (#2171) Signed-off-by: Kristen Tian <[email protected]> * adding relation between credential selection and no auth checkbox (#2175) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update getIndicesViaSearch with datasource (#2176) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add null header to child client spawn (#2188) Signed-off-by: Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD][IP]Update button position & Add UT & Add skip option (#2195) * Add UT - index pattern related Signed-off-by: Kristen Tian <[email protected]> * Update button position Signed-off-by: Kristen Tian <[email protected]> * Add skip option to allow use default os data source Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD]Refactor layout and validate input fields for listing and create pages (#2202) Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Enable datasource link in saveObjectManagement (#2209) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update configure data source per UX input (#2235) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Enable data source audit log to file (#2215) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Refactor of credential editing page layout & refactor backend field validation method (#2222) * Refactor of credential editing page layout & refactor backend field validation method * Resolved the comments & fix the multiple call for one operation Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Tweak fetch data back to original (#2238) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Revamped UX for data source management (#2239) * revamped UX for data source management Signed-off-by: mpabba3003 <[email protected]> * refactored datasource screens as per PR comments Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Credentials security redesign (#2253) 1. Data model changes for data source saved object 2. Server side changes for data source saved object a. Implement data_source_saved_objects_client_wrapper to integrate with CryptographyClient for password encryption / decryption. b. Change data_source_service to fetch credentials directly from data source (still decrypt via CryptographyClient) c. Fix unit tests accordingly Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Add step data source UI test (#2264) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD]Improve datasource server side error handling (#2236) Signed-off-by: Su <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource management new API change integrations (#2282) * changing datasource management design to integrate with new API changes Signed-off-by: mpabba3003 <[email protected]> * Update edit_data_source_form.tsx moving masked password to constants Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Update stream test to bypass CI env generate domain attribute Signed-off-by: Kristen Tian <[email protected]> * Delete credential management Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * [MD] Datasource Management - Create data source - Unit tests (#2341) * Unit test cases for data source management - create Signed-off-by: mpabba3003 <[email protected]> * adding tests to utils.ts & changing it to test Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Fix update data source & block update endpint (#2364) Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * MD datasource management-datasource table-UTs (#2350) Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * reafctor based on PR 2334 comments to merge to main (#2375) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * removing invalid urls as CI fails on link checker for inavlid urls in git (#2376) Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * Fix DS snapshot test Signed-off-by: Kristen Tian <[email protected]> * Add https://test.com/ to lychee exclude Signed-off-by: Kristen Tian <[email protected]> * Address comments Signed-off-by: Kristen Tian <[email protected]> * Remove unnessacry check Signed-off-by: Kristen Tian <[email protected]> * Remove not needed check Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Kristen Tian <[email protected]> Signed-off-by: Zhongnan Su <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Yibo Wang <[email protected]> Signed-off-by: mpabba3003 <[email protected]> Signed-off-by: Su <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Zhongnan Su <[email protected]> Co-authored-by: Yan Zeng <[email protected]> Co-authored-by: Manideep Pabba <[email protected]> Co-authored-by: Yibo Wang <[email protected]>
opensearch-project · Sep 19, 2022 · 890a4b6 · 890a4b6
1 parent b5d529a
commit 890a4b6
Show file tree

Hide file tree

Showing 142 changed files with 16,133 additions and 154 deletions.
diff --git a/.github/workflows/build_and_test_workflow.yml b/.github/workflows/build_and_test_workflow.yml
@@ -38,7 +38,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v2
         with:
-          node-version-file: ".nvmrc"
+          node-version-file: '.nvmrc'
           registry-url: 'https://registry.npmjs.org'
 
       - name: Setup Yarn
@@ -77,7 +77,7 @@ jobs:
       image: docker://opensearchstaging/ci-runner:ci-runner-rockylinux8-opensearch-dashboards-integtest-v2
       options: --user 1001
     name: Run functional tests
-    strategy: 
+    strategy:
       matrix:
         group: [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 ]
     steps:
@@ -89,7 +89,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v2
         with:
-          node-version-file: ".nvmrc"
+          node-version-file: '.nvmrc'
           registry-url: 'https://registry.npmjs.org'
 
       - name: Setup Yarn
@@ -125,7 +125,7 @@ jobs:
     defaults:
       run:
         working-directory: ./artifacts
-    strategy: 
+    strategy:
       matrix:
         include:
           - name: Linux x64
@@ -145,7 +145,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v2
         with:
-          node-version-file: "./artifacts/.nvmrc"
+          node-version-file: './artifacts/.nvmrc'
           registry-url: 'https://registry.npmjs.org'
 
       - name: Setup Yarn
@@ -173,9 +173,9 @@ jobs:
           name: ${{ matrix.suffix }}-${{ env.VERSION }}
           path: ./artifacts/target/${{ env.ARTIFACT_BUILD_NAME }}
           retention-days: 1
-    
+
   bwc-tests:
-    needs: [ build-min-artifact-tests ]
+    needs: [build-min-artifact-tests]
     runs-on: ubuntu-latest
     container:
       image: docker://opensearchstaging/ci-runner:ci-runner-rockylinux8-opensearch-dashboards-integtest-v2
@@ -184,7 +184,7 @@ jobs:
     defaults:
       run:
         working-directory: ./artifacts
-    strategy: 
+    strategy:
       matrix:
         version: [ osd-2.0.0, osd-2.1.0, osd-2.2.0 ]
     steps:
@@ -199,7 +199,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v2
         with:
-          node-version-file: "./artifacts/.nvmrc"
+          node-version-file: './artifacts/.nvmrc'
           registry-url: 'https://registry.npmjs.org'
 
       - name: Setup Yarn
@@ -221,15 +221,15 @@ jobs:
           if curl -I -L ${{ env.OPENSEARCH_URL }}; then
             echo "::set-output name=version-exists::true"
           fi
-      
+
       - name: Skipping tests
         if: steps.verify-opensearch-exists.outputs.version-exists != 'true'
         run: echo Tests were skipped because an OpenSearch release build does not exist for this version yet!
 
       - name: Setting environment variable to run tests for ${{ matrix.version }}
         if: steps.verify-opensearch-exists.outputs.version-exists == 'true'
         run: echo "BWC_VERSIONS=${{ matrix.version }}" >> $GITHUB_ENV
-    
+
       - name: Download OpenSearch Dashboards
         uses: actions/download-artifact@v3
         id: download
@@ -242,7 +242,7 @@ jobs:
         if: steps.verify-opensearch-exists.outputs.version-exists == 'true'
         run: |
           ./bwctest.sh -s false -o ${{ env.OPENSEARCH_URL }} -d ${{ steps.download.outputs.download-path }}/opensearch-dashboards-${{ env.VERSION }}-linux-x64.tar.gz
-      
+
       - uses: actions/upload-artifact@v3
         if: ${{ failure() && steps.verify-opensearch-exists.outputs.version-exists == 'true' }}
         with:

diff --git a/.lycheeexclude b/.lycheeexclude
@@ -35,6 +35,7 @@ http://noone.nowhere.none/
 http://bar
 http://foo
 http://test.com/
+https://test.com/
 https://manifest.foobar
 https://files.foobar/
 https://tiles.foobar/

diff --git a/package.json b/package.json
@@ -110,6 +110,7 @@
     ]
   },
   "dependencies": {
+    "@aws-crypto/client-node": "^3.1.1",
     "@elastic/datemath": "5.0.3",
     "@elastic/eui": "npm:@opensearch-project/[email protected]",
     "@elastic/good": "^9.0.1-kibana3",

diff --git a/packages/osd-config-schema/src/types/stream_type.test.ts b/packages/osd-config-schema/src/types/stream_type.test.ts
@@ -68,13 +68,18 @@ test('includes namespace in failure', () => {
 describe('#defaultValue', () => {
   test('returns default when undefined', () => {
     const value = new Stream();
-    expect(schema.stream({ defaultValue: value }).validate(undefined)).toMatchInlineSnapshot(`
-      Stream {
-        "_events": Object {},
-        "_eventsCount": 0,
-        "_maxListeners": undefined,
-      }
-    `);
+    expect(schema.stream({ defaultValue: value }).validate(undefined)).toHaveProperty(
+      '_events',
+      {}
+    );
+    expect(schema.stream({ defaultValue: value }).validate(undefined)).toHaveProperty(
+      '_eventsCount',
+      0
+    );
+    expect(schema.stream({ defaultValue: value }).validate(undefined)).toHaveProperty(
+      '_maxListeners',
+      undefined
+    );
   });
 
   test('returns value when specified', () => {

diff --git a/src/core/server/http/router/router.ts b/src/core/server/http/router/router.ts
@@ -300,11 +300,23 @@ export class Router implements IRouter {
       if (LegacyOpenSearchErrorHelpers.isNotAuthorizedError(e)) {
         return e;
       }
+
+      if (isDataSourceConfigError(e)) {
+        return hapiResponseAdapter.handle(
+          opensearchDashboardsResponseFactory.badRequest({ body: e.message })
+        );
+      }
+      // TODO: add legacy data source client config error handling
+
       return hapiResponseAdapter.toInternalError();
     }
   }
 }
 
+const isDataSourceConfigError = (error: any) => {
+  return error.constructor.name === 'DataSourceConfigError' && error.statusCode === 400;
+};
+
 const convertOpenSearchUnauthorized = (
   e: OpenSearchNotAuthorizedError
 ): ErrorHttpResponseOptions => {

diff --git a/src/dev/jest/config.js b/src/dev/jest/config.js
@@ -101,4 +101,7 @@ export default {
     '<rootDir>/node_modules/enzyme-to-json/serializer',
   ],
   reporters: ['default', '<rootDir>/src/dev/jest/junit_reporter.js'],
+  globals: {
+    Uint8Array: Uint8Array,
+  },
 };
diff --git a/src/fixtures/stubbed_saved_object_index_pattern.ts b/src/fixtures/stubbed_saved_object_index_pattern.ts
@@ -33,8 +33,8 @@ import stubbedLogstashFields from './logstash_fields';
 
 const mockLogstashFields = stubbedLogstashFields();
 
-export function stubbedSavedObjectIndexPattern(id: string | null = null) {
-  return {
+export function stubbedSavedObjectIndexPattern(id: string | null = null, withDataSource?: false) {
+  const indexPattern: any = {
     id,
     type: 'index-pattern',
     attributes: {
@@ -45,4 +45,16 @@ export function stubbedSavedObjectIndexPattern(id: string | null = null) {
     },
     version: '2',
   };
+
+  if (withDataSource) {
+    indexPattern.reference = [
+      {
+        id: 'id',
+        name: 'name',
+        type: 'data-source',
+      },
+    ];
+  }
+
+  return indexPattern;
 }