[Discover] Data selector enhancement

[mengweieric]

Description

Discover data selector enhancements includes:

• Add a few missing i18n
• Add id, improve data source metadata with fields addition for data sources
• Improve data source filtering
• Add in-memory cache for data selector list with refresh button
• Improve data selector rendering logics
• Removed hard coded data source type-to-display mapping from selector and read from metadata
• Increased test coverage
• Types and interfaces improvements

Issues Resolved

• [Design] Data Explorer : Discover 2.0 + Event Explorer MDS cohesion #4991
• [Datasources] Address Remaining Items in Datasource Selector #5211

Screenshot

Testing the changes

Changelog

Check List

• All tests pass
  • yarn test:jest
  • yarn test:jest_integration
• New functionality includes testing.
• New functionality has been documented.
• Update CHANGELOG.md
• Commits are signed per the DCO using --signoff

[github-actions]

ℹ️ Manual Changeset Creation Reminder

Please ensure manual commit for changeset file 6571.yml under folder changelogs/fragments to complete this PR.

If you want to use the available OpenSearch Changeset Bot App to avoid manual creation of changeset file you can install it in your forked repository following this link.

For more information about formatting of changeset files, please visit OpenSearch Auto Changeset and Release Notes Tool.

[github-actions]

❌ Changeset File Not Added Yet

Please ensure manual commit for changeset file 6571.yml under folder changelogs/fragments to complete this PR. File still missing.

[codecov]

Codecov Report

Attention: Patch coverage is 72.63158% with 26 lines in your changes are missing coverage. Please review.

Project coverage is 67.74%. Comparing base (cfe7cd1) to head (c8753d9).
Report is 2 commits behind head on main.

Files	Patch %	Lines
..._sources/datasource_services/datasource_service.ts	59.25%	9 Missing and 2 partials ⚠️
...rces/datasource_selector/datasource_selectable.tsx	78.78%	3 Missing and 4 partials ⚠️
src/plugins/data/public/plugin.ts	0.00%	4 Missing ⚠️
...lugins/discover/public/__mock__/index.test.mock.ts	25.00%	3 Missing ⚠️
.../data_explorer/public/components/sidebar/index.tsx	75.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6571   +/-   ##
=======================================
  Coverage   67.73%   67.74%           
=======================================
  Files        3415     3417    +2     
  Lines       66851    66887   +36     
  Branches    10874    10878    +4     
=======================================
+ Hits        45282    45312   +30     
- Misses      18923    18926    +3     
- Partials     2646     2649    +3     

Flag	Coverage Δ
Linux_1	33.19% <20.68%> (-0.01%)	⬇️
Linux_2	55.59% <ø> (ø)
Linux_3	?
Linux_4	?
Windows_1	33.24% <20.68%> (+0.01%)	⬆️
Windows_2	55.56% <ø> (ø)
Windows_3	45.27% <72.63%> (+0.02%)	⬆️
Windows_4	34.85% <20.68%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kavilla]

Link is not failing because of this PR. All good to me

[ashwin-pc]

Can you outline in the PR description what exactly this change is about? Currently it looks like this addresses all the feedback from the previous PR and closes the Data Sources issue. However i see in here that S3 and Spark data sources are still hard coded and the registration function is still not generic enough to handle future data sources.

[ashwin-pc]

Are we still hard coding these? Is there a plan to remove them?

[ashwin-pc]

Same as above, do we have a plan on getting away from hardcoding datasource types in this service? These should be generic enough to allow different groups of datasources.

[mengweieric]

Can you outline in the PR description what exactly this change is about? Currently it looks like this addresses all the feedback from the previous PR and closes the Data Sources issue. However i see in here that S3 and Spark data sources are still hard coded and the registration function is still not generic enough to handle future data sources.

Thanks for the review.

The removal of hardcoding in this PR is to remove previous hardcoded type-to-display-name mapping and instead reading from data source metadata. The s3 data sources are backend data sources where it doesn't have any UI specific definitions. Ideally these front end specific definitions should be retrieving from settings or front end data sources separated from backend data sources where user can configure them. Does this align with the direction you are pointing to?

As for registration function, are you point to the registerDataSources in data source service or the way data source owner register the data source?

[kavilla]

Can you outline in the PR description what exactly this change is about? Currently it looks like this addresses all the feedback from the previous PR and closes the Data Sources issue. However i see in here that S3 and Spark data sources are still hard coded and the registration function is still not generic enough to handle future data sources.

Thanks for the review.

The removal of hardcoding in this PR is to remove previous hardcoded type-to-display-name mapping and instead reading from data source metadata. The s3 data sources are backend data sources where it doesn't have any UI specific definitions. Ideally these front end specific definitions should be retrieving from settings or front end data sources separated from backend data sources where user can configure them. Does this align with the direction you are pointing to?

As for registration function, are you point to the registerDataSources in data source service or the way data source owner register the data source?

I think @ashwin-pc is saying the next iteration of this where if I am a cluster admin, I can install a plugin that adds the new data source. For example, if I want a S3 connection type I install a S3 connection plugin that wires that up. Or better yet, some type of catalogue.

[mengweieric]

Can you outline in the PR description what exactly this change is about? Currently it looks like this addresses all the feedback from the previous PR and closes the Data Sources issue. However i see in here that S3 and Spark data sources are still hard coded and the registration function is still not generic enough to handle future data sources.

Thanks for the review.
The removal of hardcoding in this PR is to remove previous hardcoded type-to-display-name mapping and instead reading from data source metadata. The s3 data sources are backend data sources where it doesn't have any UI specific definitions. Ideally these front end specific definitions should be retrieving from settings or front end data sources separated from backend data sources where user can configure them. Does this align with the direction you are pointing to?
As for registration function, are you point to the registerDataSources in data source service or the way data source owner register the data source?

I think @ashwin-pc is saying the next iteration of this where if I am a cluster admin, I can install a plugin that adds the new data source. For example, if I want a S3 connection type I install a S3 connection plugin that wires that up. Or better yet, some type of catalogue.

Yea understood that we will want this in future iterations, this PR addressed the step one mapping (updated the description), eventually we will not want those to be defined in code.

[ashwin-pc]

@mengweieric yes my goal is to have no service specific names hardcoded in the framework. If any other company wanted to extend the datasources to support their version of the service like Azure for example. They should not need to make a code change in OSD to show up as a new Datasource as long as they adhere to the schema.

e.g.

registerDataSource({
  name: 'Postgress`
  schema: any
  ...
})

today, if i wanted to add postgress or any other datasource type to this list, i will need to make a code change in OSD to show up as an option in the DataSources menu. That should not be necessary. That should all be abstracted away in the API.