[Auto Suggest] PPL & SQL Value Suggestion

[paulstn]

Description

Only done for opensearch SQL + PPL. For SQL, value completion only occurs inside of a binary comparison predicate (for example column = value, and for an in-predicate with the keyword IN (e.g. column IN ("value1", "value2", "value3")). These are the only cases in which it makes sense for SQL to suggest values for a particular column.

SQL:

Screen.Recording.2024-10-09.at.11.51.51.AM.mov

PPL:

Screen.Recording.2024-10-09.at.11.42.04.AM.mov

src/plugins/data/public/antlr/*

For PPL & SQL, uses the SQL query

SELECT <column> FROM <table> GROUP BY <column> ORDER BY COUNT(<column>) DESC LIMIT <limit>

to find the most popular column values to display within autocomplete whenever a value should be suggested.

src/plugins/data/server/ui_settings.ts

There were two UI settings added.
query:enhancements:suggestValues is a boolean that determines if value suggestion will be used
query:enhancements:suggestValuesLimit is a number that specifies how many values should be queried, defaulting to 200

src/plugins/data/public/autocomplete/autocomplete_service.ts

The ability to add a value suggestion provided was included in this pr, but isn't in use so far. Creating a value suggestion provider that would make the api call (with all of the same parameters that exist right now) could be done but would require a lot of complexity for what is essentially done in 10 lines today.

This PR also contains various fixes and quality of life updates within SQL & PPL suggestions.

Issues Resolved

Screenshot

Testing the changes

Field level security testing:

Screen.Recording.2024-10-21.at.2.22.46.PM.mov

Field level masking testing:

Screen.Recording.2024-10-21.at.2.09.44.PM.mov

Changelog

• feat: Autocomplete Value Suggestion

Check List

• All tests pass
  • yarn test:jest
  • yarn test:jest_integration
• New functionality includes testing.
• New functionality has been documented.
• Update CHANGELOG.md
• Commits are signed per the DCO using --signoff

[github-actions]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[codecov]

Codecov Report

Attention: Patch coverage is 6.79612% with 96 lines in your changes missing coverage. Please review.

Project coverage is 60.67%. Comparing base (758e4fb) to head (d11356d).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...ntlr/opensearch_sql/opensearch_sql_autocomplete.ts	0.00%	30 Missing and 1 partial ⚠️
...ata/public/antlr/opensearch_sql/code_completion.ts	0.00%	27 Missing ⚠️
...ntlr/opensearch_ppl/opensearch_ppl_autocomplete.ts	0.00%	12 Missing ⚠️
...s/data/public/autocomplete/autocomplete_service.ts	0.00%	12 Missing ⚠️
...ata/public/antlr/opensearch_ppl/code_completion.ts	0.00%	10 Missing ⚠️
src/plugins/data/public/antlr/shared/utils.ts	55.55%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8275      +/-   ##
==========================================
- Coverage   60.78%   60.67%   -0.12%     
==========================================
  Files        3798     3798              
  Lines       90701    90862     +161     
  Branches    14284    14321      +37     
==========================================
- Hits        55133    55130       -3     
- Misses      32068    32232     +164     
  Partials     3500     3500              

Flag	Coverage Δ
Linux_1	28.98% <3.88%> (-0.07%)	⬇️
Linux_2	56.39% <ø> (ø)
Linux_3	?
Linux_4	29.74% <3.88%> (-0.09%)	⬇️
Windows_1	29.00% <3.88%> (-0.07%)	⬇️
Windows_2	56.34% <ø> (ø)
Windows_3	37.54% <6.79%> (-0.12%)	⬇️
Windows_4	29.74% <3.88%> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[joshuali925]

seems it's simplifying some of the comparison grammars, any context on why they are needed?

[joshuali925]

why do we need this change?

[paulstn]

This line, along with line 307/308, was changed so that values don't get suggested for the left side of a predicate, while the right side doesn't have field names suggested and only values suggested instead. Although having a constant on the left and a field on the right does get parsed, suggesting it is practically useless/ information overload. There would be too many suggestions and it doesn't make sense to show every single possibility.

[joshuali925]

np. might be better to put args type and return type as generic, to allow extending and keeps autocomplete generic to what languages are used

[paulstn]

I'm not sure I fully understand what making the args or the return type generic would accomplish. Since this file is meant to include the implementation for functions that would do the value completion, they require specific inputs that may not be related to the output.
To keep autocomplete generic to the languages used, should I just change the name of the function to move away from "SQL"?

[mengweieric]

Just summarize some of the concerns for this PR that I feel we need to address:

Grammar Changes:
Modifying the grammar directly is not considered an optimal approach, as it introduces risks of mismatches with removed or modified grammar rules. We should prioritize leveraging visitors, token streams, and logical code adjustments to achieve the intended behavior.

Test Coverage:
There is insufficient test coverage to ensure the modified grammar behaves consistently with previous expectations. Additional tests should be added to confirm the correctness of the changes if we want to go ahead with grammar changes.

Value Suggestions:
While value suggestions are not a typical feature for SQL/PPL languages, it is acceptable in this context. However, the current implementation still has gaps in providing a meaningful suggestion experience, such as suggesting masked values.

Unaddressed Comments:
Some comments have not been addressed yet. Please ensure that all feedback is reviewed and handled accordingly.

[paulstn]

NOTE: this is reverting back a change made in a previous pr, to go back to the original grammar.