Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Audit newly added rows to the supported ops CSV files #865

Closed
20 tasks done
amahussein opened this issue Mar 22, 2024 · 0 comments · Fixed by #963
Closed
20 tasks done

[BUG] Audit newly added rows to the supported ops CSV files #865

amahussein opened this issue Mar 22, 2024 · 0 comments · Fixed by #963
Assignees
Labels
bug Something isn't working core_tools Scope the core module (scala)

Comments

@amahussein
Copy link
Collaborator

amahussein commented Mar 22, 2024

Describe the bug

Following the merge of #847, some new rows were appended to the supported-Execs/Exprs CSV files.
Those new rows have column Support value set to TNEW.

On the tools side, we need to audit the new operators and start adding them to the Qual tool support if any.

  • Some of those rows are already supported by the qualification tool but they did not have an entry in the CSV files. Example, WriteFiles and AQEShuffleReadExec. For those records, we need to:
    • remove them from the override-config file and
    • change the code to read the column Support
  • For the remaining rows one by one, we need to test their support and update the override-config file.

Additional context

Please add the list of operators task here.

Tasks

  1. bug core_tools
    amahussein
  2. bug core_tools
    amahussein
  3. bug core_tools
    amahussein
  4. bug core_tools
    amahussein
  5. bug core_tools
    parthosa
  6. bug core_tools
    amahussein
  7. bug core_tools
    parthosa
  8. bug core_tools
    parthosa
  9. bug core_tools
    amahussein
  10. audit_plugin core_tools
    amahussein
  11. bug core_tools
    cindyyuanjiang
@amahussein amahussein added bug Something isn't working ? - Needs Triage core_tools Scope the core module (scala) labels Mar 22, 2024
@amahussein amahussein self-assigned this Mar 26, 2024
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Apr 1, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to NVIDIA#865

This PR adds support to two new expressions for the bloomFilter
- Adds a new test case to check that the expressions are parsed
  correctly
amahussein added a commit that referenced this issue Apr 2, 2024
#891)

* Add Support for BloomFilterAggregate and BloomFilterMightContain exprs

Contributes to #865

This PR adds support to two new expressions for the bloomFilter
- Adds a new test case to check that the expressions are parsed
  correctly

---------

Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Apr 2, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to NVIDIA#865

Adds support to two new string expressions
- `EphermalSubstring` is supported by DB, but we add it to the score
  sheet of all other platforms since we have a single expression.csv
file anyway.
- No need for a new unit test for `EphermalSubstring` because it is the
  same as substring.
- Added a new test for `KnownNullable`
- Removed configuration override related to `Empty2Null` because the sql
  function is the lower string of the expr-name which should be
supported by default.
- Removed the entries of the expressions in `override_supported_configs.json`
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Apr 2, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to NVIDIA#865

Adds support to two new string expressions
- `EphermalSubstring` is supported by DB, but we add it to the score
  sheet of all other platforms since we have a single expression.csv
file anyway.
- No need for a new unit test for `EphermalSubstring` because it is the
  same as substring.
- Added a new test for `KnownNullable`
- Removed configuration override related to `Empty2Null` because the sql
  function is the lower string of the expr-name which should be
supported by default.
- Removed the entries of the expressions in `override_supported_configs.json`
amahussein added a commit that referenced this issue Apr 2, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to #865

Adds support to two new string expressions
- `EphermalSubstring` is supported by DB, but we add it to the score
  sheet of all other platforms since we have a single expression.csv
file anyway.
- No need for a new unit test for `EphermalSubstring` because it is the
  same as substring.
- Added a new test for `KnownNullable`
- Removed configuration override related to `Empty2Null` because the sql
  function is the lower string of the expr-name which should be
supported by default.
- Removed the entries of the expressions in `override_supported_configs.json`
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Apr 9, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to NVIDIA#865

Re-enable the support for GetJsonObject as RAPIDS 24.04 is supporting it
by default
- Update the unit-test accordingly
- Minor change in notes of `InMemoryTableScanExec`
amahussein added a commit that referenced this issue Apr 9, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to #865

Re-enable the support for GetJsonObject as RAPIDS 24.04 is supporting it
by default
- Update the unit-test accordingly
- Minor change in notes of `InMemoryTableScanExec`
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Apr 12, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to NVIDIA#865

- Fix notes column in the supported ops CSV files. Many of them should
  be set to None as the operator is marked as supported.
amahussein added a commit that referenced this issue Apr 12, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to #865

- Fix notes column in the supported ops CSV files. Many of them should
  be set to None as the operator is marked as supported.
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Apr 18, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to NVIDIA#865

- Disable Support of `GetJsonObject` with its sql function
  `get_json_object`
amahussein added a commit that referenced this issue Apr 18, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Contributes to #865

- Disable Support of `GetJsonObject` with its sql function
  `get_json_object`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
3 participants