Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug Fixes for minor issues with SQL PIT refactor #3045

Merged
merged 1 commit into from
Sep 26, 2024

Conversation

manasvinibs
Copy link
Member

@manasvinibs manasvinibs commented Sep 25, 2024

Description

Issue found during E2E testing:

  1. If join queries fail to pull results due to internal errors in the service. The API fails with a Null Pointer Exception
    Reproducible if you force the OpenSearch client to throw an exception while executing a join.
Before:

curl -X POST "http://localhost:9200/_plugins/_sql" -H 'Content-Type: application/json' -d'
> {
>   "query": "SELECT t1.field_1, t2.field_3 FROM testpit t1 INNER JOIN testjoin t2 ON t1.field_2 = t2.field_4"
> }
> '
{
  "error": {
    "reason": "There was internal problem at backend",
    "details": "Cannot invoke \"java.util.List.size()\" because \"this.results\" is null",
    "type": "NullPointerException"
  },
  "status": 500
}


After fix:

curl -X POST "http://localhost:9200/_plugins/_sql" -H 'Content-Type: application/json' -d'
{
  "query": "SELECT t1.field_1, t2.field_3 FROM testpit t1 INNER JOIN testjoin t2 ON t1.field_2 = t2.field_4"
}
'
{
  "error": {
    "reason": "There was internal problem at backend",
    "details": "Error occurred during join query run",
    "type": "IllegalStateException"
  },
  "status": 500
}

  1. Sometimes, a cursor seems to be returned even when one is not needed eg: select a.* from my-index a returns a cursor but select * from my-index a does not.

After fix:

curl -X POST "http://localhost:9200/_plugins/_sql" -H 'Content-Type: application/json' -d'
{
  "query": "SELECT a.* FROM testpit a"
}
'


Response: 
{
  "schema": [
    {
      "name": "field_1",
      "type": "text"
    },
    {
      "name": "field_2",
      "type": "integer"
    }
  ],
  "total": 10,
  "datarows": [
    [
      "Sample data 1",
      1
    ],
    [
      "Sample data 2",
      2
    ],
    [
      "Sample data 3",
      3
    ],
    [
      "Sample data 4",
      4
    ],
    [
      "Sample data 5",
      5
    ],
    [
      "Sample data 6",
      6
    ],
    [
      "Sample data 7",
      7
    ],
    [
      "Sample data 8",
      8
    ],
    [
      "Sample data 9",
      9
    ],
    [
      "Sample data 10",
      10
    ]
  ],
  "size": 10,
  "status": 200
}


curl -X POST "http://localhost:9200/_plugins/_sql" -H 'Content-Type: application/json' -d'
{
  "query": "SELECT a.* FROM testpit a"
}
'

Response: 
{
  "schema": [
    {
      "name": "field_1",
      "type": "text"
    },
    {
      "name": "field_2",
      "type": "integer"
    }
  ],
  "total": 10,
  "datarows": [
    [
      "Sample data 1",
      1
    ],
    [
      "Sample data 2",
      2
    ],
    [
      "Sample data 3",
      3
    ],
    [
      "Sample data 4",
      4
    ],
    [
      "Sample data 5",
      5
    ],
    [
      "Sample data 6",
      6
    ],
    [
      "Sample data 7",
      7
    ],
    [
      "Sample data 8",
      8
    ],
    [
      "Sample data 9",
      9
    ],
    [
      "Sample data 10",
      10
    ]
  ],
  "size": 10,
  "status": 200
}


  1. The api seems to have an unsupported "size" parameter where it generates a cursor. The cursor is unusable, attempting to use it causes an exception.
    Reproducible by specifying a "size" parameter in the request body in place of "fetch_size".
After fix, response doesn't contain incorrect default cursor in the response. It just ignores the unsupported fields in the query which is existing behavior. 
curl -X POST "http://localhost:9200/_plugins/_sql" -H 'Content-Type: application/json' -d'
{
  "query": "SELECT field_1 FROM testpit WHERE field_2 > 0",
  "size": 5
}
'
{
  "schema": [{
    "name": "field_1",
    "type": "text"
  }],
  "total": 10,
  "datarows": [
    ["Sample data 1"],
    ["Sample data 2"],
    ["Sample data 3"],
    ["Sample data 4"],
    ["Sample data 5"],
    ["Sample data 6"],
    ["Sample data 7"],
    ["Sample data 8"],
    ["Sample data 9"],
    ["Sample data 10"]
  ],
  "size": 10,
  "status": 200
}
  1. From the testing, it appears the v2 engine always appears to always create a PIT context even when one is not needed (could explain 3)
This was addressed as part of refactorign PR only in the revised iterations. Here is the code ref for v2 PIT changes where PIT is created only when Scroll was used before. 
https://github.com/opensearch-project/sql/blob/main/opensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchRequestBuilder.java#L124

Check List

  • New functionality includes testing.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

ykmr1224
ykmr1224 previously approved these changes Sep 26, 2024
Comment on lines 137 to 139
if (queryAction.getSqlRequest().fetchSize() == 0
|| searchResponse.getHits().getTotalHits().value
< queryAction.getSqlRequest().fetchSize()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test cover this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we have IT to cover legacy part of the code but I don't see existing test file to cover UT for this class. I've created one and added UT for this function.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the test! Could you test API behavior instead of internal impl in a private method? I see execute API with DefaultQueryAction can cover your changes?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I need to get this one merged and can look into this as follow-up.

@ykmr1224 ykmr1224 added the bug Something isn't working label Sep 26, 2024
@ykmr1224 ykmr1224 merged commit ec5c3b7 into opensearch-project:main Sep 26, 2024
12 of 15 checks passed
@manasvinibs manasvinibs added the v2.18.0 Issues targeting release v2.18.0 label Sep 26, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 23, 2024
Signed-off-by: Manasvini B S <[email protected]>
(cherry picked from commit ec5c3b7)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x bug Something isn't working v2.18.0 Issues targeting release v2.18.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants