Ignore name of index when fetching settings #823

tienne-B · 2024-09-18T02:30:26Z

Description

This commit fixes a situation where an index cannot be updated through this client as the index is aliased, with the client pointing to the alias.

As the GET /<index>/_settings request will only ever return the settings for the specified index (through the alias), it would only have one key, so the name of the key would not matter. We can pop the key to get the settings object for the index through the alias.

Describe what this change achieves.

Issues Resolved

Closes #822

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2024-09-18T02:32:14Z

Codecov Report

Attention: Patch coverage is 20.00000% with 8 lines in your changes missing coverage. Please review.

Project coverage is 70.34%. Comparing base (ba715b9) to head (e7532db).
Report is 63 commits behind head on main.

Files with missing lines	Patch %	Lines
opensearchpy/_async/helpers/index.py	20.00%	4 Missing ⚠️
opensearchpy/helpers/index.py	20.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #823      +/-   ##
==========================================
- Coverage   71.95%   70.34%   -1.61%     
==========================================
  Files          91      113      +22     
  Lines        8001     8896     +895     
==========================================
+ Hits         5757     6258     +501     
- Misses       2244     2638     +394

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

dblock

What's the value of _name vs. what it is after the change here?
Can you please add a test?

tienne-B · 2024-09-18T16:56:16Z

Thank you @dblock for your quick response! The value of removing self._name is that saving an index previously wouldn't work if OS-py accesses it through an alias (and using an alias seems to be a good practice for downtime prevention). All the requests that are triggered by the .save() method can work transparently through an index, however, when making the GET /<alias>/_settings request, the dict response key is the name of the true index, rather than the alias. This causes the KeyError. However, the data we're extracting doesn't depend on the name, so it'd be safe to disregard and pop the key/value.

If it's wanted, I could add a check that the alias only points to one index to raise a more specific error. Would that be appreciated?

I've added a test, but I'm a little skeptical of the coverage report as it's based off a 6-month-old commit.

Thank you!

dblock · 2024-09-19T08:12:50Z

If it's wanted, I could add a check that the alias only points to one index to raise a more specific error. Would that be appreciated?

Possibly, but doesn't have to block this PR. Give it a shot after I merge this!

dblock

This all makes sense.

I think you have to do the same for the async code (see https://github.com/opensearch-project/opensearch-py/blob/main/opensearchpy/_async/helpers/index.py#L308). Add a test for that too.

tienne-B · 2024-09-20T11:47:28Z

I've added support for async, the checks for multiple indices, and testing for these.

dblock

Looks good!

I have some small language asks for the error. Also check that the tests do run because CodeCov seems unhappy and complains that tests are missing which seems odd.

dblock · 2024-09-23T12:17:04Z

opensearchpy/helpers/test.py

@@ -34,7 +34,7 @@
 from opensearchpy import OpenSearch
 from opensearchpy.exceptions import ConnectionError

-OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "https://localhost:9200")
+OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "http://localhost:9200")


What's the reason for this change? AFAIK it should always be https.

Oh sorry, didn't mean to commit that! I had noticed that the tests did not run, and had to uncomment these list values to get the tests to run:

opensearch-py/test_opensearchpy/run_tests.py

Lines 159 to 164 in a24b9f3

ignores = [

"test_opensearchpy/test_server/",

"test_opensearchpy/test_server_secured/",

"test_opensearchpy/test_async/test_server/",

"test_opensearchpy/test_async/test_server_secured/",

]

I am a little hesitant to commit such changes as I don't have much context as to why this has been done and if it'd break anything. Also, the base commit for Codecov is quite old.

Ugh this looks like a mess. I opened #826 with a high level idea of what we should do here (maybe someone wants to help later, wink wink).

For this PR I'd like to ensure the tests you wrote actually run (and pass) in CI. Is this the case?

The tests do pass locally with the files un-ignored (although this assertion is somewhat empty)

On GitHub Actions they would be skipped as there is no OpenSearch service for the tests to use.

That's the whole point, integration tests should run on GHA in one of the workflows.

This is still here, do you want to revert this line?

Reverted, sorry.

test_opensearchpy/test_async/test_server/test_helpers/test_index.py

opensearchpy/_async/helpers/index.py

dblock · 2024-09-25T19:55:49Z

~~@tienne-B so we can't merge this without a passing CI, help out with whatever the right answer is to get it to pass and ensuring that the tests you have added do run in CI, please?~~

I debugged the test failure but couldn't find a solution. You can rebase with the workaround in #828 for now (or maybe you know how to fix that failure?).

tienne-B · 2024-09-26T13:21:02Z

@dblock Rebased.

dblock

There's still that http, want to fix it?

dblock · 2024-09-26T14:44:07Z

opensearchpy/helpers/test.py

@@ -34,7 +34,7 @@
 from opensearchpy import OpenSearch
 from opensearchpy.exceptions import ConnectionError

-OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "https://localhost:9200")
+OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "http://localhost:9200")


This is still here, do you want to revert this line?

This commit fixes a situation where an index cannot be updated through this client as the index is aliased, with the client pointing to the alias. As the `GET /<index>/_settings` request will only ever return the settings for the specified index (through the alias), it would only have one key, so the name of the key would not matter. We can pop the key to get the settings object for the index through the alias. Signed-off-by: Étienne Beaulé <[email protected]>

dblock · 2024-09-26T18:02:35Z

good work merci @tienne-B !

tienne-B requested review from VachaShah, dblock, harshavamsi, axeoman, Shephalimittal, saimedhi and florianvazelle as code owners September 18, 2024 02:30

dblock requested changes Sep 18, 2024

View reviewed changes

tienne-B force-pushed the patch-1 branch 2 times, most recently from 263b708 to 2ed7b6e Compare September 18, 2024 16:43

tienne-B requested a review from dblock September 18, 2024 16:56

dblock requested changes Sep 19, 2024

View reviewed changes

tienne-B force-pushed the patch-1 branch 2 times, most recently from e026aaf to 50771c1 Compare September 20, 2024 03:11

tienne-B requested a review from dblock September 20, 2024 11:41

tienne-B force-pushed the patch-1 branch from 50771c1 to 7da15f6 Compare September 20, 2024 11:46

dblock requested changes Sep 23, 2024

View reviewed changes

dblock mentioned this pull request Sep 23, 2024

[BUG] Test matrix difficult to grok #826

Open

tienne-B force-pushed the patch-1 branch 3 times, most recently from d0b4bed to 2edcaf3 Compare September 25, 2024 02:02

tienne-B requested a review from dblock September 25, 2024 02:09

tienne-B force-pushed the patch-1 branch from 2edcaf3 to 65b8987 Compare September 26, 2024 13:09

dblock requested changes Sep 26, 2024

View reviewed changes

tienne-B force-pushed the patch-1 branch from 65b8987 to e7532db Compare September 26, 2024 17:41

tienne-B requested a review from dblock September 26, 2024 17:42

dblock approved these changes Sep 26, 2024

View reviewed changes

dblock merged commit 1b0440a into opensearch-project:main Sep 26, 2024
32 of 34 checks passed

tienne-B deleted the patch-1 branch September 26, 2024 18:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ignore name of index when fetching settings #823

Ignore name of index when fetching settings #823

tienne-B commented Sep 18, 2024

codecov bot commented Sep 18, 2024 •

edited

Loading

dblock left a comment

tienne-B commented Sep 18, 2024

dblock commented Sep 19, 2024

dblock left a comment

tienne-B commented Sep 20, 2024

dblock left a comment

dblock Sep 23, 2024

tienne-B Sep 23, 2024

dblock Sep 23, 2024 •

edited

Loading

tienne-B Sep 25, 2024 •

edited

Loading

dblock Sep 25, 2024

dblock Sep 26, 2024

tienne-B Sep 26, 2024

dblock commented Sep 25, 2024 •

edited

Loading

tienne-B commented Sep 26, 2024

dblock left a comment

dblock Sep 26, 2024

dblock commented Sep 26, 2024

	ignores = [
	"test_opensearchpy/test_server/",
	"test_opensearchpy/test_server_secured/",
	"test_opensearchpy/test_async/test_server/",
	"test_opensearchpy/test_async/test_server_secured/",
	]

Ignore name of index when fetching settings #823

Ignore name of index when fetching settings #823

Conversation

tienne-B commented Sep 18, 2024

Description

Issues Resolved

codecov bot commented Sep 18, 2024 • edited Loading

Codecov Report

dblock left a comment

Choose a reason for hiding this comment

tienne-B commented Sep 18, 2024

dblock commented Sep 19, 2024

dblock left a comment

Choose a reason for hiding this comment

tienne-B commented Sep 20, 2024

dblock left a comment

Choose a reason for hiding this comment

dblock Sep 23, 2024

Choose a reason for hiding this comment

tienne-B Sep 23, 2024

Choose a reason for hiding this comment

dblock Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

tienne-B Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

dblock Sep 25, 2024

Choose a reason for hiding this comment

dblock Sep 26, 2024

Choose a reason for hiding this comment

tienne-B Sep 26, 2024

Choose a reason for hiding this comment

dblock commented Sep 25, 2024 • edited Loading

tienne-B commented Sep 26, 2024

dblock left a comment

Choose a reason for hiding this comment

dblock Sep 26, 2024

Choose a reason for hiding this comment

dblock commented Sep 26, 2024

codecov bot commented Sep 18, 2024 •

edited

Loading

dblock Sep 23, 2024 •

edited

Loading

tienne-B Sep 25, 2024 •

edited

Loading

dblock commented Sep 25, 2024 •

edited

Loading