[Feature] Improve the observability of integration tests #775

jasoonn · 2022-12-01T19:45:41Z

Why are these changes needed?

Add app.kubernetes.io/component: kuberay-operator label in helm-chart/kuberay-operator/templates/deployment.yaml to make operator pod can be selected in show_cluster_info().

Why not call `show_cluster_info()` in tearDown()?

The only way I found to check if the test pass in teardown() is to access the private field self._outcome in unittest.TestCase object(reference). The fields in self._outcome changes in Python 3.11. It seems like accessing self._outcome in unittest.TestCase is not a robust approach.

Why added hard code label `app.kubernetes.io/component=kuberay-operator`?

Because all of the labels app.kubernetes.io/component: for the operator is hard code with value kuberay-operato, I added app.kubernetes.io/component: kuberay-operator to the helm chart to make show_cluster_info able to get logs from the operator pod.

Related issue number

Closes #766

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

kevin85421

Thank you for this contribution!

tests/framework/prototype.py

jasoonn · 2022-12-02T05:32:02Z

@kevin85421 @DmitriGekhtman Thanks for the suggestion. I have updated my pr.

DmitriGekhtman · 2022-12-02T19:09:28Z

Was there a Git issue? I think we lost most of the changes (I see Files changed 1)

jasoonn · 2022-12-02T22:22:20Z

@DmitriGekhtman Because the label utilized in the original show_cluster_info() is app.kubernetes.io/component: kuberay-operator, we only need to add that label in the helm chart. Besides, I changed the shell_subprocess_run() back.

jasoonn · 2022-12-02T22:29:19Z

To improve the observability of integration tests, I called show_cluster_info("default") in several places in tests/compatibility-test.py. The places I choose are where the most CI tests failed in: nonzero return in the execution of script files. e.g. test_ray_serve and test_detach_actor.

DmitriGekhtman · 2022-12-02T22:39:14Z

LGTM, I'll merge this after the CI finishes running.
This will be useful to have in the release branch.

kevin85421 · 2022-12-02T22:47:25Z

tests/compatibility-test.py

@@ -198,6 +193,7 @@ def test_ray_serve(self):
        exit_code, _ = utils.exec_run_container(container, f'python3 /usr/local/test_ray_serve_1.py {ray_namespace}', timeout_sec = 180)

        if exit_code != 0:
+            show_cluster_info("default")


I found that we hard-coded a lot of "default" in both test_ray_serve and test_detached_actor. How about creating a local variable in each test to replace this hard-coded method?

For example,

utils.get_pod(namespace='default', label_selector='ray.io/node-type=head')

I have updated in d5c8fd6 and c0d8ef6. To be noted, these two functions will utilize the same raycluster which is created in setUpClass(). If we want to separate the namespace in the future, we will need to move the creation of raycluster into corresponding functions.

kevin85421 · 2022-12-02T22:49:31Z

tests/compatibility-test.py

-                'kubectl logs $(kubectl get pods | grep -e "-head" | awk "{print \$1}")')
-            shell_subprocess_run(
-                'kubectl logs -n $(kubectl get pods -A | grep -e "-operator" | awk \'{print $1 "  " $2}\')')
+            show_cluster_info("default")
        assert rtn == 0


Remove the assert and raise an exception in the if statement?

Updated 7d640b0.

jasoonn · 2022-12-02T23:55:01Z

@kevin85421 Thanks for the suggestion. I have updated the pr.

kevin85421 · 2022-12-03T05:34:13Z

tests/compatibility-test.py


    def test_ray_serve(self):
+        cluster_name_space = "default"


cluster_namespace

Update in bc948f2.

jasoonn · 2022-12-03T06:13:11Z

@kevin85421 Thanks for the suggestion. I have updated the pr.

kevin85421

LGTM. Wait for CI to pass.

…#775) Show debug information in failed test cases. Make operator label consistent across Kustomize and Helm deployment methods.

* [Feature] Improve the observability of integration tests (#775) Show debug information in failed test cases. Make operator label consistent across Kustomize and Helm deployment methods.

…#775) Show debug information in failed test cases. Make operator label consistent across Kustomize and Helm deployment methods.

jasoonn added 2 commits December 1, 2022 19:30

update show_cluster_info

aa8528e

update shell_subprocess_run

0619700

kevin85421 reviewed Dec 1, 2022

View reviewed changes

tests/framework/prototype.py Outdated Show resolved Hide resolved

tests/framework/prototype.py Outdated Show resolved Hide resolved

tests/framework/prototype.py Outdated Show resolved Hide resolved

tests/framework/prototype.py Outdated Show resolved Hide resolved

update label

5177ebf

add show_cluster_info in error handling

d024ba4

DmitriGekhtman approved these changes Dec 2, 2022

View reviewed changes

kevin85421 reviewed Dec 2, 2022

View reviewed changes

jasoonn added 3 commits December 2, 2022 23:28

replace assertt

7d640b0

add namespace

d5c8fd6

add namespace to test_ray_serve and test_detached_actor

c0d8ef6

kevin85421 reviewed Dec 3, 2022

View reviewed changes

ufix typ

bc948f2

kevin85421 approved these changes Dec 3, 2022

View reviewed changes

DmitriGekhtman approved these changes Dec 3, 2022

View reviewed changes

DmitriGekhtman merged commit c6df15e into ray-project:master Dec 3, 2022

DmitriGekhtman mentioned this pull request Dec 6, 2022

[Feature] Improve the observability of integration tests (#775) #796

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Improve the observability of integration tests #775

[Feature] Improve the observability of integration tests #775

jasoonn commented Dec 1, 2022 •

edited

Loading

kevin85421 left a comment

jasoonn commented Dec 2, 2022

DmitriGekhtman commented Dec 2, 2022

jasoonn commented Dec 2, 2022

jasoonn commented Dec 2, 2022 •

edited

Loading

DmitriGekhtman commented Dec 2, 2022

kevin85421 Dec 2, 2022

jasoonn Dec 2, 2022

kevin85421 Dec 2, 2022

jasoonn Dec 2, 2022

jasoonn commented Dec 2, 2022

kevin85421 Dec 3, 2022

jasoonn Dec 3, 2022

jasoonn commented Dec 3, 2022

kevin85421 left a comment

[Feature] Improve the observability of integration tests #775

[Feature] Improve the observability of integration tests #775

Conversation

jasoonn commented Dec 1, 2022 • edited Loading

Why are these changes needed?

Why not call show_cluster_info() in tearDown()?

Why added hard code label app.kubernetes.io/component=kuberay-operator?

Related issue number

Checks

kevin85421 left a comment

Choose a reason for hiding this comment

jasoonn commented Dec 2, 2022

DmitriGekhtman commented Dec 2, 2022

jasoonn commented Dec 2, 2022

jasoonn commented Dec 2, 2022 • edited Loading

DmitriGekhtman commented Dec 2, 2022

kevin85421 Dec 2, 2022

Choose a reason for hiding this comment

jasoonn Dec 2, 2022

Choose a reason for hiding this comment

kevin85421 Dec 2, 2022

Choose a reason for hiding this comment

jasoonn Dec 2, 2022

Choose a reason for hiding this comment

jasoonn commented Dec 2, 2022

kevin85421 Dec 3, 2022

Choose a reason for hiding this comment

jasoonn Dec 3, 2022

Choose a reason for hiding this comment

jasoonn commented Dec 3, 2022

kevin85421 left a comment

Choose a reason for hiding this comment

jasoonn commented Dec 1, 2022 •

edited

Loading

Why not call `show_cluster_info()` in tearDown()?

Why added hard code label `app.kubernetes.io/component=kuberay-operator`?

jasoonn commented Dec 2, 2022 •

edited

Loading