Feature/add k8s event interpretation #382
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Robertation256
commented
May 1, 2024
| runner.run_without_collect(resource_file_path) | ||
| cluster.delete_cluster(cluster_name, kube_config_path) | ||
|
|
||
| assert(log_test_predicate(log_file_path)) |
There was a problem hiding this comment.
Test to see if an abort is triggered by examining the log file, since I haven't been able to find a cleaner way to test that.
tylergu
reviewed
May 1, 2024
acto/utils/k8s_event_watcher.py
Outdated
| k8s_event_watcher_config = { | ||
| "default_threshold": 3, | ||
| "abort_predicates": [ | ||
| # a full list of kubelet emitted reason can be found at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/events/event.go |
|
@Robertation256 Thanks for the PR! Could you follow this doc to format the code? https://github.com/xlab-uiuc/acto/blob/main/docs/development.md If you use vscode, installing the black, pylint, and isort should be enough. The github pre-commit hook can also perform the tasks |
Done! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, Acto waits for system convergence by observing a silence of K8s events within a configured
time window. Such a wait of convergence is performed after applying each test case to ensure that the
collected states reflect the system's final states. However, such a wait can be futile when the input CR
carries faulty mutations (ie. invalid container image urls) or represents un-schedulable resources given the
test cluster topology and hardware constraints. In these cases, the cluster gets stuck in an unhealthy state
and corresponding K8s events will not cease to exist, causing Acto to wait until the configured timeout
(480 seconds by default). Therefore, this PR aims to interpret K8s events received during the
convergence wait and identify opportunities for aborting the wait to accelerate the total test runtime.