feat: store test cluster logs and improve test readme [DET-3269] #657
Conversation
webui/tests/bin/e2e-tests.py
Outdated
| @@ -63,13 +63,15 @@ def teardown_cluster(config): | |||
| @contextmanager | |||
| def det_cluster(config): | |||
| try: | |||
| yield setup_cluster(config) | |||
| with open(str(results_dir.joinpath("cluster.stdout.logs")), "w") as f: | |||
There was a problem hiding this comment.
issue: The results_dir might not exist so we run into an error when it attempts to open the cluster.stdout.logs to write. A simple check of the directory and creating it if it doesn't exist should fix this.
| ### Debugging Test Issues | ||
|
|
||
| For reproducing and catching test flakes there is a simple helper script `./bin/try-for-flake.sh`. | ||
| Just executing the script will run `make test` over and over until it hits a error. |
There was a problem hiding this comment.
suggestion: add that you can pass in a parameter to limit the number of times to run tests. eg) ./bin/try-for-flake.sh 30 to run make test 30 times.
There was a problem hiding this comment.
that would be helpful but the script doesn't support this option we can add that (separately) though.
webui/tests/README.md
Outdated
|
|
||
| To speed up this process you can: | ||
|
|
||
| 1. try to avoid the whole cluster set up and tear down on each iteration |
There was a problem hiding this comment.
question: How is this done? Current setup process creates a new db, starts the cluster, sets up the users and experiments. The tests are strongly tied to the db setup. Wondering if there is something we can skip doing during each flake test?
There was a problem hiding this comment.
hmm it depends on if the target tests are re-runnable or not. I added more description here. let me know if that helps
|
|
||
| 1. try to avoid the whole cluster set up and tear down on each iteration | ||
| 2. limit the test scope: | ||
| - use `.skip` on each unwanted test suite or `.only` on the target suite. |
There was a problem hiding this comment.
question: does this actually work with our cypress setup? It looks like it's a mix bag. I tried applying .only to the very first test in 02-loginout.spec.ts and it ran all the other test files fully, and when it came to 02-loginout.spec.ts it did run just the first test where I applied .only... Is that the correct behavior or is there a weird setup going on?
There was a problem hiding this comment.
update: here's what worked for blocking full test files. If you apply x to the top level describe (xdescribe) it stops the entire describe block in that file from running. So to isolate the tests I want to run, I need to add x to all the other files.
There was a problem hiding this comment.
it was mixed bag last time I used it. from my googling, it's about the Jest version that Cypress is using. It is supposed to work so I think it's safe to have it in the readme. I usually just delete the unwanted test suites.
cypress-io/cypress#5441
cypress-io/cypress#632
cypress-io/cypress#5345
hamidzr
force-pushed
the
e2e-tests-flake-catcher
branch
from
aa7883a
to
418430d
Compare
hamidzr
changed the title
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
My prior fix to fixed Docker packaging, but the RPM still had the wrong image name, so change the path in .goreleaser.yml to the expected binary name.
Description
Test Plan
Commentary (optional)