Refactor run_tests.sh script #6280
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6280 +/- ##
==========================================
- Coverage 26.09% 9.82% -16.28%
==========================================
Files 32 32
Lines 2698 2698
==========================================
- Hits 704 265 -439
- Misses 1948 2411 +463
+ Partials 46 22 -24
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
thomasferrandiz
approved these changes
Jul 8, 2024
dereknola
approved these changes
Jul 8, 2024
tests/e2e/scripts/run_tests.sh
Outdated
| } | ||
|
|
||
| ls createreport/rke2_"$OS".log 2>/dev/null && rm createreport/rke2_"$OS".log | ||
| ls createreport/rke2_${date}.log 2>/dev/null && rm createreport/rke2_${date}.log |
There was a problem hiding this comment.
Can we leave it as OS to avoid cleanup of s3 bucket and preferably one sharable link to results for different OS?
There was a problem hiding this comment.
ok, done! I moved all tests to Ubuntu2310 (same like in k3s)
|
|
||
| echo 'RUNNING MIXEDOS TEST' | ||
| /usr/local/go/bin/go test -v mixedos/mixedos_test.go -nodeOS="$nodeOS" -serverCount=$((servercount)) -timeout=1h -json -ci |tee -a createreport/rke2_"$OS".log | ||
| for i in ${!tests[@]}; do |
There was a problem hiding this comment.
Some tests have just server nodes and no agents. Some have HA and some single server single agent. Can we default then in Vagrantfile?
There was a problem hiding this comment.
Yes, vagrantfile defaults the number of servers and agents per test. Same for the OS
Signed-off-by: Manuel Buil <[email protected]>
manuelbuil
force-pushed
the
improveruntestssh
branch
from
0f3ab8f
to
46e2577
Compare
Signed-off-by: Manuel Buil <[email protected]>
manuelbuil
force-pushed
the
improveruntestssh
branch
from
46e2577
to
fdc40dd
Compare
dereknola
approved these changes
Jul 9, 2024
ShylajaDevadiga
approved these changes
Jul 9, 2024
iamsarthakk
pushed a commit
to iamsarthakk/rke2
that referenced
this pull request
Aug 19, 2024
* Improve run_tests script Signed-off-by: Manuel Buil <[email protected]> * Upgrade tests to ubuntu2310 and use nodeOS var Signed-off-by: Manuel Buil <[email protected]> --------- Signed-off-by: Manuel Buil <[email protected]>
dereknola
pushed a commit
to dereknola/rke2
that referenced
this pull request
Nov 6, 2024
* Improve run_tests script Signed-off-by: Manuel Buil <[email protected]> * Upgrade tests to ubuntu2310 and use nodeOS var Signed-off-by: Manuel Buil <[email protected]> --------- Signed-off-by: Manuel Buil <[email protected]>
dereknola
pushed a commit
to dereknola/rke2
that referenced
this pull request
Nov 6, 2024
* Improve run_tests script Signed-off-by: Manuel Buil <[email protected]> * Upgrade tests to ubuntu2310 and use nodeOS var Signed-off-by: Manuel Buil <[email protected]> --------- Signed-off-by: Manuel Buil <[email protected]>
dereknola
pushed a commit
to dereknola/rke2
that referenced
this pull request
Nov 6, 2024
* Improve run_tests script Signed-off-by: Manuel Buil <[email protected]> * Upgrade tests to ubuntu2310 and use nodeOS var Signed-off-by: Manuel Buil <[email protected]> --------- Signed-off-by: Manuel Buil <[email protected]>
This was referenced Nov 6, 2024
dereknola
added a commit
that referenced
this pull request
Nov 7, 2024
* Refactor run_tests.sh script (#6280) * Improve run_tests script Signed-off-by: Manuel Buil <[email protected]> * Upgrade tests to ubuntu2310 and use nodeOS var Signed-off-by: Manuel Buil <[email protected]> --------- Signed-off-by: Manuel Buil <[email protected]> * Update vagrant OS boxes Signed-off-by: Derek Nola <[email protected]> * Add cleanup to e2e tests in vagrant env (#6961) Signed-off-by: ShylajaDevadiga <[email protected]> * Remove dereknola debug action Signed-off-by: Derek Nola <[email protected]> --------- Signed-off-by: Manuel Buil <[email protected]> Signed-off-by: Derek Nola <[email protected]> Signed-off-by: ShylajaDevadiga <[email protected]> Co-authored-by: Manuel Buil <[email protected]> Co-authored-by: Shylaja Devadiga <[email protected]>
dereknola
pushed a commit
that referenced
this pull request
Nov 7, 2024
* Improve run_tests script Signed-off-by: Manuel Buil <[email protected]> * Upgrade tests to ubuntu2310 and use nodeOS var Signed-off-by: Manuel Buil <[email protected]> --------- Signed-off-by: Manuel Buil <[email protected]>
dereknola
pushed a commit
that referenced
this pull request
Nov 7, 2024
* Improve run_tests script Signed-off-by: Manuel Buil <[email protected]> * Upgrade tests to ubuntu2310 and use nodeOS var Signed-off-by: Manuel Buil <[email protected]> --------- Signed-off-by: Manuel Buil <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed Changes
This PR adds some extra logging in the latest_commit.sh script. If there was an error because of the API exceeding the limits, it was impossible to notice
Moreover, it refactors run_tests.sh script trying to make it simpler. Changes:
1 - Removes all the parameters that can be passed except for nodeOS, which is important for the results generation. Some of them depend on the test itself and should not be changed (servercount, agentcount). Some are not read by any test (db or hardened)
2 - Removes
rke2_versionandrke2_channelvariables and references as the former was always empty and the latter was alwayscommit, which is anyway the default: https://github.com/rancher/rke2/blob/master/tests/e2e/vagrantdefaults.rb#L19-L223 - Creates a loop that goes over an array with the different e2e tests available instead of calling each one in a different line
4 - Uses a 2h timeout for all e2e tests. In the end, the important timeout is the one happening in each Test Case (in the Eventually instruction)
5 - Slightly changes the "VM gargabe collection". It was detected that sometimes, VMs are not really cleaned up. That should always happen as we run the tests with the
-ci. We suspect it's a problem related to the timeout, that's why we increased it to 2h as previously explained. Just in case, before starting the e2e test, we destroy the VMs related to that test. At the beginning of the script, we also destroy any VM which might be in an invalid state withvagrant global-status --prune6 - The log file includes now the date in the name instead of the OS
7 - Upgrades all images to Ubuntu2310
Before running each test, it makes sure that VMs are destroyed.
Types of Changes
Code refactor
Verification
Testing
Linked Issues
User-Facing Change
Further Comments