Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ci improvements #1500

Merged
merged 4 commits into from
Jul 29, 2020
Merged

Ci improvements #1500

merged 4 commits into from
Jul 29, 2020

Conversation

aojea
Copy link
Contributor

@aojea aojea commented Jul 14, 2020

- What this PR does and why is it needed

It runs all the [sig-network] tests in the same job, removing current sharding. We are constrained on number of parallel jobs and the new features we are adding, require new independent jobs.

It allows to run the tests in parallel, right now we run with 4 tests in parallel, running a job in 20 minutes less

7 completed jobs in 40m 56s
vs previously that it took few minutes more than 1 hour.

It build the ovn-kubernetes image only once, and share it using github actions artifacts with the multiple jobs.

- Special notes for reviewers

- How to verify it

- Description for the changelog

@aojea
Copy link
Contributor Author

aojea commented Jul 15, 2020

/assign @dave-tucker

.github/workflows/test.yml Outdated Show resolved Hide resolved
.github/workflows/test.yml Outdated Show resolved Hide resolved
.github/workflows/test.yml Outdated Show resolved Hide resolved
@coveralls
Copy link

coveralls commented Jul 15, 2020

Coverage Status

Coverage increased (+0.05%) to 58.607% when pulling fcb25ed on aojea:ciImprovements into f592712 on ovn-org:master.

@aojea
Copy link
Contributor Author

aojea commented Jul 15, 2020

/retest
the test that failed are flake in other runs, this looks good now 32 mins 🚀

@aojea
Copy link
Contributor Author

aojea commented Jul 20, 2020

/retest

@aojea
Copy link
Contributor Author

aojea commented Jul 20, 2020

hmm only the HA tests are failing with etcd timeouts

hopefully we can get the new 3.4.10 version in kind soon with the unsafe fsync feature that will make it more suitable for testing etcd-io/etcd#11946 (cc @girishmg @dcbw )
let's hold until then
/hold

@aojea
Copy link
Contributor Author

aojea commented Jul 27, 2020

I didn't enable the parallel feature yet, it took 7 completed jobs in 1h 18m 28s running all tests serially.This is only 10 mins more than this previous https://github.com/ovn-org/ovn-kubernetes/actions/runs/183674922

This runs 7 completed jobs in 40m 56s, also this approach scales better (with the new amount of new combinations we are testing (shared,loca,ipv4,ipv6,dual,ha,noha) and the report is much cleaner (current sharding is not intuitive when debugging CI jobs, shard-p and shard-np depends on

all tests that don't have P as their sixth letter after the N, and all other tests

/assign @dave-tucker @dcbw @girishmg

.github/workflows/test.yml Outdated Show resolved Hide resolved
test/scripts/e2e-cp.sh Show resolved Hide resolved
.github/workflows/test.yml Show resolved Hide resolved
Antonio Ojea added 3 commits July 27, 2020 18:22
build the ovn-kube image only once

Signed-off-by: Antonio Ojea <[email protected]>
it panics, we have to wait until
kubernetes/kubernetes#93119
and backport to the corresponding branch

Signed-off-by: Antonio Ojea <[email protected]>
Kubernetes upstream has a helper script to run ginkgo,
that allows to can run tests in parallel, that will improve
the time to run the jobs by several orders of magnitude.
From 50 mins to 7 mins to run the whole sig-network tests

This way we don't neet to shard the tests in several jobs.

It also allows to configure retrying flake tests,
We are already retrying PRs, this at least will
give signal on totally broken tests, if a test
is not working after X times, that's something
we should investigate.

Signed-off-by: Antonio Ojea <[email protected]>
.github/workflows/test.yml Show resolved Hide resolved
# all tests that have P as the sixth letter after the N
GINKGO_ARGS="${GINKGO_ARGS} "'--ginkgo.focus=\[sig-network\]\s[Nn].{6}[Pp].*$'
shard-conformance)
FOCUS="\\[Conformance\\]"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'll need this soon

It allows to run tests in parallel, reducing the time of the job
and adding more stress to OVN so we can test its performance.

Currently set to 4 tests in parallel

Signed-off-by: Antonio Ojea <[email protected]>
@trozet trozet merged commit b739a33 into ovn-kubernetes:master Jul 29, 2020
andreaskaris pushed a commit to andreaskaris/ovn-kubernetes that referenced this pull request Jul 18, 2023
…2-fix_ds_acl_cluster

OCPBUGS-6823: [release-4.12] Fix Egress FW ACL rules in dualstack mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants