TestClusterResourceSetReconciler test is flaky

[Sunnatillo]

Which jobs are flaking?

capi-test-main

Which tests are flaking?

TestClusterResourceSetReconciler

Since when has it been flaking?

Most likely after merging this PR: #10656

Testgrid link

edited: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-test-mink8s-main/1810533764078505984

Reason for failure (if possible)

No response

Anything else we need to know?

No response

Label(s) to be applied

/kind flake
/area ci

[sbueringer]

cc @fabriziopandini Should be the same as the one I mentioned last time

[sbueringer]

@Sunnatillo I think the link is pointing to the wrong job

[Sunnatillo]

@Sunnatillo I think the link is pointing to the wrong job

Fixed now.
Also another occurence:
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-test-main/1807033132835147776

[fabriziopandini]

@jimmidyson PTAL
I will also take a look because I recently made changes on ownerReference management on the same area of the code base

It is important to nail this down before release

[jimmidyson]

I can't see results before 30th June to check for flakiness before #10756 was merged on 28th June. Is that data available? The first flake appeared in f57b8c8 which was merged after that PR too, but like I said I can't tell if the build was stable before that PR or not 😞

[chrischdi]

This page here should allow you to go further back:

https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/periodic-cluster-api-test-mink8s-main?buildId=

As alternative, you could try to filter at k8s-triage

[jimmidyson]

Thanks @chrischdi!

I've gone back to the time of merge of #10656 and I can only see failures for this test after #10756 was merged so I'm pretty sure that introduced the flakiness. I'll take a closer look if I can find time to try to help figure out what's going on.

[Sunnatillo]

Increasing timeout helps to solve the issues.
Spend some time debugging this, find out eventually resources do get created but sometimes timeout is not enough.

[fabriziopandini]

The fact that a ClusterResourceSet binding takes so long to reach a stable state isn't ideal.

The issue is on the fact that we are re-queuing in case of API conflicts, and then next reconciliations are influenced by exponential backoff delay quicky growing (+ the other side of the coin, that many reconcilation are happening in a very short sequence at the beginning of the backoff sequence).

TL;DR exponential backoff should be used to handle errors, not to handle how controllers are reaching a stable state

I have submitted #10869 to get rid of the exponential backoff and all of the noise that API conflict were adding to the logs + documented the problem

But also in this case (like for the timeout increase on the test) this is a mitigation.

[sbueringer]

Let's see if the test is stable now and then close the issue in 1-2 days if the test is stable

(xref: #10868 (comment))

[sbueringer]

Link to check for new occurences https://storage.googleapis.com/k8s-triage/index.html?text=Should_handle_applying_multiple_ClusterResourceSets_concurrently_to_the_same_cluster&job=.*cluster-api.*main&xjob=.*-e2e-.*%7C.*-provider-.*

[Sunnatillo]

Link to check for new occurences https://storage.googleapis.com/k8s-triage/index.html?text=Should_handle_applying_multiple_ClusterResourceSets_concurrently_to_the_same_cluster&job=.*cluster-api.*main&xjob=.*-e2e-.*%7C.*-provider-.*

Not occurred today. It was occurred often before the fix. We can close this issue and the PR.

[Sunnatillo]

I did the test with 200 count, all passed. I would say we are safe to close this issue.
Before it was failing under 50 test counts.

[sbueringer]

Great! Thx for testing

/close

[k8s-ci-robot]

@sbueringer: Closing this issue.

In response to this:

Great! Thx for testing

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.