OCPBUGS-36734,OCPBUGS-36733,OCPBUGS-36731,OCPBUGS-36730: 4.15 critical bugs #954

zeeke · 2024-07-05T10:05:04Z

4.15 Backport of:

The generic plugin was applying config changes only if the desired spec of interfaces was different from the last applied spec. This logic is different from the one in OnNodeStateChange where the real status of the interfaces is used to detect changes. By removing the LastState parameter (and related code), the generic plugin will also use the real status of interfaces to decide whether to apply changes or not. The SyncNodeState function has this logic.

Users could modify the settings of VFs which have been configured by the sriov operator. This PR starts the reconciliation loop when these changes are detected in the generic plugin. Signed-off-by: Marcelo Guerrero <[email protected]>

Logic to check missing kernel arguments is placed in a method to be used by both OnNodeStateChange and CheckStatusChanges.

zeeke · 2024-07-05T10:17:57Z

/hold

waiting for

OCPBUGS-36509,OCPBUGS-36507: Backport critical bugs 4.16 #953

Link the correct jira backports before merging

Webhook resources (`ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration`) in OpenShift are configured with `service.beta.openshift.io/inject-cabundle` in a way that a third component fills the ClientConfig.CABundle field of the webhook. When reconciling webhooks, do not override the field and avoid a flakiness, as there might be a time slot in which the API server is not configured with a valid client certificate: ``` Error from server (InternalError): error when creating "policies": Internal error occurred: failed calling webhook "operator-webhook.sriovnetwork.openshift.io": failed to call webhook: Post "https://operator-webhook-service.openshift-sriov-network-operator.svc:443/mutating-custom-resource?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority ``` The same behavior also happens when using CertManager Refs: - https://docs.openshift.com/container-platform/4.15/security/certificates/service-serving-certificate.html - https://issues.redhat.com/browse/OCPBUGS-32139 - https://cert-manager.io/docs/concepts/ca-injector/ Signed-off-by: Andrea Panattoni <[email protected]>

we need to be consistent with the policy order Signed-off-by: Sebastian Sch <[email protected]>

SchSeba · 2024-07-07T08:03:10Z

Hi @zeeke please remove the func test for mtu that is only for 4.16+ everything else looks good :)

When the MTU set in the SRIOV Network Node Policy is lower than the actual MTU of the PF, it triggers the reconcile loop for the Node state indefinitely, preventing the configuration from completing. Signed-off-by: amaslennikov <[email protected]>

If a Virtual Function is configured with a DPDK driver (e.g. `vfio-pci`) and it is not referred by any SriovNetworkNodePolicy, `NeedToUpdateSriov` function must not trigger a reconfiguration. This may happen if a PF is configured by multiple policies (via PF partitioning) and a policy is deleted by the user. In these cases, the VF is not reconfigured [1] and a drain loop is started The same logic applies to VDPA devices. refs: [1] https://github.com/k8snetworkplumbingwg/sriov-network-operator/blob/5f3c4e903f789aa177fe54686efd6c18576b7ab1/pkg/host/internal/sriov/sriov.go#L457 Signed-off-by: Andrea Panattoni <[email protected]>

It's possible to have a race in the VFIsReady function. vf netdevice can have a default eth0 device name and be the time we call the netlink syscall to get the device information eth0 can be a different device. this cause duplicate mac allocation on vf admin mac address Signed-off-by: Sebastian Sch <[email protected]>

Signed-off-by: Andrea Panattoni <[email protected]>

openshift-ci · 2024-07-08T09:42:31Z

@zeeke: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-openstack-nfv-hwoffload	`9c27b3f`	link	false	`/test e2e-openstack-nfv-hwoffload`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

zeeke · 2024-07-09T09:15:49Z

/jira cherrypick OCPBUGS-36507

openshift-ci-robot · 2024-07-09T09:15:57Z

@zeeke: Jira Issue OCPBUGS-36507 has been cloned as Jira Issue OCPBUGS-36730. Will retitle bug to link to clone.
/retitle OCPBUGS-36730: 4.15 critical bugs

In response to this:

/jira cherrypick OCPBUGS-36507

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

zeeke · 2024-07-09T09:16:01Z

/jira cherrypick OCPBUGS-36308

openshift-ci-robot · 2024-07-09T09:16:06Z

@zeeke: This pull request references Jira Issue OCPBUGS-36730, which is invalid:

expected dependent Jira Issue OCPBUGS-36507 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

4.15 Backport of:

Avoid reconciling field Webhook.ClientConfig.CABundle k8snetworkplumbingwg/sriov-network-operator#711

add sort for the policy on the sriovOperatorConfig controller k8snetworkplumbingwg/sriov-network-operator#721

Verify status changes on managed interfaces k8snetworkplumbingwg/sriov-network-operator#530

Fix issue preventing lowering of PF's MTU k8snetworkplumbingwg/sriov-network-operator#700

Add mtu functional test k8snetworkplumbingwg/sriov-network-operator#699

Avoid reconfiguring unmentioned DPDK VFs k8snetworkplumbingwg/sriov-network-operator#710

introduce a recheck for the VF interface name k8snetworkplumbingwg/sriov-network-operator#720

cc @SchSeba, @mlguerrero12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-07-09T09:16:09Z

@zeeke: Jira Issue OCPBUGS-36308 has been cloned as Jira Issue OCPBUGS-36731. Will retitle bug to link to clone.
/retitle OCPBUGS-36731: OCPBUGS-36730: 4.15 critical bugs

In response to this:

/jira cherrypick OCPBUGS-36308

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

SchSeba · 2024-07-09T10:15:50Z

/lgtm
/approve
/label backport-risk-assessed

openshift-ci · 2024-07-09T10:24:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SchSeba, zeeke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [SchSeba]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ajaggapa · 2024-07-09T11:48:49Z

/label cherry-pick-approved

zeeke · 2024-07-09T12:17:00Z

/hold cancel
/jira refresh

openshift-ci-robot · 2024-07-09T12:17:12Z

@zeeke: This pull request references Jira Issue OCPBUGS-36734, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.15.z) matches configured target version for branch (4.15.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-36509 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-36509 targets the "4.16.z" version, which is one of the valid target versions: 4.16.0, 4.16.z
bug has dependents

Requesting review from QA contact:
/cc @zhaozhanqi

This pull request references Jira Issue OCPBUGS-36733, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.15.z) matches configured target version for branch (4.15.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-36307 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-36307 targets the "4.16.z" version, which is one of the valid target versions: 4.16.0, 4.16.z
bug has dependents

Requesting review from QA contact:
/cc @zhaozhanqi

This pull request references Jira Issue OCPBUGS-36731, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.15.z) matches configured target version for branch (4.15.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-36308 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-36308 targets the "4.16.z" version, which is one of the valid target versions: 4.16.0, 4.16.z
bug has dependents

Requesting review from QA contact:
/cc @zhaozhanqi

This pull request references Jira Issue OCPBUGS-36730, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.15.z) matches configured target version for branch (4.15.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-36507 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-36507 targets the "4.16.z" version, which is one of the valid target versions: 4.16.0, 4.16.z
bug has dependents

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

/hold cancel
/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-07-09T12:20:16Z

@zeeke: Jira Issue OCPBUGS-36734: All pull requests linked via external trackers have merged:

openshift/sriov-network-operator#954

Jira Issue OCPBUGS-36734 has been moved to the MODIFIED state.

Jira Issue OCPBUGS-36733: All pull requests linked via external trackers have merged:

openshift/sriov-network-operator#954

Jira Issue OCPBUGS-36733 has been moved to the MODIFIED state.

Jira Issue OCPBUGS-36731: All pull requests linked via external trackers have merged:

openshift/sriov-network-operator#954

Jira Issue OCPBUGS-36731 has been moved to the MODIFIED state.

Jira Issue OCPBUGS-36730: All pull requests linked via external trackers have merged:

openshift/sriov-network-operator#954

Jira Issue OCPBUGS-36730 has been moved to the MODIFIED state.

In response to this:

4.15 Backport of:

Avoid reconciling field Webhook.ClientConfig.CABundle k8snetworkplumbingwg/sriov-network-operator#711

add sort for the policy on the sriovOperatorConfig controller k8snetworkplumbingwg/sriov-network-operator#721

Verify status changes on managed interfaces k8snetworkplumbingwg/sriov-network-operator#530

Fix issue preventing lowering of PF's MTU k8snetworkplumbingwg/sriov-network-operator#700

Add mtu functional test k8snetworkplumbingwg/sriov-network-operator#699

Avoid reconfiguring unmentioned DPDK VFs k8snetworkplumbingwg/sriov-network-operator#710

introduce a recheck for the VF interface name k8snetworkplumbingwg/sriov-network-operator#720

cc @SchSeba, @mlguerrero12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

zeeke · 2024-07-09T12:21:16Z

/cherrypick release-4.14

openshift-cherrypick-robot · 2024-07-09T12:22:03Z

@zeeke: #954 failed to apply on top of branch "release-4.14":

Applying: Remove LastState parameter of GenericPlugin
Applying: Verify status changes on managed interfaces
Applying: Add e2e test for reconciliation when status changes
Applying: Abstract logic to check and load missing KArgs
Applying: Avoid reconciling field `Webhook.ClientConfig.CABundle`
Applying: add sort for the policy on the sriovOperatorConfig controller
Applying: Fix issue with infinite reconciling higher MTU
Applying: Avoid reconfiguring unmentioned DPDK VFs
Applying: Use interface index instead of name
Applying: d/s: run `make deps-update`
.git/rebase-apply/patch:3018: trailing whitespace.
	
.git/rebase-apply/patch:3018: new blank line at EOF.
+
warning: 2 lines add whitespace errors.
Using index info to reconstruct a base tree...
M	go.mod
M	go.sum
Falling back to patching base and 3-way merge...
Removing vendor/golang.org/x/sys/windows/empty.s
Removing vendor/golang.org/x/sys/unix/fstatfs_zos.go
Removing vendor/golang.org/x/sys/unix/epoll_zos.go
Auto-merging go.sum
CONFLICT (content): Merge conflict in go.sum
Auto-merging go.mod
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0010 d/s: run `make deps-update`
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherrypick release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-bot · 2024-07-09T14:09:46Z

[ART PR BUILD NOTIFIER]

This PR has been included in build sriov-network-operator-container-v4.15.0-202407091338.p0.g944b717.assembly.stream.el9 for distgit sriov-network-operator.
All builds following this will include this PR.

mlguerrero12 added 4 commits July 5, 2024 11:48

Verify status changes on managed interfaces

785e646

Users could modify the settings of VFs which have been configured by the sriov operator. This PR starts the reconciliation loop when these changes are detected in the generic plugin. Signed-off-by: Marcelo Guerrero <[email protected]>

Add e2e test for reconciliation when status changes

7c5573e

Abstract logic to check and load missing KArgs

3e67dc8

Logic to check missing kernel arguments is placed in a method to be used by both OnNodeStateChange and CheckStatusChanges.

openshift-ci bot requested review from pliurh and SchSeba July 5, 2024 10:05

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 5, 2024

zeeke and others added 2 commits July 5, 2024 14:08

add sort for the policy on the sriovOperatorConfig controller

f6405e3

we need to be consistent with the policy order Signed-off-by: Sebastian Sch <[email protected]>

zeeke force-pushed the 415-critical-bugs branch from 8de0c5c to 51dc56f Compare July 5, 2024 12:09

almaslennikov and others added 4 commits July 8, 2024 09:54

d/s: run make deps-update

9c27b3f

Signed-off-by: Andrea Panattoni <[email protected]>

zeeke force-pushed the 415-critical-bugs branch from 51dc56f to 9c27b3f Compare July 8, 2024 07:54

openshift-ci bot changed the title ~~4.15 critical bugs~~ OCPBUGS-36730: 4.15 critical bugs Jul 9, 2024

openshift-ci bot changed the title ~~OCPBUGS-36730: 4.15 critical bugs~~ OCPBUGS-36731: OCPBUGS-36730: 4.15 critical bugs Jul 9, 2024

openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. and removed jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. labels Jul 9, 2024

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Jul 9, 2024

openshift-ci bot assigned anuragthehatter, asood-rh, huiran0826, jechen0648, mffiedler, qiowang721, rbbratta, weliang1, yingwang-0320 and zhaozhanqi Jul 9, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 9, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 9, 2024

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jul 9, 2024

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 9, 2024

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 9, 2024

openshift-merge-bot bot merged commit 944b717 into openshift:release-4.15 Jul 9, 2024
9 of 10 checks passed

zeeke mentioned this pull request Jul 9, 2024

OCPBUGS-36753,OCPBUGS-36754,OCPBUGS-36755OCPBUGS-36756: [release-4.14] Critical Bugs #961

Merged

This was referenced Jul 16, 2024

OCPBUGS-36127: Bump github.com/hashicorp/go-retryablehttp from 0.7.0 to 0.7.7 #960

Merged

[OCPBUGS-36130] Bump github.com/hashicorp/go-retryablehttp from 0.7.0 to 0.7.7 #957

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-36734,OCPBUGS-36733,OCPBUGS-36731,OCPBUGS-36730: 4.15 critical bugs #954

OCPBUGS-36734,OCPBUGS-36733,OCPBUGS-36731,OCPBUGS-36730: 4.15 critical bugs #954

zeeke commented Jul 5, 2024

zeeke commented Jul 5, 2024

SchSeba commented Jul 7, 2024

openshift-ci bot commented Jul 8, 2024

zeeke commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

zeeke commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

SchSeba commented Jul 9, 2024

openshift-ci bot commented Jul 9, 2024

ajaggapa commented Jul 9, 2024

zeeke commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

zeeke commented Jul 9, 2024

openshift-cherrypick-robot commented Jul 9, 2024

openshift-bot commented Jul 9, 2024

OCPBUGS-36734,OCPBUGS-36733,OCPBUGS-36731,OCPBUGS-36730: 4.15 critical bugs #954

OCPBUGS-36734,OCPBUGS-36733,OCPBUGS-36731,OCPBUGS-36730: 4.15 critical bugs #954

Conversation

zeeke commented Jul 5, 2024

zeeke commented Jul 5, 2024

SchSeba commented Jul 7, 2024

openshift-ci bot commented Jul 8, 2024

zeeke commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

zeeke commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

SchSeba commented Jul 9, 2024

openshift-ci bot commented Jul 9, 2024

ajaggapa commented Jul 9, 2024

zeeke commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

openshift-ci-robot commented Jul 9, 2024

zeeke commented Jul 9, 2024

openshift-cherrypick-robot commented Jul 9, 2024

openshift-bot commented Jul 9, 2024