Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix normalizing fields with empty objects/slices #2576

Merged
merged 11 commits into from
Sep 28, 2023

Conversation

rquitales
Copy link
Member

@rquitales rquitales commented Sep 22, 2023

Proposed changes

This PR further improves the resource normalization logic.

Prior to this merge, field values that consists of an empty map[string]interface{} or empty slice would be discarded. This means that normalizing {parentField: {childField: {} }} would result in the childField being unset as the empty object ({}) value for childField is discarded. In Kubernetes resources, especially when defining NetworkPolicies and ingress/egress rules, {} != unset field.

A new test case is added to validate that normalizing NetworkPolicy resources does not result in unwanted behaviour. To do this, the GKE cluster we spin up for testing also enables the Calico network enforcement.

Related issues (optional)

Fixes: #2538

@rquitales rquitales changed the title Rquitales/fix empty items Fix normalizing fields with empty objects/slices. Sep 22, 2023
@rquitales rquitales changed the title Fix normalizing fields with empty objects/slices. Fix normalizing fields with empty objects/slices Sep 22, 2023
@rquitales rquitales requested review from EronWright and a team September 22, 2023 20:32
@github-actions
Copy link

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

@codecov-commenter
Copy link

codecov-commenter commented Sep 22, 2023

Codecov Report

Merging #2576 (6e05dff) into master (8f21952) will increase coverage by 0.28%.
Report is 2 commits behind head on master.
The diff coverage is 92.85%.

❗ Current head 6e05dff differs from pull request most recent head 049b680. Consider uploading reports for the commit 049b680 to get more accurate results

@@            Coverage Diff             @@
##           master    #2576      +/-   ##
==========================================
+ Coverage   18.45%   18.73%   +0.28%     
==========================================
  Files          47       47              
  Lines        9523     9532       +9     
==========================================
+ Hits         1757     1786      +29     
+ Misses       7667     7642      -25     
- Partials       99      104       +5     
Files Coverage Δ
provider/pkg/clients/unstructured.go 59.43% <100.00%> (+0.38%) ⬆️
provider/pkg/provider/util.go 75.67% <90.00%> (+1.38%) ⬆️

... and 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@danielrbradley danielrbradley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little tricky to pick up on all the context here.

  1. What's the purpose of pruneSlice? Why are we discarding items from the array if not in the other array?
  2. Does this only affect the diffing or is the pruned result used for the application of the change?

The change itself looks safe enough to me though. Looks like a good fit for a set of unit tests to help explain why we drop certain values but not others etc.

@rquitales
Copy link
Member Author

rquitales commented Sep 26, 2023

It's a little tricky to pick up on all the context here.

  1. What's the purpose of pruneSlice? Why are we discarding items from the array if not in the other array?
  2. Does this only affect the diffing or is the pruned result used for the application of the change?

The change itself looks safe enough to me though. Looks like a good fit for a set of unit tests to help explain why we drop certain values but not others etc.

From #2445:

14916d3 added logic to prune a map based on a target map. This logic is used to prune live state to match the previous inputs before performing an input diff computation. This avoids the need to store the last-applied-configuration.

This is because the API server may add or remove fields to/from resources that results in semantically equivalent resources to inputs. As such, we normalize the shape of the resulting objects to prevent continuous diff'ing.

@@ -139,6 +139,11 @@ func getActiveClusterFromConfig(config *clientapi.Config, overrides resource.Pro
// pruneMap builds a pruned map by recursively copying elements from the source map that have a matching key in the
// target map. This is useful as a preprocessing step for live resource state before comparing it to program inputs.
func pruneMap(source, target map[string]any) map[string]any {
// If either map is nil, return nil.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good overall, but it would be good to double-check the unit tests for these functions and make sure the updated logic is adequately covered there.

Copy link
Member Author

@rquitales rquitales Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Existing unit tests passed. Have expanded on unit test coverage for these functions in a follow-up commit.

@rquitales rquitales enabled auto-merge (squash) September 28, 2023 20:30
@rquitales rquitales merged commit 8a67a13 into master Sep 28, 2023
17 checks passed
@rquitales rquitales deleted the rquitales/fix-empty-items branch September 28, 2023 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

yaml.NewConfigGroup dropping empty items in NetworkPolicy
4 participants