ConfigGroup V2 (Component)

[EronWright]

Proposed changes

This PR implements a new ConfigGroup resource (v2) as a Pulumi multi-language component (MLC). The new resource is offered side-by-side with the existing (v1) resource; it is not a drop-in replacement at this time.

Some notable differences with respect to the previous (overlay) implementations:

1. Implemented as an MLC to have a consistent implementation across SDKs and to extend support for YAML and Java.
2. The component name is used as a prefix for the child resource names. Use the resourcePrefix property to override. Note that the resourcePrefix is not applied to the Kubernetes object names.
3. Transformations aren't supported (yet). Support for Pulumi-style transformation is being tracked here, and Kubernetes-style transformation may come in future.

Examples

Here's some usage examples.

YAML

name: issue-2844
runtime: yaml
description: A minimal Kubernetes Pulumi YAML program
resources:
  cg:
    type: kubernetes:yaml/v2:ConfigGroup
    properties:
      objs:
      - apiVersion: v1
        kind: Namespace
        metadata:
          name: the-namespace

NodeJS

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";

const configGroup = new kubernetes.yaml.v2.ConfigGroup("my-config-group", {
    objs: [
        {
            apiVersion: "v1",
            kind: "ConfigMap",
            metadata: {
                name: "my-config-map",
            },
            data: {
                "my-key": "my-value",
            },
        },
    ],
});

const cm = configGroup.resources[0].apply(r => r as kubernetes.core.v1.ConfigMap);
export const configmap = pulumi.interpolate`${cm.metadata.namespace}/${cm.metadata.name}`;

Testing

1. A new test suite was developed for the "yaml/v2" package, consisting of:
   a. Unit tests for the core parsing logic (here)
   b. Unit tests for the ConfigGroup resource (here)
2. The test suite for the provider was updated:
   a. New tests for the Construct method (which delegates to a component provider) (here)

Related issues (optional)

Closes #2784

[lblackstone]

A bunch of minor comments, but the overall direction looks good.

One meta-comment: I'd encourage you to write an extended commit message that includes notes about the motivation for this change and explains any nonobvious implementation details. This is a major change that will be scrutinized by maintainers in the future who may not have the full context. Your future self will thank you! :)

[github-actions]

Does the PR have any schema changes?

Looking good! No breaking changes found.

New resources:

• yaml/v2.ConfigGroup

[codecov]

Codecov Report

Attention: Patch coverage is 81.64557% with 29 lines in your changes are missing coverage. Please review.

Project coverage is 26.62%. Comparing base (7246ed3) to head (4388af3).

Files	Patch %	Lines
provider/pkg/provider/yaml/v2/yaml.go	80.39%	11 Missing and 9 partials ⚠️
provider/pkg/provider/yaml/v2/configgroup.go	84.21%	3 Missing and 3 partials ⚠️
provider/pkg/provider/resource/provider.go	0.00%	2 Missing ⚠️
provider/pkg/gen/schema.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2844      +/-   ##
==========================================
+ Coverage   25.46%   26.62%   +1.16%     
==========================================
  Files          48       52       +4     
  Lines        7532     7688     +156     
==========================================
+ Hits         1918     2047     +129     
- Misses       5456     5471      +15     
- Partials      158      170      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lukehoban]

Transformations aren't supported (yet).

I assume they can't be supported?

I suppose when we have support for Pulumi-wide transformations on MLCs, they will work for this as an alternative. But there are some scenarios that I believe the K8s-specific transforms support that Pulumi transforms don't? Do we have a summary of common transforms use cases, and commonality of transforms, to inform whether we're comfortable that users can/will adopt this?

Some notable differences with respect to the previous (overlay) implementations:

Is this list exhaustive? My biggest questions/concern is around potential breaking changes and/or things that are going to make adoption difficult.

The resourcePrefix is applied to the object name (in addition to the resource name), as seen in today's .NET overlay.

By "the object name", does this mean the component itself? Curious why we are aligning with the .NET implementation vs. the others? Not clear apriori that the resourcePrefix should apply to the component (where the user does already have control over the name).

[EronWright]

@lukehoban sorry for the misunderstanding, by "object name" I'm referring to the Kubernetes object name, and I think the .NET implementation had it "right" in doing that. The component name is NOT mangled. I'll clarify.

This new implementation is being offered side-by-side with the existing implementation, it is not (yet) a drop-in replacement. I suppose the new implementation should be characterized as a preview, and will make some adjustments to that effect.

It is true that we have no plan (yet) to provide for Kubernetes-style transformation. I had a brief discussion with the platform team, and it seems like it would necessitate a callback system. The general transformation system is apparently based on an internal callback system that could become the basis in future.

[lukehoban]

This new implementation is being offered side-by-side with the existing implementation, it is not (yet) a drop-in replacement. I suppose the new implementation should be characterized as a preview, and will make some adjustments to that effect.

Just trying to make sure I follow the full plan - is this right?

• Ship this side-by-side as v2
• No plan to be fully compatible, but will keep divergence from v1 as minimal as possible
• We will in principle support both “forever”
• In docs we will point v1 users to suggest using v2
• We will fix bugs in both? Or we will close all bugs in v1 as fixed in v2 - with assumption that 100% of users can and will move to v2 if they want bugs fixed?
• We will consider marking v1 deprecated in a future major version bump of the kubernetes provider?

[lukehoban]

It is true that we have no plan (yet) to provide for Kubernetes-style transformation. I had a brief discussion with the platform team, and it seems like it would necessitate a callback system. The general transformation system is apparently based on an internal callback system that could become the basis in future.

That makes sense - but I’m still really interested in how much impact this will have on the question of “can all v1 users adopt v2?” What are the things someone might be doing in v1, especially with transformations, that are simply not at all possible with v2? what are technically possible but very hard - and will require detailed migration guide to help users through the migration?

[EronWright]

@lukehoban yes that's the plan. I would characterize this new effort as a "preview" until the next major release, when we'll GA the "v2" and kick off the deprecation of "v1". Regarding the bugs in the v1 implementations, I believe we intend to close many as "won't fix" without being too dogmatic.

Transformations are the basis for many use cases, e.g.

1. Resource aliasing and migration
2. Applying labels, namespaces, name prefixes, or image references.
3. Applying environment-specific changes, e.g. setting Service.spec.type to LoadBalancer, injecting environment variables or sidecars.

Pulumi transformation support will help with (1) but might not make for a good experience for (2). Also, Pulumi YAML would assumedly not have transformation support of any kind. For these reasons, I would advocate for providing more built-in transformations (e.g. a namePrefix) for the GA timeframe. For more advanced use cases, the practical option is to use a Kustomize or Helm resource.

To recap: all users will be able to move to the GA of "v2", if we have Pulumi transformations and we also deliver the other resource types.

[EronWright]

The TestEmptyItemNormalization test seems to be flaking for unrelated reasons.

   nodejs_test.go:2226: 
        	Error Trace:	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/tests/sdk/nodejs/nodejs_test.go:2226
        	            				/home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/[email protected]/testing/integration/program.go:1880
        	            				/home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/[email protected]/testing/integration/program.go:1802
        	            				/home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/[email protected]/testing/integration/program.go:1692
        	            				/home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/[email protected]/testing/integration/program.go:1569
        	Error:      	"" does not contain "Welcome to nginx!"
        	Test:       	TestEmptyItemNormalization

[rquitales]

One minor suggestion, but otherwise the code LGTM.