docs: use case - Organize long manifests into smaller ones

DevOpsHiveHQ · Sep 10, 2023 · dad79e0 · dad79e0
1 parent e1eaf91
commit dad79e0
Show file tree

Hide file tree

Showing 14 changed files with 301 additions and 12 deletions.
diff --git a/README.md b/README.md
@@ -17,10 +17,10 @@ A Kustomize generator plugin to merge YAML files seamlessly for real-world use c
 - [Why](#why)
 - [Features](#features)
 - [Options](#options)
-- [Use cases](#use-cases)
+- [Common use cases](#common-use-cases)
   - [1. Generate multiple manifests from a single base](#1-generate-multiple-manifests-from-a-single-base)
   - [2. Merge lists in manifests without schema or a unique identifier](#2-merge-lists-in-manifests-without-schema-or-a-unique-identifier)
-  - [3. Structure long manifests into smaller ones](#3-structure-long-manifests-into-smaller-ones)
+  - [3. Organize long manifests into smaller ones](#3-organize-long-manifests-into-smaller-ones)
 - [TO-DO](#to-do)
 - [Project status](#project-status)
 - [Contributing](#contributing)
@@ -82,7 +82,7 @@ spec:
       # - Patch: Produce a single output by merging all sources together then with the destination.
       method: overlay
       files:
-        # The same as in the KRM container above.
+        # The same as in the KRM container above, omit it if Exec KRM is used.
         root: /mnt
         sources:
         - src01.yaml
@@ -100,7 +100,7 @@ spec:
 ```
 
 
-## Use cases
+## Common use cases
 
 This section shows a couple of use cases where Merger can help.
 
@@ -119,10 +119,10 @@ It's possible to do that using the merge strategy `append` in Merger (later on,
 
 [Use case full example](./examples/manifest-lists-without-schema/README.md).
 
-### 3. Structure long manifests into smaller ones
+### 3. Organize long manifests into smaller ones
 
 In some use cases (e.g., [Crossplane Compositions](https://docs.crossplane.io/latest/concepts/compositions/)),
-you could have a long YAML manifest, and it's hard to read through it. You can split that file
+you could have a really long YAML manifest, and it's hard to read. You can split that file
 and use the Merger `patch` input method to make it a single manifest again.
 
 [Use case full example](./examples/long-omni-manifest/README.md).

diff --git a/dist/.gitkeep b/dist/.gitkeep
diff --git a/examples/krm-and-kustomize/merger.yaml b/examples/krm-and-kustomize/merger.yaml
@@ -22,7 +22,7 @@ spec:
     input:
       method: overlay
       files:
-        # The same as in the KRM container above.
+        # The same as in the KRM container above, omit it if Exec KRM is used.
         root: /mnt
         sources:
         - input/dev.yaml

diff --git a/examples/krm-and-kustomize/resourcelist.yaml b/examples/krm-and-kustomize/resourcelist.yaml
@@ -27,7 +27,7 @@ functionConfig:
       input:
         method: overlay
         files:
-          # The same as in the KRM container above.
+          # The same as in the KRM container above, omit it if Exec KRM is used.
           root: /mnt
           sources:
           - input/dev.yaml

diff --git a/examples/long-omni-manifest/README.md b/examples/long-omni-manifest/README.md
@@ -0,0 +1,194 @@
+<!-- omit in toc -->
+# Example - Organize long manifests into smaller ones
+
+- [Use case](#use-case)
+- [Input](#input)
+- [Manifest](#manifest)
+- [Build](#build)
+- [Output](#output)
+
+## Use case
+
+In some use cases, you could have a really long YAML manifest, and it's hard to read. You can split that file
+and use the Merger `patch` input method to make it a single manifest again.
+
+An example of that is working with [Crossplane Compositions](https://docs.crossplane.io/latest/concepts/compositions/).
+A single composition file could be hundreds of lines. Let's take an example:
+the [EKS composition](https://github.com/upbound/platform-ref-aws/tree/main/package/cluster/eks)
+at [AWS Reference Platform](https://github.com/upbound/platform-ref-aws).
+
+The EKS composition is about 300 lines with 14 resources (in some other scenarios, it could be even more!),
+so Kustomize and Merger could be used together to orchestrate producing that file.
+We will use the 1st 2 out of the 14 resources for demonstration since it's the same logic anyway.
+
+
+## Input
+
+The [package/cluster/eks/composition.yaml](https://github.com/upbound/platform-ref-aws/blob/c972e0cd2fe91ffdb0b181250832fae74e473b10/package/cluster/eks/composition.yaml).
+
+
+```yaml
+apiVersion: apiextensions.crossplane.io/v1
+kind: Composition
+metadata:
+  name: xeks.aws.platformref.upbound.io
+  labels:
+    provider: aws
+spec:
+  writeConnectionSecretsToNamespace: upbound-system
+  compositeTypeRef:
+    apiVersion: aws.platformref.upbound.io/v1alpha1
+    kind: XEKS
+  resources:
+    - base:
+        apiVersion: iam.aws.upbound.io/v1beta1
+        kind: Role
+        metadata:
+          labels:
+            role: controlplane
+        spec:
+          forProvider:
+            assumeRolePolicy: |
+              {
+                "Version": "2012-10-17",
+                "Statement": [
+                    {
+                        "Effect": "Allow",
+                        "Principal": {
+                            "Service": [
+                                "eks.amazonaws.com"
+                            ]
+                        },
+                        "Action": [
+                            "sts:AssumeRole"
+                        ]
+                    }
+                ]
+              }
+      name: controlplaneRole
+    - base:
+        apiVersion: iam.aws.upbound.io/v1beta1
+        kind: RolePolicyAttachment
+        spec:
+          forProvider:
+            policyArn: arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
+            roleSelector:
+              matchControllerRef: true
+              matchLabels:
+                role: controlplane
+      name: clusterRolePolicyAttachment
+    ##############################
+    # More 12 resources redacted #
+    ##############################
+```
+
+## Manifest
+
+```yaml
+# kustomization.yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+generators:
+- merger.yaml
+```
+
+```yaml
+# merger.yaml
+apiVersion: generators.kustomize.aabouzaid.com/v1alpha1
+kind: Merger
+metadata:
+  name: merge
+  annotations:
+    # Containerized KRM function.
+    config.kubernetes.io/function: |
+      container:
+        image: ghcr.io/aabouzaid/kustomize-generator-merger
+        mounts:
+        - type: bind
+          src: ./
+          dst: /mnt
+    # Exec KRM functions.
+    # config.kubernetes.io/function: |
+    # exec:
+    #   path: kustomize-plugin-merger
+spec:
+  resources:
+  - name: my-eks-composition
+    input:
+      # Here patch is used to produce 1 resource from all sources and destination.
+      method: patch
+      files:
+        # The same as in the KRM container above, omit it if Exec KRM is used.
+        root: /mnt
+        sources:
+        # Just the 1st 2 items in the composition resources as an example.
+        - input/controlplaneRole.yaml
+        - input/clusterRolePolicyAttachment.yaml
+        destination: input/composition-base.yaml
+    merge:
+      # Add all list items to the same path.
+      strategy: append
+    output:
+      format: raw
+```
+
+## Build
+
+```shell
+kustomize build  --enable-alpha-plugins --as-current-user .
+```
+
+## Output
+
+```yaml
+apiVersion: apiextensions.crossplane.io/v1
+kind: Composition
+metadata:
+  labels:
+    provider: aws
+  name: xeks.aws.platformref.upbound.io
+spec:
+  compositeTypeRef:
+    apiVersion: aws.platformref.upbound.io/v1alpha1
+    kind: XEKS
+  resources:
+  - base:
+      apiVersion: iam.aws.upbound.io/v1beta1
+      kind: Role
+      metadata:
+        labels:
+          role: controlplane
+      spec:
+        forProvider:
+          assumeRolePolicy: |
+            {
+              "Version": "2012-10-17",
+              "Statement": [
+                  {
+                      "Effect": "Allow",
+                      "Principal": {
+                          "Service": [
+                              "eks.amazonaws.com"
+                          ]
+                      },
+                      "Action": [
+                          "sts:AssumeRole"
+                      ]
+                  }
+              ]
+            }
+    name: controlplaneRole
+  - base:
+      apiVersion: iam.aws.upbound.io/v1beta1
+      kind: RolePolicyAttachment
+      spec:
+        forProvider:
+          policyArn: arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
+          roleSelector:
+            matchControllerRef: true
+            matchLabels:
+              role: controlplane
+    name: clusterRolePolicyAttachment
+  writeConnectionSecretsToNamespace: upbound-system
+```
diff --git a/examples/long-omni-manifest/input/clusterRolePolicyAttachment.yaml b/examples/long-omni-manifest/input/clusterRolePolicyAttachment.yaml
@@ -0,0 +1,15 @@
+apiVersion: apiextensions.crossplane.io/v1
+kind: Composition
+spec:
+  resources:
+    - base:
+        apiVersion: iam.aws.upbound.io/v1beta1
+        kind: RolePolicyAttachment
+        spec:
+          forProvider:
+            policyArn: arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
+            roleSelector:
+              matchControllerRef: true
+              matchLabels:
+                role: controlplane
+      name: clusterRolePolicyAttachment
diff --git a/examples/long-omni-manifest/input/composition-base.yaml b/examples/long-omni-manifest/input/composition-base.yaml
@@ -0,0 +1,12 @@
+apiVersion: apiextensions.crossplane.io/v1
+kind: Composition
+metadata:
+  name: xeks.aws.platformref.upbound.io
+  labels:
+    provider: aws
+spec:
+  writeConnectionSecretsToNamespace: upbound-system
+  compositeTypeRef:
+    apiVersion: aws.platformref.upbound.io/v1alpha1
+    kind: XEKS
+  resources: []
diff --git a/examples/long-omni-manifest/input/controlplaneRole.yaml b/examples/long-omni-manifest/input/controlplaneRole.yaml
@@ -0,0 +1,30 @@
+apiVersion: apiextensions.crossplane.io/v1
+kind: Composition
+spec:
+  resources:
+    - base:
+        apiVersion: iam.aws.upbound.io/v1beta1
+        kind: Role
+        metadata:
+          labels:
+            role: controlplane
+        spec:
+          forProvider:
+            assumeRolePolicy: |
+              {
+                "Version": "2012-10-17",
+                "Statement": [
+                    {
+                        "Effect": "Allow",
+                        "Principal": {
+                            "Service": [
+                                "eks.amazonaws.com"
+                            ]
+                        },
+                        "Action": [
+                            "sts:AssumeRole"
+                        ]
+                    }
+                ]
+              }
+      name: controlplaneRole
diff --git a/examples/long-omni-manifest/merger.yaml b/examples/long-omni-manifest/merger.yaml
@@ -0,0 +1,37 @@
+---
+apiVersion: generators.kustomize.aabouzaid.com/v1alpha1
+kind: Merger
+metadata:
+  name: merge
+  annotations:
+    # Containerized KRM function.
+    config.kubernetes.io/function: |
+      container:
+        image: ghcr.io/aabouzaid/kustomize-generator-merger
+        mounts:
+        - type: bind
+          src: ./
+          dst: /mnt
+    # Exec KRM functions.
+    # config.kubernetes.io/function: |
+    # exec:
+    #   path: kustomize-plugin-merger
+spec:
+  resources:
+  - name: my-eks-composition
+    input:
+      # Here patch is used to produce 1 resource from all sources and destination.
+      method: patch
+      files:
+        # The same as in the KRM container above, omit it if Exec KRM is used.
+        root: /mnt
+        sources:
+        # Just the 1st 2 items in the composition resources as an example.
+        - input/controlplaneRole.yaml
+        - input/clusterRolePolicyAttachment.yaml
+        destination: input/composition-base.yaml
+    merge:
+      # Add all list items to the same path.
+      strategy: append
+    output:
+      format: raw
diff --git a/examples/manifest-lists-without-schema/README.md b/examples/manifest-lists-without-schema/README.md
@@ -86,7 +86,7 @@ spec:
     input:
       method: overlay
       files:
-        # The same as in the KRM container above.
+        # The same as in the KRM container above, omit it if Exec KRM is used.
         root: /mnt
         sources:
         - input/iam-policy-src.yaml

diff --git a/examples/manifest-lists-without-schema/merger.yaml b/examples/manifest-lists-without-schema/merger.yaml
@@ -22,7 +22,7 @@ spec:
     input:
       method: overlay
       files:
-        # The same as in the KRM container above.
+        # The same as in the KRM container above, omit it if Exec KRM is used.
         root: /mnt
         sources:
         - input/iam-policy-src.yaml

diff --git a/examples/multiple-manifests-from-single-file/README.md b/examples/multiple-manifests-from-single-file/README.md
@@ -134,7 +134,7 @@ spec:
     input:
       method: overlay
       files:
-        # The same as in the KRM container above.
+        # The same as in the KRM container above, omit it if Exec KRM is used.
         root: /mnt
         sources:
         - input/cronjob-01.yaml

diff --git a/examples/multiple-manifests-from-single-file/merger.yaml b/examples/multiple-manifests-from-single-file/merger.yaml
@@ -22,7 +22,7 @@ spec:
     input:
       method: overlay
       files:
-        # The same as in the KRM container above.
+        # The same as in the KRM container above, omit it if Exec KRM is used.
         root: /mnt
         sources:
         - input/cronjob-01.yaml

diff --git a/pkg/merger/merge.go b/pkg/merger/merge.go
@@ -45,6 +45,7 @@ func (r *mergerResource) setInputFilesOverlay() {
 
 func (r *mergerResource) setInputFilesPatch() {
 	r.setInputFilesRoot()
+	r.Input.Files.Destination = r.Input.Files.Root + r.Input.Files.Destination
 	for index, inputFileSource := range r.Input.Files.Sources {
 		r.Input.Files.Sources[index] = r.Input.Files.Root + inputFileSource
 	}