Add helm plugin scaffolding new layout #3295

camilamacedo86 · 2020-06-25T19:05:30Z

Description of the change:
This PR adds the helm plugin scaffolding the files with the new layout and using the current sdk helm pkg implementation (image)

added e2e tests for helm new layout. See:/test/e2e-helm-new/e2e_suite.go and ./hack/tests/e2e-helm-new.sh
centralize the method ReplaceInFile in the test/internal/utils.go since it is useful for all tests
Impl e2e tests in shell for the new layout. See hack/tests/e2e-helm-new-shell.sh

Motivation

SDK is in a process to be integrated with KB which means that its project layouts will be aligned. More info : Integrating Kubebuilder and Operator SDK.

This PR adds the Helm Plugin which will provide the Helm project in the new layout by the command operator-sdk init --plugins=helm.operator-sdk.io/v1`. For a further understanding see the WIP: #329 and the doc Extensible CLI and Scaffolding Plugins.

Follow-ups

(Blocked by this one) - Fix OLM/CSV gen commands for Helm/Ansible operators - Done in Allows OLM subcommands work for Helm/Ansible new layout #3341
(Blocked by this one) Update docs to use plugins instead of the operator-sdk new --type and add api commands - Done in: docs: update helm docs for the new layout #3326
(Blocked by this one) Remove operator-sdk new --type=-helm removal: helm legacy scaffold #3343

**Description of the change:** Just moving the SDK scaffolds for plugins to the sdk dir and allow it to be re-used. **Motivation for the change:** The SDK scaffolds will be required for other plugins. See for example #3295

joelanford · 2020-07-14T03:25:16Z

hack/tests/e2e-helm-new-shell.sh

+        kubectl logs deployment.apps/nginx-operator-controller-manager -c manager --namespace=${test_namespace}
+        exit 1
+    fi
+


This test is missing the checks that curl the metrics endpoints that currently exists in the legacy shell-based test. We need to check all the same scenarios to make sure we don't accidentally break something in the transition.

hack/tests/e2e-helm-new-shell.sh

internal/plugins/helm/v1/chartutil/chart.go

internal/plugins/helm/v1/init.go

internal/plugins/helm/v1/scaffolds/api.go

internal/plugins/helm/v1/scaffolds/doc.go

internal/plugins/helm/v1/scaffolds/init.go

joelanford · 2020-07-14T03:53:37Z

test/e2e-helm-new/e2e_suite.go

+
+// Modified from https://github.com/kubernetes-sigs/kubebuilder/tree/39224f0/test/e2e/v3
+
+package e2e


This file isn't testing much more than what currently exists in the new shell based file.

Rather than adding two additional helm e2e tests (which would mean we'd have 3 total e2e tests just for helm!), I think for this PR we should stick with just the shell-based test.

Then in a follow-on, we can migrate the logic to this file and remove the shell-based tests.

internal/plugins/helm/v1/scaffolds/templates/makefile.go

joelanford · 2020-07-14T12:00:09Z

hack/lib/common.sh

+function setup_kustomize {
+  header_text "getting kustomize"
+  curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
+  chmod u+x kustomize
+  mv kustomize "$1"/bin/kustomize
+}


Do we need this? I figured that since the make targets that require kustomize depend on it, and then the kustomize target would download it if it isn't already on the path.

Also it looks like the install_kustomize.sh script downloads the latest version of kustomize, whereas the Makefile has a hardcoded version. Shouldn't we be testing with the version that is scaffolded in the Makefile?

asmacdo · 2020-07-14T14:22:06Z

internal/plugins/helm/v1/scaffolds/api.go

@@ -0,0 +1,134 @@
+/*
+Copyright 2019 The Kubernetes Authors.
+Modifications copyright 2020 The Operator-SDK Authors


This looks deliberate, but I am surprised that this file wouldnt be a full Operator-SDK copyright file

The file is copied from KB and modified then, we need to respect the license. See: https://softwareengineering.stackexchange.com/questions/157968/how-to-manage-a-copyright-notice-in-an-open-source-project

In that case this should be:

Original work Copyright 2019 The Kubernetes Authors. Modified work Copyright 2020 The Operator-SDK Authors

Along with any other file that was copied with modifications.

joelanford · 2020-07-14T14:27:05Z

internal/plugins/helm/v1/scaffolds/templates/makefile.go


 kustomize:
 ifeq (, $(shell which kustomize 2>/dev/null))
 	@{ \
 	set -e ;\
 	mkdir -p bin ;\
-	curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/{{ .KustomizeVersion }}/kustomize_{{ .KustomizeVersion }}_${OS}_${ARCH}.tar.gz | tar xzf - -C bin/ ;\
+	echo https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/{{ .KustomizeVersion }}/kustomize_{{ .KustomizeVersion }}_$(OS)_$(ARCH).tar.gz ;\


Remove this echo line?

joelanford · 2020-07-14T14:35:02Z

internal/plugins/helm/v1/scaffolds/templates/makefile.go

-	if f.Image == "" {
-		f.Image = "controller:latest"
-	}


I see that we're explicitly setting f.Image in the scaffold calls, but with this defaulter removed, it is now possible to use templates.Makefile and end up with an invalid Makefile.

We either need to return an error if f.Image is not set OR we need to keep this as the default.

My preference would be to stay aligned with Kubebuilder's Go plugin, which leaves this conditional in so that f.Image is defaulted to controller:latest if it isn't explicitly set by a caller.

joelanford · 2020-07-14T14:37:39Z

hack/tests/e2e-helm.sh

@@ -1,169 +1,29 @@
 #!/usr/bin/env bash

-set -eux
+set -o errexit


Revert this file back to the legacy Helm e2e test.

In a follow-up (when the legacy Helm scaffolding is removed) we can rename e2e-helm-new-shell.sh to e2e-helm.sh. And in a separate follow-up, we can port the actual tests from Shell-based tests to Go-based tests and update this file accordingly.

joelanford · 2020-07-14T14:39:45Z

test/e2e-helm/e2e_suite.go

@@ -0,0 +1,157 @@
+// Copyright 2020 The Operator-SDK Authors


Remove this file for now. In a follow-up, we can port the shell-based test to a Go-based test and reintroduce this when it covers all of the existing test scenarios.

joelanford · 2020-07-14T14:40:07Z

test/e2e-helm/e2e_test.go

@@ -0,0 +1,29 @@
+// Copyright 2020 The Operator-SDK Authors


Ditto: https://github.com/operator-framework/operator-sdk/pull/3295/files#r454406994

camilamacedo86 · 2020-07-14T16:59:09Z

I am closing because I am testing in my fork. I will re-open soon.

estroz · 2020-07-14T17:50:56Z

test/internal/utils.go

@@ -53,3 +57,14 @@ func (tc TestContext) UninstallOLM() {
 func (tc TestContext) KustomizeBuild(dir string) ([]byte, error) {
 	return tc.Run(exec.Command("kustomize", "build", dir))
 }
+
+// ReplaceInFile replaces all instances of old with new in the file at path.
+func ReplaceInFile(path, old, new string) {


Where else is this being used? If it isn't revert this back to being unexported in Go e2e tests.

I will revert it was added here to be used in both go tests. Now it will be part of a follow-up 👍

estroz · 2020-07-14T17:52:33Z

go.mod

@@ -55,4 +57,5 @@ replace (
 	github.com/Azure/go-autorest => github.com/Azure/go-autorest v13.3.2+incompatible // Required by OLM
 	github.com/mattn/go-sqlite3 => github.com/mattn/go-sqlite3 v1.10.0
 	k8s.io/client-go => k8s.io/client-go v0.18.2
+


Can you run make tidy before committing this?

make tidy do not remove the space. But I fixed that 👍 Tks.

estroz · 2020-07-14T17:53:25Z

hack/tests/e2e-helm-new.sh

@@ -0,0 +1,156 @@
+#!/usr/bin/env bash


IIRC we want to add all new e2e tests as ginkgo/gomega suites. Perhaps that is too much extra code for this PR which is already large, so at least do this in a follow-up if that's how we want to proceed with e2e tests.

Hi @estroz,

The first version was with the tests done in go like you wish 👍 . However, @joelanford asked for we keep here the same tests in the shell just for we ensure now that all is 100% then we can do a follow with the go tests as well.

camilamacedo86 · 2020-07-14T18:05:57Z

/reopen

openshift-ci-robot · 2020-07-14T18:05:58Z

@camilamacedo86: Failed to re-open PR: state cannot be changed. The helm-external-plugin branch was force-pushed or recreated.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 25, 2020

openshift-ci-robot requested review from hasbro17 and theishshah June 25, 2020 19:05

camilamacedo86 force-pushed the helm-external-plugin branch 6 times, most recently from 2144a66 to 006f6df Compare June 25, 2020 20:19

camilamacedo86 requested review from estroz, joelanford and asmacdo June 25, 2020 20:19

camilamacedo86 force-pushed the helm-external-plugin branch 2 times, most recently from 6371b1d to 8b7f38f Compare June 26, 2020 00:40

camilamacedo86 changed the title ~~WIP - add helm plugin using joes (not merge)~~ WIP - add helm plugin scaffolding new layout with legacy implementation Jun 27, 2020

camilamacedo86 force-pushed the helm-external-plugin branch 2 times, most recently from bcdd1b8 to 2cb554d Compare June 27, 2020 13:47

camilamacedo86 added language/helm Issue is related to a Helm operator project kubebuilder-integration Relates to rewriting the SDK in Kubebuilder plugin form labels Jun 27, 2020

camilamacedo86 force-pushed the helm-external-plugin branch 11 times, most recently from 456eef9 to 001281c Compare June 30, 2020 02:11

camilamacedo86 force-pushed the helm-external-plugin branch from d07c2f2 to 522816f Compare July 13, 2020 10:11

camilamacedo86 force-pushed the helm-external-plugin branch from 58895ee to 20e0317 Compare July 13, 2020 21:10

camilamacedo86 removed the blocked label Jul 13, 2020

joelanford requested changes Jul 14, 2020

View reviewed changes

camilamacedo86 force-pushed the helm-external-plugin branch 4 times, most recently from 050a0bf to 959cef6 Compare July 14, 2020 10:33

camilamacedo86 requested a review from joelanford July 14, 2020 11:08

joelanford reviewed Jul 14, 2020

View reviewed changes

asmacdo reviewed Jul 14, 2020

View reviewed changes

joelanford requested changes Jul 14, 2020

View reviewed changes

joelanford reviewed Jul 14, 2020

View reviewed changes

camilamacedo86 force-pushed the helm-external-plugin branch 3 times, most recently from 9a2f82a to 799c08e Compare July 14, 2020 15:46

feat: add helm plugin scaffolding new layout

0219e24

camilamacedo86 force-pushed the helm-external-plugin branch from 799c08e to 0219e24 Compare July 14, 2020 16:16

camilamacedo86 added 2 commits July 14, 2020 17:43

test

1194ff9

test 2

93f1319

camilamacedo86 closed this Jul 14, 2020

estroz reviewed Jul 14, 2020

View reviewed changes

camilamacedo86 mentioned this pull request Jul 14, 2020

feat: add helm plugin scaffolding new layout #3421

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add helm plugin scaffolding new layout #3295

Add helm plugin scaffolding new layout #3295

camilamacedo86 commented Jun 25, 2020 •

edited

Loading

joelanford Jul 14, 2020

joelanford Jul 14, 2020

joelanford Jul 14, 2020

asmacdo Jul 14, 2020

camilamacedo86 Jul 14, 2020 •

edited

Loading

estroz Jul 14, 2020

estroz Jul 14, 2020

joelanford Jul 14, 2020

joelanford Jul 14, 2020

joelanford Jul 14, 2020

joelanford Jul 14, 2020

joelanford Jul 14, 2020

camilamacedo86 commented Jul 14, 2020

estroz Jul 14, 2020

camilamacedo86 Jul 14, 2020 •

edited

Loading

estroz Jul 14, 2020

camilamacedo86 Jul 14, 2020

estroz Jul 14, 2020 •

edited

Loading

camilamacedo86 Jul 14, 2020

camilamacedo86 commented Jul 14, 2020

openshift-ci-robot commented Jul 14, 2020


		// Modified from https://github.com/kubernetes-sigs/kubebuilder/tree/39224f0/test/e2e/v3

		package e2e

Add helm plugin scaffolding new layout #3295

Add helm plugin scaffolding new layout #3295

Conversation

camilamacedo86 commented Jun 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camilamacedo86 Jul 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camilamacedo86 commented Jul 14, 2020

Choose a reason for hiding this comment

camilamacedo86 Jul 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

estroz Jul 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camilamacedo86 commented Jul 14, 2020

openshift-ci-robot commented Jul 14, 2020

camilamacedo86 commented Jun 25, 2020 •

edited

Loading

camilamacedo86 Jul 14, 2020 •

edited

Loading

camilamacedo86 Jul 14, 2020 •

edited

Loading

estroz Jul 14, 2020 •

edited

Loading