[defect] install sparkoperator by helm failed with unknown field "annotations"

[mstiunicon]

I want to install sparkoperator in IBM cloud, but failed with unknown field "annotations"
Steps:

1. login IBM cloud
   ibmcloud login -a cloud.ibm.com -r us-south -g Metering --sso -u [email protected]
2. check kubernets

PS C:\workspace_go> kubectl version --short
Client Version: v1.16.2
Server Version: v1.16.3+IKS
PS C:\workspace_go> kubectl cluster-info
Kubernetes master is running at https://xxx.us-south.containers.cloud.ibm.com:20089
CoreDNS is running at https://xxx.us-south.containers.cloud.ibm.com:20089/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
kubernetes-dashboard is running at https://xxx.us-south.containers.cloud.ibm.com:20089/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
Metrics-server is running at https://xxx.us-south.containers.cloud.ibm.com:20089/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
PS C:\workspace_go>

3. install sparkoperator by helm

PS C:\workspace_go> helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
"incubator" has been added to your repositories
PS C:\workspace_go> helm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default --generate-name
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(CustomResourceDefinition): unknown field "annotations" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinition
PS C:\workspace_go>
PS C:\workspace_go> kubectl delete CustomResourceDefinition -l app.kubernetes.io/name=sparkoperator
customresourcedefinition.apiextensions.k8s.io "scheduledsparkapplications.sparkoperator.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "sparkapplications.sparkoperator.k8s.io" deleted
PS C:\workspace_go>
PS C:\workspace_go> helm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default --generate-name
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(CustomResourceDefinition): unknown field "annotations" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinition
PS C:\workspace_go>

Errors:

1. installing failed by:
   Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(CustomResourceDefinition): unknown field "annotations" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinition

I searched keyword "annotations" by google, and also in the repository "spark-on-k8s-operator", no such error, so I submit a defect, hope any helps

Thanks

[liyinan926]

This seems caused by this commit: helm/charts@0f5eef5. There's a wrong indention for the field annotations in the cloned CRD manifest under the new crds folder. Will fix that.

[liyinan926]

Created helm/charts#19654 to fix the issue.

[liyinan926]

The fix has been merged. The new version of the Chart is 0.6.1. Please give it a try. Thanks!

[mstiunicon]

Not sure whether the installation was successful, cause there is an error message.

PS C:\workspace_go> helm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default --generate-name
Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: CustomResourceDefinition, namespace: , name: scheduledsparkapplications.sparkoperator.k8s.io
PS C:\workspace_go>
PS C:\workspace_go>
PS C:\workspace_go> kubectl delete CustomResourceDefinition sparkapplications.sparkoperator.k8s.io scheduledsparkapplications.sparkoperator.k8s.io
customresourcedefinition.apiextensions.k8s.io "sparkapplications.sparkoperator.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "scheduledsparkapplications.sparkoperator.k8s.io" deleted
PS C:\workspace_go>
PS C:\workspace_go>
PS C:\workspace_go>
PS C:\workspace_go> helm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default --generate-name
Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: CustomResourceDefinition, namespace: , name: sparkapplications.sparkoperator.k8s.io
PS C:\workspace_go>
PS C:\workspace_go> kubectl delete CustomResourceDefinition -l app.kubernetes.io/name=sparkoperator
customresourcedefinition.apiextensions.k8s.io "scheduledsparkapplications.sparkoperator.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "sparkapplications.sparkoperator.k8s.io" deleted
PS C:\workspace_go>
PS C:\workspace_go>
PS C:\workspace_go> kubectl delete CustomResourceDefinition sparkapplications.sparkoperator.k8s.io scheduledsparkapplications.sparkoperator.k8s.io
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "sparkapplications.sparkoperator.k8s.io" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "scheduledsparkapplications.sparkoperator.k8s.io" not found
PS C:\workspace_go>
PS C:\workspace_go> helm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default --generate-name
Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: CustomResourceDefinition, namespace: , name: sparkapplications.sparkoperator.k8s.io
PS C:\workspace_go>
PS C:\workspace_go>
PS C:\workspace_go> helm version
version.BuildInfo{Version:"v3.0.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}
PS C:\workspace_go> helm list
NAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION
PS C:\workspace_go> helm list -a
NAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION

PS C:\workspace_go> helm inspect chart incubator/sparkoperator
apiVersion: v1
appVersion: v1beta2-1.0.1-2.4.4
description: A Helm chart for Spark on Kubernetes operator
home: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
keywords:
- spark
maintainers:
- email: [email protected]
  name: yuchaoran2011
name: sparkoperator
version: 0.6.1

PS C:\workspace_go>

But I checked the kubernetes dashboard, the Custom Resource Definitions "sparkapplications" and "scheduledsparkapplications" can be found in list

[huineng]

see my conversation in slack https://app.slack.com/client/T09NY5SBT/CALBDHMTL/thread/CALBDHMTL-1576435096.006900

[mstiunicon]

Looks I can't access the workspace

would you please invite me ([email protected]) to the workspace?

[huineng]

this is how i solved it

First I fixed this annotations indent, then i installed with the --skip-crds flag
It still installs the crds, so there's a double install of crd in the code giving you this error

[liyinan926]

See helm/charts#19236.

[mstiunicon]

@huineng @liyinan926 thanks for help
I have installed sparkoperator successfully with --skip-crds flag

PS C:\workspace_go> helm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default --skip-crds --generate-name
NAME: sparkoperator-1576726158
LAST DEPLOYED: Thu Dec 19 11:29:22 2019
NAMESPACE: spark-operator
STATUS: deployed
REVISION: 1
TEST SUITE: None
PS C:\workspace_go>