Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubernetes.helm.v4.Chart: failed to parse kubeconfig data on preview #3027

Open
yellowhat opened this issue May 27, 2024 · 8 comments
Open
Labels
area/helm area/preview kind/bug Some behavior is incorrect or out of spec needs-design Needs input from design/UX

Comments

@yellowhat
Copy link

What happened?

Hi,
I would like to use the new kubernetes.helm.v4.Chart, to install an helm chart after the cluster creation.

But unfortunately if fails on the preview steps, as it expects the cluster to be already available:

    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1010, in do_rpc_call
        return monitor.RegisterResource(req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib64/python3.12/site-packages/grpc/_channel.py", line 1160, in __call__
        return _end_unary_response_blocking(state, call, False, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib64/python3.12/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
        raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNKNOWN
    	details = "configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig`- couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }"
    	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-05-27T11:51:11.555748701+00:00", grpc_status:2, grpc_message:"configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig`- couldn\'t get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" }"}"
    >
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/bin/pulumi-language-python-exec", line 191, in <module>
        loop.run_until_complete(coro)
      File "/usr/lib64/python3.12/asyncio/base_events.py", line 687, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 138, in run_in_stack
        await run_pulumi_func(run)
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 52, in run_pulumi_func
        await wait_for_rpcs()
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 114, in wait_for_rpcs
        await task
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1015, in do_register
        resp = await asyncio.get_event_loop().run_in_executor(None, do_rpc_call)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1012, in do_rpc_call
        handle_grpc_error(exn)
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/settings.py", line 307, in handle_grpc_error
        raise grpc_error_to_exception(exn)
    Exception: configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig`- couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

Instead the "older" v3 works:

        kubernetes.helm.v3.Release(
            "helm",
            name="gitlab-runner",
            namespace=namespace.metadata.name,
            chart="gitlab-runner",
            version="0.65.0",
            repository_opts=kubernetes.helm.v3.RepositoryOptsArgs(
                repo="https://charts.gitlab.io",
            ),
            values={},
        )

Example

   kubernetes.helm.v4.Chart(
            "helm",
            name="gitlab-runner",
            namespace=namespace.metadata.name,
            chart="gitlab-runner",
            version="0.65.0",
            repository_opts=kubernetes.helm.v3.RepositoryOptsArgs(
                repo="https://charts.gitlab.io",
            ),
            values={},
        )

Output of pulumi about

CLI          
Version      3.117.0
Go Version   go1.22.3
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  command     0.11.1
resource  hcloud      1.19.1
resource  kubernetes  4.12.0
language  python      unknown

Host     
OS       fedora
Version  40
Arch     x86_64

This project is written in python: executable='/usr/bin/python3' version='3.12.3'

Backend        
Name           f1e58f99868a
URL            file:///data/gitlab-runner/.pulumi-local-states/dev
User           root
Organizations  
Token type     personal

Dependencies:
NAME               VERSION
ansible-core       2.16.6
argcomplete        3.3.0
kubernetes         29.0.0
pip                23.3.2
pulumi_command     0.11.1
pulumi_hcloud      1.19.1
pulumi_kubernetes  4.12.0
wheel              0.41.2

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@yellowhat yellowhat added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels May 27, 2024
@EronWright
Copy link
Contributor

@yellowhat there's a problem with the provider's configuration. Are you setting the kubeconfig property, and if so could you elaborate on the specifics?

The Chart/v4 resource relies on a well-configured provider, and while we may make some improvement to the experience, I think we should focus on the provider's configuration in this case.

@yellowhat
Copy link
Author

Sure.
my current setup involves creating the ~/.kube/config directory and fetching the kubeconfig from the master node before the actual kubernetes.Provider is created:

        kubeconfig = local.Command(
            f"{name}/kubeconfig",
            create=Output.all(self.cp.ipv4_address).apply(lambda args: f"""
                mkdir -p /root/.kube
                scp {args[0]}:/etc/rancher/k3s/k3s.yaml /root/.kube/config
                sed -e "s|https://.*|https://{args[0]}:6443|" \
                    -i /root/.kube/config
            """),
            opts=ResourceOptions(
                parent=child_opts.parent,
                depends_on=[bootstrap],
            ),
        )

        self.kubeconfig = kubernetes.Provider(
            name,
            kubeconfig="/root/.kube/config",
            opts=ResourceOptions(
                parent=child_opts.parent,
                depends_on=[kubeconfig],
            ),
        )

Before and after pulumi preview the ~/.kube folder does not exist, it is created during the pulumi up process.

Thanks

@EronWright
Copy link
Contributor

Thanks for the explanation, it all makes sense. The provider is no doubt failing to find /root/.kube/config during preview, and enters a "degraded mode" that not all resource types understand well. While we explore some possible fixes, I would suggest that you either skip the previews, or create the kubeconfig by some other means such that it is available during preview.

@rquitales rquitales added area/preview and removed needs-triage Needs attention from the triage team labels May 28, 2024
@EronWright
Copy link
Contributor

EronWright commented May 28, 2024

@yellowhat another suggestion is to use the Run function, since it does run during preview. I assume that you'd want to download the kubeconfig during preview.
https://www.pulumi.com/registry/packages/command/api-docs/local/run/

@yellowhat
Copy link
Author

Thanks for the explanation.

As you suggested I have run before pulumi preview:

mkdir -p ~/.kube
touch ~/.kube/config

and it is still trying to connect to he cluster (that does not exist yet):

  pulumi:pulumi:Stack (hetzner-k8s-dev):
    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1010, in do_rpc_call
        return monitor.RegisterResource(req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib64/python3.12/site-packages/grpc/_channel.py", line 1160, in __call__
        return _end_unary_response_blocking(state, call, False, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib64/python3.12/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
        raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNKNOWN
    	details = "configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have:
    
     	• set up the provider as per https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/
    
     invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
    	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-05-29T06:06:50.986871168+00:00", grpc_status:2, grpc_message:"configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have: \n\n \t \xe2\x80\xa2 set up the provider as per https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/ \n\n invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"}"
    >
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/bin/pulumi-language-python-exec", line 191, in <module>
        loop.run_until_complete(coro)
      File "/usr/lib64/python3.12/asyncio/base_events.py", line 687, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 138, in run_in_stack
        await run_pulumi_func(run)
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 52, in run_pulumi_func
        await wait_for_rpcs()
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 114, in wait_for_rpcs
        await task
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1015, in do_register
        resp = await asyncio.get_event_loop().run_in_executor(None, do_rpc_call)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1012, in do_rpc_call
        handle_grpc_error(exn)
      File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/settings.py", line 307, in handle_grpc_error
        raise grpc_error_to_exception(exn)
    Exception: configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have:
    
     	• set up the provider as per https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/
    
     invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

pulumi up --skip-preview seems to bypass the problem, but it is not a great alternative.

@EronWright
Copy link
Contributor

I did some investigation and, so far, haven't find an ideal solution. When the cluster is unreachable, the Chart resource cannot resolve kinds so as to produce their resource representation. Today the deployment fails, but if it were to skip rendering the children then they'd be marked for deletion.

At a technical level, we would need to move this check into each component provider's Construct function, and simply return the constructed component. We'd assumedly do that on Preview only.

@EronWright EronWright added the needs-design Needs input from design/UX label Jun 4, 2024
@mareckii
Copy link

I have similar issue. In my case provider is configured that way:

kubeconfig = Output.all(resource_group.name, cluster.name).apply(
    lambda args: list_managed_cluster_admin_credentials(
        resource_group_name=args[0],
        resource_name=args[1],
    ).kubeconfigs[0].value
)
provider = Provider(
    name,
    kubeconfig=Output.secret(kubeconfig.apply(lambda enc: b64decode(enc).decode()))
)

I think it is common case when you want create cluster and install some base components f.e ArgoCD

@liam-mctague-cvs
Copy link

I'm getting the exact same issue. Trying to create VPC, Cluster etc and then install ArgoCD.

const provider = new k8s.Provider("provider", {
  kubeconfig: kubeConfigSecret
});
const argocd = new k8s.helm.v4.Chart("argo-cd", {
    chart: "argo-cd",
    version: "5.53.14",
    namespace: "argo-cd",
    repositoryOpts: {
        repo: "https://argoproj.github.io/argo-helm",
    },
    valueYamlFiles: [
      new pulumi.asset.FileAsset("./apps/argocd/values.yaml")
  ],
}, { dependsOn: argocdNamespace });

however I get the same error when trying to deploy...

Error: failed to register new resource argo-cd [kubernetes:helm.sh/v4:Chart]: 2 UNKNOWN: configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have:

         • set up the provider as per https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/

     invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/helm area/preview kind/bug Some behavior is incorrect or out of spec needs-design Needs input from design/UX
Projects
None yet
Development

No branches or pull requests

5 participants