K8s: cannot create resource "deployments"

[Artimi]

Describe the issue:

Minimal Complete Verifiable Example:

Having operator installed this way

helm install -n dask_ns dask-kubernetes-operator dask-kubernetes-operator --set image.tag=2023.6.0 --set=rbac.cluster=false --set kopfArgs="{--namespace=dask_ns}"

from dask_kubernetes.operator import KubeCluster
import time
import random
import joblib
import atexit


def square(x):
    time.sleep(random.expovariate(1.5))
    return x**2


def main():
    cluster = KubeCluster(
        name="my-dask-cluster",
        image="ghcr.io/dask/dask:2023.6.0-py3.11",
        namespace="dask_ns",
        env={"EXTRA_PIP_PACKAGES": "joblib"},
        shutdown_on_close=True,
    )
    print("Cluster created")
    cluster.scale(1)
    client = cluster.get_client()
    print("Client", client)
    joblib.parallel_backend(
        "dask", client=client, pure=False, wait_for_workers_timeout=60
    )

    results = joblib.Parallel(n_jobs=2)(
        joblib.delayed(square)(arg) for arg in range(10)
    )
    print(results)
    client.close()
    atexit.register(cluster.close)


if __name__ == "__main__":
    main()

Fails with

[2023-06-28 08:46:55,093] kopf.objects         [ERROR   ] [dask_ns/my-dask-cluster] Handler 'daskcluster_create_components/status.phase' failed with an exception. Will retry.
Traceback (most recent call last):                                                                                                                                                                          File "/usr/local/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 276, in execute_handler_once
    result = await invoke_handler(                                                                                                                                                                          File "/usr/local/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 371, in invoke_handler
    result = await invocation.invoke(                                                                                                                                                                       File "/usr/local/lib/python3.10/site-packages/kopf/_core/actions/invocation.py", line 116, in invoke
    result = await fn(**kwargs)  # type: ignore                                                                                                                                                             File "/usr/local/lib/python3.10/site-packages/dask_kubernetes/operator/controller/controller.py", line 304, in daskcluster_create_components
    await kubernetes.client.AppsV1Api(api_client).create_namespaced_deployment(                                                                                                                             File "/usr/local/lib/python3.10/site-packages/kubernetes_asyncio/client/api_client.py", line 192, in __call_api
    raise e                                                                                                                                                                                                 File "/usr/local/lib/python3.10/site-packages/kubernetes_asyncio/client/api_client.py", line 185, in __call_api
    response_data = await self.request(                                                                                                                                                                     File "/usr/local/lib/python3.10/site-packages/kubernetes_asyncio/client/rest.py", line 230, in POST
    return (await self.request("POST", url,                                                                                                                                                                 File "/usr/local/lib/python3.10/site-packages/kubernetes_asyncio/client/rest.py", line 187, in request
    raise ApiException(http_resp=r)                                                                                                                                                                       kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden                                                                                                                                                                                         HTTP response headers: <CIMultiDictProxy('Audit-Id': '7c7fd7a4-b960-4e90-bfee-4da0b303f733', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'
, 'X-Kubernetes-Pf-Flowschema-Uid': 'a3ef744b-c3e6-4bf2-9c05-b8decfa0b75a', 'X-Kubernetes-Pf-Prioritylevel-Uid': '2d88b8ba-ee58-4770-a7b4-2342c139b0cc', 'Date': 'Wed, 28 Jun 2023 08:46:55 GMT', 'Content-Length': '350')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"deployments.apps is forbidden: User \"system:serviceaccount:dask_ns:dask-kubernetes-operator\" cannot create resource \"deployments\" in API group \"apps\" in the namespace \"dask_ns\"","reason":"Forbidden","details":{"group":"apps","kind":"deployments"},"code":403}

Anything else we need to know?:
This seems to be fixed by adding this to role.yaml in operator templates

  - apiGroups: ["apps"]
    resources: [deployments]
    verbs: ["*"]

Should I create a PR with the change or is there more to it?

Environment:

• Dask version: 2023.6.0
• Python version: 3.11
• Operating System: 3.11.3
• Install method (conda, pip, source): pip

[bstadlbauer]

They're probably missing in the role as well as clusterrole

dask-kubernetes/dask_kubernetes/operator/deployment/helm/dask-kubernetes-operator/templates/role.yaml

Lines 33 to 43 in 3ea9093

	- apiGroups: [""]
	resources: [pods, pods/status]
	verbs: ["*"]
	- apiGroups: [""]
	resources: [services, services/status]
	verbs: ["*"]
	- apiGroups: ["", events.k8s.io]
	resources: [events]
	verbs: ["*"]

dask-kubernetes/dask_kubernetes/operator/deployment/helm/dask-kubernetes-operator/templates/clusterrole.yaml

Lines 31 to 43 in 3ea9093

	# Application: other resources it produces and manipulates.
	# Here, we create/delete Pods.
	- apiGroups: [""]
	resources: [pods, pods/status]
	verbs: ["*"]
	- apiGroups: [""]
	resources: [services, services/status]
	verbs: ["*"]
	- apiGroups: ["", events.k8s.io]
	resources: [events]
	verbs: ["*"]

[bstadlbauer]

@Artimi I've opened #750 which should fix this