sedona's docker image can not run successfully on k8s

[yancy-hong]

Expected behavior

run spark-pi example successfully

Actual behavior

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: 
unable to start container process: exec: "driver": executable file not found in $PATH: unknown

Steps to reproduce the problem

I plan to run a Sedona cluster on Kubernetes. Prior to this, I have successfully used the spark-operator to run a Spark cluster and obtained results from the spark-pi example:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-pi-test
  namespace: spark-tasks
spec:
  type: Scala
  mode: cluster
  image: docker.io/spark:3.5.1
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.5.1.jar"
  sparkVersion: "3.5.1"
  restartPolicy:
    type: Never
  driver:
    cores: 4
    memory: "4G"
    labels:
      version: 3.5.1
    serviceAccount: spark-task-account
    env:
      - name: SPARK_MODE
        value: "driver"
  executor:
    cores: 4
    instances: 2
    memory: "4G"
    labels:
      version: 3.5.1
  arguments:
    - "1000"

However, when I change the image to Sedona's image:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: sedona-test
  namespace: spark-tasks
spec:
  type: Scala
  mode: cluster
  image: docker.io/apache/sedona:1.6.0
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar"
  sparkVersion: "3.4.1"
  restartPolicy:
    type: Never
  driver:
    cores: 4
    memory: "4G"
    labels:
      version: 3.4.1
    serviceAccount: spark-task-account
    env:
      - name: SPARK_MODE
        value: "driver"
  executor:
    cores: 4
    instances: 2
    memory: "4G"
    labels:
      version: 3.4.1
  arguments:
    - "1000"

Submitting this YAML file results in an error:

The configMap (spark-drv-fba0809048193c11-conf-map) mentioned in the error message does exist:

kubectl get configmap -n spark-tasks
NAME                                  DATA   AGE
kube-root-ca.crt                      1      11d
spark-drv-fba0809048193c11-conf-map   2      25m

My question is: Does the Sedona image not support running on Kubernetes? Or is there a problem with my configuration? Thank you.

Settings

Sedona version = 1.6.0
Apache Spark version = 3.4.1

[Kontinuation]

Sedona docker image is not designed to be deployed to a cluster, it always runs a small standalone cluster (1 master + 1 worker) and a jupyter-lab in the container, its entrypoint is also incompatible with the official spark image.

[yancy-hong]

Thank you for your patient guidance. In that case, I have to use spark-shell --packages to build a Sedona image from Spark's official image by myself.