Custom resource configuration

[esevan]

In project roadmap,

Kernel Configuration Profile
Enable client to request different resource configurations for kernels (e.g. small, medium, large)

I'm resolving this in k8s env by adding new kernelspec (e.g. small-tensorflow, medium-tensorflow, ...) and modifying kernel-pod.yaml (e.g. setting resource request and limit).
However, every time I add a new kernel, then kernelspecs including all resource-configuration (small, medium, large) should be added to the system. This makes our system too verbose and HW spec is too coupled with SW spec.

Is there any in-progress item for this? Any idea will be welcome. I need your help!

[kevin-bates]

No, nothing in the works yet. Where this is heading is parameterized kernels (or kernelspecs more like in my opinion) - which has been floated in the community as something many would like. Today, this is accomplished (only in EG to my knowledge) via envs. However, in order to break from of the NxM issue (where N is things like small, medium and large and M is the set of kernels), we pretty much need to ask for the parameters when the kernel is launched. This can be accomplished via a nb/lab extension. Then, this could gather the parameters and convey them to the parameterized kernelspec. For k8s, the kernel-pod jinja template could then include a resources: stanza and provide placeholders for the various parameters, etc. This would at least eliminate the N multiplier.

Ultimately, I think we'd want a more formal section in the kernel request body, say something like parameters:, which contains a dict of parameters and their values. What parameters are available for a given kernel (and their meta-properties) would come from the kernelspec. Today, this could simply be placed in the metadata stanza. Taking the python_kubernetes spec as an example, you might have something like this...

{
  "language": "python",
  "display_name": "Python on Kubernetes",
  "metadata": {
    "process_proxy": {
      "class_name": "enterprise_gateway.services.processproxies.k8s.KubernetesProcessProxy",
      "config": {
        "image_name": "elyra/kernel-py:VERSION"
      }
    },
    "parameters": {
      "cpus": { "min": "0.5", "max": "16", "default": "2"},
      "memory": { "min": "1GB", "max": "32GB", "default": "4GB"}
    }
  },
...
}

Then, once prompted for, the kernel creation request would have a stanza like...

{
	"parameters": { "cpus": "8", "memory": "8GB"},
	"env": { ... }
}

The other thing is that the environment might dictate how this is accomplished as well. For example, containerized envs would probably have different sets of parameters than say a YARN/Spark environment and their means of getting them conveyed to those respective environments will vary as well.

[esevan]

Impressed. It sounds like this would be good to start for EG, right?

"parameters": {
      "cpus": { "min": "0.5", "max": "16", "default": "2"},
      "memory": { "min": "1GB", "max": "32GB", "default": "4GB"}
    }

After EG supports it first, if notebook env (extension) supports to send request with parameters user can customize resource configuration for the kernels. Thanks! I'll try this soon!

[nsingl00]

Hi Folks, we have been thinking to introduce json schema in jupyter kernelspec. The idea is to have Schema driven interfaces and UI for Jupyter Kernelspec. As part of this, we will introduce new APIs in jupyter server to getKernelSpec schema which will return the schema for kernelspec. Due to current limitation of existing jupyter kernel.json, the schema will be inside metadata field of jupyter kernel. There will be an UI plugin which will read the schema and content without having any business logic in it. This is to make UI plugin reusable if schema is different for different enterprises.
Here is an example of rendering UI from schema and schema used

render_ui_from_schema.mov

{
    "$id": "https://example.com/kernelspec.schema.json",
    "title": "KernelSpec",
    "description": "A kernelspec in juypter notebooks",
    "definitions": {
        "storages": {
            "enum": [
                "HDFS",
                "Cassandra",
                "S3"
            ]
        },
        "catalogs": {
            "enum": [
                "Hive",
                "In Memory"
            ]
        },
        "HDFS": {
            "type": "object",
            "properties": {
                "cluster.name": {
                    "type": "string"
                },
                "hdfs.impl": {
                    "type": "string"
                }
            }
        },
        "Cassandra": {
            "type": "object",
            "properties": {
                "spark.cassandra.auth.username": {
                    "type": "string",
                    "default": "${CASSANDRA_USER}"
                },
                "spark.cassandra.auth.password": {
                    "type": "string",
                    "default": "${CASSANDRA_PASSWORD}"
                },
                "spark.cassandra.connection.ssl.enabled": {
                    "type": "boolean",
                    "default": "true"
                },
                "spark.casserole.cassandra.connection.enabled": {
                    "type": "boolean",
                    "default": "true"
                },
                "spark.cassandra.connection.host": {},
                "spark.cassandra.connection.local_dc": {
                    "type": "string"
                },
                "spark.casserole.cassandra.connection.application_name": {
                    "type": "string"
                },
                "spark.casserole.cassandra.connection.cluster_name": {
                    "type": "string"
                }
            }
        },
        "S3": {
            "type": "object",
            "properties": {
                "awsrole.enabled": {
                    "type": "boolean",
                    "default": "true"
                },
                "spark.hadoop.fs.s3a.aws.credentials.provider": {
                    "type": "string",
                    "default": "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
                }
            }
        },
        "Hive": {
            "type": "object",
            "properties": {
                "spark.sql.catalogImplementation": {
                    "type": "string",
                    "default": "hive"
                },
                "spark.pie.hadoop.config.service.load.config": {
                    "type": "string",
                    "default": "CORE,HDFS,YARN,HIVE"
                }
            }
        },
        "kernelTypes": {
            "enum": [
                "Python3.8",
                "Python3.9",
                "PythonWithSpark",
                "PythonWithSparkStorage"
            ]
        },
        "compute": {
            "enum": [
                "computeId1",
                "computeId2",
                "computeId3"
            ]
        },
        "Python3.8": {
            "type": "object",
            "properties": {
                "cpuLimit": {
                    "type": "string",
                    "default": "100m"
                },
                "memoryRequest": {
                    "type": "string",
                    "default": "200Mi"
                },
                "memoryLimit": {
                    "type": "string",
                    "default": "4GB"
                },
                "cpuRequest": {
                    "type": "string",
                    "default": "100m"
                }
            }
        },
        "PythonWithSpark": {
            "type": "object",
            "properties": {
                "spark.driver.memory": {
                    "type": "string",
                    "default": "16g"
                },
                "spark.executor.memory": {
                    "type": "string",
                    "default": "24g"
                },
                "spark.driver.cores": {
                    "type": "string",
                    "default": "2"
                },
                "spark.executor.cores": {
                    "type": "string",
                    "default": "4"
                },
                "spark.driver.extraJavaOptions": {
                    "type": "string"
                },
                "spark.executor.extraJavaOptions": {
                    "type": "string"
                },
                "spark.dynamicAllocation.initialExecutors": {
                    "type": "string",
                    "default": "2"
                },
                "spark.dynamicAllocation.minExecutors": {
                    "type": "string",
                    "default": "2"
                },
                "spark.dynamicAllocation.maxExecutors": {
                    "type": "string",
                    "default": "100"
                }
            }
        },
        "PythonWithSparkStorage": {
            "type": "object",
            "properties": {
                "resource": {
                    "type": "object",
                    "properties": {
                        "spark.driver.memory": {
                            "type": "string",
                            "default": "16g"
                        },
                        "spark.executor.memory": {
                            "type": "string",
                            "default": "24g"
                        },
                        "spark.driver.cores": {
                            "type": "string",
                            "default": "2"
                        },
                        "spark.executor.cores": {
                            "type": "string",
                            "default": "4"
                        },
                    }
                },
                "extraJavaOptions": {
                    "type": "object",
                    "properties": {
                        "spark.driver.extraJavaOptions": {
                            "type": "string"
                        },
                        "spark.executor.extraJavaOptions": {
                            "type": "string"
                          }
                    }
                },
                "executorAllocation": {
                    "type": "object",
                    "properties": {
                        "spark.dynamicAllocation.initialExecutors": {
                            "type": "string",
                            "default": "2"
                        },
                        "spark.dynamicAllocation.minExecutors": {
                            "type": "string",
                            "default": "2"
                        },
                        "spark.dynamicAllocation.maxExecutors": {
                            "type": "string",
                            "default": "100"
                        }
                    }
                }
            }
        }
    },
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "kernelType": {
            "type": "string",
            "$ref": "#/definitions/kernelTypes"
        },
        "compute": {
            "type": "string",
            "$ref": "#/definitions/compute"
        }
    },
    "allOf": [
        {
            "if": {
                "required": [
                    "kernelType"
                ],
                "properties": {
                    "kernelType": {
                        "enum": [
                            "Python3.8",
                            "Python3.9"
                        ]
                    }
                }
              },
            "then": {
                "properties": {
                    "Python Properties": {
                        "$ref": "#/definitions/Python3.8"
                    }
                }
            }
        },
        {
            "if": {
                "required": [
                    "kernelType"
                ],
                "properties": {
                    "kernelType": {
                        "enum": [
                            "PythonWithSpark"
                        ]
                    }
                }
            },
            "then": {
                "properties": {
                    "PySpark Properties": {
                        "$ref": "#/definitions/PythonWithSpark"
                    }
                }
            }
        },
        {
            "if": {
                "required": [
                    "kernelType"
                ],
                "properties": {
                    "kernelType": {
                        "enum": [
                            "PythonWithSparkStorage"
                        ]
                    }
                }
            },
            "then": {
                "properties": {
                    "PySpark Properties": {
                        "$ref": "#/definitions/PythonWithSparkStorage"
                    },
                    "storage": {
                        "type": "array",
                        "uniqueItems": true,
                        "items": {
                            "$ref": "#/definitions/storages"
                        }
                    },
                    "catalogs": {
                        "title": "Catalogs",
                        "$ref": "#/definitions/catalogs"
                    }
                }
            }
        },
        {
            "if": {
                "required": [
                    "kernelType",
                    "storage",
                    "mode"
                ],
                "properties": {
                    "kernelType": {
                        "enum": [
                            "PythonWithSparkStorage"
                        ]
                    },
                    "PySpark Properties": {
                        "required": [
                            "storage"
                        ]
                    },
                    "properties": {
                        "storage": {
                            "enum": [
                                "HDFS"
                            ]
                        }
                    }
                }
            },
            "then": {
                "properties": {
                    "HDFS": {
                        "$ref": "#/definitions/HDFS"
                    }
                }
            }
        },
        {
            "required": [
                "name",
                "kernelType"
            ]
        }
    ]
}

Kernel.json metadata field will contain the data corresponding to schema.
UI plugin will consume the schema, get the data and render that on JupyterLab UI.

Note- Schemas are totally independent. Each enterprise can have their own schema defined. UI plugin will read the schema from getkernelspec schema API and render the content from kernel metadata field.

cc @Zsailer @sathishlxg

[kevin-bates]

Hi @nsingl00 - thanks for commenting on this issue.

I absolutely agree that kernel parameterization, and, per this issue, custom resource configuration, should/must be schema driven and, yes, that schema (or reference to a schema) must reside within the kernel.json metadata stanza.

I also strongly believe that the parameterization/customization associated with a kernelspec must be derived from the kernel provisioner associated with that kernelspec. As your example so clearly illustrates, most parameters will not be directly associated to the kernel itself, but, rather, to the environment in which the kernel runs - which is the precise responsibility of the kernel provisioner. However, both the kernel and the kernel-provisioner should be asked to provide their respective schemas since we simply cannot ask users to create these from "whole cloth" - which can be easily done by the KernelSpecManager.

I would really prefer to keep the schema used to render UI separate from the schema used to describe kernel/provisioner parameters and view the UI schema as orthogonal and, frankly, the purview of the application that needs to render the parameter selection in a usable manner.

There is a current JEP regarding parameterization that I personally feel falls short as it does not include the provisioner's involvement (see my comments). I had authored a previous JEP, which I retracted due to lack of input, although it seems traction for schema-based parameterization has picked up recently.

Unfortunately, I don't believe your comment will get much traction here and would encourage you to read over the various linked proposals and get involved on the current JEP or in either jupyter_client or jupyter_server. I am officially "inactive" on Jupyter due to other commitments, although my passion for kernel parameterization - that includes provisioner involvement - may cause me to become more active if that's what it takes to make this right.

Also note there has been a wave of "schema persistence" discussions that, yeah, I could see a kernel provisioner (and a kernel) posting their schemas or, generally making their schemas, available to a "schema service" or repository.

@Zsailer (hi Zach! 😄) can point you to the schema-related discussions.

Lastly, I think it makes sense to close this issue. Yes, EG (or Gateway Provisioners more likely) will adopt whatever parameterization approach is set forth by Jupyter Server. If you'd like to continue this conversation here, I'm fine with that also.