Tekton tasks and pipelines to run notebooks and generate reports #613
Comments
|
Issue-Label Bot is automatically applying the labels:
Please mark this comment with 👍 or 👎 to give our bot feedback! |
|
/cc @gabrielwen |
|
I think @gabrielwen is working on this. @gabrielwen any update on this? |
|
/area gsoc |
|
I would like to work on this issue as a part of GSOC 2020 |
|
@sarahmaddox I would like to contribute on this issue during GSoC journey. |
|
/priority p1 |
|
#622 included some initial tekton tasks and pipelines for running our E2E tests. |
|
Issue-Label Bot is automatically applying the labels:
Please mark this comment with 👍 or 👎 to give our bot feedback! |
|
Here's the current notebook task
The use of init containers is a bit brittle and leads to passing around repo references and duplicating git logic best handled by Tekton. I think a better approach is
|
* Changes pulled in from kuueflow/examples#764 * Notebook tests should print a link to the stackdriver logs for the actual notebook job. * Related to kubeflow/testing#613
* Changes pulled in from kuueflow/examples#764 * Notebook tests should print a link to the stackdriver logs for the actual notebook job. * Related to kubeflow/testing#613 Co-authored-by: Gabriel Wen <[email protected]>
* kubeflow#613 currently the way we run notebook tests is by firing off a K8s job on the KF cluster which runs the notebook. * The K8s job uses init containers to pull in source code and install dependencies like papermill. * This is a bit brittle. * To fix this we will instead use Tekton to build a docker image that takes the notebook image and then adds the notebook code to it. * Dockerfile.notebook_runner dockerfile to build the test image. * Add tekton tasks to build the image.
Notebook tests should build a docker image to run the notebook in. * kubeflow#613 currently the way we run notebook tests is by firing off a K8s job on the KF cluster which runs the notebook. * The K8s job uses init containers to pull in source code and install dependencies like papermill. * This is a bit brittle. * To fix this we will instead use Tekton to build a docker image that takes the notebook image and then adds the notebook code to it. * Dockerfile.notebook_runner dockerfile to build the test image. The pipeline to run the notebook consists of two tasks 1. A Tekton Task to build a docker image to run the notebook in 1. A tekton task that fires off a K8s job to run the notebook on the Kubeflow cluster. Here's a list of changes to make this work * tekton_client should provide methods to upload artifacts but not parse junits * Add a tekton_client method to construct the full image URL based on the digest returned from kaniko * Copy over the code for running the notebook tests from kubeflow/examples and start modifying it. * Create a simple CLI to wait for nomos to sync resources to the cluster * This is used in some syntactic sugar make rules to aid the dev-test loop The mnist test isn't completing successfully yet because GoogleCloudPlatform/kubeflow-distribution#61 means the KF deployments don't have proper GSA's to write to GCS. Related to: kubeflow#613
Notebook tests should build a docker image to run the notebook in. * kubeflow#613 currently the way we run notebook tests is by firing off a K8s job on the KF cluster which runs the notebook. * The K8s job uses init containers to pull in source code and install dependencies like papermill. * This is a bit brittle. * To fix this we will instead use Tekton to build a docker image that takes the notebook image and then adds the notebook code to it. * Dockerfile.notebook_runner dockerfile to build the test image. The pipeline to run the notebook consists of two tasks 1. A Tekton Task to build a docker image to run the notebook in 1. A tekton task that fires off a K8s job to run the notebook on the Kubeflow cluster. Here's a list of changes to make this work * tekton_client should provide methods to upload artifacts but not parse junits * Add a tekton_client method to construct the full image URL based on the digest returned from kaniko * Copy over the code for running the notebook tests from kubeflow/examples and start modifying it. * Create a simple CLI to wait for nomos to sync resources to the cluster * This is used in some syntactic sugar make rules to aid the dev-test loop The mnist test isn't completing successfully yet because GoogleCloudPlatform/kubeflow-distribution#61 means the KF deployments don't have proper GSA's to write to GCS. Related to: kubeflow#613
* Revamp how Tekton pipelines to run notebooks work. Notebook tests should build a docker image to run the notebook in. * #613 currently the way we run notebook tests is by firing off a K8s job on the KF cluster which runs the notebook. * The K8s job uses init containers to pull in source code and install dependencies like papermill. * This is a bit brittle. * To fix this we will instead use Tekton to build a docker image that takes the notebook image and then adds the notebook code to it. * Dockerfile.notebook_runner dockerfile to build the test image. The pipeline to run the notebook consists of two tasks 1. A Tekton Task to build a docker image to run the notebook in 1. A tekton task that fires off a K8s job to run the notebook on the Kubeflow cluster. Here's a list of changes to make this work * tekton_client should provide methods to upload artifacts but not parse junits * Add a tekton_client method to construct the full image URL based on the digest returned from kaniko * Copy over the code for running the notebook tests from kubeflow/examples and start modifying it. * Create a simple CLI to wait for nomos to sync resources to the cluster * This is used in some syntactic sugar make rules to aid the dev-test loop The mnist test isn't completing successfully yet because GoogleCloudPlatform/kubeflow-distribution#61 means the KF deployments don't have proper GSA's to write to GCS. Related to: #613 * tekton_client.py can't use format strings yet because we are still running under python2. * Remove f-style strings. * Fix typo. * Address PR comments. * * copy-buckets should not abort on error as this prevents artifacts from being copied and thus the results from showing up in testgrid see #703
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions. |
|
This issue has been closed due to inactivity. |
Kubeflow relies heavily on notebooks for reporting and e2e testing.
We'd like to make it easier to run notebooks and generate reports for them.
One way to do that would be to create reusable Tekton tasks to run notebooks (e.g. using papermill) and then upload an HTML version of the output to an object store like GCS or S3.
Then users could easily automate the running and generation of notebook reports just by adding tasks to a Tekton pipeline.
The task should be parameterized so that users just have to specify the parameters defining their notebook. Possible parameters
inputs:
Outputs:
We should define a catalog of tasks inside kubeflow/testing and put the task there. Similar to
https://github.com/tektoncd/catalog.
kubeflow/examples#735 provides an example of using papermill to execute a notebook, then running nbconvert to convert to html, and then uploading it to GCS. That could be as a model of what a Tekton task might do.
As a follow on issue we might want to investigate using commuter to view these notebooks
The text was updated successfully, but these errors were encountered: