Automate ensembler image building with Turing API #170

deadlycoconuts · 2022-02-22T05:11:32Z

Context

In PR #164 and most notably PR #165, a real-time ensembler engine was introduced to process live ensembling requests, using an ensembler that is predefined in the mlflow pyfunc flavour. This engine can then be run in a container to serve real-time ensembling requests. However, the creation of such a container is a manual process which required the user to build a Docker image, followed by spinning up his/hero own container from the template files provided.

Hence, this PR introduces a process to automatically allow Turing API to orchestrate the Docker image building process when a user specifies a pre-defined pyfunc ensembler (identified by its given project_id and ensembler_id) when creating a Turing Router. Turing API then pushes the newly created image to a specified registry, and saves the image name as part of the configuration of the said Turing Router, which will then use this image to run a container containing the real-time ensembler web service with the specified pyfunc ensembler.

Features

Modification to the existing router creation/update process that builds a Docker image of the real-time ensembler web service with the ensembling logic given by the specified pyfunc ensembler
Introduction of a new field pyfunc_config/PyfuncConfig that contains resource request and timeout information regarding a pyfunc ensembler to be automatically deployed as a containerised web service

Fixes

Minor bug fixes to the Dockerfiles and makefiles specified for the web service
Changing fo the port that the ensembler web service listens on from 8080 to 8083, since Turing API already listens on 8080

Main Modifications

api/turing/api/deployment_controller.go - Addition of steps to orchestrate the building of the web service image when deploying a new router (version)
api/turing/api/appcontext.go - Addition of an EnsemblerServiceBuilder to the appContext
api/turing/api/request/request.go - Addition of checks to verify HTTP requests to Turing API that specify pyfunc ensemblers contain a valid project_id and ensembler_id (one that points to a pyfunc ensembler that has already been stored in Turing API)
api/turing/models/ensembler.go - Addition of a new field for pyfunc_config/PyfuncConfig
api/turing/imagebuilder/ensembler.go - Addition of new utility functions to create ensembler builder and service names
api/config-dev.yaml - Addition of new configuration for the web service building
api/db-migrations/* - Addition of SQL scripts for up/down migrations due to addition of pyfunc_config/PyfuncConfig
api/api/* - Changes to the OpenAPI specs from the addition of pyfunc_config/PyfuncConfig
infra/charts/turing/values.yaml - Addition of new default configuration for web service building to helm charts

api/turing/api/deployment_controller.go

api/turing/models/ensembler.go

krithika369

Thanks for the detailed summary, @deadlycoconuts ! I took a quick look and left some comments / questions. You may also want to get another set of eyes on this MR.

Could you also help clean up the README? There is a section showing the config-dev.yaml contents. But I think it can be removed, with a link to the config file, which you have already updated with the required configs for this feature.

Lastly, I imagine there will some user docs being updated, for the Pyfunc ensembler creation and router deployment (similar to batch ensemblers, both for the public docs and our internal docs; although batch docs are currently only internal - we can sort that out later). Will this be a part of the UI card? If not, please capture this in the backlog.

Thank you!

api/api/openapi.bundle.yaml

api/config-dev.yaml

api/db-migrations/000010_add_py_func_ref_config.down.sql

api/turing/api/deployment_controller.go

krithika369 · 2022-02-28T01:45:56Z

api/turing/api/request/request.go

+			}
+
+			// Verify if the ensembler given by its ProjectID and EnsemblerID exist
+			ensembler, err := ensemblersSvc.FindByID(


Question - can the ensembler be deleted while it is being referenced by a router? Can it be deleted at all (I can't seem to find the delete method in the codebase, for the real or the batch ensemblers). May be @ashwinath has some context too?

Nope there doesn't seem to be an API endpoint for deleting ensemblers. Not sure if this was a conscious design choice though.

Ok.. I'm not sure either. Could you create a card for this (delete for batch / real-time ensemblers) in the backlog, just to track it?

I can't remember why there isn't a delete ensembler API for this or maybe I wasn't kept in the loop. Ensemblers I believe was done by Roman where as I did the orchestration. There is a delete for the orchestration part.

api/turing/cluster/servicebuilder/router.go

api/turing/models/ensembler.go

api/turing/api/deployment_controller.go

api/turing/api/ensemblers_api_it_test.go

api/turing/api/ensembling_job_api_it_test.go

api/turing/database/integration_config.go

api/turing/service/router_deployment_service.go

engines/pyfunc-ensembler-service/app.Dockerfile

deadlycoconuts · 2022-03-02T08:53:00Z

api/api/openapi.bundle.yaml

@@ -551,7 +551,7 @@ paths:
          content:
            application/json:
              schema:
-                $ref: '#/components/schemas/inline_response_200'
+                $ref: '#/components/schemas/IdObject'


Cleaned up all these stuff in the latest update

deadlycoconuts · 2022-03-02T08:56:47Z

api/turing/cluster/servicebuilder/router.go

@@ -199,7 +199,7 @@ func (sb *clusterSvcBuilder) buildRouterEnvs(
 			{Name: envEnricherTimeout, Value: ver.Enricher.Timeout},
 		}...)
 	}
-	if ver.Ensembler != nil && ver.Ensembler.Type == models.EnsemblerDockerType {
+	if ver.HasDockerConfig() {


Simplified this to make the intention of the check explicit

api/turing/imagebuilder/ensembler.go

deadlycoconuts · 2022-03-02T09:08:49Z

api/turing/service/router_deployment_service.go

 	experimentConfig json.RawMessage,
 	eventsCh *EventChannel,
 ) (string, error) {
 	var endpoint string

+	// If pyfunc ensembler is specified as an ensembler service, build/retrieve its image
+	if pyfuncEnsembler != nil {
+		err := ds.buildEnsemblerServiceImage(pyfuncEnsembler, project, routerVersion, eventsCh)


DeploymentService now does the weightlighting for image building.

ashwinath · 2022-03-07T04:34:40Z

api/api/openapi.bundle.yaml

@@ -551,7 +551,7 @@ paths:
          content:
            application/json:
              schema:
-                $ref: '#/components/schemas/inline_response_200'
+                $ref: '#/components/schemas/IdObject_1'


This is automatically generated by the Makefile right?

Yep that's right. I corrected the source openapi specs that was generated with make bundle-openapi

deadlycoconuts · 2022-03-07T05:15:10Z

Thanks a lot for all the comments everyone! I'll rebase this on @leonlnj 's latest changes and it'll be good to go!

…mage

…mbler_service

deadlycoconuts marked this pull request as ready for review February 25, 2022 11:49

deadlycoconuts requested a review from a team February 25, 2022 11:49

deadlycoconuts commented Feb 25, 2022

View reviewed changes

api/turing/api/deployment_controller.go Outdated Show resolved Hide resolved

deadlycoconuts commented Feb 25, 2022

View reviewed changes

api/turing/models/ensembler.go Show resolved Hide resolved

krithika369 reviewed Feb 28, 2022

View reviewed changes