Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automate ensembler image building with Turing API #170

Merged
merged 57 commits into from
Mar 7, 2022
Merged

Automate ensembler image building with Turing API #170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
cd2ea55
Fix bug is dockerfile and makefile involving passing of env variables
deadlycoconuts Feb 16, 2022
556548d
Remove dockerfile command to install ensembler dependencies in base i…
deadlycoconuts Feb 16, 2022
5003688
Add new config field to store references to saved pyfunc ensemblers
deadlycoconuts Feb 16, 2022
bc54c37
Add ensembler type and config verification
deadlycoconuts Feb 16, 2022
a41de52
Add resource request and timeout limits to pyfunc ensembler option
deadlycoconuts Feb 16, 2022
0f1f279
Add template for ensembler image building option
deadlycoconuts Feb 16, 2022
a3023dc
Move getensemblerdirectory method from ensembling_job_service to ense…
deadlycoconuts Feb 17, 2022
8b58fca
Add new configs for pyfunc ensembler service building
deadlycoconuts Feb 17, 2022
879020d
Add config field for ensembler service
deadlycoconuts Feb 17, 2022
e38b454
Add holders in appcontext to carry an ensembler service builder
deadlycoconuts Feb 18, 2022
3f1d75f
Make router deploy pyfunc ensembler as if it were a normal ensembler …
deadlycoconuts Feb 18, 2022
31aaf7d
Make deployment controller utilise pyfunc configs to generate docker …
deadlycoconuts Feb 18, 2022
16bba2d
Refactor tests with changes to ensembler config schema
deadlycoconuts Feb 21, 2022
7a81abe
Add tests for imagebuilder for ensembler service
deadlycoconuts Feb 21, 2022
1e0ce0e
Revert changes to get ensembler directory function
deadlycoconuts Feb 21, 2022
0ccb233
Refactor appcontext test
deadlycoconuts Feb 21, 2022
54bb059
Create config class for resource request and timeout
deadlycoconuts Feb 22, 2022
2b04eda
Fix migration script for new py_func_ref_config
deadlycoconuts Feb 22, 2022
d427e23
Fix lint comments
deadlycoconuts Feb 22, 2022
2cfe195
Fix lint import formatting
deadlycoconuts Feb 22, 2022
784ebbf
Rename pyfunc ensembler type and regen openapi
deadlycoconuts Feb 23, 2022
15ca209
Fix bug in ensembler Dockerfile
deadlycoconuts Feb 24, 2022
421c2f0
Refactor Dockerfile and fix 8083 port for ensembler
deadlycoconuts Feb 24, 2022
99f608e
Fix port for ensembler in turing api
deadlycoconuts Feb 24, 2022
9f7023f
Fix output message
deadlycoconuts Feb 24, 2022
51afe52
Update router deployment logs messages
deadlycoconuts Feb 24, 2022
4e7f1d0
Increase specificity of error log in router deployment
deadlycoconuts Feb 24, 2022
c41a03e
Refactor imagerequest and use a name that reflects each unique ensemb…
deadlycoconuts Feb 25, 2022
d47160e
Specify pyfunc ensembler images to have name based on their versions
deadlycoconuts Feb 25, 2022
18fe38a
Add additional checks on ensemblers to api handlers
deadlycoconuts Feb 25, 2022
a1be975
Add mocks to existing tests after refactoring
deadlycoconuts Feb 25, 2022
c4013f3
Add ensembler service configs to turing chart
deadlycoconuts Feb 25, 2022
47372d1
Add test for requests with ensemblers that cannot be found
deadlycoconuts Feb 25, 2022
954592d
Update e2e test data
deadlycoconuts Feb 25, 2022
9183dc6
Revert combination of resource request and timeout into a single object
deadlycoconuts Feb 25, 2022
83f678c
Regenerate openapi objects
deadlycoconuts Feb 25, 2022
d42008b
Revert changes to e2e test data
deadlycoconuts Feb 25, 2022
2a1bc9d
Shift variables to uppercase for consistency
deadlycoconuts Feb 25, 2022
a0aa462
Add ensembler service to example config
deadlycoconuts Feb 26, 2022
c1fbe72
Simplify yaml configs with anchors and aliases
deadlycoconuts Feb 28, 2022
c17245d
Rename py_func_ref_config to pyfunc_config
deadlycoconuts Feb 28, 2022
e0845c2
Rename pyfunc_config to make it consistent with camelcase PyfuncConfig
deadlycoconuts Feb 28, 2022
e2ec340
Refactor naming of pyfunc config in openapi
deadlycoconuts Feb 28, 2022
683d7dd
Clean up openapi specs
deadlycoconuts Feb 28, 2022
1f35e29
Remove redundant schema from openapi spec
deadlycoconuts Feb 28, 2022
dd32e81
Remove redundant check on dockerconfig
deadlycoconuts Feb 28, 2022
7f4467b
Remove unnecessary logical checks for router deployment
deadlycoconuts Mar 1, 2022
88166fa
Remove modifications left by IDE
deadlycoconuts Mar 1, 2022
a3d344e
Remove go build tags
deadlycoconuts Mar 1, 2022
534bbe3
Remove go build tag from integration_config
deadlycoconuts Mar 1, 2022
9c0e34b
Refactor image building step into DeployRouterVersion process
deadlycoconuts Mar 2, 2022
81d29f6
Reformat code after lint tests
deadlycoconuts Mar 2, 2022
5e3cdd9
Rework image/docker naming convention to include run ids
deadlycoconuts Mar 2, 2022
b7cf92e
Remove redundant dockerfile layer
deadlycoconuts Mar 2, 2022
3a3c310
Clean up OpenAPI specs
deadlycoconuts Mar 3, 2022
f2767fc
Add default values to docker_config when using pyfunc ensemblers
deadlycoconuts Mar 3, 2022
e61cb8b
Fix broken test
deadlycoconuts Mar 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 111 additions & 26 deletions api/api/openapi.bundle.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -551,7 +551,7 @@ paths:
content:
application/json:
schema:
$ref: '#/components/schemas/inline_response_200'
$ref: '#/components/schemas/IdObject_1'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is automatically generated by the Makefile right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep that's right. I corrected the source openapi specs that was generated with make bundle-openapi

description: OK
"400":
description: Invalid project_id or router_id
Expand Down Expand Up @@ -651,7 +651,7 @@ paths:
content:
application/json:
schema:
$ref: '#/components/schemas/inline_response_202'
$ref: '#/components/schemas/RouterIdAndVersion'
description: Accepted
"400":
description: Invalid project_id, router_id or deploy request
Expand Down Expand Up @@ -682,7 +682,7 @@ paths:
content:
application/json:
schema:
$ref: '#/components/schemas/inline_response_200'
deadlycoconuts marked this conversation as resolved.
Show resolved Hide resolved
$ref: '#/components/schemas/IdObject_1'
description: OK
"400":
description: Invalid project_id or router_id
Expand Down Expand Up @@ -751,11 +751,11 @@ paths:
format: int32
type: integer
responses:
"200":
"202":
content:
application/json:
schema:
$ref: '#/components/schemas/inline_response_202'
$ref: '#/components/schemas/RouterIdAndVersion'
description: OK
"400":
description: Invalid project_id, router_id or version
Expand Down Expand Up @@ -832,7 +832,7 @@ paths:
content:
application/json:
schema:
$ref: '#/components/schemas/inline_response_202'
$ref: '#/components/schemas/RouterIdAndVersion'
description: Accepted
"400":
description: Invalid project_id, router_id, version_id or deploy request
Expand Down Expand Up @@ -1346,6 +1346,7 @@ components:
error: error
infra_config:
service_account_name: service_account_name
run_id: run_id
ensembler_name: ensembler_name
resources:
driver_cpu_request: driver_cpu_request
Expand Down Expand Up @@ -1400,6 +1401,7 @@ components:
EnsemblerInfraConfig:
example:
service_account_name: service_account_name
run_id: run_id
ensembler_name: ensembler_name
resources:
driver_cpu_request: driver_cpu_request
Expand All @@ -1423,6 +1425,9 @@ components:
x-go-custom-tag: validate:"required"
resources:
$ref: '#/components/schemas/EnsemblingResources'
run_id:
readOnly: true
type: string
env:
items:
$ref: '#/components/schemas/EnvVar'
Expand Down Expand Up @@ -1900,6 +1905,15 @@ components:
cpu_request: cpu_request
id: 0
ensembler:
pyfunc_config:
project_id: 7
ensembler_id: 9
resource_request:
min_replica: 0
max_replica: 6
memory_request: memory_request
cpu_request: cpu_request
timeout: timeout
updated_at: 2000-01-23T04:56:07.000+00:00
standard_config:
experiment_mappings:
Expand Down Expand Up @@ -1961,6 +1975,7 @@ components:
resource_request:
$ref: '#/components/schemas/ResourceRequest'
timeout:
pattern: ^[0-9]+(ms|s|m|h)$
type: string
log_config:
$ref: '#/components/schemas/RouterVersion_log_config'
Expand Down Expand Up @@ -2001,6 +2016,7 @@ components:
endpoint:
type: string
timeout:
pattern: ^[0-9]+(ms|s|m|h)$
type: string
annotations:
nullable: true
Expand All @@ -2023,8 +2039,10 @@ components:
max_replica:
type: integer
cpu_request:
pattern: ^(\d{1,3}(\.\d{1,3})?)$|^(\d{2,5}m)$
deadlycoconuts marked this conversation as resolved.
Show resolved Hide resolved
type: string
memory_request:
pattern: ^\d+(Ei?|Pi?|Ti?|Gi?|Mi?|Ki?)?$
type: string
type: object
LogLevel:
Expand All @@ -2047,8 +2065,10 @@ components:
batch_load: true
table: table
service_account_secret: service_account_secret
nullable: true
properties:
table:
pattern: ^[a-zA-Z][a-zA-Z0-9-]+\.\w+([_]?\w)+\.\w+([_]?\w)+$
type: string
service_account_secret:
type: string
Expand All @@ -2064,12 +2084,15 @@ components:
brokers: brokers
topic: topic
serialization_format: json
nullable: true
properties:
brokers:
description: Comma-separated list of host and port pairs that are the addresses
of the Kafka brokers.
pattern: ^([a-zA-Z]+:\/\/)?\[?([0-9a-zA-Z\-%._:]*)\]?:([0-9]+)(,([a-zA-Z]+:\/\/)?\[?([0-9a-zA-Z\-%._:]*)\]?:([0-9]+))*$
type: string
topic:
pattern: ^[A-Za-z0-9_.-]{1,249}$
type: string
serialization_format:
enum:
Expand Down Expand Up @@ -2112,6 +2135,7 @@ components:
endpoint:
type: string
timeout:
pattern: ^[0-9]+(ms|s|m|h)$
type: string
port:
type: integer
Expand Down Expand Up @@ -2142,6 +2166,15 @@ components:
type: object
RouterEnsemblerConfig:
example:
pyfunc_config:
project_id: 7
ensembler_id: 9
resource_request:
min_replica: 0
max_replica: 6
memory_request: memory_request
cpu_request: cpu_request
timeout: timeout
updated_at: 2000-01-23T04:56:07.000+00:00
standard_config:
experiment_mappings:
Expand Down Expand Up @@ -2179,11 +2212,14 @@ components:
enum:
- standard
- docker
- pyfunc
type: string
standard_config:
$ref: '#/components/schemas/EnsemblerStandardConfig'
docker_config:
$ref: '#/components/schemas/EnsemblerDockerConfig'
pyfunc_config:
$ref: '#/components/schemas/EnsemblerPyfuncConfig'
created_at:
format: date-time
readOnly: true
Expand Down Expand Up @@ -2235,12 +2271,14 @@ components:
nullable: true
properties:
image:
pattern: ^([a-zA-Z0-9]+(?:[._-][a-zA-Z0-9]+)*(?::\d{2,5})?\/)?([a-zA-Z0-9]+(?:[._-][a-zA-Z0-9]+)*\/)*([a-zA-Z0-9]+(?:[._-][a-zA-Z0-9]+)*)(?::[a-zA-Z0-9]+(?:[._-][a-zA-Z0-9]+)*)?$
type: string
resource_request:
$ref: '#/components/schemas/ResourceRequest'
endpoint:
type: string
timeout:
pattern: ^[0-9]+(ms|s|m|h)$
type: string
port:
type: integer
Expand All @@ -2261,6 +2299,34 @@ components:
- resource_request
- timeout
type: object
EnsemblerPyfuncConfig:
description: ensembler config when ensembler type is pyfunc
example:
project_id: 7
ensembler_id: 9
resource_request:
min_replica: 0
max_replica: 6
memory_request: memory_request
cpu_request: cpu_request
timeout: timeout
nullable: true
properties:
project_id:
type: integer
ensembler_id:
type: integer
resource_request:
$ref: '#/components/schemas/ResourceRequest'
timeout:
pattern: ^[0-9]+(ms|s|m|h)$
type: string
required:
- ensembler_id
- project_id
- resource_request
- timeout
type: object
TrafficRule:
example:
routes:
Expand Down Expand Up @@ -2390,6 +2456,15 @@ components:
memory_request: memory_request
cpu_request: cpu_request
ensembler:
pyfunc_config:
project_id: 7
ensembler_id: 9
resource_request:
min_replica: 0
max_replica: 6
memory_request: memory_request
cpu_request: cpu_request
timeout: timeout
updated_at: 2000-01-23T04:56:07.000+00:00
standard_config:
experiment_mappings:
Expand Down Expand Up @@ -2436,6 +2511,7 @@ components:
environment_name:
type: string
name:
pattern: ^[a-z0-9-]*$
type: string
config:
$ref: '#/components/schemas/RouterConfig_config'
Expand All @@ -2444,6 +2520,20 @@ components:
- environment_name
- name
type: object
IdObject_1:
$ref: '#/components/schemas/IdObject'
RouterIdAndVersion:
example:
router_id: 0
version: 6
properties:
router_id:
format: int32
type: integer
version:
format: int32
type: integer
type: object
Event:
example:
event_type: info
Expand Down Expand Up @@ -2767,13 +2857,18 @@ components:
value: value
properties:
name:
pattern: ^[a-zA-Z0-9_]*$
type: string
value:
type: string
required:
- name
type: object
ExperimentConfig:
additionalProperties:
nullable: true
readOnly: true
type: object
example:
type: nop
config: '{}'
Expand All @@ -2794,26 +2889,6 @@ components:
- header
- payload
type: string
inline_response_200:
example:
router_id: 0
properties:
router_id:
format: int32
type: integer
type: object
inline_response_202:
example:
router_id: 0
version: 6
properties:
router_id:
format: int32
type: integer
version:
format: int32
type: integer
type: object
EnsemblersPaginatedResults_allOf:
properties:
paging:
Expand Down Expand Up @@ -3023,6 +3098,15 @@ components:
memory_request: memory_request
cpu_request: cpu_request
ensembler:
pyfunc_config:
project_id: 7
ensembler_id: 9
resource_request:
min_replica: 0
max_replica: 6
memory_request: memory_request
cpu_request: cpu_request
timeout: timeout
updated_at: 2000-01-23T04:56:07.000+00:00
standard_config:
experiment_mappings:
Expand Down Expand Up @@ -3080,6 +3164,7 @@ components:
resource_request:
$ref: '#/components/schemas/ResourceRequest'
timeout:
pattern: ^[0-9]+(ms|s|m|h)$
type: string
log_config:
$ref: '#/components/schemas/RouterConfig_config_log_config'
Expand Down
3 changes: 3 additions & 0 deletions api/api/specs/jobs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,9 @@ components:
x-go-custom-tag: validate:"required"
resources:
$ref: "#/components/schemas/EnsemblingResources"
run_id:
type: string
readOnly: true
env:
type: array
items:
Expand Down
Loading