Version Normalization as if it were part of core #1208
Conversation
| @@ -0,0 +1,10 @@ | |||
| version: "3.7" | |||
There was a problem hiding this comment.
it is still valuable to leverage docker-compose here to guarantee we get the same env variables as the rest of the core app. but ultimately we want the config to be separate because the normalization image is not run as part of compose up.
There was a problem hiding this comment.
keep in mind we will kill this normalization container at some point too and roll it into the destination, so this will hopefully be short-ish lived. but definitely around for several weeks.
There was a problem hiding this comment.
Since Normalization has the same lifecycle as the rest of core, it shouldn't be separate. I don't see docker-compose.build.yaml as something that tells us what to run but what to build for core. So it is ok if it is declared as a service.
Having every "Core" artifacts listed in the same place will save us a lot of headache or incomplete releases.
There was a problem hiding this comment.
Normalization does not have the same lifecycle as the rest for core. All of core is run on docker-compose up. Normalization is not. Normalization is run when it is invoked from the worker.
There was a problem hiding this comment.
If we add it to docker-compose.yaml docker-compose up will fail it. It is not meant to run at that time.
There was a problem hiding this comment.
i think michel is suggesting adding it to the docker-compose.build.yaml file not docker-compose.yaml
There was a problem hiding this comment.
discussed offline and agreed to keep it separate. the issue with adding to the docker-compose.build.yaml file is that when we push to docker hub to release we rely on both docker-compose.yaml AND docker-compose.build.yaml. They stack on each other, so docker-compose.yaml is part of the build / release process, but it is also how the containers get run. Agreed for now to keep it separate. But we all think it is sad. 😭
| def composeDeps = [ | ||
| "airbyte-config:init", |
There was a problem hiding this comment.
anything that had a : wasn't working in this because project.name just returns the name no prefix (e.g. init).
| @@ -0,0 +1,10 @@ | |||
| version: "3.7" | |||
There was a problem hiding this comment.
Since Normalization has the same lifecycle as the rest of core, it shouldn't be separate. I don't see docker-compose.build.yaml as something that tells us what to run but what to build for core. So it is ok if it is declared as a service.
Having every "Core" artifacts listed in the same place will save us a lot of headache or incomplete releases.
| @@ -0,0 +1,5 @@ | |||
| version: "3.7" | |||
cgardens
force-pushed
the
cgardens/version_normalization
branch
from
096aeab
to
ad8ece3
Compare
| @@ -18,7 +18,8 @@ dependencies { | |||
| implementation project(':airbyte-queue') | |||
|
|
|||
| integrationTestImplementation project(':airbyte-integrations:bases:standard-destination-test') | |||
| integrationTestImplementation files(project(':airbyte-integrations:bases:base-normalization').airbyteDocker.outputs) | |||
| // depends on normalization docker image creation, which is handled in composeBuild. | |||
| integrationTestImplementation files(task('composeBuild').outputs) | |||
There was a problem hiding this comment.
unfortunately i think this is pretty intolerable. i think it pushes me far enough to think that we should scrap this approach.
There was a problem hiding this comment.
this is needed because now the only way that the normalization image gets built is by composeBuild instead of :airbyte-integrations:bases:base-normalization:airbyteDocker. We could keep the airbyteDocker task as well for normalization, but then we'd have 2 sources of truth for how the container gets built which also seems really terrible.
|
I think we should pin the version. Closing this. |
Closes #1190