refactor concrete configs out of openeo-aggregator

[soxofaan]

As we're starting to play with multiple aggregator instances, it's confusing to have the original concrete (openeo platform) configs still hardcoded in this repo. It's probably better to refactor them out.
(would also help with #94)

[soxofaan]

related to https://github.com/eu-cdse/openeo-cdse-infra/issues/22

[soxofaan]

I tried to work with more explicit env vars for clarity (replace ENV with OPENEO_AGGREGATOR_CONFIG), e.g. here:

openeo-aggregator/Jenkinsfile

Lines 15 to 16 in d057d00

	docker_run_options_dev = ['-e OPENEO_AGGREGATOR_CONFIG=/home/openeo/aggregator/conf/aggregator.dev.py', '-p 8081:8080']
	docker_run_options_prod = ['-e OPENEO_AGGREGATOR_CONFIG=/home/openeo/aggregator/conf/aggregator.prod.py', '-p 8081:8080']

but after a lot of head scratching I suspect these "docker_run_options" don't do anything anymore, the real env settings are now in puptap/hiera-resenv repo, as far as I understand

[soxofaan]

started with new project openeo-platform-aggregator on VITO bitbucket

[soxofaan]

Also to take into account here: https://github.com/Open-EO/openeo-aggregator/tree/master/integration-tests is currently openeo.cloud oriented as well. Make it more reusable or split off to standalone project?

[soxofaan]

Strongly related with this ticket is refactoring out the concrete dockerfile (which has hard references to configs):

• refactor Dockerfile out of generic openeo-aggregator repo #143

[soxofaan]

started working some more on this, but got sidetracked by #151 due to change in openeo_driver

[soxofaan]

Hi @tcassaert,

I already took various steps in various repo's, but now I'm a bit unsure about the next steps. Can you help me? I'll try to explain the situation below:

Current setup of openEO Platform (aka openeo.cloud) build/deploy workflow:

• Jenkins job openEO/job/openeo-aggregator
  • defined in openeo-aggregator repo: github.com/Open-EO/openeo-aggregator:Jenkinsfile
  • builds docker image openeo-aggregator
  • pushed to vito-docker-private-dev.artifactory.vgt.vito.be/openeo-aggregator
  • triggers a "dockerDeploy" with dev_hosts = 'docker-services-dev-01.vgt.vito.be'
    • involves sudo /bin/systemctl restart docker-openeo-aggregator.service
    • as far as I understand the details of the docker run are defined at hiera-resenv/resenv_generic_int/nodes/docker-services-dev-01.vgt.vito.be.yaml, e.g.

 openeo-aggregator:
   image: "vito-docker-private-dev.artifactory.vgt.vito.be/openeo-aggregator"
   ports:
     - 8081:8080
   env:
     - "OPENEO_BACKEND_CONFIG=/home/openeo/aggregator/conf/aggregator.dev.py"
     - "ENV=dev"
   subscribe: "Docker::Image[openeo-aggregator]"

• create promotion job at openEO/promotion_jobs/openeo-aggregator

This has to/is being split up a bit. Target situation:

• Jenkins job openEO/openeo-aggregator defined at github.com/Open-EO/openeo-aggregator:Jenkinsfile juist builds a wheel. No docker images are to be built here anymore
• Separate job openEO/openeo-platform-aggregator-image (already set up), defined at git.vito.be/BIGGEO/openeo-platform-aggregator (already set up)
  • builds docker image openeo-platform-aggregator with a containerImageBuildPipeline (already set up)
  • pushed to vito-docker-private-dev.artifactory.vgt.vito.be/openeo-platform-aggregator (already set up)
  • there is a "dockerDeploy" trigger that
    • pulls from (new) vito-docker-private-dev.artifactory.vgt.vito.be/openeo-platform-aggregator
    • uses a different OPENEO_BACKEND_CONFIG env vars.
      • for dev: "OPENEO_BACKEND_CONFIG=/home/openeo/aggregator/conf/openeo-platform-aggregator.dev.py"
      • for prod: "OPENEO_BACKEND_CONFIG=/home/openeo/aggregator/conf/openeo-platform-aggregator.prod.py"
  • create promotion job somewhere

I'm a bit unsure on how to do these lasts steps correctly in the appropriate repos, with the appropriate tools and workflows.

To avoid the risk of having to "hot swap" several things, I think it makes sense to:

• keep exisiting "openeo-aggregator" resources in a first step
• introduce new resources named "openeo-platform-aggregator" and make sure it works properly
• switch over (e.g. DNS)
• remove the now unused "openeo-aggregator" resources

(Once the above is done, I can continue removing the then unused resources from the openeo-aggregator repo)

[tcassaert]

The flow that's implemented now:

1. On push to openeo-aggregator, openeo-aggregator pipeline is triggered
2. On succes, openeo-platform-aggregator-image pipeline is triggered

• Creates image
• Deploys image in dev
• Creates promotion job for image and deploy in prod

3. On success, openeo-aggregator-integrationtests pipeline is triggered
4. The promotion job can be manually triggered to deploy to prod

https://openeocloud-dev.vito.be is already using the new deployment and the integrationtests ran successfully against it.

[soxofaan]

@tcassaert I see you added a requirements.txt to the root of https://github.com/Open-EO/openeo-aggregator (which is referenced from the Dockerfile)

this is quite confusing, as toplevel requirements.txt typically list dependencies for the python package itself, but this is just for the docker build that installs the package. Is it possible to move the docker resources, e.g., to a docker subfolder?

[tcassaert]

Good suggestion, fixed!

[tcassaert]

Added the pipeline flow documentation to https://git.vito.be/projects/TPT/repos/openeo_aggregator_cdse/browse/README.md and also added a link to it on confluence.