[REPOSITORY] New repository for 'uber' image

[planetf1]

Name

egeria-image-full

Owner

planetf1

Deliverable

Provide an 'uber image' to run the egeria server chassis. This will include all the useful content that we need server-chassis side to run in a container environment.

This WILL NOT contain content that is

• client only
• only used by containers OTHER than core Egeria (ie UI, jupyter notebooks etc)
• OCI Container Image
• Secure - based on UBI8
• Contains all content currently in our full assembly, and therefore base egeria image
• Adds connectors from other repositories
• Adds content packs from other repositories
• publishes a new image 'egeria-full' to quay.io & docker.io

This provides the following benefits

• No requirement for user (or chart defaults) to specify additional connectors and content packs
• improved reliability - as the required content will already be present
• allows for more content to be removed from core egeria repo, and yet still be present in our container environments
• act as another example for orgs creating custom images

The new image will be used by our Helm charts

• egeria-base
• odpi-egeria-lab

They will not be used by the following, since we need to be totally clear on what is present and tested - so these will use the existing image

• egeria-pts
• egeria-cts

There is no intention to remove the capability to dynamically load connectors in our helm charts - this is still useful for 3rd party content & testing, it's just that it will be unnecessary in more cases

The existing egeria repository will continue to create the 'egeria' image, and it's expected this new image will add additional layers on top of that image (an alternative would be to reconstruct using jars or the tar.gz archive)

Build, test and CI-CD process

• github actions similar to those used in core egeria to build/publish the new image for arm64 & amd64 to both quay.io & docker.io
• Dockerfile which will incorporate additional content
• No triggers to rebuild based on source repo content - will only rebuild when new repo modified + on a daily schedule
• versioning will follow egeria?

Dependencies

odpi/egeria

• other repos we source content from

Justification

• faciitates repo refactoring
• improve ease of use and reliability
• easier to develop new scenarios using our charts

Assumptions

yes - all true

Additional Information

Versioning needs discussion

^ cc: @mandy-chessell @dwo

Work Plan

Before creating the repo

• review overall request & get clarifications
• get approval on developer/TSC call from maintainers

Creating the repo

• Create the repo under the odpi organization (default gitignore, license, readme)
• Setup branch protection rules
• Set pull request options (allow merge, squash, rebase, suggest updating, allow automerge, do not delete head)
• Update security settings in repo (policy, advisories, alerts)
• set up permissions

First steps

• Initial code-drop (author)

Getting CI/CD started & refining settings

• Add initial build script for PR (including gradle wrapper if required)
• Add initial build script for merge

Further Refinement

• Add link in egeria docs to new repo describing purpose ( https://github.com/odpi/egeria-docs/blob/main/site/docs/education/tutorials/git-and-git-hub-tutorial/egeria-git-repositories.md )
• Add to connector catalog, if new repo is for connector ( https://github.com/odpi/egeria-docs/blob/main/site/docs/connectors/index.md )
• Add to release process docs ( https://github.com/odpi/egeria-docs/blob/main/site/docs/guides/contributor/release-process/index.md )
• Add required credentials for publishing to container repos, maven central etc
• Add artifact signing if needed
• Add dependabot config
• Add CodeQL
• Add to LFX ( https://projectadmin.lfx.linuxfoundation.org/project/a092M00001IV4JtQAL ) for
  • LFXSecurity
  • LFAnalytics
• Add check for stale defects
• Add any required issue tags

Release

• Add release pipeline

[dwo]

👋 Not sure you meant to tag me: I don't believe I have anything to do with this project. 🙇

[planetf1]

In cloud native discussions, the request to ALSO have Docker containers including JUST base egeria, plus the single connector in question, have been requested - for example for HMS, Strimzi. Over time there is also a desire to make the footprint of this image (image size - for security attach surface)

We did in fact used to do this, so I think we have requriements for both single and combined images .. so we should do both.

How should we handle these images?

Options

Build all images in same repo

• Easier to maintain build process - all in a single place for containers
• Consistent with egeria
• Consistent with each other in terms of base egeria level
• X the connector may not be built with this version of egeria
• An aggregate container is only a small variation
• X Aggregate container has more risk of dependency clashes
• Primary trigger could be egeria - but still a seperate repo so harder to manage except via schedule (limited value in triggering off each merge etc)

Distributed build of images

• Keep Dockerfile definition close to actual connector
• More representative of what users of the connector may do in their own build processes
• Closer to future cloud native model
• More to manage (mitigation: look at how to share build fragments?) in pipelines - but easier to customize for unique repo needs
• Very easy to sync with connector updates (same repo)
• connector runs with the version of egeria it's been tested with. Typical deployment being a single container just running the integration service means it can still participate in a cohort, even with mixed versions

On balance I think we should continue with

• Building connector specific images in the relevant connector repo. We can handle snapshots and releases, and the runtime image will always be as the connector developer intends. there is an onus on the repository owner to keep currency with egeria
• Building the aggregate connector in a new repo. This will be scheduled for snapshot, and can be synchronized with release for final versions. Point releases could be added inbetween main egeria releases to pick up on new connector versions if we wish. It's primary use case is evaluatino and demos where it's more convenient to have an 'all in one'

[planetf1]

Our connectors do not all follow the same release strategy, and tend to be built off varying levels of egeria.
In general connectors aim to avoid dependencies that are present in the chassis, but may still have overlap with other connectors in third party libraries, or where their contents are not closely verified

The docker container build in place for some repos will build a container with egeria+that connector, using the version of egeria it's built with.

To combine many different connectors together has the risk of adding varying level of dependencies, risks of clashes between connectors, risk of different egeria libraries. They may change at any time, on their own schedule.

Connectors today are not isolated from egeria core - any of the classes they include are added onto the classpath with no protection.

I think As we deploy demo infrastructure with helm charts, the container is already paramerized (albeit across all platforms). If we want new scenarios we should add in additional containers built with the appropriate connector. For example we add a platform to run the JDBC integration connector, and deploy the right image within the helm charts. This will JUST run jdbc. Similarly if we want to add the HMS connector ,we have another container for that. The definition of extra containers could be better parameterized in the helm charts if needed

It also remains an idea to use isolated class loading to provide in-process protection - though can be tricky in the spring boot environment. This provides the java application with appropriate control albeit with a looser isolation than container based

As part of the cloud native workgroup we need to work on creating slimmer container images per-deployment, as well as a default one to publish for demos etc

So in summary, I'm not convinced an image containing many connectors is a good idea

[planetf1]

cc: @mandy-chessell @dwolfson

[planetf1]

I think we should revisit this only if/when #6043 is implemented

[planetf1]

Closing as not planned