Revert to full jdk image for egeria container

[dwolfson]

Is there a change to the base image used for containers? It seems that basic commands which I thought used to be there are missing - like Vi, Tar etc.

I tried to copy a file into the image from the host command line and it failed..
(base) ➜ egeriaConnectors kubectl cp ./cocoGlossary.csv lab-odpi-egeria-lab-core-0:/deployments
Defaulted container "egeria" out of: egeria, init-connector (init)
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "e9a1457c133e9f7d322dce3d8228da24b3f7e93f3fd30248f141816dc494d452": OCI runtime exec failed: exec failed: unable to start container process: exec: "tar": executable file not found in $PATH: unknown

Is this expected?

[planetf1]

The 'FROM' line is registry.access.redhat.com/ubi9/openjdk-17-runtime
See redhat catalog entry https://catalog.redhat.com/software/containers/ubi9/openjdk-17-runtime/61ee7d45384a3eb331996bee

In 3.15 we used registry.access.redhat.com/ubi8/openjdk-11

So there are two main changes (as well as constant updates to those core images themselves)

• Moved from Java 11 to Java 17
• Moved from jdk to runtime

We used UBI images, in particular to aim for a platform that is secure as possible. See https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image. The same logic was why the choice of runtime -- minimize the footprint, minimize the attacks. We do not compile java code within the container.

Hwever, The -runtime image has lost 'tar' as you mention. I am inclined to agree this is a step so far as it makes debugging harder.

The 'kubectl debug' command in theory offers much more flexibility in adding all kinds of tools to the existing environment to debug. However I note that at least on openshift this isn't effective:

kubectl debug -it lab-odpi-egeria-lab-core-0  --image=busybox:latest --target=egeria

However whilst this helps in viewing processes (with more tools), it does not give access to the filesystem. This is down to userids - the new container would need to run with the same user...

This is due to several factors followign best practice -

• The minimization of the base image to reduce change of security issues
• The enforcement of non-root processes
• Standard, secure security policies that prevent priv escalation

This demonstrates we have more secure containers :-)

Actions we need to consider

• is it appropriate to stick with runtime
• how to get 'debug' to work -- and document

[planetf1]

Here's one article suggestion a possibility: https://fabianlee.org/2022/09/10/kubernetes-copying-files-into-and-out-of-containers-without-kubectl-cp/

And here's an example (from lib not extralib, as this is a clean image):

➜  Downloads kubectl exec -i lab-odpi-egeria-lab-core-0 -- ls /deployments/server/lib
Defaulted container "egeria" out of: egeria, debugger-tsmpt (ephem), debugger-trvvz (ephem)
audit-log-console-connector-4.0.jar
audit-log-event-topic-connector-4.0.jar
audit-log-file-connector-4.0.jar
audit-log-slf4j-connector-4.0.jar
avro-file-connector-4.0.jar
basic-file-connector-4.0.jar
cohort-registry-file-store-connector-4.0.jar
configuration-encrypted-file-store-connector-4.0.jar
configuration-file-store-connector-4.0.jar
csv-file-connector-4.0.jar
data-folder-connector-4.0.jar
discovery-service-connectors-4.0.jar
dynamic-archiver-connectors-4.0.jar
elasticsearch-integration-connector-4.0.jar
files-integration-connectors-4.0.jar
governance-action-connectors-4.0.jar
governance-services-sample-4.0.jar
graph-repository-connector-jar-with-dependencies-4.0.jar
inmemory-open-metadata-topic-connector-4.0.jar
inmemory-repository-connector-4.0.jar
kafka-integration-connector-4.0.jar
kafka-open-metadata-topic-connector-4.0.jar
omrs-rest-repository-connector-4.0.jar
open-lineage-janus-connector-4.0.jar
open-metadata-archive-directory-connector-4.0.jar
open-metadata-archive-file-connector-4.0.jar
open-metadata-security-samples-4.0.jar
openapi-integration-connector-4.0.jar
openlineage-integration-connectors-4.0.jar
spring-rest-client-connector-4.0.jar
➜  Downloads kubectl exec -i lab-odpi-egeria-lab-core-0 -- cat /deployments/server/lib/spring-rest-client-connector-4.0.jar > file.jar
Defaulted container "egeria" out of: egeria, debugger-tsmpt (ephem), debugger-trvvz (ephem)
➜  Downloads file file.jar
file.jar: Zip archive data, at least v2.0 to extract, compression method=deflate
➜  Downloads jar -tf file.jar  head
➜  Downloads
➜  Downloads jar -tf file.jar |head
META-INF/
META-INF/MANIFEST.MF
org/
org/odpi/
org/odpi/openmetadata/
org/odpi/openmetadata/adapters/
org/odpi/openmetadata/adapters/connectors/
org/odpi/openmetadata/adapters/connectors/restclients/
org/odpi/openmetadata/adapters/connectors/restclients/spring/
org/odpi/openmetadata/adapters/connectors/restclients/spring/SpringRESTClientConnectorProvider.class
➜  Downloads

That article also seems to suggest this is becoming more common, so we are probably doing the right thing for security....

[planetf1]

@lpalashevski @T4ylan @juergenhemelt Interesting question for you -- for typical images you run, what's your view on how tight these images need to be? What's the right balance of usability vs security? Do you think a switch to a fuller image is ok.

My inclination is that we maybe should take one step back to use the fuller image (it's otherwise very much the same base)

Dan - what do you think? We are very late for v4 to change, although it is quick & should be fairly safe.

[planetf1]

We're in the process of needing to better understand how egeria is functioning.

There are many tools in the jdk to help with this -- see standard java tools documentation java 17

These tools are generally not available in the runtime image, which will make further debugging really hard.

Given the fact that

• we know we will need them 'in situ' when we see issues occur
• the full jdk image is still maintained & as secure as it can be

I am going to switch the image back to the full jdk.

We can continue to discuss if this should be reverted long term, but at this point in time it seems appropriate

[T4ylan]

From the security perspective, it should be fine because the images are getting updated "regularly", even with the greater attack vector. But to gain access through these tools in a kubernetes environment is unlikely.

Imho, the images should be as minimal as possible (aside from a security perspective, e.g., for better startup times if the pod gets moved or killed).
If those tools don't have anything to do with the inner workings of the egeria, then they shouldn't be included. There are other ways, as @planetf1 clearly explained.
If they are needed to achieve some functions, then these functions should be in form of a new microservices (or functions in egeria).

[planetf1]

@T4ylan thanks for the feedback. I think it is the right direction, but we need to ensure we have clear documented approach to work through the alternatives in the short term. Practically speaking, orgs will quite likely build their own image in any case, with ours providing a sample. We could also build both a prod and dev image with the two different bases

For now, I needed to make a quick decision for v4, & reverting to the fuller image is required as we need to be able to productively work through some of our challenges with footprint & app behaviour, and most current usage is in dev/poc environments.

As we progress with cloud native we should aim to get back to a more minimalist approach & figure out how to address the analysis use cases in the best way.