✨ Move Earthly logic into Dockerfiles #2008

mauromorales · 2023-11-16T17:06:34Z

mauromorales · 2023-11-28T14:54:33Z

I cannot test the Nvidia image, but I've uploaded it here to quay.io/mauromorales/ubuntu:20.04-core-arm64-nvidia-jetson-agx-orin-v2.4.2-33-gbdd8615-dirty-img and @mudler will help me test

mauromorales · 2023-11-29T08:33:14Z

.github/flavors.json

@@ -338,15 +338,5 @@
        "baseImage": "rockylinux:9",
        "arch": "amd64",
        "worker": "self-hosted"
-    },
-    {
-        "family": "nvidia",


I removed it from the flavors because this one is no longer a family/distribution of its own but instead it uses the ubuntu family via the BYOI. See image-arm.yaml and release.yaml to see how the two work together

mauromorales · 2023-11-29T08:38:19Z

.github/workflows/image-arm.yaml

@@ -57,6 +57,84 @@ jobs:
          # end of optional handling for multi line json
          echo "::set-output name=matrix::{\"include\": $content }"

+  build-nvidia-base:


the nvidia base image will only be built if its dockerfile changed or if it's a release (see release-arm.yaml). This will save some runners time and I would actually like to do something similar with all other base images so I created a ticket for this #2034

mauromorales · 2023-11-29T08:39:06Z

.github/workflows/image-arm.yaml

+        run: |
+          docker push quay.io/kairos/ubuntu:nvidia-jetson-agx-orin-master
+
+  nvidia-arm-core:


nvidia now is just consuming the factory with the ubuntu family and passing its base image

mauromorales · 2023-11-29T08:39:59Z

.github/workflows/reusable-docker-arm-build.yaml

              --IMG_COMPRESSION=${{env.IMG_COMPRESSION}}
      - name: Show img sizes
        run: |
          ls -ltra build
          ls -ltrh build
+      - name: Convert all json files into a reports.tar.gz file


extended it so it can also be used on release-arm.yaml

mauromorales · 2023-11-29T08:42:27Z

images/Dockerfile.alpine

-    rc-update add fail2ban
-
-# Symlinks to make elemental installer work
-RUN ln -s /usr/sbin/grub-install /usr/sbin/grub2-install && \


done on the kairos section now

mauromorales · 2023-11-29T08:44:30Z

images/Dockerfile.debian

 ###############################################################
 ####               Post-Process Common to All              ####
 ###############################################################
-FROM all
-RUN ln -s /usr/sbin/grub-install /usr/sbin/grub2-install


done in kairos section now

mauromorales · 2023-11-29T08:47:40Z

images/Dockerfile.kairos

@@ -0,0 +1,86 @@
+# WARNING: Do not build this image on its own, use the different Dockerfile.kairos-* instead


this is not an independent image but the section that needs to run after the different family/distribution dockerfiles. At first I had this as an independent dockerfile that can run by passing a base image, however Earthly does not play well with local images, so for now I am using the target +kairos-dockerfile to concat the necessary family/distro dockerfile and this file. This is not ideal but I think it's a good step in the right direction, if we then publish base images #2034 then we can convert this one to a proper dockerfile and remove the target, plus the whole thing should be much faster

mauromorales · 2023-11-29T08:47:57Z

images/Dockerfile.nvidia

-RUN rm -rf /etc/apt/sources.list.d/nvidia-l4t-apt-source.list
-
-# Symlinks to make installer work
-RUN ln -s /usr/sbin/grub-install /usr/sbin/grub2-install && \


this is now done in the kairos section

mauromorales · 2023-11-29T08:48:52Z

images/Dockerfile.opensuse

+ARG FRAMEWORK_VERSION=master
+# Not to be confused with the concept of MODEL, this is either fips or generic
+ARG SECURITY_PROFILE=generic


these are only here because I need to concat the files using the +kairos-dockerfile target, once #2034 is address they can be removed

images/Dockerfile.ubuntu

.github/workflows/image-arm.yaml

images/Dockerfile.kairos

.github/workflows/reusable-docker-arm-build.yaml

images/Dockerfile.kairos

Itxaka · 2023-11-29T14:21:52Z

images/Dockerfile.kairos

+# Regenerate initrd if necessary
+RUN if [ -f "/usr/bin/dracut" ]; then \
+      kernel=$(ls /lib/modules | head -n1) && \
+      dracut -f "/boot/initrd-${kernel}" "${kernel}" && \


should we just force dracut to overwrite the existing /boot/initrd directly?? that avoids one extra step here to link it.

@Itxaka hmm, sounds like a good idea, how does that work? for alpine if I understood correctly we would just need to remove the --no-scripts flag right?

No, I meant that on alpine we generate the file directly overwriting anything while on dracut we generate a file and then soft link it for some reason? Dracut should be able to overwrite the file directly, so we don't need to link it afterwards

.github/workflows/reusable-docker-arm-build.yaml

.hadolint.yaml

Itxaka

looking pretty good!

Could you link the issue for the docker images from earthly so we can track it down the line?

Earthfile

Signed-off-by: Mauro Morales <[email protected]>

mauromorales force-pushed the 1897-full-kairos-Dockerfile branch from 8bca420 to ac2bdbe Compare November 16, 2023 17:07

mauromorales self-assigned this Nov 16, 2023

mauromorales force-pushed the 1897-full-kairos-Dockerfile branch from 9ea4c4d to cb90777 Compare November 18, 2023 12:51

mauromorales force-pushed the 1897-full-kairos-Dockerfile branch 2 times, most recently from 85d279a to d10ef9d Compare November 27, 2023 13:25

mauromorales commented Nov 29, 2023

View reviewed changes

images/Dockerfile.ubuntu Show resolved Hide resolved

mauromorales changed the title ~~[WIP] ✨ Move Earthly logic into Dockerfiles~~ ✨ Move Earthly logic into Dockerfiles Nov 29, 2023

mauromorales marked this pull request as ready for review November 29, 2023 08:52

mauromorales requested a review from a team November 29, 2023 08:53

jimmykarily reviewed Nov 29, 2023

View reviewed changes

.github/workflows/image-arm.yaml Outdated Show resolved Hide resolved

Itxaka reviewed Nov 29, 2023

View reviewed changes

images/Dockerfile.kairos Outdated Show resolved Hide resolved

jimmykarily reviewed Nov 29, 2023

View reviewed changes

.github/workflows/reusable-docker-arm-build.yaml Outdated Show resolved Hide resolved

Itxaka reviewed Nov 29, 2023

View reviewed changes

images/Dockerfile.kairos Outdated Show resolved Hide resolved

Itxaka reviewed Nov 29, 2023

View reviewed changes

images/Dockerfile.kairos Outdated Show resolved Hide resolved

Itxaka reviewed Nov 29, 2023

View reviewed changes

jimmykarily reviewed Nov 29, 2023

View reviewed changes

.github/workflows/reusable-docker-arm-build.yaml Outdated Show resolved Hide resolved

jimmykarily reviewed Nov 29, 2023

View reviewed changes

.hadolint.yaml Show resolved Hide resolved

Itxaka approved these changes Nov 29, 2023

View reviewed changes

jimmykarily reviewed Nov 29, 2023

View reviewed changes

Earthfile Outdated Show resolved Hide resolved

jimmykarily reviewed Nov 29, 2023

View reviewed changes

Earthfile Outdated Show resolved Hide resolved

mauromorales added 24 commits November 29, 2023 16:31

Add debian & rhel

1cd5bba

Signed-off-by: Mauro Morales <[email protected]>

Fix ubuntu arm generic

0b02bca

Signed-off-by: Mauro Morales <[email protected]>

Framework changes and luet versions

32c5eee

Signed-off-by: Mauro Morales <[email protected]>

hadolint

9fc5ae8

Signed-off-by: Mauro Morales <[email protected]>

yamllint

f2ab9b1

Signed-off-by: Mauro Morales <[email protected]>

test building nvidia on pr

02f6eee

Signed-off-by: Mauro Morales <[email protected]>

fix push

ed973f7

Signed-off-by: Mauro Morales <[email protected]>

fix path

a03234e

Signed-off-by: Mauro Morales <[email protected]>

use quay

5f364ac

Signed-off-by: Mauro Morales <[email protected]>

login quay

97c91b1

Signed-off-by: Mauro Morales <[email protected]>

🤦

ed81c65

Signed-off-by: Mauro Morales <[email protected]>

define the nvidia jetson strategy in the ubuntu file

879e19a

Signed-off-by: Mauro Morales <[email protected]>

Only run build of nvidia if dockerfile changed

e675ad0

Signed-off-by: Mauro Morales <[email protected]>

same for all other steps

4da51b5

Signed-off-by: Mauro Morales <[email protected]>

No need to push latest

038a784

Signed-off-by: Mauro Morales <[email protected]>

process nvidia on master and release

77decc5

Signed-off-by: Mauro Morales <[email protected]>

remove no-base-image

6be9f97

Signed-off-by: Mauro Morales <[email protected]>

extract kairos common & remove non-hwe

78d2c4d

Signed-off-by: Mauro Morales <[email protected]>

Remove Dockerfile.kairos-*

a737ad0

Signed-off-by: Mauro Morales <[email protected]>

hadolint

2581bba

Signed-off-by: Mauro Morales <[email protected]>

forgot to remove this section on debian

2b42473

Signed-off-by: Mauro Morales <[email protected]>

move non-hwe to examples

a075499

Signed-off-by: Mauro Morales <[email protected]>

feedback

3c025c4

Signed-off-by: Mauro Morales <[email protected]>

add name generation for base-images

5ff0d97

Signed-off-by: Mauro Morales <[email protected]>

mauromorales force-pushed the 1897-full-kairos-Dockerfile branch from 316ffc6 to 5ff0d97 Compare November 29, 2023 15:31

mauromorales added 3 commits November 29, 2023 16:31

shoot

864ad68

Signed-off-by: Mauro Morales <[email protected]>

lint

2c9a7f9

Signed-off-by: Mauro Morales <[email protected]>

oops

b2dc647

Signed-off-by: Mauro Morales <[email protected]>

mauromorales merged commit 84f1eaa into master Nov 30, 2023
34 checks passed

mauromorales deleted the 1897-full-kairos-Dockerfile branch November 30, 2023 09:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Move Earthly logic into Dockerfiles #2008

✨ Move Earthly logic into Dockerfiles #2008

mauromorales commented Nov 16, 2023

mauromorales commented Nov 28, 2023

mauromorales Nov 29, 2023

mauromorales Nov 29, 2023

mauromorales Nov 29, 2023

mauromorales Nov 29, 2023

mauromorales Nov 29, 2023

mauromorales Nov 29, 2023

mauromorales Nov 29, 2023

mauromorales Nov 29, 2023

mauromorales Nov 29, 2023

Itxaka Nov 29, 2023

mauromorales Nov 29, 2023

Itxaka Nov 29, 2023

Itxaka left a comment

		@@ -0,0 +1,86 @@
		# WARNING: Do not build this image on its own, use the different Dockerfile.kairos-* instead

✨ Move Earthly logic into Dockerfiles #2008

✨ Move Earthly logic into Dockerfiles #2008

Conversation

mauromorales commented Nov 16, 2023

mauromorales commented Nov 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Itxaka left a comment

Choose a reason for hiding this comment