Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade ubuntu image to 24.04 (LTS) and ffmpeg to 7.1 #408

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from

Conversation

jsheffie
Copy link

@jsheffie jsheffie commented Oct 3, 2024

resolves: #407 , #406

  • add in support for ubuntu 2404 TLS
  • if using ffmpeg 7.1 or greater build with 2404.
  • added logic in update.py to build ffmpeg 7.1 and greater with new lib dependancies
  • added templates for Dockerfile-env-ffmpeg-7.1-plus and Dockerfile-run-ffmpeg-7.1-plus
  • these new templates updated these libs ( to make build work )
    • OPENCOREAMR_VERSION 0.1.5 -> 0.1.6
    • OPENJPEG_VERSION 2.1.2 -> 2.5.2
    • X264_VERSION 20170226-2245-stable -> 20191217-2245-stable
    • X265_VERSION 3.4 -> 4.0
    • XVID_VERSION 1.3.4 -> 1.3.7
    • LIBZMQ_VERSION 4.3.2 -> 4.3.5
    • LIBSRT_VERSION 1.4.1 -> `1.5.31
    • LIBVMAF_VERSION 2.1.1 -> 2.3.1
  • added template Dockerfile-template.ubuntu2404 ( updated buildDeps to use python3 )
  • removed template Dockerfile-template.ubuntu2204 ( which was really pointed to 20.04 )
    Decided not to remove 2204 at this time, so that this PR does not get too large.

Developer notes
I was building this on my desktop w/ ( because I have apple silicon I needed the --platform argument )

./update.py; docker build --no-cache --platform linux/amd64 -t ffmpeg-7.1-ubuntu2404-desktop-build docker-images/7.1/ubuntu2404

Screenshot 2024-10-08 at 2 35 08 PM

Initial Description: was point Dockerfile-template.ubuntu2204 at correct version number

Docker template Dockerfile-template.ubuntu2204 is pointed at

FROM ubuntu:20.04 AS base

Which is old (March 2023), and misleading, since the template name specifies 2204.

should be
FROM ubuntu:22.04 AS base

@jsheffie
Copy link
Author

jsheffie commented Oct 3, 2024

For reference I found/realized because of a SOC2 Vulnerabilities Scan on the docker image.

@jsheffie jsheffie marked this pull request as draft October 3, 2024 16:05
@jsheffie
Copy link
Author

jsheffie commented Oct 3, 2024

converted to draft as I think this is going to require a
X265_VERSION update to 4.0

Working through that now.

@jsheffie
Copy link
Author

jsheffie commented Oct 4, 2024

alright I got 7.1-ubuntu docker-images/7.1/ubuntu2404 to build with no vulnerabilities.
I need to split it out into its own Pull-Request ( probably ) and circle back to making 2204 work.
Note: I did not check in anything into the docker-images directory at the moment. As it just
makes the PR large. Is there a reason that we don't have a
.gitignore configured to ignore the docker-images directory?

./update.py; docker build --no-cache -t 7.1-ubuntu docker-images/7.1/ubuntu2404

Screenshot 2024-10-04 at 8 53 21 AM

@jrottenberg
Copy link
Owner

2404 is the current LTS, https://endoflife.date/ubuntu at this point if it's simpler to upgrade to Noble Numbat, I'd say let's drop 2204.

Thank you very much for the contribution !

@jsheffie
Copy link
Author

jsheffie commented Oct 7, 2024

2404 is the current LTS, https://endoflife.date/ubuntu at this point if it's simpler to upgrade to Noble Numbat, I'd say let's drop 2204.

@jrottenberg concur, I will get this issue and pull-request cleaned up today to reflect the decision to add support for Ubuntu 2404 Noble Numbat, and I will leave all of the 20.04 and 22.04 stuff alone in this pull-request.

@jsheffie jsheffie marked this pull request as ready for review October 8, 2024 19:35
@jsheffie jsheffie changed the title fix dockerfile template for ubuntu point at correct base 'ubuntu:22.04' Upgrade ubuntu images to 24.04 (LTS) and ffmpeg to 7.1 Oct 8, 2024
@jsheffie jsheffie changed the title Upgrade ubuntu images to 24.04 (LTS) and ffmpeg to 7.1 Upgrade ubuntu image to 24.04 (LTS) and ffmpeg to 7.1 Oct 8, 2024
@jsheffie
Copy link
Author

jsheffie commented Oct 8, 2024

2404 is the current LTS, https://endoflife.date/ubuntu at this point if it's simpler to upgrade to Noble Numbat, I'd say let's drop 2204.

Thank you very much for the contribution !

@jrottenberg you are welcome. Alright I cleaned up the PR and I think this is ready to go.
I considered removing 2204 altogether, but I decided that should be done in a different PR probably.
Note: I did run update.py before checking in. update.py now has a bit more control over building newer versions of ffmpeg. Let me know your thoughts on this.

@jrottenberg
Copy link
Owner

Very nice, running it on https://dev.azure.com/video-tools/ffmpeg/_build/results?buildId=690&view=results

@jrottenberg
Copy link
Owner

Why 7.1 has its own env and run files ? I needed to change the download mirror for xvid, but that should be the main change required, rest should work for every version of ffmpeg, ideally.

Trying to not have to do too much per version.

@jsheffie
Copy link
Author

Very nice, running it on https://dev.azure.com/video-tools/ffmpeg/_build/results?buildId=690&view=results

  • Looks like Build_Docker_Images ubuntu2404_7.1 succeeded.
  • there were 32 errors, 16 of which were 2204. Maybe it is worth pulling out the 2204 build now. ( I figured we would do it after this, but after seeing the build results maybe it makes since to do it now )
    Let me know if you want me to pull 2204 out and I will do it.

@jsheffie
Copy link
Author

jsheffie commented Oct 10, 2024

Why 7.1 has its own env and run files ? I needed to change the download mirror for xvid, but that should be the main change required, rest should work for every version of ffmpeg, ideally.

Trying to not have to do too much per version.

I just chose the newest thing. ( just trying to make one thing work, for starters )
We can probably pull this back to a couple of revisions ( ffmpeg 5.1 ) and it will probably work.

so: Build_Docker_Images ubuntu2404_7.1 now works.

We can turn the knobs in update.py
- VARIANTS
- SKIP_VARIANTS
- and the new logic ffmpeg > 7.1 lib deps. Dockerfile-{env_or_run}-ffmpeg-7.1-plus

To dial this back to get everything building.

Looking Again

Very nice, running it on https://dev.azure.com/video-tools/ffmpeg/_build/results?buildId=690&view=results

  • Looks like Build_Docker_Images ubuntu2404_7.1 succeeded.
  • there were 32 errors, 16 of which were 2204. Maybe it is worth pulling out the 2204 build now. ( I figured we would do it after this, but after seeing the build results maybe it makes since to do it now )
    Let me know if you want me to pull 2204 out and I will do it.

🤔 actually: this build (of the same branch): https://dev.azure.com/video-tools/ffmpeg/_build/results?buildId=691&view=results
Did much better, it had the following problems:

  • Build_Docker_Images alpine313_5.1.6 == error: ran longer than the maximum time of 60 minutes
  • Build_Docker_Images vaapi2204_5.1.6 == Push docker image, failed with received unexpected HTTP status: 500 Internal Server Error
  • Build_Docker_Images alpine313_4.4.5 == error: ran longer than the maximum time of 60 minutes
  • Build_Docker_Images alpine313_3.4.13 == error: ran longer than the maximum time of 60 minutes
  • Build_Docker_Images scratch313_3.4.13 == error: ran longer than the maximum time of 60 minutes
  • Build_Docker_Images ubuntu2204_2.8.22 == error: ran longer than the maximum time of 60 minutes ( also a: gent Hosted Agent did not respond to a cancelation request )
  • Build_Docker_Images ubuntu2204_2.8.22 == error: here I am seeing the ffmpeg dep libs build errors.
  • Build_Docker_Images vaapi2204_2.8.22 == error: here I am seeing the ffmpeg dep libs build
  • `Build_Docker_Images nvidia2204_2.8.22`` == error: here I am seeing the ffmpeg dep libs build

based on this information: I am going to tweek the update.py to try and get us better results.

@jsheffie
Copy link
Author

jsheffie commented Oct 10, 2024

Doing some local builds to see if this works:

for debuging the deps
$ docker run -it --rm alpine:3.20 sh

$ ./update.py; docker build --no-cache --platform linux/amd64 -t ffmpeg-5.1-alpine320-desktop-build docker-images/5.1/alpine320

@jsheffie jsheffie marked this pull request as draft October 10, 2024 20:23
@jrottenberg
Copy link
Owner

jrottenberg commented Oct 17, 2024

Thank you so much for all the pushes and effort on it, I'll catch up and rebase on the weekend, but one thing I've noticed, the build doesn't pass pre-commit, can you run the step locally (pre-commit install ; pre-commit run -a) to fix the minor lint issues ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade ubuntu image to 24.04 (LTS) and ffmpeg to 7.1
2 participants