Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for arm64 archives #1333

Closed
wants to merge 2 commits into from
Closed

Add support for arm64 archives #1333

wants to merge 2 commits into from

Conversation

anthonydahanne
Copy link
Member

@anthonydahanne anthonydahanne commented Oct 17, 2023

Summary

Add support for arm64 archives

  • bellsoft liberica
  • watchexec (via github-release-dependency)
  • syft (via github-release-dependency)
  • upx (via github-release-dependency)

Use Cases

We need buildpacks to have arm64 metadata, to enable dual arch support

Checklist

  • I have viewed, signed, and submitted the Contributor License Agreement.
  • I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
  • I have added an integration test, if necessary.
  • I have reviewed the styleguide for guidance on my code quality.
  • I'm happy with the commit history on this PR (I have rebased/squashed as needed).

@anthonydahanne anthonydahanne requested a review from a team as a code owner October 17, 2023 02:31
@anthonydahanne
Copy link
Member Author

This comment is for bellsoft
INPUT_PRODUCT=nik;INPUT_TYPE=core;INPUT_VERSION=17

source<<ac29c6c2662f5dcd6436f4e4cbc99df2
https://download.bell-sw.com/vm/23.0.1/bellsoft-liberica-vm-openjdk17.0.8.1+1-23.0.1+2-src.tar.gz
ac29c6c2662f5dcd6436f4e4cbc99df2
arm64_sha256<<ac29c6c2662f5dcd6436f4e4cbc99df2
61d351553408bc19be6c66100033663728daa114af0cb20115a5e5abbb1af07d
ac29c6c2662f5dcd6436f4e4cbc99df2
arm64-uri<<ac29c6c2662f5dcd6436f4e4cbc99df2
https://download.bell-sw.com/vm/23.0.1/bellsoft-liberica-vm-core-openjdk17.0.8.1+1-23.0.1+2-linux-aarch64.tar.gz
ac29c6c2662f5dcd6436f4e4cbc99df2
sha256<<ac29c6c2662f5dcd6436f4e4cbc99df2
44c5e9c4488957613fbebce189b58d867ab27c132cd5b723e79efc78674d7724
ac29c6c2662f5dcd6436f4e4cbc99df2
uri<<ac29c6c2662f5dcd6436f4e4cbc99df2
https://download.bell-sw.com/vm/23.0.1/bellsoft-liberica-vm-core-openjdk17.0.8.1+1-23.0.1+2-linux-amd64.tar.gz
ac29c6c2662f5dcd6436f4e4cbc99df2
version<<ac29c6c2662f5dcd6436f4e4cbc99df2
17.0.8
ac29c6c2662f5dcd6436f4e4cbc99df2
purl<<ac29c6c2662f5dcd6436f4e4cbc99df2
23.0.1
ac29c6c2662f5dcd6436f4e4cbc99df2
source_sha256<<ac29c6c2662f5dcd6436f4e4cbc99df2
0f9baae8f965ffe0dc01b66df848be35893568a51536dc5298fd3c4e61d46efa
ac29c6c2662f5dcd6436f4e4cbc99df2

INPUT_PRODUCT=liberica;INPUT_TYPE=jdk;INPUT_VERSION=17

sha256<<119fef28479fd9fe4c4788d19d03da34
8b2a27a576d0c8afc4d41f8e836c60ff4c04bbf747fc491d240975396d4e8cb8
119fef28479fd9fe4c4788d19d03da34
uri<<119fef28479fd9fe4c4788d19d03da34
https://github.com/bell-sw/Liberica/releases/download/17.0.8.1+1/bellsoft-jdk17.0.8.1+1-linux-amd64.tar.gz
119fef28479fd9fe4c4788d19d03da34
version<<119fef28479fd9fe4c4788d19d03da34
17.0.8
119fef28479fd9fe4c4788d19d03da34
source_sha256<<119fef28479fd9fe4c4788d19d03da34
91d759054a5ba8df95fdedf69acf3976189d1ca141858f5160954237dcb1a711
119fef28479fd9fe4c4788d19d03da34
source<<119fef28479fd9fe4c4788d19d03da34
https://github.com/bell-sw/Liberica/releases/download/17.0.8.1+1/bellsoft-jdk17.0.8.1+1-src.tar.gz
119fef28479fd9fe4c4788d19d03da34
arm64_sha256<<119fef28479fd9fe4c4788d19d03da34
87d92de354dcc47bb5c2af4254ca9c508b98a3fa4248e9805f86183f4c5bf944
119fef28479fd9fe4c4788d19d03da34
arm64-uri<<119fef28479fd9fe4c4788d19d03da34
https://github.com/bell-sw/Liberica/releases/download/17.0.8.1+1/bellsoft-jdk17.0.8.1+1-linux-aarch64.tar.gz
119fef28479fd9fe4c4788d19d03da34

INPUT_PRODUCT=liberica;INPUT_TYPE=jdk;INPUT_VERSION=21

arm64-uri<<b818fe8c1b4c15fed8555b18062f18a1
https://github.com/bell-sw/Liberica/releases/download/21+37/bellsoft-jdk21+37-linux-aarch64.tar.gz
b818fe8c1b4c15fed8555b18062f18a1
sha256<<b818fe8c1b4c15fed8555b18062f18a1
22005e92897fa5ee3d2baaafca6242fd408a54e3d093127359b00e26330fcb02
b818fe8c1b4c15fed8555b18062f18a1
uri<<b818fe8c1b4c15fed8555b18062f18a1
https://github.com/bell-sw/Liberica/releases/download/21+37/bellsoft-jdk21+37-linux-amd64.tar.gz
b818fe8c1b4c15fed8555b18062f18a1
version<<b818fe8c1b4c15fed8555b18062f18a1
21.0.0
b818fe8c1b4c15fed8555b18062f18a1
source_sha256<<b818fe8c1b4c15fed8555b18062f18a1
283f57e1b37b7eccc371a7edfc46d961cf568c50b308922b6447c716004fb705
b818fe8c1b4c15fed8555b18062f18a1
source<<b818fe8c1b4c15fed8555b18062f18a1
https://github.com/bell-sw/Liberica/releases/download/21+37/bellsoft-jdk21+37-src.tar.gz
b818fe8c1b4c15fed8555b18062f18a1
arm64_sha256<<b818fe8c1b4c15fed8555b18062f18a1
3034a0d0e6cabcb530d1dc69510519299e907142886b15d08a7820e8163f0242
b818fe8c1b4c15fed8555b18062f18a1

@anthonydahanne anthonydahanne added semver:minor A change requiring a minor version bump type:enhancement A general enhancement labels Oct 17, 2023
@anthonydahanne
Copy link
Member Author

This comment is for watchexec, or any github-release-dependency component buildpack

INPUT_GLOB=watchexec-.+-x86_64-unknown-linux-musl.tar.xz;INPUT_OWNER=watchexec;INPUT_REPOSITORY=watchexec;INPUT_TAG_FILTER=cli-v(1.*)

uri<<ba69d3a3bd8e1d4fd72cd28358c42794
https://github.com/watchexec/watchexec/releases/download/cli-v1.20.5/watchexec-1.20.5-x86_64-unknown-linux-musl.tar.xz
ba69d3a3bd8e1d4fd72cd28358c42794
version<<ba69d3a3bd8e1d4fd72cd28358c42794
1.20.5
ba69d3a3bd8e1d4fd72cd28358c42794
source_sha256<<ba69d3a3bd8e1d4fd72cd28358c42794
2bc04c7ecf58d34a48c3eeea54a76b7e621717cb93305497bea2f6399dd119c6
ba69d3a3bd8e1d4fd72cd28358c42794
source<<ba69d3a3bd8e1d4fd72cd28358c42794
https://github.com/watchexec/watchexec/archive/refs/tags/cli-v1.20.5.tar.gz
ba69d3a3bd8e1d4fd72cd28358c42794
sha256<<ba69d3a3bd8e1d4fd72cd28358c42794
33e091870736833b77b103a8e7268c0ae66d756b8d71203405353eb9d40774f2
ba69d3a3bd8e1d4fd72cd28358c42794

INPUT_ARM_GLOB=watchexec-.+-aarch64-unknown-linux-musl.tar.xz;INPUT_GLOB=watchexec-.+-x86_64-unknown-linux-musl.tar.xz;INPUT_OWNER=watchexec;INPUT_REPOSITORY=watchexec;INPUT_TAG_FILTER=cli-v(1.*)

source<<2e5e069fd353c14fcb601bb8f20453ad
https://github.com/watchexec/watchexec/archive/refs/tags/cli-v1.20.5.tar.gz
2e5e069fd353c14fcb601bb8f20453ad
arm64_sha256<<2e5e069fd353c14fcb601bb8f20453ad
8598eac8099e7b8df2237317b56d216c236742f37b684a91ec630fdc695da6d8
2e5e069fd353c14fcb601bb8f20453ad
arm64-uri<<2e5e069fd353c14fcb601bb8f20453ad
https://github.com/watchexec/watchexec/releases/download/cli-v1.20.5/watchexec-1.20.5-aarch64-unknown-linux-musl.tar.xz
2e5e069fd353c14fcb601bb8f20453ad
sha256<<2e5e069fd353c14fcb601bb8f20453ad
33e091870736833b77b103a8e7268c0ae66d756b8d71203405353eb9d40774f2
2e5e069fd353c14fcb601bb8f20453ad
uri<<2e5e069fd353c14fcb601bb8f20453ad
https://github.com/watchexec/watchexec/releases/download/cli-v1.20.5/watchexec-1.20.5-x86_64-unknown-linux-musl.tar.xz
2e5e069fd353c14fcb601bb8f20453ad
version<<2e5e069fd353c14fcb601bb8f20453ad
1.20.5
2e5e069fd353c14fcb601bb8f20453ad
source_sha256<<2e5e069fd353c14fcb601bb8f20453ad
2bc04c7ecf58d34a48c3eeea54a76b7e621717cb93305497bea2f6399dd119c6
2e5e069fd353c14fcb601bb8f20453ad

@dmikusa
Copy link
Contributor

dmikusa commented Oct 17, 2023

Is introducing new fields like arm64-uri and arm64_sha256 here going to impact the structure of the dependencies in buildpack.toml? I don't think we can modify the structure of the dependency without having an RFC. Or are these changes isolated within the pipeline-builder/CI code?

@anthonydahanne
Copy link
Member Author

Is introducing new fields like arm64-uri and arm64_sha256 here going to impact the structure of the dependencies in buildpack.toml?

Yes, that's the idea, that each "arm64 compatible BP" has a both arm64-uri and arm64_sha256

I don't think we can modify the structure of the dependency without having an RFC

well, there's this one that's still open: paketo-buildpacks/rfcs#297 ; but we also discussed with you to rather have a map for uri and sha256 fields

Anyways, the idea here is to ease the early adopters experience (ie have a Paketo option rather than some personal repos); could we have those 2 fields as "alpha" fields that could be removed any day? Or we could just hack around libpak to replace some amd with some arm and ignore sha256 with an "experimental" flag?

WDYT?

@dmikusa
Copy link
Contributor

dmikusa commented Oct 19, 2023

well, there's this one that's still open: paketo-buildpacks/rfcs#297 ; but we also discussed with you to rather have a map for uri and sha256 fields

I think this RFC should be for what we do in the future with the metadata, outside of buildpack.toml. Possibly the url/sha256 as maps of data or some other format that might be more compact.

Anyways, the idea here is to ease the early adopters experience (ie have a Paketo option rather than some personal repos); could we have those 2 fields as "alpha" fields that could be removed any day? Or we could just hack around libpak to replace some amd with some arm and ignore sha256 with an "experimental" flag?

My worry about introducing these is that if we don't get team buy-in first, others may implement things differently and then we have a fragmented implementation.

We could probably do it without any structure changes while taking on a little bit of duplication/bloat in buildpack.toml. I'm Ok with that in the short term, with the same goal as you, which is getting people able to use this more quickly.

What if we go from this:

api = "0.7"

[buildpack]
  description = "A Cloud Native Buildpack that provides the Watchexec binary tool"
  homepage = "https://github.com/paketo-buildpacks/watchexec"
  id = "paketo-buildpacks/watchexec"
  keywords = ["watchexec", "reloadable", "processes"]
  name = "Paketo Buildpack for Watchexec"
  sbom-formats = ["application/vnd.syft+json", "application/vnd.cyclonedx+json"]
  version = "{{.version}}"

  [[buildpack.licenses]]
    type = "Apache-2.0"
    uri = "https://github.com/paketo-buildpacks/watchexec/blob/main/LICENSE"

[metadata]
  include-files = ["LICENSE", "NOTICE", "README.md", "bin/build", "bin/detect", "bin/main", "buildpack.toml"]
  pre-package = "scripts/build.sh"

  [[metadata.dependencies]]
    cpes = ["cpe:2.3:a:watchexec:watchexec:1.20.5:*:*:*:*:*:*:*"]
    id = "watchexec"
    name = "Watchexec"
    purl = "pkg:generic/[email protected]?arch=amd64"
    sha256 = "33e091870736833b77b103a8e7268c0ae66d756b8d71203405353eb9d40774f2"
    source = "https://github.com/watchexec/watchexec/archive/refs/tags/cli-v1.20.5.tar.gz"
    source-sha256 = "2bc04c7ecf58d34a48c3eeea54a76b7e621717cb93305497bea2f6399dd119c6"
    stacks = ["*"]
    uri = "https://github.com/watchexec/watchexec/releases/download/cli-v1.20.5/watchexec-1.20.5-x86_64-unknown-linux-musl.tar.xz"
    version = "1.20.5"

    [[metadata.dependencies.licenses]]
      type = "Apache-2.0"
      uri = "https://github.com/watchexec/watchexec/blob/main/LICENSE"

[[stacks]]
  id = "*"

to this...

api = "0.7"

[buildpack]
  description = "A Cloud Native Buildpack that provides the Watchexec binary tool"
  homepage = "https://github.com/paketo-buildpacks/watchexec"
  id = "paketo-buildpacks/watchexec"
  keywords = ["watchexec", "reloadable", "processes"]
  name = "Paketo Buildpack for Watchexec"
  sbom-formats = ["application/vnd.syft+json", "application/vnd.cyclonedx+json"]
  version = "{{.version}}"

  [[buildpack.licenses]]
    type = "Apache-2.0"
    uri = "https://github.com/paketo-buildpacks/watchexec/blob/main/LICENSE"

[metadata]
  include-files = ["LICENSE", "NOTICE", "README.md", "bin/build", "bin/detect", "bin/main", "buildpack.toml"]
  pre-package = "scripts/build.sh"

  [[metadata.dependencies]]
    cpes = ["cpe:2.3:a:watchexec:watchexec:1.20.5:*:*:*:*:*:*:*"]
    id = "watchexec"
    name = "Watchexec"
    purl = "pkg:generic/[email protected]?arch=amd64"
    sha256 = "33e091870736833b77b103a8e7268c0ae66d756b8d71203405353eb9d40774f2"
    source = "https://github.com/watchexec/watchexec/archive/refs/tags/cli-v1.20.5.tar.gz"
    source-sha256 = "2bc04c7ecf58d34a48c3eeea54a76b7e621717cb93305497bea2f6399dd119c6"
    stacks = ["*"]
    uri = "https://github.com/watchexec/watchexec/releases/download/cli-v1.20.5/watchexec-1.20.5-x86_64-unknown-linux-musl.tar.xz"
    version = "1.20.5"

    [[metadata.dependencies.licenses]]
      type = "Apache-2.0"
      uri = "https://github.com/watchexec/watchexec/blob/main/LICENSE"

  [[metadata.dependencies]]
    cpes = ["cpe:2.3:a:watchexec:watchexec:1.20.5:*:*:*:*:*:*:*"]
    id = "watchexec-arm64"
    name = "Watchexec"
    purl = "pkg:generic/[email protected]?arch=arm64"
    sha256 = "8598eac8099e7b8df2237317b56d216c236742f37b684a91ec630fdc695da6d8"
    source = "https://github.com/watchexec/watchexec/archive/refs/tags/cli-v1.20.5.tar.gz"
    source-sha256 = "2bc04c7ecf58d34a48c3eeea54a76b7e621717cb93305497bea2f6399dd119c6"
    stacks = ["*"]
    uri = "https://github.com/watchexec/watchexec/releases/download/cli-v1.20.5/watchexec-1.20.5-aarch64-unknown-linux-musl.tar.xz"
    version = "1.20.5"

    [[metadata.dependencies.licenses]]
      type = "Apache-2.0"
      uri = "https://github.com/watchexec/watchexec/blob/main/LICENSE"

[[stacks]]
  id = "*"

The differences:

  1. We're duplicating the dependency info & assigning a different ID. The ID needs to be unique, so we make it unique to the arch.
  2. We update the PURL which has the arch in it.
  3. Update the URI to point to an ARM64 download & we update the sha256 hash.

I think that would actually work without any changes to libpak. It might even with within the context of pipeline builder updates/actions without any changes too. You'd just need to add another entry in pipeline-descriptor.yml for that id (it'll work the same as how we do this for different major versions).

I think that we'd need to change, and I can't recall if this ever got into pipeline-builder, is to give it the ability to filter out some dependencies when you package offline. Then you could pass it a regex when you package that forces it to only bundle some of the dependencies. With that flag, you could build one set for AMD64 and one for ARM64.

Thoughts on that approach?

@anthonydahanne
Copy link
Member Author

closed in favor of #1348

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver:minor A change requiring a minor version bump type:enhancement A general enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants