Add support for querying OCI graph #99
Open
+234
−117
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It's confusingly named `stream_metadata_url` but actually is the URL to the update guidance.
Found this helpful in debugging.
Currently, we're starting a total of N x M scrapers, where N is the number of streams, and M the number of arches. So right now, this would be 3 x 4 = 12 scrapers. This is very wasteful because each scraper individually downloads the release index and update metadata every 30 seconds, even though that metadata is not different per architecture. I think the reason it was set up this way is in case we wanted to host separate e.g. release index or update files _per_ architecture in S3 instead of all together. This can be seen by the fact the code supports templating those URLs with `basearch`. However, it's unlikely we'll be changing that design decision, so let's just do the saner thing and rework the scraping to be stream-based. This is done by changing the scraper to host not one single `Graph` object, but instead a `HashMap<String, Graph>` which maps architectures to graphs. Then, when a request for a graph comes in, we lookup in our cache keying off of the requested architecture. This is prep for adding another dimension to the matrix, which is whether the OCI version of the graph was requested. If we didn't do this cleanup first, it would have blown up the number of scrapers to 24. Part of coreos/fedora-coreos-tracker#1823.
When parsing the release index, check for the new `oci-images` key. If present, also build up a separate graph with only nodes containing OCI information. In that case, the node payload is the pullspec and the scheme declared in the node metadata is `oci`. When a client requests a graph, check if the `oci=` URL parameter was set. If so, return back the OCI graph instead of the OSTree one. Part of coreos/fedora-coreos-tracker#1823.
This is a mix of both new and old clippy warnings. Just doing it in one commit to make it easier.
dustymabe
reviewed
Nov 4, 2024
Comment on lines
+6
to
+8
| // pub static RELEASES_JSON: &str = | ||
| // "https://builds.coreos.fedoraproject.org/prod/streams/${stream}/releases.json"; | ||
| pub static RELEASES_JSON: &str = "http://localhost:8000/releases.json"; |
There was a problem hiding this comment.
will want to change this back before final merge.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When parsing the release index, check for the new
oci-imageskey. Ifpresent, also build up a separate graph with only nodes containing OCI
information. In that case, the node payload is the pullspec and the
scheme declared in the node metadata is
oci.When a client requests a graph, check if the
oci=URL parameter wasset. If so, return back the OCI graph instead of the OSTree one.
Part of coreos/fedora-coreos-tracker#1823.