Scope and structure of this repository, godot-build-scripts and build-containers

[umarcor]

I was about to reply in godotengine/godot#75401 when this repository was created, so let's continue the dialogue here.

For context, the motivation of godotengine/godot#75401 was not only to provide tip/nightly/latest builds of godot's master branch in binary format. That's the foundation for providing container images that game/app developers can use for building, testing and deploying their projects in CI (see godotengine/godot-proposals#1411). There should be containers available for both master and stable releases, so they can use the same flow for testing upcoming changes in the tool and for generating the releases with stable versions. For instance:

- uses: docker://ghcr.io/godotengine/export:master
  with:
    args: --export-release '${{ matrix.preset }}' '${{ matrix.path }}'

In godotengine/godot-proposals#1411, the assumption was that maintainers would need to create a user in docker.io and setup the secrets. However, that's not the case anymore. Nowadays, the default github.token allows pushing containers to ghcr.io/godotengine (which is the "Packages" section of this organisation: https://github.com/orgs/godotengine/packages). Therefore, it's only a matter of deciding in which repository we want to have the workflow that builds and pushes the containers.

In reply to godotengine/godot#75401 (comment), I was to ask whether the correct approach would be to have a workflow execute scripts from godot-build-scripts using containers from build-containers. None of those repos has any CI workflow. The default registry in them is registry.prehensile-tales.com, but I don't know whether that's the official registry used for hosting the images used to generate official Godot releases. I believe it would be sensible to have a (maybe weekly or monthly) workflow for building containers from build-containers and pushing them to ghcr.io/godotengine/build. Then, have another workflow to run godot-build-script and push to ghcr.io/godotengine/editor and ghcr.io/godotengine/export.

In this context, I saw godotengine/build-containers#96. I agree with merging build-containers and godot-build-script being sensible. Both have very little content and traffic, are tightly related, and having separated bugtrackers does not provide much benefit. It might be named godot-builds. Since GitHub supports having multiple decoupled workflows in a single repo, the one building the images would be independent from the one using them.

Then, this repository was created. Currently, it provides redirects to artifacts. As a result, a GitHub login is needed. That's not a problem in GitHub Actions, because the default token allows read access to artifacts. However, in order to retrieve those nightlies from e.g. GitLab, credentials need to be managed. Conversely, neither containers nor release assets do require any login at all. Therefore, although this is a nice foundation (particularly given how long it took to bring it up ❤️), I believe we should evaluate how to improve it in relation to build-containers and godot-build-scripts.
Assuming that those two repos are merged into godot-builds, and that we use those to build proper (with regard to flags, signing, etc.) nightly (daily) binaries, would it be feasible to publish them as release assets of that secondary repo and make this page point to those? Should then this page/repo be also merged into godot-builds as the frontend?

Summarising:

• Two workflows need to be created in order to automate building nightly artifacts of the same "quality" as the regular releases. Shall those be contributed to godot-build-scripts and build-containers? Or shall those two repos be merged first?
• Is this repo to be considered standalone or is it to be merge into godot-builds along with the two other?

For context, MSYS2 has been using dummy pre-release assets to host temporary/staging packages for two years now. See msys2/msys2-autobuild, github.com/msys2/msys2-autobuild/releases and https://github.com/msys2/msys2-autobuild/tree/6788467670f5ce71e62db593c2960f051e457f76#automated-build-process.

[YuriSizov]

Okay, so there are a few things in here. And I'd like to mention beforehand that most of the building stuff is pretty niche, so it's on a backburner, so concreteness of any plans or their timeframe is not guaranteed at the moment.

With that said, I do see the merit of putting our build containers into some docker registry properly. registry.prehensile-tales.com is what we use, as Prehensile Tales, managed by @hpvb, are providing us with a build server. But I can see how trusting a 3rd party registry on an unrelated to Godot domain can be an issue for people. So yeah, if we can put them somewhere where people could easily fetch them through various docker-powered tools, let's do it. Adding a CI action to publish them directly from the repo makes sense to me.

As for providing pre-compiled binaries as containers ready for testing, this can also be useful, of course. For official public builds creation and deployment of those can be a part of our normal build routine, so the overhead should be pretty low. For the rolling version based on the dev branches it's trickier. Without a process to compile nightlies in place, we only have CI builds. And those CI builds, as I mentioned, are not suitable for publishing in an official capacity. We don't provide all the versions, and they are not compiled the same, so putting them on some latest tag wouldn't work. I'd wait for future development on nightlies or other more regular dev builds before considering this.

The reason build scripts don't have any CI is because we do not use GitHub to create the official builds, and I don't think we ever will. We have a build server which is dedicated to our needs and is powerful to handle these big builds with optimizations and other expensive steps. And it's isolated, so signing and whatnot can be done without any risk of exposure. So anything related to building Godot is not going to be a part of the public infrastructure, I'm afraid.

I also don't see much point in merging the repos, unless there is an issue with keeping them in sync. But we update them so rarely, it's probably not a big deal regardless.

And finally, if we have official nightlies or other sources that I can link in this repo, I surely will. For official releases we can add links to the download page on the main website too.

[umarcor]

But I can see how trusting a 3rd party registry on an unrelated to Godot domain can be an issue for people.

I believe that trusting a 3rd party is not much of a problem, as long as the process is public, thus auditable.

• The sources of the environments (containers).
• The scripts.
• The logs of how containers are built and how scripts are run, including the SHAs of the commits, and the sha256shums of the results.
• The public keys of the credentials used for signing.

I'm trusting GitHub or GitLab when I run workflows on the servers provided by them, so why not trust Prehensile Tales (particularly given their obvious interest in the good health of Godot). However, trust and faith are not the same. Whenever I run something on GitHub's or GitLab's runners I do have access to all the information I need to ensure it's correct. I can very easily reproduce everything on my own infrastructure and check that they are building the code I expect them to build and not something else. I don't need to do it every day, but I can do it whenever I need to debug something or just periodically for hygiene.

Without a process to compile nightlies in place, we only have CI builds. And those CI builds, as I mentioned, are not suitable for publishing in an official capacity. We don't provide all the versions, and they are not compiled the same, so putting them on some latest tag wouldn't work.

As commented, the proposal is to have a workflow in godot-build-scripts (or godot-builds) which builds the master branch the same way all regular releases are built and uploads the artifacts as assets of a pre-release in that secondary repository.
So, they would be compiled the same and the would not be in the main repository (to avoid misunderstandings).

For clarification:

• A workflow in build-containers builds the containers and pushes them to ghcr.io/godotengine/build and registry.prehensile-tales.com.
• Another workflow in godot-build-scripts:
  • Builds the master branch of godot and pushes the artifacts to a tip/nightly pre-release in godot-build-script.
  • And, builds the containers for users and pushes them to ghcr.io/godotengine/editor and ghcr.io/godotengine/export.
• The workflows or the artifacts from godot are unused/unrelated.

The reason build scripts don't have any CI is because we do not use GitHub to create the official builds, and I don't think we ever will. We have a build server which is dedicated to our needs and is powerful to handle these big builds with optimizations and other expensive steps. And it's isolated, so signing and whatnot can be done without any risk of exposure. So anything related to building Godot is not going to be a part of the public infrastructure, I'm afraid.

This is really unfortunate, because it requires faith. AFAIAA, the logs of that build server are not public. Hence, we cannot see which sources or optimisations are used. We need to believe that whatever is in the repos is not outdated/incomplete.

I think that nightlies should be built on GHA. Apart from improving the transparency, it would serve as continous testing of those resources.

Should performance be an issue, a GitHub runner (see https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) might be executed in a sandbox on the machine provided by Prehensile Tales and attached to godot-build-scripts. Then, the scheduled workflow would use that runner instead of the ones provided by GitHub. As a result, logs would be shown in the Actions tabs and the same workflow used by Godot for official and nightly releases might be used by any other user on their own runners or using the ones provided by GitHub.

Alternatively, the logs of the runs on the machine provided by Prehensile Tales might be made public, but I'm afraid that might be more effort than setting up a runner.

I also don't see much point in merging the repos, unless there is an issue with keeping them in sync. But we update them so rarely, it's probably not a big deal regardless.

Since scripts are to be run on the containers and containers are to be used for running the scripts, issues and PRs are likely to be related. For instance, 30% of the closed PRs in godot-build-scripts are related to build-containers: https://github.com/godotengine/godot-build-scripts/pulls?q=is%3Apr+is%3Aclosed+build-containers. That's increased effort for both developers and readers with no obvious benefit (wrt requiring separated issues, projects, discussions, actions, etc.).