Documentation on custom builds

[aekiss]

Is there clear, step-by-step documentation on how users can

• find the MOM5, CICE5, libaccessom2 source code used by a given executable, and
• build custom executables with modified sources?

There are some users (e.g. @kialstewart) who will need to build their own executables, but I couldn't find this info documented anywhere. Apologies if I didn't look carefully enough...

[aidanheerdegen]

Sorry Andrew, only just seen this. Thanks for bringing this up.

Will respond with something of substance ASAP.

[aekiss]

Thanks @aidanheerdegen et al for all your work on documenting this.

I couldn't find anything on my first point - apologies if I overlooked it

• find the MOM5, CICE5, libaccessom2 source code used by a given executable

-- is there a way to go from an exe entry in a config.yaml to the corresponding commit in the source repository for that model component? This is something users need quite frequently, to see exactly what code they're running. If it's complicated to do that with Spack, perhaps the configs could contain a tool (e.g. python script) that automates it by reading config.yaml and doing the required spack magic? Or perhaps README.md for each config could include links to the specific commits of MOM5, CICE5 and libaccessom2, auto-updated by CI?

[aidanheerdegen]

You're right, I didn't address your question about finding the source code used by an executable.

We're now capturing all the build information from deployed models

ACCESS-NRI/build-cd#33

The intention is to expose this information in an experiment metadata view, but also make it viewable standalone as record of all builds. A initial prototype that @utkarshgupta95 just created on Friday is available here:

https://experiment-metadb-seven.vercel.app/release-provenance

You can see the individual components in each model by toggling the arrow in the left most "components" column.

[aekiss]

Great! This is awesome.

Some suggestions:

1. is it possible to go to a particular view of https://experiment-metadb-seven.vercel.app/release-provenance (ie the view you get when clicking the toggle arrow) via a direct URL? That would make it easier to link to the relevant table entry.
2. following from 1, this direct link to the table entry could be put in the README for each config
3. it would be one less click to see the source if the table links to the tree view rather than a release tag, eg
   https://github.com/ACCESS-NRI/MOM5/tree/2023.11.09
   rather than
   https://github.com/ACCESS-NRI/mom5/releases/tag/2023.11.09

[aidanheerdegen]

is it possible to go to a particular view of https://experiment-metadb-seven.vercel.app/release-provenance (ie the view you get when clicking the toggle arrow) via a direct URL? That would make it easier to link to the relevant table entry.

Should be possible. I'll ask. The intention was that we can excerpt a subset of model versions and just display those so that it is flexible, which covers your use case I think.

following from 1, this direct link to the table entry could be put in the README for each config

Yes I suppose it could. Good idea.

it would be one less click to see the source if the table links to the tree view rather than a release tag

Good point. I'll ask @utkarshgupta95 to make those changes.

[aidanheerdegen]

@aekiss The first iteration of the developer docs have been merged and are available here

https://github.com/ACCESS-NRI/ACCESS-OM2/blob/main/DEVELOPERS.md

This covers only the most basic use-case of altering a single model component, and doesn't cover the following use cases:

• Altering dependency versions (e.g. netcdf)
• Developing more than one component simultaneously in the same environment
• Developing in more than one environment with the same spack instance (the environments can interfere with each other)

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/developer-docs/2133/1

[aekiss]

Great, thanks for the documentation @aidanheerdegen and team, hopefully I'll have a chance to look through it again next week.

A couple more thoughts:

• will it be possible for users to use the release provenance tool to keep track of the sources used in their local custom builds? Or will it only support the ACCESS-NRI official releases?
• (especially if the latter) can the component repo git hash be included in each component's executable name or path to help users track provenance of their custom builds, as we're doing with ACCESS-OM3? Executable path in configuration branches COSIMA/access-om3#133 (comment)

[aidanheerdegen]

will it be possible for users to use the release provenance tool to keep track of the sources used in their local custom builds? Or will it only support the ACCESS-NRI official releases?

Currently this is only capturing builds deployed by ACCESS-NRI.

can the component repo git hash be included in each component's executable name or path to help users track provenance of their custom builds, as we're doing with ACCESS-OM3?

Micael's solution required specifying the exact commit to install. I don't know how that will work for a developer workflow which isn't doing this. The developer workflow specifies a version to clone and after that it will built whatever is in the local source clone, regardless if it has been git committed or not. Isn't that the same situation that existed with the COSIMA ACCESS-OM2 build scripts?

We could provide some guidance for developers about best practices for building their modified code and providing build provenance. e.g. committing their changes and pushing to a fork and building directly from there, and/or adding their spack.lock file to their model repo.

[aidanheerdegen]

Something that is missing from the first version of the developer docs is how to clean up when development has paused, or finished, i.e. spack env deactivate