Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test pre-release for MOM5 with generic tracer WOMBAT #76

Draft
wants to merge 27 commits into
base: main
Choose a base branch
from

Conversation

dougiesquire
Copy link
Contributor

Do not merge

This PR is to deploy a pre-release with the MOM5 changes required to run with the rewritten (generic tracer) WOMBAT.

@dougiesquire dougiesquire marked this pull request as draft July 26, 2024 00:17
Copy link
Contributor

The model version in the spack.yaml has not been updated.
Either update it manually, or comment the following to have it updated and committed automatically:

  • !bump major for feature releases
  • !bump minor for bugfixes

@dougiesquire
Copy link
Contributor Author

!bump major

Copy link
Contributor

❌ Failed to bump version or commit changes, see https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/10103452736

@dougiesquire
Copy link
Contributor Author

@CodeGat?

@CodeGat
Copy link
Contributor

CodeGat commented Jul 26, 2024

!bump major

Copy link
Contributor

❌ Failed to bump version or commit changes, see https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/10103695385

Copy link
Contributor

🚀 Deploying access-om2 2024.07.0 as prerelease pr76-3

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.07.0 as a Release (when merged).
  • pr76-3 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr76-3

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr76-3 environment.

🛠️ Using: spack-packages `2024.04.20`, spack-config `2024.04.23`
Details

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

Copy link
Contributor

🚀 Deploying access-om2 2024.07.0 as prerelease pr76-4

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.07.0 as a Release (when merged).
  • pr76-4 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr76-4

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr76-4 environment.

🛠️ Using: spack-packages `2024.04.20`, spack-config `2024.04.23`
Details

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

@dougiesquire
Copy link
Contributor Author

There's an issue with the access-om2/pr76-26 environment module. The prerelease deployment did successfully build the MOM binaries but the environment module includes the wrong binaries on the $PATH.

The following are the correct MOM binaries built by the pr76-26 deployment:

$ ls -alh /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2024.08.23_2024.08.23-oi2lpoedckoyp742smul5qcuoxwu2o3g/bin
total 73M
drwxr-sr-x+ 2 tm70_ci vk83 4.0K Oct 21 20:48 .
drwxr-sr-x+ 4 tm70_ci vk83 4.0K Oct 21 20:48 ..
-rwxr-xr-x+ 1 tm70_ci vk83  73M Oct 21 20:49 fms_ACCESS-OM.x
-rwxr-xr-x+ 1 tm70_ci vk83 174K Oct 21 20:49 mppnccombine.spack

But:

$ module use /g/data/vk83/prerelease/modules
$ module load access-om2/pr76-26
$ which fms_ACCESS-OM.x
/usr/bin/which: no fms_ACCESS-OM.x in ...

Instead of the correct MOM binaries, the access-om2/pr76-26 environment module includes binaries from an ACCESS-ESM1.6 prerelease:

$ which fms_ACCESS-ESM.x
/g/data/vk83/prerelease/apps/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/mom5-git.2024.08.23_access-esm1.6-oycrxfrimqhonk4i6iowchxru4hzfvm5/bin/fms_ACCESS-ESM.x

This is the same version of MOM, but it's a different "type" variant. Perhaps that's why wires are getting crossed?

Halp @aidanheerdegen, @harshula, @CodeGat

@harshula
Copy link
Contributor

Thanks @dougiesquire , We'll look at it.

@harshula
Copy link
Contributor

harshula commented Oct 25, 2024

Hi @aidanheerdegen , This looks like the Pre-release namespace clash you were anticipating.

$ find /g/data/vk83/prerelease/apps/spack/0.22/release/modules/ -iname '*2024.08.23*'
/g/data/vk83/prerelease/apps/spack/0.22/release/modules/linux-rocky8-x86_64/mom5/2024.08.23
/g/data/vk83/prerelease/apps/spack/0.22/release/modules/linux-rocky8-x86_64_v4/mom5/2024.08.23
/g/data/vk83/prerelease/apps/spack/0.22/release/modules/linux-rocky8-x86_64_v4/mom5/access-esm1.5_2024.08.23

@harshula
Copy link
Contributor

Copy link
Contributor

🚀 Deploying access-om2 2024.08.0 as prerelease pr76-27

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.08.0 as a Release (when merged).
  • pr76-27 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr76-27

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-om2- environment.

🛠️ Using: spack 0.22, spack-packages 2024.08.21, spack-config 2024.10.17

Details

It will be deployed using:

  • access-nri/spack on branch 0.22
  • access-nri/spack-packages version 2024.08.21
  • access-nri/spack-config version 2024.10.17

If this is not what was expected, commit changes to config/versions.json.

@harshula
Copy link
Contributor

@chrisb13
Copy link

chrisb13 commented Oct 28, 2024

@dougiesquire @aidanheerdegen, @harshula, @CodeGat. Following from the Slack chat and some Friday discussion I missed..

The following are the correct MOM binaries built by the pr76-26 deployment:

$ ls -alh /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2024.08.23_2024.08.23-oi2lpoedckoyp742smul5qcuoxwu2o3g/bin
total 73M
drwxr-sr-x+ 2 tm70_ci vk83 4.0K Oct 21 20:48 .
drwxr-sr-x+ 4 tm70_ci vk83 4.0K Oct 21 20:48 ..
-rwxr-xr-x+ 1 tm70_ci vk83  73M Oct 21 20:49 fms_ACCESS-OM.x
-rwxr-xr-x+ 1 tm70_ci vk83 174K Oct 21 20:49 mppnccombine.spack

@dougiesquire, how do you know it's these ones? Following @anton-seaice's suggestion, I was going to run a test with the exact paths to the four executables but I don't know how to choose which ones are the ones built by pr76-26. Doing find in here:

[cyb561.gadi-login-05: linux-rocky8-x86_64]$ pwd
/g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64

Gives many options:

[1284][cyb561.gadi-login-05: linux-rocky8-x86_64]$ find ./ -iname "*fms_ACCESS*"
./intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-jiaz6eo2n2ovh6nnv6b6v4dg3byspefx/bin/fms_ACCESS-OM-BGC.x
./intel-2021.10.0/mom5-git.dougiesquire_issue388-accessom-gtracer_2024.06.27-git.8-y45ahkw2oxthkp5wefrjsynqwq4ubwy5/bin/fms_ACCESS-OM.x
./intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-ysj247r6owjfwcralshyy4yqpges7n6o/bin/fms_ACCESS-OM.x
./intel-19.0.5.281/mom5-git.development_2024.06.27-git.3-ze3ogc622mubqjzgw7wedpeehdf6fyoj/bin/fms_ACCESS-OM.x
./intel-19.0.5.281/mom5-git.dev_2024.08.14_2024.08.14-4wmrmgtcarc332xbdp34b4dm3ho7gg7c/bin/fms_ACCESS-OM.x
./intel-19.0.5.281/mom5-git.development_2024.06.27-git.3-qb5tdudpdvupxmorqhawlxrjtmbmpsdz/bin/fms_ACCESS-OM.x
./intel-19.0.5.281/mom5-git.2024.08.23_2024.08.23-oi2lpoedckoyp742smul5qcuoxwu2o3g/bin/fms_ACCESS-OM.x
[1285][cyb561.gadi-login-05: linux-rocky8-x86_64]$ find ./ -iname "*yatm*"
./intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/bin/yatm.exe
./intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/include/yatm_version_mod.mod
./intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-hsvb52bxolscdz6klzydntoay7yf7nlz/bin/yatm.exe
./intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-hsvb52bxolscdz6klzydntoay7yf7nlz/include/yatm_version_mod.mod
./intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-tlis5urth5tc2snznv6uv2oqlelxwbrr/bin/yatm.exe
./intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-tlis5urth5tc2snznv6uv2oqlelxwbrr/include/yatm_version_mod.mod
./intel-19.0.5.281/libaccessom2-git.2023.10.26_2023.10.26-l6hmyszag4j7dey6j5wuahzy7mnh4nl3/bin/yatm.exe
./intel-19.0.5.281/libaccessom2-git.2023.10.26_2023.10.26-l6hmyszag4j7dey6j5wuahzy7mnh4nl3/include/yatm_version_mod.mod
./intel-19.0.5.281/libaccessom2-git.2023.10.26_2023.10.26-xarvbgn4yf4mxy5hxixm4gfbp7rbzsfh/bin/yatm.exe
./intel-19.0.5.281/libaccessom2-git.2023.10.26_2023.10.26-xarvbgn4yf4mxy5hxixm4gfbp7rbzsfh/include/yatm_version_mod.mod
[1286][cyb561.gadi-login-05: linux-rocky8-x86_64]$ find ./ -iname "*cice_auscom_360x300_24x1_24p.exe*"
./intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-3khrmuxma7p4wifa5av2biukx4iomk2l/bin/cice_auscom_360x300_24x1_24p.exe
./intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-a7nepxaj3durabe7ndhkmyxl4gint4r7/bin/cice_auscom_360x300_24x1_24p.exe
./intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-kwi3fa7zzkrkzjqciusglmud7q4hwcpl/bin/cice_auscom_360x300_24x1_24p.exe
./intel-19.0.5.281/cice5-git.2023.10.19_2023.10.19-4tttbvaxg4afg3hjcrfaah6qngp43e5e/bin/cice_auscom_360x300_24x1_24p.exe
./intel-19.0.5.281/cice5-git.2023.10.19_2023.10.19-epnbkescatwhxi2pvcjlj5eflcaupj3t/bin/cice_auscom_360x300_24x1_24p.exe
[cyb561.gadi-login-05: linux-rocky8-x86_64]$ find ./ -iname "*mppnccombine.spack*"
./intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-jiaz6eo2n2ovh6nnv6b6v4dg3byspefx/bin/mppnccombine.spack
./intel-2021.10.0/mom5-git.dougiesquire_issue388-accessom-gtracer_2024.06.27-git.8-y45ahkw2oxthkp5wefrjsynqwq4ubwy5/bin/mppnccombine.spack
./intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-ysj247r6owjfwcralshyy4yqpges7n6o/bin/mppnccombine.spack
./intel-19.0.5.281/mom5-git.development_2024.06.27-git.3-ze3ogc622mubqjzgw7wedpeehdf6fyoj/bin/mppnccombine.spack
./intel-19.0.5.281/mom5-git.dev_2024.08.14_2024.08.14-4wmrmgtcarc332xbdp34b4dm3ho7gg7c/bin/mppnccombine.spack
./intel-19.0.5.281/mom5-git.development_2024.06.27-git.3-qb5tdudpdvupxmorqhawlxrjtmbmpsdz/bin/mppnccombine.spack
./intel-19.0.5.281/mom5-git.2024.08.23_2024.08.23-oi2lpoedckoyp742smul5qcuoxwu2o3g/bin/mppnccombine.spack

Posting here as I thought it would be helpful to have the answer in a public spot for the next naive person.

@anton-seaice
Copy link
Contributor

anton-seaice commented Oct 28, 2024

You can check the build log e.g. here and check via the spack hash

In general, if you have a spack installation of the environment, I think you could find the spack hashes and check them by searching the results of spack find --variants

Copy link
Contributor

🚀 Deploying access-om2 2024.08.0 as prerelease pr76-28

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.08.0 as a Release (when merged).
  • pr76-28 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr76-28

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-om2- environment.

🛠️ Using: spack 0.22, spack-packages 2024.08.21, spack-config 2024.10.17

Details

It will be deployed using:

  • access-nri/spack on branch 0.22
  • access-nri/spack-packages version 2024.08.21
  • access-nri/spack-config version 2024.10.17

If this is not what was expected, commit changes to config/versions.json.

Copy link
Contributor

🚀 Deploying access-om2 2024.08.0 as prerelease pr76-29

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.08.0 as a Release (when merged).
  • pr76-29 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr76-29

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-om2- environment.

🛠️ Using: spack 0.22, spack-packages 2024.08.21, spack-config 2024.10.17

Details

It will be deployed using:

  • access-nri/spack on branch 0.22
  • access-nri/spack-packages version 2024.08.21
  • access-nri/spack-config version 2024.10.17

If this is not what was expected, commit changes to config/versions.json.

Copy link
Contributor

🚀 Deploying access-om2 2024.08.0 as prerelease pr76-30

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.08.0 as a Release (when merged).
  • pr76-30 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr76-30

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-om2- environment.

🛠️ Using: spack 0.22, spack-packages 2024.08.21, spack-config 2024.10.17

Details

It will be deployed using:

  • access-nri/spack on branch 0.22
  • access-nri/spack-packages version 2024.08.21
  • access-nri/spack-config version 2024.10.17

If this is not what was expected, commit changes to config/versions.json.

@chrisb13
Copy link

chrisb13 commented Nov 7, 2024

Update for this PR. Now compared this run with pr76-26 as suggested by @dougiesquire here. Due to this issue, full path executables in config.yaml.txt were used. TL;DR differences exist but are very small, I think we are ok.

Details. md5sums are different..

[1747][cyb561.gadi-login-02: ocean]$ pwd #Dougie's
/home/561/cyb561/access-om2/1deg_jra55_ryf4_dougie/archive/output000/ocean
[cyb561.gadi-login-02: ocean]$ md5sum oceanbgc-3d-zprod_gross-1-monthly-mean-y_1900.nc
bc09f28137688ba6b62acff00aa2c4b9  oceanbgc-3d-zprod_gross-1-monthly-mean-y_1900.nc
[1742][cyb561.gadi-login-05: ocean]$ pwd #Chris'
/home/561/cyb561/access-om2/1deg_jra55_ryf3_cb/archive/output000/ocean
[cyb561.gadi-login-05: ocean]$ md5sum oceanbgc-3d-zprod_gross-1-monthly-mean-y_1900.nc
8b50e69cbc72d9f01ee0612b96b7f162  oceanbgc-3d-zprod_gross-1-monthly-mean-y_1900.nc

Actual fields have very small differences.. Visualising the ncdiff ([cyb561.gadi-login-02: ocean]$ ncdiff /home/561/cyb561/access-om2/1deg_jra55_ryf4_dougie/archive/output000/ocean/oceanbgc-3d-zprod_gross-1-monthly-mean-y_1900.nc /home/561/cyb561/access-om2/1deg_jra55_ryf3_cb/archive/output000/ocean/oceanbgc-3d-zprod_gross-1-monthly-mean-y_1900.nc oceanbgc-3d-zprod_gross-1-monthly-mean-y_1900_diffd.nc) gives:
image

Copy link
Contributor

github-actions bot commented Nov 7, 2024

🚀 Deploying access-om2 2024.08.0 as prerelease pr76-31

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.08.0 as a Release (when merged).
  • pr76-31 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr76-31

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-om2- environment.

🛠️ Using: spack 0.22, spack-packages 2024.08.21, spack-config 2024.10.17

Details

It will be deployed using:

  • access-nri/spack on branch 0.22
  • access-nri/spack-packages version 2024.08.21
  • access-nri/spack-config version 2024.10.17

If this is not what was expected, commit changes to config/versions.json.

Copy link
Contributor

🚀 Deploying access-om2 2024.08.0 as prerelease pr76-28

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.08.0 as a Release (when merged).
  • pr76-28 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr76-28

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-om2- environment.

🛠️ Using: spack 0.22, spack-packages 2024.08.21, spack-config 2024.10.17

Details

It will be deployed using:

  • access-nri/spack on branch 0.22
  • access-nri/spack-packages version 2024.08.21
  • access-nri/spack-config version 2024.10.17

If this is not what was expected, commit changes to config/versions.json.

Copy link
Contributor

🚀 Deploying access-om2 2024.08.0 as prerelease pr76-28

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.08.0 as a Release (when merged).
  • pr76-28 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr76-28

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-om2- environment.

🛠️ Using: spack 0.22, spack-packages 2024.08.21, spack-config 2024.10.17

Details

It will be deployed using:

  • access-nri/spack on branch 0.22
  • access-nri/spack-packages version 2024.08.21
  • access-nri/spack-config version 2024.10.17

If this is not what was expected, commit changes to config/versions.json.

* This change moves us closer to upstream behaviour.
* Remove all the duplication of spack-config's common/modules.yaml.
Copy link
Contributor

🚀 Deploying access-om2 2024.08.0 as prerelease pr76-27

Details and usage instructions

This access-om2 model will be deployed as:

  • 2024.08.0 as a Release (when merged).
  • pr76-27 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om2/pr76-27

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-om2- environment.

🛠️ Using: spack 0.22, spack-packages 2024.08.21, spack-config 2024.11.11

Details

It will be deployed using:

  • access-nri/spack on branch 0.22
  • access-nri/spack-packages version 2024.08.21
  • access-nri/spack-config version 2024.11.11

If this is not what was expected, commit changes to config/versions.json.

@CodeGat
Copy link
Contributor

CodeGat commented Nov 15, 2024

Hey @dougiesquire, due to:

Some of the spack environment names may clash for future deployments in this PR. If possible, can we move this to a new PR, and close this one? Happy to help out if needed.

There are conflicts currently on this branch due to the above, so you'll need to rebase onto main before opening the next PR - let me know if you need a hand!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants