`hdf5` Bug Avoidance #75

CodeGat · 2024-07-03T08:24:16Z

This PR principally updates the spack-config used to deploy the model, so we avoid the buggy hdf5 1.13.3 that caused 2023.03.1 to be unreleased. The spack-packages repo was also updated to keep it relatively up to date, although the changes shouldn't have much of a bearing on the deployment.

References #73
Closes #74

…2024.04.23

…onents from include: array as part of #60

…n a preference

`ACCESS-OM2` Dependencies and Compiler Update

…ck-packages

github-actions · 2024-07-03T08:25:13Z

🚀 Deploying access-om2 2024.03.2 as prerelease pr75-3

Details and usage instructions

This access-om2 model will be deployed as:

2024.03.2 as a Release (when merged).
pr75-3 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr75-3

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr75-3 environment.

🛠️ Using: spack-packages `2024.07.03`, spack-config `2024.07.02`

Details

It will be deployed using:

access-nri/spack-packages version 2024.07.03
access-nri/spack-config version 2024.07.02

If this is not what was expected, commit changes to config/versions.json.

CodeGat · 2024-07-03T22:19:44Z

This current build is using hdf5 1.14.2 - https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/9774408940/job/26982640737?pr=75#step:8:366

dougiesquire · 2024-07-03T23:57:14Z

Thanks @CodeGat!

Some very basic testing indicates that the prerelease runs successfully but there is an impact on performance from these updates 😞

In the table below:

Exp 1 uses release-1deg_jra55_ryf modified to run for 1 year at a time;
Exp 2 uses the same configuration but the executable paths are changed to those built in prerelease pr75-3.

I ran three years of each. The experiments were run at the same time on Gadi. Times in the table are from the PAYU_WALLTIME field in job.yaml.

	Exp 1	Exp 2
`output000`	820.90 s	841.14 s
`output001`	830.14 s	850.29 s
`output002`	819.19 s	868.89 s
average	823.41 s	853.44

Below is the (selective) output of a diff between the two experiments, confirming that (this time) the executables were indeed changed in Exp 2 to the prerelease versions:

$ git diff -- om2_1deg_jra55_ryf_pr75-orig om2_1deg_jra55_ryf_pr75-3
diff --git a/om2_1deg_jra55_ryf_pr75-orig/config.yaml b/om2_1deg_jra55_ryf_pr75-3/config.yaml
index de774f2..3c38ae4 100644
--- a/om2_1deg_jra55_ryf_pr75-orig/config.yaml
+++ b/om2_1deg_jra55_ryf_pr75-3/config.yaml
@@ -30,7 +30,7 @@ input:
 submodels:
     - name: atmosphere
       model: yatm
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
       input:
             - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
             - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -38,7 +38,7 @@ submodels:

     - name: ocean
       model: mom
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/grid_spec.nc
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc
@@ -57,7 +57,7 @@ submodels:

     - name: ice
       model: cice5
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_360x300_24x1_24p.exe
       input:
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.05.30/grid.nc
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.10.22/kmt.nc
@@ -74,7 +74,7 @@ collate:
   mem: 30GB
   ncpus: 4
   queue: normal
-  exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/mppnccombine.spack
+  exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/mppnccombine.spack

 # Misc
 runlog: false
diff --git a/om2_1deg_jra55_ryf_pr75-orig/manifests/exe.yaml b/om2_1deg_jra55_ryf_pr75-3/manifests/exe.yaml
index 80ed3f5..0d9571d 100644
--- a/om2_1deg_jra55_ryf_pr75-orig/manifests/exe.yaml
+++ b/om2_1deg_jra55_ryf_pr75-3/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
   hashes:
-    binhash: 4e8b4ef76e971c4af3b26cfac632e160
-    md5: 5baa1d417fe6708fc30cbeaa57d82f96
+    binhash: b21a1ef267acae687c3c4cbc0ae51a8e
+    md5: f434bd5b517481ae27b4c8f47f95b94d
 work/ice/cice_auscom_360x300_24x1_24p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_360x300_24x1_24p.exe
   hashes:
-    binhash: 3a65f67d21152e77034da28c22a94c66
-    md5: 37866455b057c85c3ea50c0ef0ea840b
+    binhash: 6f048474c92f2b2cc4ce14d159d9c3ad
+    md5: f74eb0ac0ca781c62d45c3a5e3f2fde7
 work/ocean/fms_ACCESS-OM.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
   hashes:
-    binhash: 92ce1ff1a38f44f92ceafd67e8e7142c
-    md5: a3f10baeadb88e813b4a8121f61a6226
+    binhash: f4eaa918e5e24afa8050d4a5c9ae2382
+    md5: df8bec1f690c24cdfbeced2481352c38

aidanheerdegen · 2024-07-04T05:24:45Z

This is roughly a 4% degradation in performance. My supposition is that it's the parallelio, as this showed a degradation in performance.

Options at this point:

Live with the change
Get more data (I'm not too concerned about 1 degree, but if it has a big impact on 0.1 that could be expensive)
Try reverting parallelio and see if that fixes it
Don't bother with this right now, and in the BGC case just add the bug fixes

dougiesquire · 2024-07-04T10:37:09Z

I also tested a 0.1 deg configuration. In the table below:

Exp 1 uses release-01deg_jra55_ryf modified to run for 1 month at a time;
Exp 2 uses the same configuration but the executable paths are changed to those built in prerelease pr75-3.

I ran three months of each. The experiments were run at the same time on Gadi. Times in the table are from the PAYU_WALLTIME field in job.yaml.

	Exp 1	Exp 2
`output000`	4946.68 s	5040.16 s
`output001`	4517.40 s	4630.98 s
`output002`	4991.54 s	5098.09 s
average	4818.54 s	4923.07 s

So Exp 2 looks ~2-3% slower...

Below is the (selective) output of a diff between the two experiments:

$ git diff -- om2_01deg_jra55_ryf_pr75-orig om2_01deg_jra55_ryf_pr75-3
diff --git a/om2_01deg_jra55_ryf_pr75-orig/config.yaml b/om2_01deg_jra55_ryf_pr75-3/config.yaml
index f8c3249..6da9ffd 100644
--- a/om2_01deg_jra55_ryf_pr75-orig/config.yaml
+++ b/om2_01deg_jra55_ryf_pr75-3/config.yaml
@@ -28,7 +28,7 @@ input:
 submodels:
     - name: atmosphere
       model: yatm
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
       input:
             - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.01deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
             - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -36,7 +36,7 @@ submodels:

     - name: ocean
       model: mom
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.01deg/2020.05.30/grid_spec.nc
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.01deg/2020.05.30/ocean_hgrid.nc
@@ -55,7 +55,7 @@ submodels:

     - name: ice
       model: cice5
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_3600x2700_90x90_722p.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_3600x2700_90x90_722p.exe
       input:
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.01deg/2020.05.30/grid.nc
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.01deg/2020.05.30/kmt.nc
diff --git a/om2_01deg_jra55_ryf_pr75-orig/manifests/exe.yaml b/om2_01deg_jra55_ryf_pr75-3/manifests/exe.yaml
index fe1c2e5..3e3de46 100644
--- a/om2_01deg_jra55_ryf_pr75-orig/manifests/exe.yaml
+++ b/om2_01deg_jra55_ryf_pr75-3/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
   hashes:
-    binhash: 4e8b4ef76e971c4af3b26cfac632e160
-    md5: 5baa1d417fe6708fc30cbeaa57d82f96
+    binhash: b21a1ef267acae687c3c4cbc0ae51a8e
+    md5: f434bd5b517481ae27b4c8f47f95b94d
 work/ice/cice_auscom_3600x2700_90x90_722p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_3600x2700_90x90_722p.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_3600x2700_90x90_722p.exe
   hashes:
-    binhash: 67dd37413d389fc240c8ca24e3337fba
-    md5: f2787a34fcedaedc63a979c260e1c82d
+    binhash: 81832be2447772591f06152856921a42
+    md5: 7edfdd3d4d3c28109bb31321976389a8
 work/ocean/fms_ACCESS-OM.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
   hashes:
-    binhash: 92ce1ff1a38f44f92ceafd67e8e7142c
-    md5: a3f10baeadb88e813b4a8121f61a6226
+    binhash: f4eaa918e5e24afa8050d4a5c9ae2382
+    md5: df8bec1f690c24cdfbeced2481352c38

github-actions · 2024-07-04T10:47:33Z

🚀 Deploying access-om2 2024.03.2 as prerelease pr75-4

Details and usage instructions

This access-om2 model will be deployed as:

2024.03.2 as a Release (when merged).
pr75-4 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr75-4

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr75-4 environment.

🛠️ Using: spack-packages `2024.07.03`, spack-config `2024.07.02`

Details

It will be deployed using:

access-nri/spack-packages version 2024.07.03
access-nri/spack-config version 2024.07.02

If this is not what was expected, commit changes to config/versions.json.

aidanheerdegen · 2024-07-05T07:31:04Z

So better (Exp 2 < 1% slower).

@aidanheerdegen are we okay with this?

I think it's your call. Well mostly your call. Yeah I think that is in the noise. Sorry I didn't see this earlier, we'd kinda decided to shelve the updates to the dependencies until we could work out what the issue was. Looks like you found it.

As long as that version of parallelio is fit for purpose then yeah, I think we should plough on and update everything but parallelio. It would be good to report something to the devs about impaired performance, but that would require a lot more work and who has the time?

I recall @anton-seaice had some issue with a parallelio bug in the MOM6-CICE6 world, but I suppose it doesn't affect OM2?

anton-seaice · 2024-07-07T23:09:38Z

PIO 2.6.2 does a bit more validation of compatibility between dependencies and file formats than 2.5.2 did, this might cause some delay.

I haven't found any issues with 2.6.2, it did fix some issues we had with backward compatibility in 2.5.10 (NCAR/ParallelIO#1985).

@micaeljtoliveira found a floating point exception with hdf5, which we think has been fixed in the hdf5 main branch but not released yet. It would stop compilation when using debug flags, but normal release build compiles ok.

…s) and spack-config 2024.07.05 (remove om2 includes from base modules.yaml

github-actions · 2024-07-08T00:04:01Z

🚀 Deploying access-om2 2024.03.2 as prerelease pr75-5

Details and usage instructions

This access-om2 model will be deployed as:

2024.03.2 as a Release (when merged).
pr75-5 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr75-5

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr75-5 environment.

🛠️ Using: spack-packages `2024.07.07`, spack-config `2024.07.05`

Details

It will be deployed using:

access-nri/spack-packages version 2024.07.07
access-nri/spack-config version 2024.07.05

If this is not what was expected, commit changes to config/versions.json.

github-actions · 2024-07-08T02:31:20Z

🚀 Deploying access-om2 2024.03.2 as prerelease pr75-6

Details and usage instructions

This access-om2 model will be deployed as:

2024.03.2 as a Release (when merged).
pr75-6 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr75-6

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr75-6 environment.

🛠️ Using: spack-packages `2024.07.07`, spack-config `2024.07.05`

Details

It will be deployed using:

access-nri/spack-packages version 2024.07.07
access-nri/spack-config version 2024.07.05

If this is not what was expected, commit changes to config/versions.json.

github-actions · 2024-07-08T04:47:23Z

🚀 Deploying access-om2 2024.03.2 as prerelease pr75-7

Details and usage instructions

This access-om2 model will be deployed as:

2024.03.2 as a Release (when merged).
pr75-7 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr75-7

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr75-7 environment.

🛠️ Using: spack-packages `2024.07.07`, spack-config `2024.07.05`

Details

It will be deployed using:

access-nri/spack-packages version 2024.07.07
access-nri/spack-config version 2024.07.05

If this is not what was expected, commit changes to config/versions.json.

github-actions · 2024-07-08T05:59:40Z

🚀 Deploying access-om2 2024.03.2 as prerelease pr75-7

Details and usage instructions

This access-om2 model will be deployed as:

2024.03.2 as a Release (when merged).
pr75-7 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr75-7

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr75-7 environment.

🛠️ Using: spack-packages `2024.07.07`, spack-config `2024.07.05`

Details

It will be deployed using:

access-nri/spack-packages version 2024.07.07
access-nri/spack-config version 2024.07.05

If this is not what was expected, commit changes to config/versions.json.

github-actions · 2024-07-08T07:23:14Z

🚀 Deploying access-om2 2024.03.2 as prerelease pr75-8

Details and usage instructions

This access-om2 model will be deployed as:

2024.03.2 as a Release (when merged).
pr75-8 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr75-8

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr75-8 environment.

🛠️ Using: spack-packages `2024.07.07`, spack-config `2024.07.05`

Details

It will be deployed using:

access-nri/spack-packages version 2024.07.07
access-nri/spack-config version 2024.07.05

If this is not what was expected, commit changes to config/versions.json.

github-actions · 2024-07-08T07:52:59Z

🚀 Deploying access-om2 2024.03.2 as prerelease pr75-9

Details and usage instructions

This access-om2 model will be deployed as:

2024.03.2 as a Release (when merged).
pr75-9 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules/access-models/
module load access-om2/pr75-9

where the binaries shall be on your $PATH.
This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr75-9 environment.

🛠️ Using: spack-packages `2024.07.08`, spack-config `2024.07.05`

Details

It will be deployed using:

access-nri/spack-packages version 2024.07.08
access-nri/spack-config version 2024.07.05

If this is not what was expected, commit changes to config/versions.json.

dougiesquire · 2024-07-09T00:53:23Z

In the table below:

Exp 1 uses release-1deg_jra55_ryf modified to run for 1 year at a time;
Exp 2 uses the same configuration but the executable paths are changed to those built in prerelease pr75-9

I ran three years of each. The experiments were run at the same time on Gadi. Times in the table are from the PAYU_WALLTIME field in job.yaml.

	Exp 1	Exp 2
`output000`	818.15 s	835.54 s
`output001`	827.26 s	833.76 s
`output002`	826.75 s	833.64 s
average	824.05 s	834.31 s

So the prerelease runs (~ 1% slower).

Below is the (selective) output of a diff between the two experiments:

$ git diff -- orig pr75-9
diff --git a/orig/config.yaml b/pr75-9/config.yaml
index de774f2..8e48816 100644
--- a/orig/config.yaml
+++ b/pr75-9/config.yaml
@@ -30,7 +30,7 @@ input:
 submodels:
     - name: atmosphere
       model: yatm
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/bin/yatm.exe
       input:
             - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
             - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -38,7 +38,7 @@ submodels:

     - name: ocean
       model: mom
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+      exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-ysj247r6owjfwcralshyy4yqpges7n6o/bin/fms_ACCESS-OM.x
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/grid_spec.nc
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc
@@ -57,7 +57,7 @@ submodels:

     - name: ice
       model: cice5
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-3khrmuxma7p4wifa5av2biukx4iomk2l/bin/cice_auscom_360x300_24x1_24p.exe
       input:
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.05.30/grid.nc
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.10.22/kmt.nc
@@ -74,7 +74,7 @@ collate:
   mem: 30GB
   ncpus: 4
   queue: normal
-  exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/mppnccombine.spack
+  exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-ysj247r6owjfwcralshyy4yqpges7n6o/bin/mppnccombine.spack

 # Misc
 runlog: false
diff --git a/orig/manifests/exe.yaml b/pr75-9/manifests/exe.yaml
index 80ed3f5..241395f 100644
--- a/orig/manifests/exe.yaml
+++ b/pr75-9/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/bin/yatm.exe
   hashes:
-    binhash: 4e8b4ef76e971c4af3b26cfac632e160
-    md5: 5baa1d417fe6708fc30cbeaa57d82f96
+    binhash: d14806ad4b0b890d4c2f78a7e34258ab
+    md5: 651ea586afa441f3d94376f28f26b54d
 work/ice/cice_auscom_360x300_24x1_24p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-3khrmuxma7p4wifa5av2biukx4iomk2l/bin/cice_auscom_360x300_24x1_24p.exe
   hashes:
-    binhash: 3a65f67d21152e77034da28c22a94c66
-    md5: 37866455b057c85c3ea50c0ef0ea840b
+    binhash: 000959a21e84ba4baa66c32e61c03d39
+    md5: 237c0c1838fd072e43f7b632315cc2bb
 work/ocean/fms_ACCESS-OM.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-ysj247r6owjfwcralshyy4yqpges7n6o/bin/fms_ACCESS-OM.x
   hashes:
-    binhash: 92ce1ff1a38f44f92ceafd67e8e7142c
-    md5: a3f10baeadb88e813b4a8121f61a6226
+    binhash: 66f01cf1c5e589ba8a253bc09bb365c7
+    md5: 04e7b7594760b7072ff828dd463acf5e

dougiesquire · 2024-07-09T01:45:51Z

We decided not to merge this yet, given that there still seems to be a performance hit and further investigation is needed to work out what is causing this.

The merge-base changed after approval.

CodeGat · 2024-09-24T04:46:32Z

Going to have to rebase this PR to the new main - done some history rewriting here.

CodeGat · 2024-10-30T23:15:46Z

Will be closing this for now. The PR and branch will be here for future work, and it can always be reopened.

CodeGat added 17 commits April 24, 2024 10:57

spack.yaml: Reformatted file

dcb447b

spack.yaml: Blanket bump for all components

8eabdb1

versions.json: Updated spack-packages to 2024.04.20, spack-config to …

742ebbe

…2024.04.23

spack.yaml: Compiler updated to 2021.6.0

bb76f97

spack.yaml: Downgraded to [email protected] as it is not on Gadi yet

ffab2b6

spack.yaml: Updated to [email protected]

f357de8

spack.yaml: renamed exclude_implicits to hide_implicits, removed comp…

8e8670f

…onents from include: array as part of #60

spack.yaml: Added spack.packages.all.target: x86_64

a9ca2c5

config: versions.json uses 2-0-0 schema, deleted local schema

d46d599

spack.yaml: Updated model version to 2024.06.0

6ab1bd7

spack.yaml: Updated version to 2024.03.1 from 2024.06.0

4ed4c7b

spack.yaml: added cppflags arg to openmpi package

f87c8f6

spack.yaml: Updated spack.packages.all to be a requirement rather tha…

c49a105

…n a preference

Merge pull request #73 from ACCESS-NRI/om2-dependencies-update

361b976

`ACCESS-OM2` Dependencies and Compiler Update

spack.yaml: Added comments

0bbf2a6

versions.json: Updated spack-config to avoid hdf5 1.14.3, updated spa…

b43b930

…ck-packages

spack.yaml: Updated model version

0af6b65

CodeGat added priority:high type:bug type:release labels Jul 3, 2024

CodeGat self-assigned this Jul 3, 2024

CodeGat temporarily deployed to Gadi Prerelease July 3, 2024 08:25 — with GitHub Actions Inactive

Revert parallelio to 2.5.2

0785b56

dougiesquire temporarily deployed to Gadi Prerelease July 4, 2024 10:47 — with GitHub Actions Inactive

versions.json: Update spack-packages 2024.07.07 (run type dependencie…

178d325

…s) and spack-config 2024.07.05 (remove om2 includes from base modules.yaml

CodeGat temporarily deployed to Gadi Prerelease July 8, 2024 00:04 — with GitHub Actions Inactive

spack.yaml: Updated comments, rebuild using ACCESS-NRI/spack@42e3100

aef150d

CodeGat had a problem deploying to Gadi Prerelease July 8, 2024 02:31 — with GitHub Actions Failure

CodeGat temporarily deployed to Gadi Prerelease July 8, 2024 03:21 — with GitHub Actions Inactive

spack.yaml: exclude_implicits and autoload: run changes

4826f1e

CodeGat temporarily deployed to Gadi Prerelease July 8, 2024 04:47 — with GitHub Actions Inactive

CodeGat temporarily deployed to Gadi Prerelease July 8, 2024 05:59 — with GitHub Actions Inactive

spack.yaml: include dependencies

b92a62c

CodeGat temporarily deployed to Gadi Prerelease July 8, 2024 07:23 — with GitHub Actions Inactive

versions.json: spack-packages 2024.07.08

e799f0a

CodeGat had a problem deploying to Gadi Prerelease July 8, 2024 07:53 — with GitHub Actions Error

CodeGat temporarily deployed to Gadi Prerelease July 8, 2024 08:02 — with GitHub Actions Inactive

dougiesquire previously approved these changes Jul 9, 2024

View reviewed changes

harshula force-pushed the main branch from 1f5e4c6 to 3b0a7e4 Compare September 24, 2024 04:35

CodeGat marked this pull request as draft October 23, 2024 21:39

CodeGat closed this Oct 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`hdf5` Bug Avoidance #75

`hdf5` Bug Avoidance #75

CodeGat commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024

CodeGat commented Jul 3, 2024

dougiesquire commented Jul 3, 2024

aidanheerdegen commented Jul 4, 2024

dougiesquire commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024

aidanheerdegen commented Jul 5, 2024

anton-seaice commented Jul 7, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

dougiesquire commented Jul 9, 2024

dougiesquire commented Jul 9, 2024

CodeGat commented Sep 24, 2024

CodeGat commented Oct 30, 2024

hdf5 Bug Avoidance #75

hdf5 Bug Avoidance #75

Conversation

CodeGat commented Jul 3, 2024 • edited Loading

github-actions bot commented Jul 3, 2024

CodeGat commented Jul 3, 2024

dougiesquire commented Jul 3, 2024

aidanheerdegen commented Jul 4, 2024

dougiesquire commented Jul 4, 2024 • edited Loading

github-actions bot commented Jul 4, 2024

aidanheerdegen commented Jul 5, 2024

anton-seaice commented Jul 7, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

github-actions bot commented Jul 8, 2024

dougiesquire commented Jul 9, 2024

dougiesquire commented Jul 9, 2024

CodeGat commented Sep 24, 2024

CodeGat commented Oct 30, 2024

`hdf5` Bug Avoidance #75

`hdf5` Bug Avoidance #75

CodeGat commented Jul 3, 2024 •

edited

Loading

dougiesquire commented Jul 4, 2024 •

edited

Loading