Upgrade the GSI to Spack-Stack version 1.6.0 #684

DavidHuber-NOAA · 2024-01-18T12:46:39Z

DUE DATE for merger of this PR into develop is 2/29/2024 (six weeks after PR creation).

Description
This upgrades the spack-stack version to 1.6.0 which also upgrades

netCDF-Fortran 4.6.0 -> 4.6.1
sp 2.3.3 -> 2.5.0
CRTM 2.4.0 -> 2.4.0.1
prod_util 1.2.2 -> 2.1.1

Depends on #672
Resolves #674

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?
Built on Hera and Orion. Regression tests performed on Hera. Four tests currently crash (those running CRTM) due to an issue with the way HIRS is versioned in the CRTM. This will be retested once #672 is merged.

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
New and existing tests pass with my changes
Any dependent changes have been merged and published

DavidHuber-NOAA · 2024-01-19T19:59:27Z

Reran regression tests on Hera. All tests pass except the following:

global_4denvar_loproc_updat and global_4denvar_loproc_contrl produced different step sizes on the 6th iteration starting at 10^-16 decimal point.
- Additionally, the log shows ~60,000 Remove_AntCorr errors: Remove_AntCorr(FAILURE) : Input iFOV inconsistent with AC data that seem to be coming from CRTM as well.
- @RussTreadon-NOAA @ADCollard do you know the cause of these errors? Are the differences cause for concern? All J terms were identical to the precision available in the logs.
rrfs_3denvar_glbens_hiproc_updat exceeded the time threshold of 91.64s at 92.87s
hafs_4denvar_glbens_loproc_updat exceeded the time threshold of 353.55s at 358.36s

DavidHuber-NOAA · 2024-01-22T20:55:31Z

@ADCollard @RussTreadon-NOAA I am seeing copious warning messages from the CRTM v2.4.0.1 (~60,000 lines of Remove_AntCorr(FAILURE) : Input iFOV inconsistent with AC data). The CRTM output seems to be nearly identical for the global_4denvar test case (see above for details), so it doesn't seem to be a significant problem numerically. Do you know what may be causing this or who I can ask about it? And should this be resolved before attempting to upgrade to 2.4.0.1?

ADCollard · 2024-01-23T15:40:14Z

@DavidHuber-NOAA Sorry just saw this. I will check out your branch and run it myself.

RussTreadon-NOAA · 2024-01-23T18:29:08Z

I ran global_4denvar on Hera and WCOSS2 (Cactus). The 60K+ lines of Remove_AntCorr(FAILURE) are, as @DavidHuber-NOAA , reports only on Hera. No such output is generated from the Cactus un.

Comparison of the amsua coefficients in the runtime crtm_coeffs directories on Hera and Cactus show differences for amsua_metop-c.SpcCoeff.bin

Hera

-rw-r--r-- 1 role.epic nems  1252 Jan  4 21:17 amsua_aqua.SpcCoeff.bin
-rw-r--r-- 1 role.epic nems 97092 Jan  4 21:17 amsua_aqua.TauCoeff.bin
-rw-r--r-- 1 role.epic nems 12196 Jan  4 21:17 amsua_metop-a.SpcCoeff.bin
-rw-r--r-- 1 role.epic nems 97092 Jan  4 21:17 amsua_metop-a.TauCoeff.bin
-rw-r--r-- 1 role.epic nems 12196 Jan  4 21:17 amsua_metop-a_v2.SpcCoeff.bin
-rw-r--r-- 1 role.epic nems 12196 Jan  4 21:17 amsua_metop-b.SpcCoeff.bin
-rw-r--r-- 1 role.epic nems 97532 Jan  4 21:17 amsua_metop-b.TauCoeff.bin
-rw-r--r-- 1 role.epic nems  1252 Jan  4 21:17 amsua_metop-c.SpcCoeff.bin
-rw-r--r-- 1 role.epic nems 97092 Jan  4 21:17 amsua_metop-c.TauCoeff.bin
-rw-r--r-- 1 role.epic nems 12196 Jan  4 21:17 amsua_n15.SpcCoeff.bin
-rw-r--r-- 1 role.epic nems 97092 Jan  4 21:17 amsua_n15.TauCoeff.bin
-rw-r--r-- 1 role.epic nems 12196 Jan  4 21:17 amsua_n18.SpcCoeff.bin
-rw-r--r-- 1 role.epic nems 97092 Jan  4 21:17 amsua_n18.TauCoeff.bin
-rw-r--r-- 1 role.epic nems 12196 Jan  4 21:17 amsua_n19.SpcCoeff.bin
-rw-r--r-- 1 role.epic nems 97092 Jan  4 21:17 amsua_n19.TauCoeff.bin

Cactus

-rw-r--r-- 1 ops.prod prod  1252 Nov 13 15:03 amsua_aqua.SpcCoeff.bin
-rw-r--r-- 1 ops.prod prod 97092 Nov 13 15:03 amsua_aqua.TauCoeff.bin
-rw-r--r-- 1 ops.prod prod 12196 Nov 13 15:03 amsua_metop-a.SpcCoeff.bin
-rw-r--r-- 1 ops.prod prod 97092 Nov 13 15:03 amsua_metop-a.TauCoeff.bin
-rw-r--r-- 1 ops.prod prod 12196 Nov 13 15:03 amsua_metop-a_v2.SpcCoeff.bin
-rw-r--r-- 1 ops.prod prod 12196 Nov 13 15:03 amsua_metop-b.SpcCoeff.bin
-rw-r--r-- 1 ops.prod prod 97532 Nov 13 15:03 amsua_metop-b.TauCoeff.bin
-rw-r--r-- 1 ops.prod prod 12196 Nov 13 15:03 amsua_metop-c.SpcCoeff.bin
-rw-r--r-- 1 ops.prod prod 97092 Nov 13 15:03 amsua_metop-c.TauCoeff.bin
-rw-r--r-- 1 ops.prod prod 12196 Nov 13 15:03 amsua_n15.SpcCoeff.bin
-rw-r--r-- 1 ops.prod prod 97092 Nov 13 15:03 amsua_n15.TauCoeff.bin
-rw-r--r-- 1 ops.prod prod 12196 Nov 13 15:03 amsua_n18.SpcCoeff.bin
-rw-r--r-- 1 ops.prod prod 97092 Nov 13 15:03 amsua_n18.TauCoeff.bin
-rw-r--r-- 1 ops.prod prod 12196 Nov 13 15:03 amsua_n19.SpcCoeff.bin
-rw-r--r-- 1 ops.prod prod 97092 Nov 13 15:03 amsua_n19.TauCoeff.bin

The Hera amsua_metop-c.SpcCoeff.bin is too small. The Hera spack-stack-1.6.0 crtm/2.4.0.1 fix file directory is missing the amsua_metop-c.SpcCoeff.bin with the antenna correction.

The crtm/2.4.0.1 fix directory on Cactus has five files.

-rw-r--r-- 1 ops.prod prod 12196 Nov 13 15:03 /apps/ops/prod/libs/intel/19.1.3.304/crtm/2.4.0.1/fix/amsua_metop-c.SpcCoeff.bin
-rw-r--r-- 1 ops.prod prod  1252 Nov 13 15:03 /apps/ops/prod/libs/intel/19.1.3.304/crtm/2.4.0.1/fix/amsua_metop-c.SpcCoeff.noACC.bin
-rw-r--r-- 1 ops.prod prod 97092 Nov 13 15:03 /apps/ops/prod/libs/intel/19.1.3.304/crtm/2.4.0.1/fix/amsua_metop-c.TauCoeff.bin
-rw-r--r-- 1 ops.prod prod 20109 Nov 13 15:03 /apps/ops/prod/libs/intel/19.1.3.304/crtm/2.4.0.1/fix/amsua_metop-c_v2.ACCoeff.nc
-rw-r--r-- 1 ops.prod prod  4576 Nov 13 15:03 /apps/ops/prod/libs/intel/19.1.3.304/crtm/2.4.0.1/fix/amsua_metop-c_v2.SpcCoeff.nc

whereas on Hera we have four files

-rw-r--r-- 1 role.epic nems  1252 Jan  4 21:17 /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/intel/2021.5.0/crtm-fix-2.4.0.1_emc-sl2g2hw/fix/amsua_metop-c.SpcCoeff.bin
-rw-r--r-- 1 role.epic nems 97092 Jan  4 21:17 /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/intel/2021.5.0/crtm-fix-2.4.0.1_emc-sl2g2hw/fix/amsua_metop-c.TauCoeff.bin
-rw-r--r-- 1 role.epic nems 20109 Jan  4 21:17 /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/intel/2021.5.0/crtm-fix-2.4.0.1_emc-sl2g2hw/fix/amsua_metop-c_v2.ACCoeff.nc
-rw-r--r-- 1 role.epic nems  4576 Jan  4 21:17 /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/intel/2021.5.0/crtm-fix-2.4.0.1_emc-sl2g2hw/fix/amsua_metop-c_v2.SpcCoeff.nc

The Cactus amsua_metop-c.SpcCoeff.noACC.bin is the same size as the Hera amsua_metop-c.SpcCoeff.bin.

DavidHuber-NOAA · 2024-01-23T18:40:48Z

@RussTreadon-NOAA Thank you for the detailed investigation. I will report this to the spack-stack team.

ADCollard · 2024-01-23T19:03:40Z

@DavidHuber-NOAA @RussTreadon-NOAA I also did the same test and noted the difference between cactus and hera. I also checked the coefficient files in the github and tarred versions of the CRTM fix files.

These are:

https://github.com/JCSDA/crtm  tag v2.4.0_emc.3

and

https://bin.ssec.wisc.edu/pub/s4/CRTM/fix_REL-2.4.0_emc_07112023.tgz

In both cases the metop-c coefficient files are small (lacking the antenna correction information). So the WCOSS2 coefficient files appear inconsistent with the official release, but are in fact the correct ones to use.

@Hang-Lei-NOAA worked with NCO to install the coefficient files on the WCOSS machines, so maybe he can explain the difference.

My guess is that as this was supposed to only be an incremental change (only six files needed to be added) only these files were updated on WCOSS2 (which is definitely the safest way to manage this change).

jswhit · 2024-01-23T20:23:39Z

can this PR support gaea C5?

DavidHuber-NOAA · 2024-01-24T12:45:57Z

@jswhit First, we will need to resolve this CRTM fix file issue. If that can be resolved, then I'm fine with updating the Gaea module to point to the C5/F5 stack -- I think it was just installed on F5 yesterday. Would you be willing to run the regression tests there?

ulmononian · 2024-01-29T18:14:18Z

just wanted to leave a note here that several versions of spack-stack have been installed in a new location to accommodate the decommissioning of c4 and the transition to the f5 filesystem. the available versions and their locations are as follows:

/ncrc/proj/epic/spack-stack/spack-stack-1.4.1/envs/unified-env/install/modulefiles/Core
/ncrc/proj/epic/spack-stack/spack-stack-1.5.0/envs/unified-env/install/modulefiles/Core
/ncrc/proj/epic/spack-stack/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
/ncrc/proj/epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core

there is a PR in the ufs-wm to update gaea to use /ncrc/proj/epic/spack-stack/spack-stack-1.5.1, but testing is not yet completed for this (ufs-community/ufs-weather-model#2115 ). for more information on the f2/f5 transition on the ufs-wm side, please see ufs-community/ufs-weather-model#2101.

DavidHuber-NOAA · 2024-01-31T15:39:41Z

The CRTM-fix file was fixed on Orion and regression tests were run. All tests passed except RTMA, which encountered a non-fatal time exceedance failure:

The runtime for rtma_loproc_updat is 240.516202 seconds.  This has exceeded maximum allowable threshold time of 237.953284 seconds,
resulting in Failure time-thresh of the regression test.

When the fix files are fixed on Hera, Hercules, and Jet, I will run RTs on each machine then mark this ready for review. I will also update the Gaea modulefiles to point to those suggested by @ulmononian so they can be tested by @jswhit.

RussTreadon-NOAA · 2024-01-31T16:23:03Z

@DavidHuber-NOAA: I cloned DavidHuber-NOAA:feature/ss_160 on Hera. It's three commits behind the head of NOAA-EMC/GSI:develop. A manual merge of the upstream develop into the cloned ss_160 did not encounter any merge conflicts. I did this in /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/pr684.

DavidHuber-NOAA · 2024-01-31T18:07:09Z

@RussTreadon-NOAA Apologies, I had pulled develop in but not pushed it back to GitHub. It has been now. Thanks.

DavidHuber-NOAA · 2024-01-31T18:13:02Z

@jswhit I have given a stab at updating the module file and regression test variables and parameters for Gaea-C5. I verified that the GSI builds. The CRTM-fix files have also been fixed on Gaea. Would you mind running the RTs to verify everything is OK?

RussTreadon-NOAA · 2024-01-31T18:17:50Z

GSI PR #691 has not yet been merged into DavidHuber-NOAA:feature/ss_160. GSI PR #691 entered develop this morning (1/31/2024).

DavidHuber-NOAA · 2024-02-13T12:58:51Z

@RussTreadon-NOAA @edwardhartnett @TingLei-NOAA @climbfuji I know now what is causing Hercules to fail with netCDF errors. The errors only appear with the flag I_MPI_EXTRA_FILESYSTEM=1 is set. Spack-stack version 1.6.0 sets this flag in /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/gsi-addon-env/install/modulefiles/Core/stack-intel/2021.9.0.lua and that is why when switching from v1.5.1 to v1.6.0, these failures started to appear. They continued to appear after switching back to v1.5.1 because I added this flag to sub_hercules. Removing this flag from sub_hercules allows the tests to run to completion.

That said, I believe this still indicates an issue with the way the regional GSI is implementing its parallel I/O as this flag allows Intel MPI to distribute I/O across a parallel filesystem. If it is disabled, then native support for parallel I/O is disabled.

My next question is, should I upgrade to v1.6.0 and add unset I_MPI_EXTRA_FILESYSTEM to sub_hercules? This will (continue to) hide the problem and regional workflows using spack-stack v1.6.0 will also need to add this flag when running the GSI.

RussTreadon-NOAA · 2024-02-13T13:29:40Z

Excellent detective work @DavidHuber-NOAA!

I agree that we need to re-examine and likely refactor netcdf parallel i/o. You opened GSI issue #694 for this purpose. Let's capture your findings in this issue. @TingLei-NOAA, issue #694 has been assigned you to you. I also added @ShunLiu-NOAA and @hu5970 to this issue. We need to resolve this problem.

Since ctests pass on Hercules from a spack-stack/1.6.0 build with unset I_MPI_EXTRA_FILESYSTEM, I recommend adding this to sub_hercules. As @DavidHuber-NOAA notes, this hides the problem. It is true that hidden problems are easily forgotten or ignored. This noted, my preference is to see the authoritative GSI move to spack-stack/1.6.0 on Hercules now instead of holding at spack-stack/1.5.1 until regional parallel netcdf i/o is refactored and/or we jump to a later spack-stack release.

TingLei-NOAA · 2024-02-13T14:07:15Z

@DavidHuber-NOAA That is great! Yes "export I_MPI_EXTRA_FILESYSTEM=0" makes my previously failed test succeed now. I have two questions, first , will this make those previously failed hdf5 tests pass now?
Second, is this setup needed for the same packages while on other machines?

DavidHuber-NOAA · 2024-02-13T14:10:20Z

@TingLei-NOAA I'll respond to this in #694.

DavidHuber-NOAA · 2024-02-13T16:29:58Z

Tests were successful on Hercules with the exception of hafs_3denvar_glbens, which failed to produce identical fv3_dynvars files between the hafs_3denvar_glbens_loproc_updat and hafs_3denvar_glbens_hiproc_updat test cases. I repeated this test on the current develop branch on Hercules and found the same result, so this is not a problem with spack-stack v1.6.0, but with Hercules/HAFS DA, so I think that this PR is ready to go. I will create a separate issue to track this.

RussTreadon-NOAA · 2024-02-13T16:55:51Z

@DavidHuber-NOAA , I found similar behavior on Hercules while running ctests for PR #695. PR #695 builds gsi.x and enkf.x using spack-stack/1.5.1. Perhaps resolution of GSI issue #694 will remove this behavior.

RussTreadon-NOAA

Changes look good to me. Thank you @DavidHuber-NOAA for persistently working through various problems.

RussTreadon-NOAA · 2024-02-13T21:01:31Z

@DavidHuber-NOAA , we need to peer reviews for this PR. I signed off as the Handling Reviewer.

RussTreadon-NOAA · 2024-02-13T21:47:08Z

@TingLei-NOAA , since you have already commented on this PR can you serve as a peer reviewer?

RussTreadon-NOAA · 2024-02-13T21:48:12Z

@DavidHuber-NOAA , since this is a spack-stack PR, can we get someone from the g-w or spack-stack teams to serve as a peer reviewer?

AlexanderRichert-NOAA · 2024-02-13T22:15:52Z

@RussTreadon-NOAA this looks good to me from the spack-stack perspective. If it's been tested on the systems in question then those configs look good.

RussTreadon-NOAA · 2024-02-14T00:14:23Z

Thank you @AlexanderRichert-NOAA for your comment. I added you as a reviewer so we can formally capture your input as a reviewer. Your review will help move this PR forward.

AlexanderRichert-NOAA · 2024-02-14T00:27:44Z

modulefiles/gsi_hera.gnu.lua

 local stack_gnu_ver=os.getenv("stack_gnu_ver") or "9.2.0"
 local stack_openmpi_ver=os.getenv("stack_openmpi_ver") or "4.1.5"
 local cmake_ver=os.getenv("cmake_ver") or "3.23.1"
-local prod_util_ver=os.getenv("prod_util_ver") or "1.2.2"
+local prod_util_ver=os.getenv("prod_util_ver") or "2.1.1"
 local openblas_ver=os.getenv("openblas_ver") or "0.3.19"


I believe openblas version should be 0.3.24, @DavidHuber-NOAA can you confirm?

@AlexanderRichert-NOAA You are correct, it should be 0.3.24. I updated it and ran a test build. Thanks!

TingLei-NOAA · 2024-02-14T00:43:31Z

Hi Russ I think I am not qualified to review this pr since I know little about spack-stack (yet). Ting

…

______________________________ Ting Lei Physical Scientist, Contractor with Lynker in support of EMC/NCEP/NWS/NOAA 5830 University Research Ct., Cubicle 2765 College Park, MD 20740 ***@***.*** 301-683-3624

On Tue, Feb 13, 2024 at 4:47 PM RussTreadon-NOAA ***@***.***> wrote: @TingLei-NOAA <https://github.com/TingLei-NOAA> , since you have already commented on this PR can you serve as a peer reviewer? — Reply to this email directly, view it on GitHub <#684 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APEFS7CTXZ2Q4S524G4W3QDYTPNOTAVCNFSM6AAAAABCAG5ZM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBSGY3TSMRYHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

RussTreadon-NOAA · 2024-02-14T00:49:04Z

@TingLei-NOAA , like you, I am not a spack-stack expert. We are both GSI, and now JEDI, developers. Can you review this PR as one who is knowledgeable about the GSI? @AlexanderRichert-NOAA is reviewing this PR as one knowledgeable about spack-stack.

TingLei-NOAA · 2024-02-14T01:59:54Z

Russ Sure if you think my knowlege of gsi would allow me help with this pr as a reviewer I will be happy to do so Ting

…

______________________________ Ting Lei Physical Scientist, Contractor with Lynker in support of EMC/NCEP/NWS/NOAA 5830 University Research Ct., Cubicle 2765 College Park, MD 20740 ***@***.*** 301-683-3624

On Tue, Feb 13, 2024 at 7:49 PM RussTreadon-NOAA ***@***.***> wrote: @TingLei-NOAA <https://github.com/TingLei-NOAA> , like you I am not a spack-stack expert. We are both GSI, and now JEDI, developers. Can you review this PR as one who is knowledgeable about the GSI? @AlexanderRichert-NOAA <https://github.com/AlexanderRichert-NOAA> is reviewing this PR as one knowledgeable about spack-stack. — Reply to this email directly, view it on GitHub <#684 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APEFS7DN32IERIERNSFZ4I3YTQCYZAVCNFSM6AAAAABCAG5ZM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBSHEYTOOJYHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

AlexanderRichert-NOAA

spack-stack paths and versions look good.

TingLei-NOAA

Thanks to @DavidHuber-NOAA for revealing the issues related to fv3reg GSI IO and demonstrating them are not results from the current PR with helps from other colleagues. Also Dave's work in this PR helps a lot in identifying the culprit in the parallel IO issue on hercules by showing the impacts of I_MPI_EXTRA_FILESYSTEM. As a developer for GSI, albeit not an expert in Spack-stack, I find this PR to be good and approve it.

RussTreadon-NOAA

Approve.

DavidHuber-NOAA added 3 commits January 17, 2024 17:54

Initial upgrade to spack-stack v1.6.0. NOAA-EMC#674

c23b510

Update CI library versions.

2136882

Update CI library versions to SS versions.

89d8bf8

RussTreadon-NOAA assigned DavidHuber-NOAA Jan 18, 2024

DavidHuber-NOAA added 2 commits January 18, 2024 19:19

Merge remote-tracking branch 'origin/develop' into feature/ss_160

598c9f7

Remove Cheyenne.

955eedf

Remove Cheyenne submit script.

7f3969a

DavidHuber-NOAA mentioned this pull request Jan 19, 2024

Update to spack-stack 1.6.0 NOAA-EMC/global-workflow#2239

Merged

7 tasks

RussTreadon-NOAA self-requested a review January 20, 2024 12:54

RussTreadon-NOAA self-assigned this Jan 20, 2024

Fix gsi-addon paths for hercules, orion, and S4.

339bf16

DavidHuber-NOAA mentioned this pull request Jan 23, 2024

CRTM-fix differs from WCOSS2, CRTM produces copious errors in the GSI JCSDA/spack-stack#963

Closed

25 tasks

Merge remote-tracking branch 'origin/develop' into feature/ss_160

2ae8c3c

DavidHuber-NOAA added 2 commits January 31, 2024 12:53

Update Gaea-C5 support.

04a737d

Remove references to Cheyenne.

bfb46e2

Merge remote-tracking branch 'origin/develop' into feature/ss_160

2efa4ba

ADCollard mentioned this pull request Feb 13, 2024

Add GFS v16.3.11[12] changes to develop NOAA-EMC/global-workflow#1918

Closed

9 tasks

Re-update Hercules to SS/1.6.0.

e56921c

DavidHuber-NOAA mentioned this pull request Feb 13, 2024

hafs_3denvar_glbens ctest fails on Hercules #697

Closed

Merge remote-tracking branch 'origin/develop' into feature/ss_160

3951289

RussTreadon-NOAA previously approved these changes Feb 13, 2024

View reviewed changes

RussTreadon-NOAA requested a review from AlexanderRichert-NOAA February 14, 2024 00:13

AlexanderRichert-NOAA reviewed Feb 14, 2024

View reviewed changes

RussTreadon-NOAA requested a review from TingLei-NOAA February 14, 2024 02:02

Update openblas version for GNU/SS v1.6.0.

11a4ffd

DavidHuber-NOAA dismissed RussTreadon-NOAA’s stale review via 11a4ffd February 14, 2024 13:29

AlexanderRichert-NOAA approved these changes Feb 14, 2024

View reviewed changes

TingLei-NOAA approved these changes Feb 14, 2024

View reviewed changes

RussTreadon-NOAA self-requested a review February 14, 2024 17:05

RussTreadon-NOAA approved these changes Feb 14, 2024

View reviewed changes

RussTreadon-NOAA merged commit 74ac594 into NOAA-EMC:develop Feb 14, 2024
4 checks passed

RussTreadon-NOAA mentioned this pull request Feb 15, 2024

Issue 694: Upgrade/refactoring for U and V write-out sub for FV3REG GSI failure … #698

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade the GSI to Spack-Stack version 1.6.0 #684

Upgrade the GSI to Spack-Stack version 1.6.0 #684

DavidHuber-NOAA commented Jan 18, 2024 •

edited

Loading

DavidHuber-NOAA commented Jan 19, 2024

DavidHuber-NOAA commented Jan 22, 2024

ADCollard commented Jan 23, 2024

RussTreadon-NOAA commented Jan 23, 2024

DavidHuber-NOAA commented Jan 23, 2024

ADCollard commented Jan 23, 2024

jswhit commented Jan 23, 2024

DavidHuber-NOAA commented Jan 24, 2024

ulmononian commented Jan 29, 2024

DavidHuber-NOAA commented Jan 31, 2024

RussTreadon-NOAA commented Jan 31, 2024

DavidHuber-NOAA commented Jan 31, 2024

DavidHuber-NOAA commented Jan 31, 2024

RussTreadon-NOAA commented Jan 31, 2024

DavidHuber-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 13, 2024

TingLei-NOAA commented Feb 13, 2024

DavidHuber-NOAA commented Feb 13, 2024

DavidHuber-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 13, 2024

RussTreadon-NOAA left a comment

RussTreadon-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 13, 2024

AlexanderRichert-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 14, 2024

AlexanderRichert-NOAA Feb 14, 2024

DavidHuber-NOAA Feb 14, 2024

TingLei-NOAA commented Feb 14, 2024 via email

RussTreadon-NOAA commented Feb 14, 2024 •

edited

Loading

TingLei-NOAA commented Feb 14, 2024 via email

AlexanderRichert-NOAA left a comment

TingLei-NOAA left a comment

RussTreadon-NOAA left a comment

Upgrade the GSI to Spack-Stack version 1.6.0 #684

Upgrade the GSI to Spack-Stack version 1.6.0 #684

Conversation

DavidHuber-NOAA commented Jan 18, 2024 • edited Loading

DavidHuber-NOAA commented Jan 19, 2024

DavidHuber-NOAA commented Jan 22, 2024

ADCollard commented Jan 23, 2024

RussTreadon-NOAA commented Jan 23, 2024

DavidHuber-NOAA commented Jan 23, 2024

ADCollard commented Jan 23, 2024

jswhit commented Jan 23, 2024

DavidHuber-NOAA commented Jan 24, 2024

ulmononian commented Jan 29, 2024

DavidHuber-NOAA commented Jan 31, 2024

RussTreadon-NOAA commented Jan 31, 2024

DavidHuber-NOAA commented Jan 31, 2024

DavidHuber-NOAA commented Jan 31, 2024

RussTreadon-NOAA commented Jan 31, 2024

DavidHuber-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 13, 2024

TingLei-NOAA commented Feb 13, 2024

DavidHuber-NOAA commented Feb 13, 2024

DavidHuber-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 13, 2024

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 13, 2024

AlexanderRichert-NOAA commented Feb 13, 2024

RussTreadon-NOAA commented Feb 14, 2024

AlexanderRichert-NOAA Feb 14, 2024

Choose a reason for hiding this comment

DavidHuber-NOAA Feb 14, 2024

Choose a reason for hiding this comment

TingLei-NOAA commented Feb 14, 2024 via email

RussTreadon-NOAA commented Feb 14, 2024 • edited Loading

TingLei-NOAA commented Feb 14, 2024 via email

AlexanderRichert-NOAA left a comment

Choose a reason for hiding this comment

TingLei-NOAA left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

DavidHuber-NOAA commented Jan 18, 2024 •

edited

Loading

RussTreadon-NOAA commented Feb 14, 2024 •

edited

Loading