Snow quantities change a lot from restart values in the first physics step

[AnnaKwa]

It looks like the data in the restarts for snow cover and depth aren't being used to initialize the model. To test this I overwrote the surface restart data quantities sncovr, sheleg, snwdph as zeroes in the restart files and did a short test run.

[AnnaKwa]

Elynn suggested checking that the surface restarts were indeed being read- confirmed that they are read. Also, the subroutines that can calculate sncovr in the absence of a restart value are not being used.

[AnnaKwa]

Edit: the previous statement was incorrect- restarts are read in correctly, and zeroed out restarts stay as zeroes in the state for at least a couple timestepts. The issue that motivated testing the this is still unresolved: why do the snow-related restart quantities saved out in the first couple hours look so different from the restarts? Below is the diff of the sncovr, sheleg, and sndph between the first coarse timestep saved (30 min after init) and the restart.

Somewhat notably, the coarse run snow coverage is always adjusted upwards from the restart values

Albedo (as inferred by saved downward and upward SW sfc flux) is significantly different between the start of the coarse run and the corresponding time range of the restarts (coarsened fine res), not just in areas of differing snow coverage but across snow covered regions.

[AnnaKwa]

Verified that when initializing using a saved restart from the coarse run, the above quantities do not change significantly. i.e. The updates to the snow cover seem to be due to the model wanting to adjust it towards a particular state.

Left: the difference between first timestep sncovr, sheleg, swndph when initializing from a coarse model restart. Right: same when initializing from coarsened PIRE restart.

[AnnaKwa]

Also, verified that this happens in the physics step (not in the reading in of restarts or initialization). I did a test with the runtime physics disabled (just commented out the physics compute and apply calls) and the quantities saved out from the state agree with the restarts.

[AnnaKwa]

The large adjustments to surface properties at runtime seems to be an issue so far for coarsened fine res restarts. Above figures were using coarsened PIRE data but this also seems to occur for coarsened C384 -> C48 restarts initialized in N. winter. This does not at the start of coarse runs initialized off GFS analysis data from 2016. (https://github.com/ai2cm/explore/blob/master/annak/2021-12-10-PIRE-restart-sfc-changes/2021-12-10-do-other-restarts-have-issues.ipynb)

[AnnaKwa]

If the coarse model run is initialized during northern summer and run into winter, the resulting snow cover over northern land during winter is much more similar to the coarsened fine res reference. Albedo over snow covered regions is still quite different though.

https://github.com/ai2cm/explore/blob/master/annak/2021-10-05-PIRE-rad-investigation/2021-12-08-does-coarse-run-accumulate-too-much-snow-in-fall.ipynb

top: coarse run snow cover in Dec (for run started in Aug from PIRE restart)
middle: coarsened fine res snow cover in corresponding time
bottom: diff

Albedo difference in Dec: