Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Picking up a simulation fails with VerticallyStretchedGrids #2018

Closed
tomchor opened this issue Oct 17, 2021 · 8 comments · Fixed by #2019 or #2028
Closed

Picking up a simulation fails with VerticallyStretchedGrids #2018

tomchor opened this issue Oct 17, 2021 · 8 comments · Fixed by #2019 or #2028
Labels
bug 🐞 Even a perfect program still has bugs grids 🗺️

Comments

@tomchor
Copy link
Collaborator

tomchor commented Oct 17, 2021

I'm finding that simulations with VerticallyStretchedGrids don't get picked up and instead throw an error saying that the grid doesn't match. Here's a MWE:

using Oceananigans

grid = VerticallyStretchedRectilinearGrid(size=(2, 2, 2),
                                              x=(0, 1), y=(0, 1), z_faces=k -> k,
                                              halo=(3,3,3),
                                              )

model = NonhydrostaticModel(grid=grid)

progress(sim) = @info "Iteration: $(sim.model.clock.iteration), time: $(round(Int, sim.model.clock.time))"
simulation = Simulation(model, Δt=1, stop_time=10, progress=progress)
simulation.output_writers[:chk_writer] = Checkpointer(model;
                                         dir=".",
                                         prefix = "mwe",
                                         schedule = TimeInterval(2),
                                         force = false,
                                         cleanup = true,
                                         )

run!(simulation)

simulation.stop_time = 20

run!(simulation, pickup=true)

This fails with:

[ Info: Updating model auxiliary state before the first time step...
[ Info:     ... updated in 400.714 μs.
[ Info: Executing first time step...
[ Info: Iteration: 1, time: 1
[ Info: Iteration: 2, time: 2
[ Info: Iteration: 3, time: 3
[ Info: Iteration: 4, time: 4
[ Info: Iteration: 5, time: 5
[ Info: Iteration: 6, time: 6
[ Info: Iteration: 7, time: 7
[ Info: Iteration: 8, time: 8
[ Info: Iteration: 9, time: 9
[ Info: Iteration: 10, time: 10
[ Info: Simulation is stopping. Model time 10 seconds has hit or exceeded simulation stop time 10 seconds.
ERROR: LoadError: The grid associated with ./mwe_iteration10.jld2 and model.grid are not the same!
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] (::Oceananigans.OutputWriters.var"#52#53"{NonhydrostaticModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float64, NamedTuple{(:u, :v, :w, :T, :S), Tuple{Field{Face, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Face, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Face, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, Nothing}, Nothing, CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, Buoyancy{SeawaterBuoyancy{Float64, LinearEquationOfState{Float64}, Nothing, Nothing}, Oceananigans.Grids.ZDirection}, Nothing, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Face, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Face, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, NamedTuple{(:T, :S), Tuple{Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, NamedTuple{(:pHY′, :pNHS), Tuple{Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, NamedTuple{(:u, :v, :w, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, CenteredSecondOrder, Oceananigans.Solvers.FourierTridiagonalPoissonSolver{CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Oceananigans.Solvers.BatchedTridiagonalSolver{Vector{Float64}, Array{Float64, 3}, Vector{Float64}, Array{Float64, 3}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, CPU, Nothing}, Array{ComplexF64, 3}, Array{ComplexF64, 3}, Nothing, NamedTuple{(:forward, :backward), Tuple{Tuple{Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Vector{Int64}}, Oceananigans.Solvers.Forward, CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}}, Tuple{Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Vector{Int64}}, Float64}, Oceananigans.Solvers.Backward, CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}, Nothing, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}, NamedTuple{(:T, :S), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}}}, Nothing, Nothing, NamedTuple{(), Tuple{}}}, String})(file::JLD2.JLDFile{JLD2.MmapIO})
   @ Oceananigans.OutputWriters /glade/work/tomasc/.julia_bkp/packages/Oceananigans/52CTk/src/OutputWriters/checkpointer.jl:197
 [3] jldopen(::Oceananigans.OutputWriters.var"#52#53"{NonhydrostaticModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float64, NamedTuple{(:u, :v, :w, :T, :S), Tuple{Field{Face, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Face, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Face, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, Nothing}, Nothing, CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, Buoyancy{SeawaterBuoyancy{Float64, LinearEquationOfState{Float64}, Nothing, Nothing}, Oceananigans.Grids.ZDirection}, Nothing, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Face, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Face, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, NamedTuple{(:T, :S), Tuple{Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, NamedTuple{(:pHY′, :pNHS), Tuple{Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, NamedTuple{(:u, :v, :w, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, CenteredSecondOrder, Oceananigans.Solvers.FourierTridiagonalPoissonSolver{CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Oceananigans.Solvers.BatchedTridiagonalSolver{Vector{Float64}, Array{Float64, 3}, Vector{Float64}, Array{Float64, 3}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, CPU, Nothing}, Array{ComplexF64, 3}, Array{ComplexF64, 3}, Nothing, NamedTuple{(:forward, :backward), Tuple{Tuple{Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Vector{Int64}}, Oceananigans.Solvers.Forward, CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}}, Tuple{Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Vector{Int64}}, Float64}, Oceananigans.Solvers.Backward, CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}, Nothing, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}, NamedTuple{(:T, :S), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}}}, Nothing, Nothing, NamedTuple{(), Tuple{}}}, String}, ::String, ::Vararg{String, N} where N; kws::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ JLD2 /glade/work/tomasc/.julia_bkp/packages/JLD2/DcnTD/src/loadsave.jl:4
 [4] jldopen
   @ /glade/work/tomasc/.julia_bkp/packages/JLD2/DcnTD/src/loadsave.jl:2 [inlined]
 [5] set!
   @ /glade/work/tomasc/.julia_bkp/packages/Oceananigans/52CTk/src/OutputWriters/checkpointer.jl:193 [inlined]
 [6] run!(sim::Simulation{NonhydrostaticModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float64, NamedTuple{(:u, :v, :w, :T, :S), Tuple{Field{Face, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Face, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Face, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, Nothing}, Nothing, CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, Buoyancy{SeawaterBuoyancy{Float64, LinearEquationOfState{Float64}, Nothing, Nothing}, Oceananigans.Grids.ZDirection}, Nothing, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Face, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Face, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, NamedTuple{(:T, :S), Tuple{Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, NamedTuple{(:pHY′, :pNHS), Tuple{Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Field{Center, Center, Center, CPU, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}}, NamedTuple{(:u, :v, :w, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, CenteredSecondOrder, Oceananigans.Solvers.FourierTridiagonalPoissonSolver{CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Oceananigans.Solvers.BatchedTridiagonalSolver{Vector{Float64}, Array{Float64, 3}, Vector{Float64}, Array{Float64, 3}, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, CPU, Nothing}, Array{ComplexF64, 3}, Array{ComplexF64, 3}, Nothing, NamedTuple{(:forward, :backward), Tuple{Tuple{Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Vector{Int64}}, Oceananigans.Solvers.Forward, CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}}, Tuple{Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Vector{Int64}}, Float64}, Oceananigans.Solvers.Backward, CPU, VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}, Nothing, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}, NamedTuple{(:T, :S), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}}}, Nothing, Nothing, NamedTuple{(), Tuple{}}}, Int64, Vector{Any}, Float64, Int64, Float64, Float64, typeof(progress), Int64, Nothing}; pickup::Bool, callbacks::Vector{Any})
   @ Oceananigans.Simulations /glade/work/tomasc/.julia_bkp/packages/Oceananigans/52CTk/src/Simulations/run.jl:140
 [7] top-level scope
   @ /glade/scratch/tomasc/tilted_bbl4/mwe.jl:25
 [8] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [9] top-level scope
   @ REPL[11]:1
in expression starting at /glade/scratch/tomasc/tilted_bbl4/mwe.jl:25

The same code works if grid is regular rectilinear.

Not very familiar with jld2, but I'm guessing this has to do with the fact that VerticallyStretchedGrid has an array associated with it that has to be serialized?

@tomchor tomchor added the bug 🐞 Even a perfect program still has bugs label Oct 17, 2021
@tomchor
Copy link
Collaborator Author

tomchor commented Oct 17, 2021

Apparently "identical" VerticallyStretchedGrids aren't actually equal to each other for some reason:

julia> grid1 = VerticallyStretchedRectilinearGrid(size=(2, 2, 2),
                                                  x=(0, 1), y=(0, 1), z_faces=k -> k,
                                                  halo=(3,3,3),
                                                  )
VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded}
                   domain: x  [0.0, 1.0], y  [0.0, 1.0], z  [1.0, 3.0]
                 topology: (Periodic, Periodic, Bounded)
        size (Nx, Ny, Nz): (2, 2, 2)
        halo (Hx, Hy, Hz): (3, 3, 3)
grid spacing (Δx, Δy, Δz): (0.5, 0.5, [min=1.0, max=1.0])

julia> grid2 = VerticallyStretchedRectilinearGrid(size=(2, 2, 2),
                                                  x=(0, 1), y=(0, 1), z_faces=k -> k,
                                                  halo=(3,3,3),
                                                  )
VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded}
                   domain: x  [0.0, 1.0], y  [0.0, 1.0], z  [1.0, 3.0]
                 topology: (Periodic, Periodic, Bounded)
        size (Nx, Ny, Nz): (2, 2, 2)
        halo (Hx, Hy, Hz): (3, 3, 3)
grid spacing (Δx, Δy, Δz): (0.5, 0.5, [min=1.0, max=1.0])

julia> grid1 == grid2
false

I don't know why this happens. In fact even using deepcopy doesn't seem to work:

julia> grid1 = VerticallyStretchedRectilinearGrid(size=(2, 2, 2),
                                                  x=(0, 1), y=(0, 1), z_faces=k -> k,
                                                  halo=(3,3,3),
                                                  )
VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded}
                   domain: x  [0.0, 1.0], y  [0.0, 1.0], z  [1.0, 3.0]
                 topology: (Periodic, Periodic, Bounded)
        size (Nx, Ny, Nz): (2, 2, 2)
        halo (Hx, Hy, Hz): (3, 3, 3)
grid spacing (Δx, Δy, Δz): (0.5, 0.5, [min=1.0, max=1.0])

julia> grid2 = deepcopy(grid1)
VerticallyStretchedRectilinearGrid{Float64, Periodic, Periodic, Bounded}
                   domain: x  [0.0, 1.0], y  [0.0, 1.0], z  [1.0, 3.0]
                 topology: (Periodic, Periodic, Bounded)
        size (Nx, Ny, Nz): (2, 2, 2)
        halo (Hx, Hy, Hz): (3, 3, 3)
grid spacing (Δx, Δy, Δz): (0.5, 0.5, [min=1.0, max=1.0])

julia> grid1 == grid2
false

@tomchor
Copy link
Collaborator Author

tomchor commented Oct 17, 2021

This apparently has to do with the fact that

julia> grid1.zᵃᵃᶜ ===  grid2.zᵃᵃᶜ
false

julia> grid1.zᵃᵃᶠ ===  grid2.zᵃᵃᶠ
false

I don't think this is a due to the grid design as this appears to be standard Julia behavior for arrays:

julia> zeros(3) == zeros(3)
true

julia> zeros(3) === zeros(3)
false

which seems pretty counter-intuitive to me and apparently there's some discussion about this behavior in julia already: JuliaLang/julia#4648

@glwagner
Copy link
Member

This apparently has to do with the fact that

julia> grid1.zᵃᵃᶜ ===  grid2.zᵃᵃᶜ
false

julia> grid1.zᵃᵃᶠ ===  grid2.zᵃᵃᶠ
false

I don't think this is a due to the grid design as this appears to be standard Julia behavior for arrays:

julia> zeros(3) == zeros(3)
true

julia> zeros(3) === zeros(3)
false

which seems pretty counter-intuitive to me and apparently there's some discussion about this behavior in julia already: JuliaLang/julia#4648

=== means that "no program can distinguish" between the objects. For arrays this basically means they point to the same space in memory; changes to one imply changes in another. == is a weaker statement, usually about numerical equality. What's counter intuitive?

@glwagner
Copy link
Member

I think there are probably a few other things to fix here, because we can't "restore" a grid with a GPU architecture.

Fixing this fully really requires finishing #1998 first, and overhauling the checkpointing infrastructure to match so that the checkpointer is "architecture aware". We should also add tests for checkpointing with other grid types.

@tomchor
Copy link
Collaborator Author

tomchor commented Oct 18, 2021

=== means that "no program can distinguish" between the objects. For arrays this basically means they point to the same space in memory; changes to one imply changes in another. == is a weaker statement, usually about numerical equality. What's counter intuitive?

Oh yeah I know about ===. What I mean is that you test two grids with ==, but Julia then tests each property with ===. So the user is actually performing a much stricter operation than what it seems at first sight. That's the un-intuitive part to me. Although I may be missing something.

@glwagner
Copy link
Member

That's because == falls back to ===, right?

@tomchor
Copy link
Collaborator Author

tomchor commented Oct 18, 2021

That's because == falls back to ===, right?

No idea! Like I said, it's not very intuitive behavior to me. You probably understand this way better.

@glwagner
Copy link
Member

Base has

https://github.com/JuliaLang/julia/blob/806b12880cc8c89c751c19e7b5a89f0b45df42e5/base/Base.jl#L115

or just ==(x, y) = x === y.

So unless there's a special definition for == we fallback to ===. This is what the docstring says:

==(x, y)

Generic equality operator. Falls back to ===. Should be implemented for all types with a notion of equality,
based on the abstract value that an instance represents. For example, all numeric types are compared by
numeric value, ignoring type. Strings are compared as sequences of characters, ignoring encoding. For
collections, == is generally called recursively on all contents, though other properties (like the shape for
arrays) may also be taken into account.

So basically for any type at all, we'll get ==(x, y) = x === y unless we've invested the effort to define a particular == for our type.

Also based on the above I think strictly speaking a vertically-stretched and rectilinear grid should be identical if the vertically stretched grid is regularly spaced. This would be a pain to implement though --- its really a problem with our grids. We should only have one RectilinearGrid :-/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐞 Even a perfect program still has bugs grids 🗺️
Projects
None yet
3 participants