set!() using function fails in Julia 1.6 GPU

[mukund-gupta]

I am using Oceananigans v0.54.0 with Julia v1.6 on GPU. I tried setting an initial condition in salinity as follows:

Stop = 29.5; Sbot = 32
ztop = -35; zbot = -70
dSdz = (Sbot-Stop)/ztop
S_func(x, y, z) = Stop + dSdz*(z > ztop)*z 
set!(model, S=S_func)

and got the error copied below. Note that this works fine in Julia v1.5.4.

┌ Warning: You appear to be using MPI.jl with the default MPI binary on a cluster.
│ We recommend using the system-provided MPI, see the Configuration section of the MPI.jl docs.
└ @ MPI ~/.julia/packages/MPI/3q18R/deps/deps.jl:15
ERROR: LoadError: MethodError: no method matching set!(::Field{Center, Center, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, CuArray{Float64, 2}}}}}}, ::typeof(S_func))
Closest candidates are:
  set!(::Oceananigans.Fields.AbstractField, !Matched::Number) at /home/guptam/.julia/packages/Oceananigans/SPGnT/src/Fields/set!.jl:14
  set!(::Oceananigans.Fields.AbstractField{X, Y, Z, A, G} where G, !Matched::Oceananigans.Fields.AbstractField{X, Y, Z, A, G} where G) where {X, Y, Z, A} at /home/guptam/.julia/packages/Oceananigans/SPGnT/src/Fields/set!.jl:16
  set!(!Matched::Oceananigans.Fields.AbstractField{X, Y, Z, A, G} where {X, Y, Z, A<:(OffsetArrays.OffsetArray{T, D, var"#s203"} where {T, D, var"#s203"<:Array}), G}, ::Function) at /home/guptam/.julia/packages/Oceananigans/SPGnT/src/Fields/set!.jl:44
  ...
Stacktrace:
 [1] set!(model::IncompressibleModel{Oceananigans.TimeSteppers.RungeKutta3TimeStepper{Float64, NamedTuple{(:u, :v, :w, :T, :S), Tuple{Field{Face, Center, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Face, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Face, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}, Field{Center, Center, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}}, Tuple{AnisotropicDiffusivity{Float64, Float64, Float64, NamedTuple{(:T, :S), Tuple{Float64, Float64}}, NamedTuple{(:T, :S), Tuple{Float64, Float64}}, NamedTuple{(:T, :S), Tuple{Float64, Float64}}}, AnisotropicBiharmonicDiffusivity{Float64, NamedTuple{(:T, :S), Tuple{Float64, Float64}}, NamedTuple{(:T, :S), Tuple{Float64, Float64}}, NamedTuple{(:T, :S), Tuple{Float64, Float64}}}}, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, Buoyancy{SeawaterBuoyancy{Float64, SeawaterPolynomials.BoussinesqEquationOfState{TEOS10SeawaterPolynomial{Float64}, Int64}, Nothing, Nothing}, Oceananigans.BuoyancyModels.ZDirection}, FPlane{Float64}, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, CuArray{Float64, 2}}}}}}, Field{Center, Face, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, CuArray{Float64, 2}}}}}}, Field{Center, Center, Face, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}}}, NamedTuple{(:T, :S), Tuple{Field{Center, Center, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, CuArray{Float64, 2}}}}}}, Field{Center, Center, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, CuArray{Float64, 2}}}}}}}}, NamedTuple{(:pHY′, :pNHS), Tuple{Field{Center, Center, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, NamedTuple{(:u, :v, :w, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, CenteredFourthOrder, Oceananigans.Solvers.FFTBasedPoissonSolver{GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:λx, :λy, :λz), Tuple{CuArray{Float64, 3}, CuArray{Float64, 3}, CuArray{Float64, 3}}}, CuArray{ComplexF64, 3}, CuArray{ComplexF64, 3}, NamedTuple{(:forward, :backward), Tuple{Tuple{Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.cCuFFTPlan{ComplexF64, -1, true, 3}, Oceananigans.Solvers.Forward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Int64, Bounded, Int64, NamedTuple{(:forward, :backward), Tuple{CuArray{ComplexF64, 3}, CuArray{ComplexF64, 3}}}, Nothing}, Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.cCuFFTPlan{ComplexF64, -1, true, 3}, Oceananigans.Solvers.Forward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}}, Tuple{Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.cCuFFTPlan{ComplexF64, 1, true, 3}, Float64}, Oceananigans.Solvers.Backward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.cCuFFTPlan{ComplexF64, 1, true, 3}, Float64}, Oceananigans.Solvers.Backward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Int64, Bounded, Int64, NamedTuple{(:forward, :backward), Tuple{CuArray{ComplexF64, 3}, CuArray{ComplexF64, 3}}}, Nothing}}}}}, Tuple{Nothing, Nothing}, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}, NamedTuple{(:T, :S), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}}}, Nothing, Nothing}; enforce_incompressibility::Bool, kwargs::Base.Iterators.Pairs{Symbol, Function, Tuple{Symbol, Symbol}, NamedTuple{(:S, :T), Tuple{typeof(S_func), typeof(T_func)}}})
   @ Oceananigans.Models.IncompressibleModels ~/.julia/packages/Oceananigans/SPGnT/src/Models/IncompressibleModels/set_incompressible_model.jl:41
 [2] top-level scope
   @ /central~/ocean_floes/test_stratif/static_size64_1floe.jl:133
in expression starting at /central/home/guptam/ocean_floes/test_stratif/static_size64_1floe.jl:133

[ali-ramadhan]

Thanks for reporting this!

Actually @gianlisi encountered this issue yesterday but I couldn't reproduce locally. Maybe he was using Julia 1.6 and I was using Julia 1.5...

I would say especially for the GPU, Julia 1.5 might be more stable right now until we sort out any issues in PR #1514 (we should also look at this).

[glwagner]

@mukund-gupta it looks like your CUDA installation may not be working on 1.6. Can you report the output of CUDA.has_cuda()?

[ali-ramadhan]

Oh if this is an issue with our usage of @hascuda (e.g.

Oceananigans.jl/src/Fields/set!.jl

Line 50 in 5c4f493

@hascuda begin

) we should probably stop using it now that it's not needed anymore.

Yesterday might have also been caused by CUDA.has_cuda() == false on the machine.

[mukund-gupta]

I get: CUDA.has_cuda() = true on Julia v1.6

[glwagner]

Bah, fail. How about typing

julia> using Oceananigans.Fields: AbstractGPUField

For example on my laptop I get:

julia> using Oceananigans.Fields: AbstractGPUField
[ Info: Oceananigans will use 8 threads
ERROR: UndefVarError: AbstractGPUField not defined

whereas on a machine with a GPU I get no error.

As @ali-ramadhan noted it looks like the method for set! that's defined in a @hascuda block is not defined for your version of Oceananigans. (You may also need to rebuild Oceananigans after you verify that CUDA.has_cuda() returns true? I've never encountered this issue but some others may have.)

[navidcy]

Is this still ongoing issue?

[glwagner]

This appears to be stale, so I'm closing it.