Many TEQUILA failures during optimization run

[daveweisberg]

I've started testing TEQUILA as the primary equilibrium actor in my optimization runs, and I'm seeing a LOT of errors (>50% of total) that FUSE categorizes as "task_exception". Below is one example of a run that failed due to this error:

load_dir = "/fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/playground/weisberg_FPP_opt/nominal/opt_betaN_cost__TEQUILA_Kr_flattop48_HTS0.1_qpol2.75_ds0.9/opt_runs/2023-06-12T09:17:55.808__2824312"
ini = FUSE.json2ini(joinpath(load_dir, "ini.json"))
act = FUSE.json2act(joinpath(load_dir, "act.json"))
dd0 = IMAS.json2imas(joinpath(load_dir, "dd.json"));

dd = IMAS.dd();
FUSE.init(dd, ini, act; do_plot=false);

TaskFailedException

    nested task error: ArgumentError: Initial x[(5,)]=1.6734030381044294 is outside of [-1.5707963267948966, 1.5707963267948966]
    Stacktrace:
     [1] optimize(df::NLSolversBase.OnceDifferentiable{Float64, Vector{Float64}, Vector{Float64}}, l::Vector{Float64}, u::Vector{Float64}, initial_x::Vector{Float64}, F::Optim.Fminbox{Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.HagerZhang{Float64, Base.RefValue{Bool}}, Optim.var"#19#21"}, Float64, Optim.var"#49#51"}, options::Optim.Options{Float64, Nothing})
       @ Optim ~/.julia/packages/Optim/dBGGV/src/multivariate/solvers/constrained/fminbox.jl:310
     [2] #optimize#65
       @ ~/.julia/packages/Optim/dBGGV/src/multivariate/solvers/constrained/fminbox.jl:269 [inlined]
     [3] optimize_fit!(flat::Vector{Float64}, pr::SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, pz::SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}; inner_optimizer::Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.HagerZhang{Float64, Base.RefValue{Bool}}, Optim.var"#19#21"}, debug::Bool)
       @ MillerExtendedHarmonic ~/.julia/dev/MillerExtendedHarmonic/src/MXH.jl:483
     [4] optimize_fit!
       @ ~/.julia/dev/MillerExtendedHarmonic/src/MXH.jl:441 [inlined]
     [5] macro expansion
       @ ~/.julia/dev/TEQUILA/src/fit_MXH.jl:130 [inlined]
     [6] (::TEQUILA.var"#65#threadsfor_fun#118"{TEQUILA.var"#65#threadsfor_fun#114#119"{Matrix{Float64}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Tuple{Int64, Contour.ContourLevel{Tuple{Float64, Float64}, Float64}}}}})(tid::Int64; onethread::Bool)
       @ TEQUILA ./threadingconstructs.jl:84
     [7] #65#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [8] (::Base.Threads.var"#1#2"{TEQUILA.var"#65#threadsfor_fun#118"{TEQUILA.var"#65#threadsfor_fun#114#119"{Matrix{Float64}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Tuple{Int64, Contour.ContourLevel{Tuple{Float64, Float64}, Float64}}}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30

Stacktrace:
  [1] wait
    @ ./task.jl:345 [inlined]
  [2] threading_run(fun::TEQUILA.var"#65#threadsfor_fun#118"{TEQUILA.var"#65#threadsfor_fun#114#119"{Matrix{Float64}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Vector{Tuple{Int64, Contour.ContourLevel{Tuple{Float64, Float64}, Float64}}}}}, static::Bool)
    @ Base.Threads ./threadingconstructs.jl:38
  [3] macro expansion
    @ ./threadingconstructs.jl:89 [inlined]
  [4] refit(shot::TEQUILA.Shot{Int64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, FiniteElementHermite.FE_rep{Vector{Float64}, Vector{Float64}}, Nothing, Nothing, Nothing, FiniteElementHermite.FE_rep{Vector{Float64}, Vector{Float64}}, Float64, Float64, Float64, FiniteElementHermite.FE_rep{Vector{Float64}, Vector{Float64}}, Vector{FiniteElementHermite.FE_rep{Vector{Float64}, Vector{Float64}}}, Vector{PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}}, BandedMatrices.BandedLU{Float64, BandedMatrices.BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}}, lvls::Vector{Float64})
    @ TEQUILA ~/.julia/dev/TEQUILA/src/fit_MXH.jl:122
  [5] solve!(refill::TEQUILA.Shot{Int64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, FiniteElementHermite.FE_rep{Vector{Float64}, Vector{Float64}}, Nothing, Nothing, Nothing, FiniteElementHermite.FE_rep{Vector{Float64}, Vector{Float64}}, Float64, Float64, Float64, FiniteElementHermite.FE_rep{Vector{Float64}, Vector{Float64}}, Vector{FiniteElementHermite.FE_rep{Vector{Float64}, Vector{Float64}}}, Vector{PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}}, BandedMatrices.BandedLU{Float64, BandedMatrices.BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}}, its::Int64; debug::Bool)
    @ TEQUILA ~/.julia/dev/TEQUILA/src/solve.jl:35
  [6] #solve#170
    @ ~/.julia/dev/TEQUILA/src/solve.jl:5 [inlined]
  [7] _step(actor::FUSE.ActorTEQUILA{Float64, Float64})
    @ FUSE /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/equilibrium/tequila_actor.jl:83
  [8] macro expansion
    @ /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/abstract_actors.jl:41 [inlined]
  [9] macro expansion
    @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined]
 [10] step(::FUSE.ActorTEQUILA{Float64, Float64}; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FUSE /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/abstract_actors.jl:38
 [11] step
    @ /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/abstract_actors.jl:33 [inlined]
 [12] _step(actor::FUSE.ActorEquilibrium{Float64, Float64})
    @ FUSE /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/equilibrium/equilibrium_actor.jl:51
 [13] macro expansion
    @ /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/abstract_actors.jl:41 [inlined]
 [14] macro expansion
    @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined]
 [15] step(::FUSE.ActorEquilibrium{Float64, Float64}; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FUSE /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/abstract_actors.jl:38
 [16] step
    @ /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/abstract_actors.jl:33 [inlined]
 [17] #ActorEquilibrium#149
    @ /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/equilibrium/equilibrium_actor.jl:25 [inlined]
 [18] ActorEquilibrium
    @ /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/actors/equilibrium/equilibrium_actor.jl:23 [inlined]
 [19] macro expansion
    @ /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/ddinit/init_equilibrium.jl:141 [inlined]
 [20] macro expansion
    @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined]
 [21] init_equilibrium(dd::IMASDD.dd{Float64}, ini::FUSE.ParametersInits{Float64}, act::FUSE.ParametersActors{Float64})
    @ FUSE /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/ddinit/init_equilibrium.jl:11
 [22] macro expansion
    @ /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/ddinit/init.jl:21 [inlined]
 [23] macro expansion
    @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined]
 [24] init(dd::IMASDD.dd{Float64}, ini::FUSE.ParametersInits{Float64}, act::FUSE.ParametersActors{Float64}; do_plot::Bool)
    @ FUSE /fusion/ga/projects/ird/ptp/weisbergd/julia/dev/FUSE/src/ddinit/init.jl:12
 [25] top-level scope
    @ In[129]:2

[bclyons12]

Good to know. I've been working on an update that would bypass this (#36). Today or tomorrow I'll have a version of that you could try. Seems like this would be a good stress test for the new method

[orso82]

May be related to ProjectTorreyPines/FUSE.jl#348 (comment)

[bclyons12]

@daveweisberg @TimSlendebroek I've merged #36 which should be more robust, more accurate, and faster. Please give it a try on the problems that previously failed. If anything is still wrong, I'll get to debugging those.

[orso82]

@bclyons12 there are still some issues. You can get to them with:

ini, act = FUSE.case_parameters(:FPP; version=:v1_demount, init_from=:scalars, STEP=true);

act.ActorEquilibrium.model = :TEQUILA;

dd=IMAS.dd()
dd = FUSE.init(dd,ini, act);

detail:

[bclyons12]

@orso82 Is that running everything on the master branches?

[orso82]

yes, everything was on master

[bclyons12]

@orso82 TEQUILA was having some trouble fitting surfaces near the axis. I've fixed the problem in two ways.

1. It actually just needed more iterations to get through the initial difficulty. 5 was too few. This FUSE PR increases the number of iterations, but uses the new capability to terminate by tolerance to end early if desired: Allow user to terminate by TEQUILA by tolerance FUSE.jl#363, which makes use of 66e9007.
2. I improved the robustness of finding extrema for a surface with 459d27a, so this case converges more rapidly now.

FUSE.init is working for me now. There's a bit of a kink in the current profile. Perhaps it's a resolution issue, but more than sufficient for FUSE I imagine.