Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

solve and concrete_solve crash when called through sciml_train #153

Closed
lindnemi opened this issue Feb 11, 2020 · 3 comments · Fixed by SciML/SciMLSensitivity.jl#194
Closed

Comments

@lindnemi
Copy link

lindnemi commented Feb 11, 2020

This is just some simple example code for the problem. When calling predict directly, x1 is an Array{Float64,2}, when called through sciml_train it is Float64 and i get an index error in the line below. I have no idea why this happens

function f!(du,u,p,t)
    du .= p.*u
end

N=10
p=1
globalx = zeros(N, N)

function predict(x,p)
    for i = 1:N
        globalx[:,i] = [x[i] * i for j in 1:N]
    end
    prob = ODEProblem(f!, globalx , (0.,1.), p)
    x1 = concrete_solve(prob, Tsit5(), globalx, p, save_everystep=false)[end]
    println(typeof(x1)) # direct call: Array{Float64,2}, call through sciml_train: Float64
    for i = 1:N
        x1[:,i] .*= globalx[:,i]
    end
    return x1
end

x = ones(N)


function loss_adjoint(p)
  prediction = predict(x,p)
  loss = sum(abs2, prediction)
  loss, prediction
end

loss_adjoint(1.)

res = DiffEqFlux.sciml_train(loss_adjoint, [1.], BFGS(initial_stepnorm=0.01))

when I use solve instead of concrete_solve by replacing the relevant line with

    x1 = solve(prob, Tsit5(), save_everystep=false)[end]

julia crashes altogether. This is the error message

julia: /buildworker/worker/package_linux64/build/src/codegen.cpp:4345: jl_cgval_t emit_expr(jl_codectx_t&, jl_value_t*, ssize_t): Assertion `token.V->getType()->isTokenTy()' failed.

signal (6): Aborted
in expression starting at /home/FOLDERNAMES/diffeqfluxbug.jl:34
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__assert_fail_base.cold at /lib64/libc.so.6 (unknown line)
__assert_fail at /lib64/libc.so.6 (unknown line)
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4345
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3839
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:4032 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6658
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1257
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3311
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4127
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3839
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:4032 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6658
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1257
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3311
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4127
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3839
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:4032 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6658
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1257
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3311
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4127
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3839
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:4032 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6658
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1257
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3311
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:4127
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3839
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:4032 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6658
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1257
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1889
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2153 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
#__solve#303 at /home/micha/.julia/packages/OrdinaryDiffEq/nV9bA/src/solve.jl:5 [inlined]
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface2.jl:0
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
adjoint at /home/micha/.julia/packages/Zygote/tJj2w/src/lib/lib.jl:164
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_pullback at /home/micha/.julia/packages/ZygoteRules/6nssF/src/adjoint.jl:47 [inlined]
__solve##kw at /home/micha/.julia/packages/OrdinaryDiffEq/nV9bA/src/solve.jl:4 [inlined]
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface2.jl:0
adjoint at /home/micha/.julia/packages/Zygote/tJj2w/src/lib/lib.jl:164 [inlined]
_pullback at /home/micha/.julia/packages/ZygoteRules/6nssF/src/adjoint.jl:47 [inlined]
#solve_call#440 at /home/micha/.julia/packages/DiffEqBase/avuk1/src/solve.jl:38 [inlined]
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface2.jl:0
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
adjoint at /home/micha/.julia/packages/Zygote/tJj2w/src/lib/lib.jl:164
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_pullback at /home/micha/.julia/packages/ZygoteRules/6nssF/src/adjoint.jl:47 [inlined]
solve_call##kw at /home/micha/.julia/packages/DiffEqBase/avuk1/src/solve.jl:37 [inlined]
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface2.jl:0
unknown function (ip: 0x7f73e37753ec)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
adjoint at /home/micha/.julia/packages/Zygote/tJj2w/src/lib/lib.jl:164
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_pullback at /home/micha/.julia/packages/ZygoteRules/6nssF/src/adjoint.jl:47 [inlined]
#solve#441 at /home/micha/.julia/packages/DiffEqBase/avuk1/src/solve.jl:57 [inlined]
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface2.jl:0
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
adjoint at /home/micha/.julia/packages/Zygote/tJj2w/src/lib/lib.jl:164
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_pullback at /home/micha/.julia/packages/ZygoteRules/6nssF/src/adjoint.jl:47 [inlined]
solve##kw at /home/micha/.julia/packages/DiffEqBase/avuk1/src/solve.jl:45 [inlined]
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface2.jl:0
unknown function (ip: 0x7f73e377292c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
predict at /home/micha/git/graphneuralode/diffeqfluxbug.jl:15 [inlined]
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface2.jl:0
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
loss_adjoint at /home/micha/git/graphneuralode/diffeqfluxbug.jl:27 [inlined]
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface2.jl:0
optim_loss at /home/micha/.julia/packages/DiffEqFlux/4NwUK/src/train.jl:80 [inlined]
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface2.jl:0
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface.jl:31
pullback at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface.jl:37
gradient at /home/micha/.julia/packages/Zygote/tJj2w/src/compiler/interface.jl:46
optim_loss_gradient! at /home/micha/.julia/packages/DiffEqFlux/4NwUK/src/train.jl:85
fg! at /home/micha/.julia/packages/NLSolversBase/mGaJg/src/objective_types/abstract.jl:13
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
value_gradient!! at /home/micha/.julia/packages/NLSolversBase/mGaJg/src/interface.jl:82
initial_state at /home/micha/.julia/packages/Optim/SFpsz/src/multivariate/solvers/first_order/bfgs.jl:66
optimize at /home/micha/.julia/packages/Optim/SFpsz/src/multivariate/optimize/optimize.jl:33 [inlined]
#optimize#94 at /home/micha/.julia/packages/Optim/SFpsz/src/multivariate/optimize/interface.jl:123
optimize at /home/micha/.julia/packages/Optim/SFpsz/src/multivariate/optimize/interface.jl:121
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
#sciml_train#28 at /home/micha/.julia/packages/DiffEqFlux/4NwUK/src/train.jl:88
unknown function (ip: 0x7f73e3a8bd02)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
sciml_train at /home/micha/.julia/packages/DiffEqFlux/4NwUK/src/train.jl:70
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:369
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:458
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:409 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:817
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:911
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:814
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:872
include_string at /home/micha/bin/julia-1.4.0-rc1/lib/julia/sys.so (unknown line)
include_string at /home/micha/.julia/packages/CodeTools/sf1Tz/src/eval.jl:30
unknown function (ip: 0x7f73d8656ef2)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
#127 at /home/micha/.julia/packages/Atom/lBERI/src/eval.jl:94
withpath at /home/micha/.julia/packages/CodeTools/sf1Tz/src/utils.jl:30
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
withpath at /home/micha/.julia/packages/Atom/lBERI/src/eval.jl:47
#126 at /home/micha/.julia/packages/Atom/lBERI/src/eval.jl:93 [inlined]
with_logstate at ./logging.jl:396
with_logger at ./logging.jl:503 [inlined]
#125 at /home/micha/.julia/packages/Atom/lBERI/src/eval.jl:92 [inlined]
hideprompt at /home/micha/.julia/packages/Atom/lBERI/src/repl.jl:85
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
macro expansion at /home/micha/.julia/packages/Atom/lBERI/src/eval.jl:91 [inlined]
macro expansion at /home/micha/.julia/packages/Media/ItEPc/src/dynamic.jl:24 [inlined]
#124 at /home/micha/.julia/packages/Atom/lBERI/src/eval.jl:86
#94 at /home/micha/.julia/packages/Revise/S7mrl/src/Revise.jl:1205
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
handlemsg at /home/micha/.julia/packages/Atom/lBERI/src/comm.jl:164
unknown function (ip: 0x7f741926a07b)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:643
#19 at ./task.jl:358
unknown function (ip: 0x7f7419260484)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
Allocations: 756743140 (Pool: 756607598; Big: 135542); GC: 438

I am on julia 1.4 but the problem has been observed on 1.3 as well.

@lindnemi
Copy link
Author

I managed to clean up the example

using DiffEqFlux
using OrdinaryDiffEq
using Flux
using Optim

function f!(du,u,p,t)
    du .= p.*u
end

N       = 10
p       = 1
globalx = zeros(N)
x       = ones(N)
prob    = ODEProblem(f!, globalx , (0.,1.), p)

function predict(x,p)    
    x1 = concrete_solve(prob, Tsit5(), x .* collect(1:N), p, save_everystep=false)[end]
    println(typeof(x1)) # direct call: Array{Float64,1}, call through sciml_train: Float64
    println(x1[:])    
    return x1
end

function loss_adjoint(p)
  prediction = predict(x,p)
  loss = sum(abs2, prediction)
  loss, prediction
end

loss_adjoint(-30.)

res = DiffEqFlux.sciml_train(loss_adjoint, [1.], BFGS(initial_stepnorm=0.01))

Zygote seems to handle the output of concrete_solve differently. A workaround is to cast the concrete_solve solution to an Array, by replacing the corresponding line with

    x1 = Array(concrete_solve(prob, Tsit5(), x .* collect(1:N), p, save_everystep=false))[:,end]

Is this expected/desired behaviour? Is it documented somewhere why this happens?

ChrisRackauckas added a commit to SciML/SciMLSensitivity.jl that referenced this issue Feb 13, 2020
The adjoints should all commit to returning the same DiffEqArray as the forward pass, otherwise the semantics get confusing and different during the gradient passes. Fixes SciML/DiffEqFlux.jl#153
@ChrisRackauckas
Copy link
Member

Undesirable and confusing! Fixed here: SciML/SciMLSensitivity.jl#194

@ChrisRackauckas
Copy link
Member

Thanks for the report and sorry for the late response.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants