Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WaterLily using ForwardDiff and Enzyme through DifferentiationInterface #355

Closed
b-fg opened this issue Jul 15, 2024 · 19 comments · Fixed by #357
Closed

WaterLily using ForwardDiff and Enzyme through DifferentiationInterface #355

b-fg opened this issue Jul 15, 2024 · 19 comments · Fixed by #357

Comments

@b-fg
Copy link

b-fg commented Jul 15, 2024

Hey! I have been playing with this package to implement AD for WaterLily.jl. The ForwardDiff backend works as expected, and it provides the same result as using ForwardDiff without DI, only if compilation is performed through the appropriate Dual type. For example, the following MWE works

using WaterLily
using StaticArrays
using DifferentiationInterface
import ForwardDiff

function make_sim(θ;L=32,U=1,Re=100,mem=Array)
    function map(x,_)
        s,c = sincos(θ)
        SA[c -s; s c]*(x-SA[L,L])
    end
    function sdf(ξ,_)
        p = ξ-SA[0,clamp(ξ[1],-L/2,L/2)]
        √(p'*p)-2
    end
    Simulation((2L,2L),(U,0),L,ν=U*L/Re,body=AutoBody(sdf,map),T=typeof(θ),mem=mem)
end
function dsim(θ)
    sim = make_sim(θ)
    sim_step!(sim)
    sum(sim.flow.p)
end

function main()
    FT = Float64
    θ = FT(π/36)

    # FD
    θ1 = θ*0.999
    θ2 = θ*1.001
    println("FD value and gradient\n",
        dsim(θ),",",(dsim(θ2)-dsim(θ1))/(θ2-θ1)
    )

    # ForwardDiff
    T = typeof(ForwardDiff.Tag(dsim,FT))
    θAD = ForwardDiff.Dual{T}(θ,one(FT))
    println("\nForwardDiff value and gradient\n", dsim(θAD))

    # ForwardDiff via DifferentiationInterface
    println("\nDifferentiationInterface ForwardDiff value and gradient\n",
        value_and_gradient(dsim, AutoForwardDiff(), θ)
    )
end

main()

but commenting out the ForwardDiff lines

  # T = typeof(ForwardDiff.Tag(dsim,FT))
  # θAD = ForwardDiff.Dual{T}(θ,one(FT))
  # println("\nForwardDiff value and gradient\n", dsim(θAD))

makes the DI call to error because of the ordering of Dual tags.

Furthermore, switching to Enzyme with

value_and_gradient(dsim, AutoEnzyme(), θ)

throws an error (with a rather large stacktrace...). I am not sure how to specify forward or reverse mode when using the AutoEnzyme() backend, so it could be that by default tries to do backward mode and we have some incompatibility with that. Any help would be appreciated, thanks! :)
@gdalle
Copy link
Member

gdalle commented Jul 15, 2024 •
edited
Loading

I can't reproduce the ForwardDiff bug, are you perhaps using an older version of DI? Proper tag handling wasn't always present but it should work in the latest version. The two versions below have the exact same output for me, while keeping the lines you mention commented out. Incidentally, you want derivative and not gradient here, because your input is scalar.

# ForwardDiff
dsim(θ), ForwardDiff.derivative(dsim, θ)

# ForwardDiff via DI
value_and_derivative(dsim, AutoForwardDiff(), θ)

@b-fg
Copy link
Author

b-fg commented Jul 15, 2024

Currently using v0.5.7

(WaterLily.jl-Examples) pkg> st DifferentiationInterface
Status `~/WaterLily.jl-Examples/Project.toml`
  [a0c0ee7d] DifferentiationInterface v0.5.7

I think that's the latest, right? So not sure what's going on.
Also, thanks for that remark. Could you please clarify the practical difference in this case?

@gdalle
Copy link
Member

gdalle commented Jul 15, 2024 •
edited
Loading

As for the Enzyme issue, I don't think it is related to DI: I get the error too when I call Enzyme directly. Presumably something in the WaterLily Simulation object makes Enzyme mad.

julia> Enzyme.autodiff(Enzyme.Forward, dsim, Enzyme.Duplicated, Enzyme.Duplicated(θ, Enzyme.make_zero(θ)))
ERROR: Enzyme execution failed.
Enzyme cannot deduce type
Current scope: 
; Function Attrs: mustprogress willreturn
define "enzyme_type"="{[0]:Pointer, [8]:Pointer, [16]:Pointer, [32]:Pointer}" "enzymejl_parmtype"="138889797305424" "enzymejl_parmtype_ref"="1" [6 x {} addrspace(10)*] @preprocess_julia__make_sim_10_8465_inner.1(i64 signext "enzyme_inactive" "enzyme_type"="{[-1]:Integer}" "enzymejl_parmtype"="138891479608832" "enzymejl_parmtype_ref"="0" %0, i64 signext "enzyme_inactive" "enzyme_type"="{[-1]:Integer}" "enzymejl_parmtype"="138891479608832" "enzymejl_parmtype_ref"="0" %1, i64 signext "enzyme_inactive" "enzyme_type"="{[-1]:Integer}" "enzymejl_parmtype"="138891479608832" "enzymejl_parmtype_ref"="0" %2, {} addrspace(10)* nofree noundef nonnull readonly "enzyme_type"="{[-1]:Pointer}" "enzymejl_parmtype"="138891414058144" "enzymejl_parmtype_ref"="2" %3, double "enzyme_type"="{[-1]:Float@double}" "enzymejl_parmtype"="138891479608032" "enzymejl_parmtype_ref"="0" %4) local_unnamed_addr #11 !dbg !57 {
entry:
  %newstruct7.i.sroa.5 = alloca [7 x i8], align 1
  %newstruct7.i.sroa.5.0.sroa_idx27 = getelementptr inbounds [7 x i8], [7 x i8]* %newstruct7.i.sroa.5, i64 0, i64 0
  %newstruct7.i.sroa.5.0.sroa_idx2 = getelementptr inbounds [7 x i8], [7 x i8]* %newstruct7.i.sroa.5, i64 0, i64 0
  call void @llvm.lifetime.start.p0i8(i64 7, i8* %newstruct7.i.sroa.5.0.sroa_idx2)
  %5 = call {}*** @julia.get_pgcstack() #12, !noalias !58
  %current_task1.i49 = getelementptr inbounds {}**, {}*** %5, i64 -14
  %current_task1.i = bitcast {}*** %current_task1.i49 to {}**
  %ptls_field.i50 = getelementptr inbounds {}**, {}*** %5, i64 2
  %6 = bitcast {}*** %ptls_field.i50 to i64***
  %ptls_load.i5152 = load i64**, i64*** %6, align 8, !tbaa !11, !noalias !58
  %7 = getelementptr inbounds i64*, i64** %ptls_load.i5152, i64 2
  %safepoint.i = load i64*, i64** %7, align 8, !tbaa !15, !noalias !58
  fence syncscope("singlethread") seq_cst
  call void @julia.safepoint(i64* %safepoint.i) #12, !dbg !61, !noalias !58
  fence syncscope("singlethread") seq_cst
  %8 = shl i64 %0, 1, !dbg !63
  %9 = mul i64 %1, %0, !dbg !63
  %10 = sitofp i64 %9 to double, !dbg !65
  %11 = sitofp i64 %2 to double, !dbg !65
  %12 = fdiv double %10, %11, !dbg !69
  %box.i = call noalias nonnull dereferenceable(8) "enzyme_type"="{[-1]:Pointer, [-1,-1]:Float@double}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 8, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891479608032 to {}*) to {} addrspace(10)*)) #13, !dbg !70
  %13 = bitcast {} addrspace(10)* %box.i to double addrspace(10)*, !dbg !70
  store double %12, double addrspace(10)* %13, align 8, !dbg !70, !tbaa !35, !alias.scope !39, !noalias !71
  %box10.i = call noalias nonnull dereferenceable(48) "enzyme_type"="{[-1]:Pointer, [-1,0]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Float@double}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 48, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891597143504 to {}*) to {} addrspace(10)*)) #13, !dbg !70
  %14 = bitcast {} addrspace(10)* %box10.i to i8 addrspace(10)*, !dbg !70
  store i8 1, i8 addrspace(10)* %14, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.5.0..sroa_raw_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 1, !dbg !70
  call void @llvm.memcpy.p10i8.p0i8.i64(i8 addrspace(10)* nocapture nofree noundef writeonly align 1 dereferenceable(7) %newstruct7.i.sroa.5.0..sroa_raw_idx, i8* noundef nonnull readonly align 1 dereferenceable(7) %newstruct7.i.sroa.5.0.sroa_idx27, i64 noundef 7, i1 noundef false) #12, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.58.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 8, !dbg !70
  %newstruct7.i.sroa.58.0..sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.58.0..sroa_idx to i64 addrspace(10)*, !dbg !70
  store i64 %0, i64 addrspace(10)* %newstruct7.i.sroa.58.0..sroa_cast, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 16, !dbg !70
  %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_idx to i64 addrspace(10)*, !dbg !70
  store i64 %0, i64 addrspace(10)* %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_cast, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 24, !dbg !70
  %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_idx to double addrspace(10)*, !dbg !70
  store double %4, double addrspace(10)* %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_cast, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 32, !dbg !70
  %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_idx to i64 addrspace(10)*, !dbg !70
  store i64 %0, i64 addrspace(10)* %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_cast, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 40, !dbg !70
  %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_idx to double addrspace(10)*, !dbg !70
  store double %4, double addrspace(10)* %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_cast, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %15 = call noalias nonnull "enzyme_type"="{[-1]:Pointer, [-1,-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @jl_f_tuple, {} addrspace(10)* noundef null, {} addrspace(10)* nofree nonnull %box.i, {} addrspace(10)* nofree nonnull %box10.i, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891479608032 to {}*) to {} addrspace(10)*), {} addrspace(10)* nofree nonnull %3) #14, !dbg !70
  %typeof.i = call "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* @julia.typeof({} addrspace(10)* nonnull %15) #15, !dbg !75
  %16 = call nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @jl_f_apply_type, {} addrspace(10)* noundef null, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891479609504 to {}*) to {} addrspace(10)*), {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891567787088 to {}*) to {} addrspace(10)*), {} addrspace(10)* nonnull %typeof.i) #16, !dbg !75
  %box12.i = call noalias nonnull dereferenceable(8) "enzyme_type"="{[-1]:Pointer, [-1,-1]:Float@double}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 8, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891479608032 to {}*) to {} addrspace(10)*)) #13, !dbg !75
  %17 = bitcast {} addrspace(10)* %box12.i to double addrspace(10)*, !dbg !75
  store double %12, double addrspace(10)* %17, align 8, !dbg !75, !tbaa !35, !alias.scope !39, !noalias !71
  %box14.i = call noalias nonnull dereferenceable(48) "enzyme_type"="{[-1]:Pointer, [-1,0]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Float@double}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 48, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891597143504 to {}*) to {} addrspace(10)*)) #13, !dbg !75
  %18 = bitcast {} addrspace(10)* %box14.i to i8 addrspace(10)*, !dbg !75
  store i8 1, i8 addrspace(10)* %18, align 8, !dbg !75, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.5.0..sroa_raw_idx4 = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 1, !dbg !75
  call void @llvm.memcpy.p10i8.p0i8.i64(i8 addrspace(10)* nocapture nofree noundef align 1 dereferenceable(7) %newstruct7.i.sroa.5.0..sroa_raw_idx4, i8* noundef nonnull align 1 dereferenceable(7) %newstruct7.i.sroa.5.0.sroa_idx27, i64 noundef 7, i1 noundef false) #12, !dbg !75, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.58.0..sroa_idx9 = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 8, !dbg !75
  %newstruct7.i.sroa.58.0..sroa_cast10 = bitcast i8 addrspace(10)* %newstruct7.i.sroa.58.0..sroa_idx9 to i64 addrspace(10)*, !dbg !75
  store i64 %0, i64 addrspace(10)* %newstruct7.i.sroa.58.0..sroa_cast10, align 8, !dbg !75, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 16, !dbg !75
  %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_idx to i64 addrspace(10)*, !dbg !75
  store i64 %0, i64 addrspace(10)* %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_cast, align 8, !dbg !75, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 24, !dbg !75
  %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_idx to double addrspace(10)*, !dbg !75
  store double %4, double addrspace(10)* %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_cast, align 8, !dbg !75, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 32, !dbg !75
  %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_idx to i64 addrspace(10)*, !dbg !75
  store i64 %0, i64 addrspace(10)* %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_cast, align 8, !dbg !75, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 40, !dbg !75
  %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_idx to double addrspace(10)*, !dbg !75
  store double %4, double addrspace(10)* %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_cast, align 8, !dbg !75, !tbaa !49, !alias.scope !50, !noalias !74
  %19 = call noalias nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @ijl_new_structv, {} addrspace(10)* nonnull %16, {} addrspace(10)* nofree nonnull %box12.i, {} addrspace(10)* nofree nonnull %box14.i, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891479608032 to {}*) to {} addrspace(10)*), {} addrspace(10)* nofree nonnull %3) #14, !dbg !75
  %box16.i = call noalias nonnull dereferenceable(16) "enzyme_inactive" "enzyme_type"="{[-1]:Pointer, [-1,-1]:Integer}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 16, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891407532896 to {}*) to {} addrspace(10)*)) #13, !dbg !70
  %20 = bitcast {} addrspace(10)* %box16.i to i8 addrspace(10)*, !dbg !70
  %newstruct3.i.sroa.0.0..sroa_cast = bitcast {} addrspace(10)* %box16.i to i64 addrspace(10)*, !dbg !70
  store i64 %8, i64 addrspace(10)* %newstruct3.i.sroa.0.0..sroa_cast, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct3.i.sroa.4.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %20, i64 8, !dbg !70
  %newstruct3.i.sroa.4.0..sroa_cast = bitcast i8 addrspace(10)* %newstruct3.i.sroa.4.0..sroa_idx to i64 addrspace(10)*, !dbg !70
  store i64 %8, i64 addrspace(10)* %newstruct3.i.sroa.4.0..sroa_cast, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %box18.i = call noalias nonnull dereferenceable(16) "enzyme_inactive" "enzyme_type"="{[-1]:Pointer, [-1,-1]:Integer}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 16, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891407532896 to {}*) to {} addrspace(10)*)) #13, !dbg !70
  %21 = bitcast {} addrspace(10)* %box18.i to i8 addrspace(10)*, !dbg !70
  %newstruct4.i.sroa.0.0..sroa_cast = bitcast {} addrspace(10)* %box18.i to i64 addrspace(10)*, !dbg !70
  store i64 %1, i64 addrspace(10)* %newstruct4.i.sroa.0.0..sroa_cast, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %newstruct4.i.sroa.4.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %21, i64 8, !dbg !70
  %newstruct4.i.sroa.4.0..sroa_cast = bitcast i8 addrspace(10)* %newstruct4.i.sroa.4.0..sroa_idx to i64 addrspace(10)*, !dbg !70
  store i64 0, i64 addrspace(10)* %newstruct4.i.sroa.4.0..sroa_cast, align 8, !dbg !70, !tbaa !49, !alias.scope !50, !noalias !74
  %22 = call noalias nonnull "enzyme_inactive" "enzyme_type"="{[-1]:Pointer, [-1,-1]:Integer}" {} addrspace(10)* @ijl_box_int64(i64 signext %0) #17, !dbg !70, !noalias !58
  %23 = call nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @ijl_apply_generic, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891468552880 to {}*) to {} addrspace(10)*), {} addrspace(10)* nonnull %19, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138889797305424 to {}*) to {} addrspace(10)*), {} addrspace(10)* nofree nonnull %box16.i, {} addrspace(10)* nofree nonnull %box18.i, {} addrspace(10)* nonnull %22) #18, !dbg !70
  %24 = addrspacecast {} addrspace(10)* %23 to i8 addrspace(11)*, !dbg !70
  %innersret.sroa.0.0..sroa_cast = addrspacecast {} addrspace(10)* %23 to {} addrspace(10)* addrspace(11)*, !dbg !70
  %innersret.sroa.0.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.0.0..sroa_cast, align 1, !dbg !70, !noalias !76
  %innersret.sroa.2.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 8, !dbg !70
  %innersret.sroa.2.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.2.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !70
  %innersret.sroa.2.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.2.0..sroa_cast, align 1, !dbg !70, !noalias !76
  %innersret.sroa.3.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 16, !dbg !70
  %innersret.sroa.3.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.3.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !70
  %innersret.sroa.3.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.3.0..sroa_cast, align 1, !dbg !70, !noalias !76
  %innersret.sroa.4.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 24, !dbg !70
  %innersret.sroa.4.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.4.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !70
  %innersret.sroa.4.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.4.0..sroa_cast, align 1, !dbg !70, !noalias !76
  %innersret.sroa.5.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 32, !dbg !70
  %innersret.sroa.5.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.5.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !70
  %innersret.sroa.5.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.5.0..sroa_cast, align 1, !dbg !70, !noalias !76
  %innersret.sroa.6.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 40, !dbg !70
  %innersret.sroa.6.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.6.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !70
  %innersret.sroa.6.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.6.0..sroa_cast, align 1, !dbg !70, !noalias !76
  %newstruct7.i.sroa.5.0.sroa_idx3 = getelementptr inbounds [7 x i8], [7 x i8]* %newstruct7.i.sroa.5, i64 0, i64 0, !dbg !70
  call void @llvm.lifetime.end.p0i8(i64 7, i8* %newstruct7.i.sroa.5.0.sroa_idx3), !dbg !70
  %.fca.0.insert = insertvalue [6 x {} addrspace(10)*] poison, {} addrspace(10)* %innersret.sroa.0.0.copyload, 0, !dbg !77
  %.fca.1.insert = insertvalue [6 x {} addrspace(10)*] %.fca.0.insert, {} addrspace(10)* %innersret.sroa.2.0.copyload, 1, !dbg !77
  %.fca.2.insert = insertvalue [6 x {} addrspace(10)*] %.fca.1.insert, {} addrspace(10)* %innersret.sroa.3.0.copyload, 2, !dbg !77
  %.fca.3.insert = insertvalue [6 x {} addrspace(10)*] %.fca.2.insert, {} addrspace(10)* %innersret.sroa.4.0.copyload, 3, !dbg !77
  %.fca.4.insert = insertvalue [6 x {} addrspace(10)*] %.fca.3.insert, {} addrspace(10)* %innersret.sroa.5.0.copyload, 4, !dbg !77
  %.fca.5.insert = insertvalue [6 x {} addrspace(10)*] %.fca.4.insert, {} addrspace(10)* %innersret.sroa.6.0.copyload, 5, !dbg !77
  ret [6 x {} addrspace(10)*] %.fca.5.insert, !dbg !77
}

 Type analysis state: 
<analysis>
  %14 = bitcast {} addrspace(10)* %box10.i to i8 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,0]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Float@double}, intvals: {}
  %innersret.sroa.2.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.2.0..sroa_cast, align 1, !dbg !34, !noalias !55: {[-1]:Pointer}, intvals: {}
  %.fca.0.insert = insertvalue [6 x {} addrspace(10)*] poison, {} addrspace(10)* %innersret.sroa.0.0.copyload, 0, !dbg !56: {[0]:Pointer, [8]:Anything, [9]:Anything, [10]:Anything, [11]:Anything, [12]:Anything, [13]:Anything, [14]:Anything, [15]:Anything, [16]:Anything, [17]:Anything, [18]:Anything, [19]:Anything, [20]:Anything, [21]:Anything, [22]:Anything, [23]:Anything, [24]:Anything, [25]:Anything, [26]:Anything, [27]:Anything, [28]:Anything, [29]:Anything, [30]:Anything, [31]:Anything, [32]:Anything, [33]:Anything, [34]:Anything, [35]:Anything, [36]:Anything, [37]:Anything, [38]:Anything, [39]:Anything, [40]:Anything, [41]:Anything, [42]:Anything, [43]:Anything, [44]:Anything, [45]:Anything, [46]:Anything, [47]:Anything}, intvals: {}
  %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 16, !dbg !52: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Float@double, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double}, intvals: {}
  %newstruct4.i.sroa.4.0..sroa_cast = bitcast i8 addrspace(10)* %newstruct4.i.sroa.4.0..sroa_idx to i64 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
  %.fca.4.insert = insertvalue [6 x {} addrspace(10)*] %.fca.3.insert, {} addrspace(10)* %innersret.sroa.5.0.copyload, 4, !dbg !56: {[0]:Pointer, [8]:Pointer, [16]:Pointer, [32]:Pointer, [40]:Anything, [41]:Anything, [42]:Anything, [43]:Anything, [44]:Anything, [45]:Anything, [46]:Anything, [47]:Anything}, intvals: {}
  %15 = call noalias nonnull "enzyme_type"="{[-1]:Pointer, [-1,-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @jl_f_tuple, {} addrspace(10)* noundef null, {} addrspace(10)* nofree nonnull %box.i, {} addrspace(10)* nofree nonnull %box10.i, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891479608032 to {}*) to {} addrspace(10)*), {} addrspace(10)* nofree nonnull %3) #14, !dbg !34: {[-1]:Pointer, [-1,-1]:Pointer}, intvals: {}
  %box18.i = call noalias nonnull dereferenceable(16) "enzyme_inactive" "enzyme_type"="{[-1]:Pointer, [-1,-1]:Integer}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 16, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891407532896 to {}*) to {} addrspace(10)*)) #13, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
  %newstruct7.i.sroa.58.0..sroa_idx9 = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 8, !dbg !52: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Float@double, [-1,24]:Integer, [-1,25]:Integer, [-1,26]:Integer, [-1,27]:Integer, [-1,28]:Integer, [-1,29]:Integer, [-1,30]:Integer, [-1,31]:Integer, [-1,32]:Float@double}, intvals: {}
  %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_idx to i64 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Float@double, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double}, intvals: {}
  %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_idx to double addrspace(10)*, !dbg !52: {[-1]:Pointer, [-1,0]:Float@double}, intvals: {}
  %innersret.sroa.0.0..sroa_cast = addrspacecast {} addrspace(10)* %23 to {} addrspace(10)* addrspace(11)*, !dbg !34: {[-1]:Pointer, [-1,0]:Pointer, [-1,8]:Pointer, [-1,16]:Pointer, [-1,32]:Pointer}, intvals: {}
  %8 = shl i64 %0, 1, !dbg !19: {[-1]:Integer}, intvals: {}
  %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_idx to double addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,0]:Float@double}, intvals: {}
  %9 = mul i64 %1, %0, !dbg !19: {[-1]:Integer}, intvals: {}
  %innersret.sroa.4.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.4.0..sroa_cast, align 1, !dbg !34, !noalias !55: {}, intvals: {}
  %box16.i = call noalias nonnull dereferenceable(16) "enzyme_inactive" "enzyme_type"="{[-1]:Pointer, [-1,-1]:Integer}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 16, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891407532896 to {}*) to {} addrspace(10)*)) #13, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
  %.fca.1.insert = insertvalue [6 x {} addrspace(10)*] %.fca.0.insert, {} addrspace(10)* %innersret.sroa.2.0.copyload, 1, !dbg !56: {[0]:Pointer, [8]:Pointer, [16]:Anything, [17]:Anything, [18]:Anything, [19]:Anything, [20]:Anything, [21]:Anything, [22]:Anything, [23]:Anything, [24]:Anything, [25]:Anything, [26]:Anything, [27]:Anything, [28]:Anything, [29]:Anything, [30]:Anything, [31]:Anything, [32]:Anything, [33]:Anything, [34]:Anything, [35]:Anything, [36]:Anything, [37]:Anything, [38]:Anything, [39]:Anything, [40]:Anything, [41]:Anything, [42]:Anything, [43]:Anything, [44]:Anything, [45]:Anything, [46]:Anything, [47]:Anything}, intvals: {}
  %ptls_field.i50 = getelementptr inbounds {}**, {}*** %5, i64 2: {[-1]:Pointer, [-1,0]:Pointer}, intvals: {}
  %newstruct3.i.sroa.4.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %20, i64 8, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
  %newstruct7.i.sroa.5.0..sroa_raw_idx4 = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 1, !dbg !52: {[-1]:Pointer, [-1,7]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Float@double, [-1,31]:Integer, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Float@double}, intvals: {}
  %newstruct7.i.sroa.58.0..sroa_cast10 = bitcast i8 addrspace(10)* %newstruct7.i.sroa.58.0..sroa_idx9 to i64 addrspace(10)*, !dbg !52: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Float@double, [-1,24]:Integer, [-1,25]:Integer, [-1,26]:Integer, [-1,27]:Integer, [-1,28]:Integer, [-1,29]:Integer, [-1,30]:Integer, [-1,31]:Integer, [-1,32]:Float@double}, intvals: {}
i64 %0: {[-1]:Integer}, intvals: {}
i64 %1: {[-1]:Integer}, intvals: {}
i64 %2: {[-1]:Integer}, intvals: {}
{} addrspace(10)* %3: {[-1]:Pointer}, intvals: {}
double %4: {[-1]:Float@double}, intvals: {}
  %box10.i = call noalias nonnull dereferenceable(48) "enzyme_type"="{[-1]:Pointer, [-1,0]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Float@double}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 48, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891597143504 to {}*) to {} addrspace(10)*)) #13, !dbg !34: {[-1]:Pointer, [-1,0]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Float@double}, intvals: {}
  %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 32, !dbg !34: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Float@double}, intvals: {}
  %box12.i = call noalias nonnull dereferenceable(8) "enzyme_type"="{[-1]:Pointer, [-1,-1]:Float@double}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 8, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891479608032 to {}*) to {} addrspace(10)*)) #13, !dbg !52: {[-1]:Pointer, [-1,-1]:Float@double}, intvals: {}
{} addrspace(10)* addrspacecast ({}* inttoptr (i64 138889797305424 to {}*) to {} addrspace(10)*): {[-1]:Anything}, intvals: {}
  %.fca.5.insert = insertvalue [6 x {} addrspace(10)*] %.fca.4.insert, {} addrspace(10)* %innersret.sroa.6.0.copyload, 5, !dbg !56: {[0]:Pointer, [8]:Pointer, [16]:Pointer, [32]:Pointer}, intvals: {}
  %12 = fdiv double %10, %11, !dbg !32: {[-1]:Float@double}, intvals: {}
  %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_idx to double addrspace(10)*, !dbg !52: {[-1]:Pointer, [-1,0]:Float@double, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Float@double}, intvals: {}
{}* inttoptr (i64 138891468552880 to {}*): {[-1]:Anything}, intvals: {}
[6 x {} addrspace(10)*] poison: {[-1]:Anything}, intvals: {}
i64 1: {[-1]:Integer}, intvals: {1,}
  %18 = bitcast {} addrspace(10)* %box14.i to i8 addrspace(10)*, !dbg !52: {[-1]:Pointer, [-1,0]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Float@double}, intvals: {}
{}* inttoptr (i64 138891597143504 to {}*): {[-1]:Anything}, intvals: {}
i64 0: {[-1]:Anything}, intvals: {0,}
  %innersret.sroa.3.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 16, !dbg !34: {[-1]:Pointer, [-1,0]:Pointer, [-1,16]:Pointer}, intvals: {}
  %innersret.sroa.0.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.0.0..sroa_cast, align 1, !dbg !34, !noalias !55: {[-1]:Pointer}, intvals: {}
  %newstruct7.i.sroa.5.0.sroa_idx2 = getelementptr inbounds [7 x i8], [7 x i8]* %newstruct7.i.sroa.5, i64 0, i64 0: {[-1]:Pointer}, intvals: {}
  %innersret.sroa.4.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.4.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !34: {[-1]:Pointer, [-1,8]:Pointer}, intvals: {}
  %newstruct7.i.sroa.5.0.sroa_idx27 = getelementptr inbounds [7 x i8], [7 x i8]* %newstruct7.i.sroa.5, i64 0, i64 0: {[-1]:Pointer}, intvals: {}
{} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891468552880 to {}*) to {} addrspace(10)*): {[-1]:Anything}, intvals: {}
  %typeof.i = call "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* @julia.typeof({} addrspace(10)* nonnull %15) #15, !dbg !52: {[-1]:Pointer}, intvals: {}
{}* inttoptr (i64 138891479608032 to {}*): {[-1]:Anything}, intvals: {}
  %newstruct7.i.sroa.5 = alloca [7 x i8], align 1: {[-1]:Pointer}, intvals: {}
  %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_idx to i64 addrspace(10)*, !dbg !52: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Float@double}, intvals: {}
  %innersret.sroa.5.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.5.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !34: {[-1]:Pointer, [-1,0]:Pointer}, intvals: {}
  %safepoint.i = load i64*, i64** %7, align 8, !tbaa !15, !noalias !8: {}, intvals: {}
  %innersret.sroa.2.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 8, !dbg !34: {[-1]:Pointer, [-1,0]:Pointer, [-1,8]:Pointer, [-1,24]:Pointer}, intvals: {}
  %22 = call noalias nonnull "enzyme_inactive" "enzyme_type"="{[-1]:Pointer, [-1,-1]:Integer}" {} addrspace(10)* @ijl_box_int64(i64 signext %0) #17, !dbg !34, !noalias !8: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
{} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891479609504 to {}*) to {} addrspace(10)*): {[-1]:Anything}, intvals: {}
  %innersret.sroa.5.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.5.0..sroa_cast, align 1, !dbg !34, !noalias !55: {[-1]:Pointer}, intvals: {}
{} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891567787088 to {}*) to {} addrspace(10)*): {[-1]:Anything}, intvals: {}
  %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 24, !dbg !52: {[-1]:Pointer, [-1,0]:Float@double, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Float@double}, intvals: {}
{}* inttoptr (i64 138891567787088 to {}*): {[-1]:Anything}, intvals: {}
  %box14.i = call noalias nonnull dereferenceable(48) "enzyme_type"="{[-1]:Pointer, [-1,0]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Float@double}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 48, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891597143504 to {}*) to {} addrspace(10)*)) #13, !dbg !52: {[-1]:Pointer, [-1,0]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Float@double}, intvals: {}
  %innersret.sroa.6.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 40, !dbg !34: {[-1]:Pointer}, intvals: {}
  %box.i = call noalias nonnull dereferenceable(8) "enzyme_type"="{[-1]:Pointer, [-1,-1]:Float@double}" {} addrspace(10)* @julia.gc_alloc_obj({}** nonnull %current_task1.i, i64 noundef 8, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891479608032 to {}*) to {} addrspace(10)*)) #13, !dbg !34: {[-1]:Pointer, [-1,-1]:Float@double}, intvals: {}
  %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx14.sroa_idx to i64 addrspace(10)*, !dbg !52: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Float@double, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double}, intvals: {}
  %10 = sitofp i64 %9 to double, !dbg !23: {[-1]:Float@double}, intvals: {}
{}* inttoptr (i64 138891407532896 to {}*): {[-1]:Anything}, intvals: {}
  %16 = call nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @jl_f_apply_type, {} addrspace(10)* noundef null, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891479609504 to {}*) to {} addrspace(10)*), {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891567787088 to {}*) to {} addrspace(10)*), {} addrspace(10)* nonnull %typeof.i) #16, !dbg !52: {[-1]:Pointer}, intvals: {}
  %innersret.sroa.3.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.3.0..sroa_cast, align 1, !dbg !34, !noalias !55: {[-1]:Pointer}, intvals: {}
  %newstruct7.i.sroa.6.sroa.0.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 16, !dbg !34: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Float@double, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Float@double}, intvals: {}
  %11 = sitofp i64 %2 to double, !dbg !23: {[-1]:Float@double}, intvals: {}
  %newstruct7.i.sroa.5.0.sroa_idx3 = getelementptr inbounds [7 x i8], [7 x i8]* %newstruct7.i.sroa.5, i64 0, i64 0, !dbg !34: {[-1]:Pointer}, intvals: {}
  %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 24, !dbg !34: {[-1]:Pointer, [-1,0]:Float@double, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Float@double}, intvals: {}
  %innersret.sroa.4.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 24, !dbg !34: {[-1]:Pointer, [-1,8]:Pointer}, intvals: {}
  %newstruct7.i.sroa.58.0..sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.58.0..sroa_idx to i64 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Float@double, [-1,24]:Integer, [-1,25]:Integer, [-1,26]:Integer, [-1,27]:Integer, [-1,28]:Integer, [-1,29]:Integer, [-1,30]:Integer, [-1,31]:Integer, [-1,32]:Float@double}, intvals: {}
{} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891597143504 to {}*) to {} addrspace(10)*): {[-1]:Anything}, intvals: {}
  %ptls_load.i5152 = load i64**, i64*** %6, align 8, !tbaa !11, !noalias !8: {[-1]:Pointer}, intvals: {}
  %23 = call nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @ijl_apply_generic, {} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 138891468552880 to {}*) to {} addrspace(10)*), {} addrspace(10)* nonnull %19, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138889797305424 to {}*) to {} addrspace(10)*), {} addrspace(10)* nofree nonnull %box16.i, {} addrspace(10)* nofree nonnull %box18.i, {} addrspace(10)* nonnull %22) #18, !dbg !34: {[-1]:Pointer, [-1,0]:Pointer, [-1,8]:Pointer, [-1,16]:Pointer, [-1,32]:Pointer}, intvals: {}
  %6 = bitcast {}*** %ptls_field.i50 to i64***: {[-1]:Pointer, [-1,0]:Pointer}, intvals: {}
  %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 40, !dbg !34: {[-1]:Pointer, [-1,0]:Float@double}, intvals: {}
  %19 = call noalias nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @ijl_new_structv, {} addrspace(10)* nonnull %16, {} addrspace(10)* nofree nonnull %box12.i, {} addrspace(10)* nofree nonnull %box14.i, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891479608032 to {}*) to {} addrspace(10)*), {} addrspace(10)* nofree nonnull %3) #14, !dbg !52: {[-1]:Pointer}, intvals: {}
  %newstruct7.i.sroa.5.0..sroa_raw_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 1, !dbg !34: {[-1]:Pointer, [-1,7]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Float@double, [-1,31]:Integer, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Float@double}, intvals: {}
  %7 = getelementptr inbounds i64*, i64** %ptls_load.i5152, i64 2: {[-1]:Pointer}, intvals: {}
  %newstruct4.i.sroa.0.0..sroa_cast = bitcast {} addrspace(10)* %box18.i to i64 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
  %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.6.sroa.5.0.newstruct7.i.sroa.6.0..sroa_raw_idx.sroa_idx to double addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,0]:Float@double, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Float@double}, intvals: {}
  %newstruct4.i.sroa.4.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %21, i64 8, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
  %newstruct7.i.sroa.7.sroa.5.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 40, !dbg !52: {[-1]:Pointer, [-1,0]:Float@double}, intvals: {}
  %20 = bitcast {} addrspace(10)* %box16.i to i8 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
{} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891407532896 to {}*) to {} addrspace(10)*): {[-1]:Anything}, intvals: {}
{}* inttoptr (i64 138889797305424 to {}*): {[-1]:Anything}, intvals: {}
  %newstruct3.i.sroa.0.0..sroa_cast = bitcast {} addrspace(10)* %box16.i to i64 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
  %innersret.sroa.6.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.6.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !34: {[-1]:Pointer}, intvals: {}
{} addrspace(10)* addrspacecast ({}* inttoptr (i64 138891479608032 to {}*) to {} addrspace(10)*): {[-1]:Anything}, intvals: {}
  %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_cast = bitcast i8 addrspace(10)* %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx.sroa_idx to i64 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Float@double}, intvals: {}
  %21 = bitcast {} addrspace(10)* %box18.i to i8 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
{}* inttoptr (i64 138891479609504 to {}*): {[-1]:Anything}, intvals: {}
  %current_task1.i = bitcast {}*** %current_task1.i49 to {}**: {[-1]:Pointer}, intvals: {}
  %13 = bitcast {} addrspace(10)* %box.i to double addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,-1]:Float@double}, intvals: {}
  %innersret.sroa.2.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.2.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !34: {[-1]:Pointer, [-1,0]:Pointer, [-1,8]:Pointer, [-1,24]:Pointer}, intvals: {}
  %current_task1.i49 = getelementptr inbounds {}**, {}*** %5, i64 -14: {[-1]:Pointer}, intvals: {}
  %newstruct7.i.sroa.7.sroa.0.0.newstruct7.i.sroa.7.0..sroa_raw_idx19.sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %18, i64 32, !dbg !52: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Float@double}, intvals: {}
  %newstruct7.i.sroa.58.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(10)* %14, i64 8, !dbg !34: {[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Float@double, [-1,24]:Integer, [-1,25]:Integer, [-1,26]:Integer, [-1,27]:Integer, [-1,28]:Integer, [-1,29]:Integer, [-1,30]:Integer, [-1,31]:Integer, [-1,32]:Float@double}, intvals: {}
  %.fca.3.insert = insertvalue [6 x {} addrspace(10)*] %.fca.2.insert, {} addrspace(10)* %innersret.sroa.4.0.copyload, 3, !dbg !56: {[0]:Pointer, [8]:Pointer, [16]:Pointer, [32]:Anything, [33]:Anything, [34]:Anything, [35]:Anything, [36]:Anything, [37]:Anything, [38]:Anything, [39]:Anything, [40]:Anything, [41]:Anything, [42]:Anything, [43]:Anything, [44]:Anything, [45]:Anything, [46]:Anything, [47]:Anything}, intvals: {}
  %17 = bitcast {} addrspace(10)* %box12.i to double addrspace(10)*, !dbg !52: {[-1]:Pointer, [-1,-1]:Float@double}, intvals: {}
  %5 = call {}*** @julia.get_pgcstack() #12, !noalias !8: {[-1]:Pointer, [-1,16]:Pointer}, intvals: {}
  %innersret.sroa.3.0..sroa_cast = bitcast i8 addrspace(11)* %innersret.sroa.3.0..sroa_idx to {} addrspace(10)* addrspace(11)*, !dbg !34: {[-1]:Pointer, [-1,0]:Pointer, [-1,16]:Pointer}, intvals: {}
  %newstruct3.i.sroa.4.0..sroa_cast = bitcast i8 addrspace(10)* %newstruct3.i.sroa.4.0..sroa_idx to i64 addrspace(10)*, !dbg !34: {[-1]:Pointer, [-1,-1]:Integer}, intvals: {}
  %.fca.2.insert = insertvalue [6 x {} addrspace(10)*] %.fca.1.insert, {} addrspace(10)* %innersret.sroa.3.0.copyload, 2, !dbg !56: {[0]:Pointer, [8]:Pointer, [16]:Pointer, [24]:Anything, [25]:Anything, [26]:Anything, [27]:Anything, [28]:Anything, [29]:Anything, [30]:Anything, [31]:Anything, [32]:Anything, [33]:Anything, [34]:Anything, [35]:Anything, [36]:Anything, [37]:Anything, [38]:Anything, [39]:Anything, [40]:Anything, [41]:Anything, [42]:Anything, [43]:Anything, [44]:Anything, [45]:Anything, [46]:Anything, [47]:Anything}, intvals: {}
  %24 = addrspacecast {} addrspace(10)* %23 to i8 addrspace(11)*, !dbg !34: {[-1]:Pointer, [-1,0]:Pointer, [-1,8]:Pointer, [-1,16]:Pointer, [-1,32]:Pointer}, intvals: {}
  %innersret.sroa.5.0..sroa_idx = getelementptr inbounds i8, i8 addrspace(11)* %24, i64 32, !dbg !34: {[-1]:Pointer, [-1,0]:Pointer}, intvals: {}
  %innersret.sroa.6.0.copyload = load {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %innersret.sroa.6.0..sroa_cast, align 1, !dbg !34, !noalias !55: {}, intvals: {}
</analysis>

Cannot deduce type of copy   call void @llvm.memcpy.p10i8.p0i8.i64(i8 addrspace(10)* nocapture nofree noundef writeonly align 1 dereferenceable(7) %newstruct7.i.sroa.5.0..sroa_raw_idx, i8* noundef nonnull readonly align 1 dereferenceable(7) %newstruct7.i.sroa.5.0.sroa_idx27, i64 noundef 7, i1 noundef false) #12, !dbg !34, !tbaa !49, !alias.scope !50, !noalias !51

Caused by:
Stacktrace:
 [1] #make_sim#10
   @ ~/Work/GitHub/Julia/Sandbox/waterlily.jl:15
 [2] #make_sim#10
   @ ~/Work/GitHub/Julia/Sandbox/waterlily.jl:0
within MethodInstance for var"#make_sim#10"(::Int64, ::Int64, ::Int64, ::Type{Array}, ::typeof(make_sim), ::Float64)


Stacktrace:
  [1] throwerr(cstr::Cstring)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/Pljwm/src/compiler.jl:1678
  [2] #make_sim#10
    @ ~/Work/GitHub/Julia/Sandbox/waterlily.jl:15 [inlined]
  [3] fwddiffejulia__make_sim_10_8465_inner_1wrap
    @ ~/Work/GitHub/Julia/Sandbox/waterlily.jl:0
  [4] macro expansion
    @ ~/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6630 [inlined]
  [5] enzyme_call(::Val{…}, ::Ptr{…}, ::Type{…}, ::Val{…}, ::Val{…}, ::Type{…}, ::Type{…}, ::EnzymeCore.Const{…}, ::Type{…}, ::EnzymeCore.Const{…}, ::EnzymeCore.Const{…}, ::EnzymeCore.Const{…}, ::EnzymeCore.Const{…}, ::EnzymeCore.Const{…}, ::EnzymeCore.Duplicated{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6231
  [6] (::Enzyme.Compiler.ForwardModeThunk{…})(::EnzymeCore.Const{…}, ::EnzymeCore.Const{…}, ::Vararg{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6111
  [7] runtime_generic_fwd(activity::Type{…}, width::Val{…}, RT::Val{…}, f::var"##make_sim#10", df::Nothing, primal_1::Int64, shadow_1_1::Nothing, primal_2::Int64, shadow_2_1::Nothing, primal_3::Int64, shadow_3_1::Nothing, primal_4::Type{…}, shadow_4_1::Nothing, primal_5::typeof(make_sim), shadow_5_1::Nothing, primal_6::Float64, shadow_6_1::Float64)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/Pljwm/src/rules/jitrules.jl:210
  [8] make_sim
    @ ~/Work/GitHub/Julia/Sandbox/waterlily.jl:6 [inlined]
  [9] dsim
    @ ~/Work/GitHub/Julia/Sandbox/waterlily.jl:19 [inlined]
 [10] fwddiffejulia_dsim_8822wrap
    @ ~/Work/GitHub/Julia/Sandbox/waterlily.jl:0
 [11] macro expansion
    @ ~/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6630 [inlined]
 [12] enzyme_call
    @ ~/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6231 [inlined]
 [13] ForwardModeThunk
    @ ~/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6111 [inlined]
 [14] autodiff
    @ ~/.julia/packages/Enzyme/Pljwm/src/Enzyme.jl:427 [inlined]
 [15] autodiff(mode::EnzymeCore.ForwardMode{…}, f::typeof(dsim), ::Type{…}, args::EnzymeCore.Duplicated{…})
    @ Enzyme ~/.julia/packages/Enzyme/Pljwm/src/Enzyme.jl:326
 [16] top-level scope
    @ ~/Work/GitHub/Julia/Sandbox/waterlily.jl:33
Some type information was truncated. Use `show(err)` to see complete types.

The AutoEnzyme() backend object picks the best mode based on the operator you're applying. In the case of derivative, it will be forward mode, while in the case of gradient, it will be reverse mode. You can force a given mode with a keyword argument:

help?> AutoEnzyme
search: AutoEnzyme

  AutoEnzyme{M}


  Struct used to select the Enzyme.jl (https://github.com/EnzymeAD/Enzyme.jl) backend for automatic differentiation.

  Defined by ADTypes.jl (https://github.com/SciML/ADTypes.jl).

  Constructors
  ≡≡≡≡≡≡≡≡≡≡≡≡

  AutoEnzyme(; mode=nothing)


  Fields
  ≡≡≡≡≡≡

    •  mode::M: can be either
       • an object subtyping EnzymeCore.Mode (like EnzymeCore.Forward or EnzymeCore.Reverse) if a specific mode is required
       • nothing to choose the best mode automatically

@b-fg
Copy link
Author

b-fg commented Jul 15, 2024

Thanks for the info! And yes, that's the error I get with enzyme. It points to a memcpy when creating the Simulation struct and says that "Enzyme cannot deduce type", but I cannot see information of where this is exactly happening.

@gdalle
Copy link
Member

gdalle commented Jul 15, 2024 •
edited
Loading

I think that's the latest, right? So not sure what's going on.

Can you run the full code below in a fresh Julia REPL (the initial environment doesn't matter) and see if it still errors?

using Pkg
Pkg.activate(temp=true)
Pkg.add(["WaterLily", "StaticArrays", "DifferentiationInterface", "ForwardDiff"])

using WaterLily
using StaticArrays
using DifferentiationInterface
import ForwardDiff

function make_sim(θ; L=32, U=1, Re=100, mem=Array)
    function map(x, _)
        s, c = sincos(θ)
        SA[c -s; s c] * (x - SA[L, L])
    end
    function sdf(ξ, _)
        p = ξ - SA[0, clamp(ξ[1], -L / 2, L / 2)]
        √(p' * p) - 2
    end
    Simulation((2L, 2L), (U, 0), L, ν=U * L / Re, body=AutoBody(sdf, map), T=typeof(θ), mem=mem)
end

function dsim(θ)
    sim = make_sim(θ)
    sim_step!(sim)
    sum(sim.flow.p)
end

θ = Float64(π / 36)

# ForwardDiff
dsim(θ), ForwardDiff.derivative(dsim, θ)

# ForwardDiff via DI
value_and_derivative(dsim, AutoForwardDiff(), θ)

Also, thanks for that remark. Could you please clarify the practical difference in this case?

As for the operators:

  • derivative is for functions f: R -> anything (shortcut for pushforward with dx=1)
  • gradient is for functions f: anything -> R (shortcut for pullback with dy=1)

For scalar input and output, both should work, but I didn't make sure that gradient does since this case is covered by derivative already and it will be more efficient

@gdalle
Copy link
Member

gdalle commented Jul 15, 2024

Thanks for the info! And yes, that's the error I get with enzyme. It points to a memcpy when creating the Simulation struct and says that "Enzyme cannot deduce type", but I cannot see information of where this is exactly happening.

I would open an issue on the Enzyme.jl repo with my MWE (not using DifferentiationInterface at all) to see what @wsmoses thinks.

@b-fg
Copy link
Author

b-fg commented Jul 15, 2024

The code you shared does not error indeed. But note that it contains ForwardDiff.derivative(dsim, θ), which I assume it forces compilation of dsim with the appropriate ForwardDiff.Dual type. If this line is removed, this code produces an error for me

using Pkg
Pkg.activate(temp=true)
Pkg.add(["WaterLily", "StaticArrays", "DifferentiationInterface", "ForwardDiff"])

using WaterLily
using StaticArrays
using DifferentiationInterface
import ForwardDiff

function make_sim(θ; L=32, U=1, Re=100, mem=Array)
    function map(x, _)
        s, c = sincos(θ)
        SA[c -s; s c] * (x - SA[L, L])
    end
    function sdf(ξ, _)
        p = ξ - SA[0, clamp(ξ[1], -L / 2, L / 2)]
        √(p' * p) - 2
    end
    Simulation((2L, 2L), (U, 0), L, ν=U * L / Re, body=AutoBody(sdf, map), T=typeof(θ), mem=mem)
end

function dsim(θ)
    sim = make_sim(θ)
    sim_step!(sim)
    sum(sim.flow.p)
end

θ = Float64(π / 36)

# ForwardDiff via DI
value_and_derivative(dsim, AutoForwardDiff(), θ)

And noted about Enzyme, I will open an issue there. Thanks!

@gdalle
Copy link
Member

gdalle commented Jul 15, 2024

Okay now I got the error. It is very weird because the first call to ForwardDiff.derivative should be completely independent of the second one to DI.derivative. Investigating

@gdalle
Copy link
Member

gdalle commented Jul 15, 2024

Is there a version of the code without the KernelAbstractions? I'm not familiar with that library and its macros may mess with my understanding of what goes on inside WaterLily.measure

@b-fg
Copy link
Author

b-fg commented Jul 15, 2024 •
edited
Loading

Oh, note that we use ForwardDiff within the code:

https://github.com/WaterLily-jl/WaterLily.jl/blob/68b7c6b5658e6b3ca06b69ffa35fcdd5aabbebee/src/AutoBody.jl#L129-L130

So I think that the tags are getting mixed up there. That's why we sometimes create the Dual type with the appropriate tag, eg

T = typeof(ForwardDiff.Tag(dsim, Float64))
θ = ForwardDiff.Dual{T}(θ, one(Float64))

@gdalle
Copy link
Member

gdalle commented Jul 15, 2024

Actually, it seems KernelAbstractions may also mess with the tagging of the arguments, see for example:

@b-fg
Copy link
Author

b-fg commented Jul 15, 2024

To get KA out of the loop, just run with julia -t 1 and WaterLily defaults to non-KA kernels.

@gdalle
Copy link
Member

gdalle commented Jul 15, 2024

Okay, we now have a simpler stacktrace, but the error is still here. Weirdly enough, in this order both calls succeed

ForwardDiff.derivative(dsim, θ)
derivative(dsim, AutoForwardDiff(), θ)

but in this order both calls error

derivative(dsim, AutoForwardDiff(), θ)
ForwardDiff.derivative(dsim, θ)

That's beyond superweird. Is there any global state in WaterLily that might be altered?

@b-fg
Copy link
Author

b-fg commented Jul 15, 2024

Yes, that is weird indeed :/ If you could please clarify what do you mean by global state?

@gdalle
Copy link
Member

gdalle commented Jul 15, 2024 •
edited
Loading

Maybe a global variable of some form that the first call could modify. I think you're right and it has to do with compilation though.
I have removed KA, removed the @loop macro in measure!, still getting the same bug.

@gdalle
Copy link
Member

gdalle commented Jul 15, 2024 •
edited
Loading

I think I know what is happening. If I'm right, the global state I'm looking for is ForwardDiff.TAGCOUNT. I was creating tags in a way that doesn't increment it properly. Let my try a quick fix, by creating tags with Tag(f, eltype(x)) (which increments the counter) instead of Tag{typeof(f), eltype(x)} (which doesn't).

@b-fg
Copy link
Author

b-fg commented Jul 15, 2024 •
edited
Loading

There is no such thing AFAIK Sure, thanks!. The only thing is that we use ForwardDiff within measure, and I have been doing some checks and that is indeed the problem. When I don't use the functions in there, ie ForwardDiff.jacobian, ForwardDiff.derivative, ForwardDiff.gradient the error does not show up.

@gdalle gdalle linked a pull request Jul 15, 2024 that will close this issue
Fix tag construction in ForwardDiff #357
Merged
@gdalle
Copy link
Member

gdalle commented Jul 15, 2024

Can you try it with the branch from #357?

@b-fg
Copy link
Author

b-fg commented Jul 15, 2024

Works as intended! Thank you very much for the quick fix. And I will open an issue in Enzyme.jl regarding the other stuff. Cheers :)

@b-fg b-fg closed this as completed Jul 15, 2024
This was referenced Jul 17, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix tag construction in ForwardDiff JuliaDiff/DifferentiationInterface.jl
2 participants
@b-fg @gdalle