DEQ layer

[rkurchin]

Want to get this capability working (reference) eventually.

[rkurchin]

Food for thought: may need to be careful and do some preprocessing/rounding on weight matrices here, because sometimes graph edge weights that should be identical aren't quite...take the methane molecule for instance:

julia> Matrix(ag2.graph.weights)
5×5 Array{Float32,2}:
 0.0       0.999988  0.99999   1.0       0.999998
 0.999988  0.0       0.375009  0.374997  0.374993
 0.99999   0.375009  0.0       0.374993  0.374997
 1.0       0.374997  0.374993  0.0       0.374997
 0.999998  0.374993  0.374997  0.374997  0.0

And obviously a SteadyStateSolution would be a lot more sensitive to this than the result of just a couple discrete convolutions...

[rkurchin]

@ChrisRackauckas @YingboMa any updates on whatever's needed under the hood (something with SteadyStateAdjoint I think?) for this to work? I just rebased the branch so all the syntax, etc. is up to date for the stuff that got renamed. deq.jl on the deq branch now runs up through building the model, but the forward pass just hangs forever and so obviously we can't train either.

[ChrisRackauckas]

I don't think it's the implementation I think it's the method. A different rootfinding method would be needed her for a faster forward pass. The Newton iterations are just unstable so it needs to somehow be stabilized.

[rkurchin]

Are you saying a new method needs to be implemented or just that I should try other methods?

Also, IIRC even when the forward pass worked, the backwards still didn't.