Function to find the Differential type for a given primal type

[oxinabox]

When declaring rules for constructors of parameteric types (e.g. Arrays)
one needs too know what the parameter type of the elements are.

We should define a helper for this.

Its probaly something like:

differential_type(::Type{<:Any})) = Any
differential_type(T::Type{<:Real}) = Union(AbstractZero, T)
differential_type(::Array{T}) where T = Array{differential_type(T)}

This is primarily for use for type parameters of zero arg constructors.
Rather than the general way to do it

[willtebbutt]

Good point. Probably the most performance critical case is going to be constructing a differential for something like

Vector{Float64}(undef, N)

One obvious choice for the element type of the differential is Union{Float64, AbstractZero}. Being an unboundedly-large union, this is probably bad for performance.

It might be that we genuinely do need to do things like convert AbstractZero to Float64 in such cases for the sake of performance. Feels like there's a case to be made for sensible (but generally correct) heuristics in such cases.

[oxinabox]

If rather than Vector{Union{Float64, AbstractZero}} it was Vector{Union{Float64, Zero, DoesNotExist}} then it hits the small union optimization.
Andd so works just as well as Vector{Union{Missing, Nothing, Float64}}
which is to say: "pretty well"

But yes, for the Float64 case, it might well be reasonable to convert to 0.0
ideally we would expost the option to the AD system somehow

[willtebbutt]

If rather than Vector{Union{Float64, AbstractZero}} it was Vector{Union{Float64, Zero, DoesNotExist}} then it hits the small union optimization.
Andd so works just as well as Vector{Union{Missing, Nothing, Float64}}
which is to say: "pretty well"

Makes more sense.

ideally we would expost the option to the AD system somehow

Agreed. Tbh this might even live in the realms of requiring user-intervention, as it's not clear to me that any particular AD system will have the information necessary to make this optimisation automatically.

[oxinabox]

Agreed. Tbh this might even live in the realms of requiring user-intervention, as it's not clear to me that any particular AD system will have the information necessary to make this optimisation automatically.

Yeah, one situation is if the user knows for example that in this case its never going to be mutated we don't need to allow the possibility that it could become Zero if it wasn't already.

But in the short term we can do the marginally sub-optimal but always correct thing.

[willtebbutt]

Yeah, one situation is if the user knows for example that in this case its never going to be mutated we don't need to allow the possibility that it could become Zero if it wasn't already.

Exactly. Or if they happen to know that it's never going to become Zero.