Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a kwarg to turn off data healthy check. #156

Open
GiggleLiu opened this issue Apr 24, 2020 · 9 comments
Open

Provide a kwarg to turn off data healthy check. #156

GiggleLiu opened this issue Apr 24, 2020 · 9 comments

Comments

@GiggleLiu
Copy link

GiggleLiu commented Apr 24, 2020

Great tool!

For some user defined types, the following check would break for not having isinf and isnan.

function check_data_health(xdata, ydata)
    if any(ismissing, xdata) || any(ismissing, ydata)
        error("Data contains `missing` values and a fit cannot be performed")
    end
    if any(isinf, xdata) || any(isinf, ydata) || any(isnan, xdata) || any(isnan, ydata)
        error("Data contains `Inf` or `NaN` values and a fit cannot be performed")
    end
end

An example is StaticArray. By overloading this check (so called prirating), everything works fine.
I guess it would be good to provide a kwarg to allow user to switch off this check?

@pkofod
Copy link
Member

pkofod commented May 28, 2020

Does it not work for a static array?

julia> using StaticArrays

julia> any(isnan, @SVector([1,2,3]))
false

@GiggleLiu
Copy link
Author

GiggleLiu commented May 29, 2020

I mean sometimes, it could be helpful to use a StaticArray as a scalar. isnan and isinf are originally designed for Floating point numbers.

julia> isnan(SMatrix{2,2}(1.0, 2.0, 3.0, 5.0))
ERROR: MethodError: no method matching isnan(::SArray{Tuple{2,2},Float64,2,4})
Closest candidates are:
  isnan(::BigFloat) at mpfr.jl:893
  isnan(::Missing) at missing.jl:100
  isnan(::Float16) at float.jl:537
  ...
Stacktrace:
 [1] top-level scope at REPL[4]:1

julia> isnan(SVector{4}(1.0, 2.0, 3.0, 5.0))
ERROR: MethodError: no method matching isnan(::SArray{Tuple{4},Float64,1,4})
Closest candidates are:
  isnan(::BigFloat) at mpfr.jl:893
  isnan(::Missing) at missing.jl:100
  isnan(::Float16) at float.jl:537
  ...
Stacktrace:
 [1] top-level scope at REPL[5]:1

(@v1.4) pkg> st StaticArrays
Status `~/.julia/environments/v1.4/Project.toml`
  [90137ffa] StaticArrays v0.12.3

@pkofod
Copy link
Member

pkofod commented Dec 16, 2020

Let's try to fix this :) Are you using something like a static array of static arrays?

@GiggleLiu
Copy link
Author

Sounds great, yes, I am using a static vector as the element.

@pkofod
Copy link
Member

pkofod commented Dec 17, 2020

What are you using as the container?

@pkofod pkofod mentioned this issue Dec 17, 2020
@GiggleLiu
Copy link
Author

GiggleLiu commented Dec 17, 2020

A regular array, I should probably write a MWE.

@pkofod
Copy link
Member

pkofod commented Dec 18, 2020

That would be helpful. Can you use RecursiveArrayTools?

@gustaphe
Copy link

In #202 , I used something like

is_unhealthy(x) = ismissing(x) || isnan(x) || isinfinite(x)
is_unhealthy(x::AbstractArray) = any(is_unhealthy, x)

which recursively does the right thing for vectors. Could be interesting to implement for this whether or not #202 happens?

@m-wells
Copy link

m-wells commented Oct 13, 2022

I ran into this issue. I have a custom LazyTable that I pass as xdata. I only consume certain columns but some unused columns contain NaN. As this is a lazy source the presence of the extra columns do not harm performance. The check_data_health function is eager (and ignorant of the data source). I can disable this check easily enough.

function LsqFit.check_data_health(::MyData, ::Any) end

I imagine there is probably a more flexible way of erroring out such as waiting to do any ismissing, isnan, isfinite such as waiting until after the first evaluation and checking df.

function levenberg_marquardt(
df::OnceDifferentiable,
initial_x::AbstractVector{T};
x_tol::Real=1e-8,
g_tol::Real=1e-12,
maxIter::Integer=1000,
maxTime::Float64=Inf,
lambda=T(10),
tau=T(Inf),
lambda_increase::Real=10.0,
lambda_decrease::Real=0.1,
min_step_quality::Real=1e-3,
good_step_quality::Real=0.75,
show_trace::Bool=false,
store_trace::Bool=false,
lower::AbstractVector{T}=Array{T}(undef, 0),
upper::AbstractVector{T}=Array{T}(undef, 0),
avv!::Union{Function,Nothing,Avv}=nothing,
) where {T}
# First evaluation
value_jacobian!!(df, initial_x)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants