Avoid error with MATLAB classes that we can't handle #129

jebej · 2019-11-26T04:52:23Z

This method simply avoids classes that cannot currently be handled without crashing the read, and at least allows looking at the rest of the file.

Maybe logging a warning would be better than just replacing the object with a string. Happy to change this to whatever is preferred.

timholy · 2019-11-29T09:46:49Z

Thanks! Yes, I think it would be better to @warn or @error. Can you paste what happens now? And ideally, add a test for this new functionality? You'd need to contribute a small file that has an unreadable data type.

jebej · 2019-11-30T17:11:32Z

Right now you get an error, as in #127.

What do you suggest we do with the problematic variable? With this PR, the variable's contents get replaced with a string which I thought was useful, because then you can see that a variable was supposed to be there, rather than silently ignoring it.

timholy · 2019-12-02T13:08:05Z

Maybe return missing? Along with the warning that gets printed to the REPL?

jebej · 2019-12-02T23:22:00Z

Ok, so this now works, with a caveat that I hadn't noticed before.

If using matread, or read without specifying a variable, the read will fail. This seems to be due to a weird key in the names of the file returned by matopen. See below.

When using read with the variable, things work out, and this is how I am running the test currently (using a closure with matopen).

julia> mfile = matopen(path)
MAT.MAT_HDF5.MatlabHDF5File(HDF5 data file: C:\Users\Jeremy\.julia\dev\MAT\test\v7.3\struct_table_datetime.mat, true, false, 36, false)

julia> read(mfile)
HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
  #000: E:/mingwbuild/mingw-w64-hdf5/src/hdf5-1.10.5/src/H5A.c line 425 in H5Aopen(): unable to load attribute info from object header for attribute: 'MATLAB_class'
    major: Attribute
    minor: Can't open object
  #001: E:/mingwbuild/mingw-w64-hdf5/src/hdf5-1.10.5/src/H5Aint.c line 433 in H5A__open(): unable to load attribute info from object header for attribute: 'MATLAB_class'
    major: Attribute
    minor: Can't open object
  #002: E:/mingwbuild/mingw-w64-hdf5/src/hdf5-1.10.5/src/H5Oattribute.c line 515 in H5O__attr_open_by_name(): can't locate attribute: 'MATLAB_class'
    major: Attribute
    minor: Object not found
ERROR: Error opening attribute /#subsystem#/MATLAB_class
Stacktrace:
 [1] h5a_open at C:\Users\Jeremy\.julia\packages\HDF5\Zh9on\src\HDF5.jl:2279 [inlined]
 [2] a_open(::HDF5.HDF5Group, ::String) at C:\Users\Jeremy\.julia\packages\HDF5\Zh9on\src\HDF5.jl:852
 [3] a_read(::HDF5.HDF5Group, ::String) at C:\Users\Jeremy\.julia\packages\HDF5\Zh9on\src\HDF5.jl:1269
 [4] m_read(::HDF5.HDF5Group) at C:\Users\Jeremy\.julia\dev\MAT\src\MAT_HDF5.jl:201
 [5] read(::MAT.MAT_HDF5.MatlabHDF5File, ::String) at C:\Users\Jeremy\.julia\dev\MAT\src\MAT_HDF5.jl:264
 [6] read(::MAT.MAT_HDF5.MatlabHDF5File) at C:\Users\Jeremy\.julia\packages\HDF5\Zh9on\src\datafile.jl:45
 [7] top-level scope at none:0

julia> read(mfile,"s")
┌ Warning: MATLAB datetime values are currently not supported
└ @ MAT.MAT_HDF5 C:\Users\Jeremy\.julia\dev\MAT\src\MAT_HDF5.jl:173
┌ Warning: MATLAB table values are currently not supported
└ @ MAT.MAT_HDF5 C:\Users\Jeremy\.julia\dev\MAT\src\MAT_HDF5.jl:173
Dict{String,Any} with 2 entries:
  "testDatetime" => nothing
  "testTable"    => nothing

julia> names(mfile)
2-element Array{String,1}:
 "#subsystem#"
 "s"

jebej · 2019-12-03T02:18:58Z

I am locally getting HDF5 read errors (Error reading dataset /imaginary) with check("complex.mat", result), though not consistently.

I can see see that the same thing is happening on master.

jebej · 2019-12-03T02:26:34Z

Ok, it seems that the error does not happen right after recompiling, but if I exit julia after running the tests, and then try running the tests again in a fresh session there it crashes.

Then it keeps crashing, until I trigger a recompilation at which points it works once and the cycle begins again...

jebej · 2019-12-03T02:56:45Z

The error does seem to be amplified by my changes, but I can't tell why... The code path is clearly not being called or we would see the warning happening.

timholy · 2019-12-03T11:51:27Z

If using matread, or read without specifying a variable, the read will fail.

Where does #subsystem# come from? Is it something you added to the file, or did Matlab add that itself? If it's a reproducible name one could systematically exclude it.

The error does seem to be amplified by my changes, but I can't tell why... The code path is clearly not being called or we would see the warning happening.

Oh dear, I hate these kind of things. I saw it on Travis for Julia 0.7, but you're getting it on a more recent version? Might be an HDF5.jl bug, I would start by looking into that line h5d_read(::Int64, ::Int64, ::Int64, ::Int64, ::Int64, ::Array{Complex{Float64},1}) at /home/travis/.julia/packages/HDF5/Zh9on/src/HDF5.jl:2279

jebej · 2020-02-25T22:07:30Z

I've beeen trying to rebase this but the Github action tagbot is preventing me from pushing changes... Do you know how this is supposed to work?

jebej · 2020-02-25T22:51:35Z

Sigh, I had to add everything as SSH connections for it to work. Let's see what CI says now.

jebej · 2020-02-25T23:23:02Z

Ok, it seems that the only error is the same one that happens on master. Locally tests pass for me.

jebej · 2020-04-28T16:28:26Z

Ok, so I went down the rabbit hole with this, and ended up seeing PR #23. The #subsystem# entry is described there.

I have excluded the entry from the names function, meaning everything works now.

I have also adapted the above PR to allow reading datetime values, which I will submit once this is in.

jebej · 2020-04-29T14:57:23Z

@timholy I believe this is ready.

The 1 CI failure has been seen on master, e.g.
https://travis-ci.org/github/JuliaIO/MAT.jl/jobs/647798944
https://travis-ci.org/github/JuliaIO/MAT.jl/jobs/619762537
https://travis-ci.org/github/JuliaIO/MAT.jl/jobs/619738192

jebej · 2020-06-07T22:47:06Z

Small bump here: I think this is ready

jebej · 2020-07-07T22:11:05Z

@timholy or @simonster Would you mind merging this PR? I have been using it for the past months without issues. It would help with #128 and #133.

musm · 2020-09-04T04:10:25Z

@jebej sorry this got missed, thanks!

jebej · 2020-09-04T11:58:12Z

Thanks! ~~Presumably #133 & #128 can be closed.~~

Edit: nevermind

This is similar to JuliaIO#129 which returns `missing` for unhandled data types in HDF5 Datasets; function handles are stored as HDF5 groups, so a different explicit check is required to skip over them.

This is similar to #129 which returns `missing` for unhandled data types in HDF5 Datasets; function handles are stored as HDF5 groups, so a different explicit check is required to skip over them.

jebej force-pushed the patch-1 branch from 5dd9675 to ed85c4a Compare December 2, 2019 23:15

jebej force-pushed the patch-1 branch from ed85c4a to bc6dfe3 Compare December 2, 2019 23:30

jebej force-pushed the patch-1 branch from bc6dfe3 to 1a2bee6 Compare February 25, 2020 22:49

jebej and others added 3 commits June 8, 2020 12:25

Avoid error with MATLAB classes that we can't handle

2087930

Add test for MATLAB classes that can't be read

693ea0d

fix reading variable names in mfiles with a subsystem entry

661a2ed

jebej force-pushed the patch-1 branch from 613f158 to 661a2ed Compare June 8, 2020 16:25

This was referenced Jul 7, 2020

Can't correctly convert datetime type #133

Open

MAT.jl can't find 'MATLAB_class' #128

Open

musm merged commit 33123dc into JuliaIO:master Sep 4, 2020

jebej deleted the patch-1 branch September 4, 2020 11:56

jmert mentioned this pull request Nov 1, 2020

Skip over function handles in v7.3 format #146

Merged

musm mentioned this pull request Dec 8, 2020

Matlab datetime / H5Aopen(): unable to load attribute info from object header for attribute: 'MATLAB_class' JuliaIO/HDF5.jl#626

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid error with MATLAB classes that we can't handle #129

Avoid error with MATLAB classes that we can't handle #129

jebej commented Nov 26, 2019

timholy commented Nov 29, 2019

jebej commented Nov 30, 2019

timholy commented Dec 2, 2019

jebej commented Dec 2, 2019

jebej commented Dec 3, 2019

jebej commented Dec 3, 2019

jebej commented Dec 3, 2019

timholy commented Dec 3, 2019

jebej commented Feb 25, 2020

jebej commented Feb 25, 2020

jebej commented Feb 25, 2020

jebej commented Apr 28, 2020

jebej commented Apr 29, 2020

jebej commented Jun 7, 2020

jebej commented Jul 7, 2020

musm commented Sep 4, 2020

jebej commented Sep 4, 2020 •

edited

Loading

Avoid error with MATLAB classes that we can't handle #129

Avoid error with MATLAB classes that we can't handle #129

Conversation

jebej commented Nov 26, 2019

timholy commented Nov 29, 2019

jebej commented Nov 30, 2019

timholy commented Dec 2, 2019

jebej commented Dec 2, 2019

jebej commented Dec 3, 2019

jebej commented Dec 3, 2019

jebej commented Dec 3, 2019

timholy commented Dec 3, 2019

jebej commented Feb 25, 2020

jebej commented Feb 25, 2020

jebej commented Feb 25, 2020

jebej commented Apr 28, 2020

jebej commented Apr 29, 2020

jebej commented Jun 7, 2020

jebej commented Jul 7, 2020

musm commented Sep 4, 2020

jebej commented Sep 4, 2020 • edited Loading

jebej commented Sep 4, 2020 •

edited

Loading