Fail to create HDF5 output file on 64bit Ubuntu 14.04 LTS

[starrynight]

I've added following to the basic Off-the-shelf SGD for classification
layers {
name: "hdf5out"
type: HDF5_OUTPUT
bottom: "fc1"
bottom: "label"
hdf5_output_param {
file_name: "output.h5"
}
}

However, the HDF5 file creation always fails. Error message :
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
#000: H5F.c line 1512 in H5Fcreate(): unable to create file
major: File accessibilty
minor: Unable to open file
#1: H5F.c line 1322 in H5F_open(): unable to truncate a file which is already open
major: File accessibilty
minor: Unable to open file
F0930 21:00:09.435128 10359 hdf5_output_layer.cpp:21] Check failed: file_id_ >= 0 (-1 vs. 0) Failed to open HDF5 filepred.h5
*** Check failure stack trace: ***
@ 0x7f61f4754daa (unknown)
@ 0x7f61f4754ce4 (unknown)
@ 0x7f61f47546e6 (unknown)
@ 0x7f61f4757687 (unknown)
@ 0x4e7c6d caffe::HDF5OutputLayer<>::HDF5OutputLayer()
@ 0x488368 caffe::GetLayer<>()
@ 0x475b16 caffe::Net<>::Init()
@ 0x47771e caffe::Net<>::Net()
@ 0x499620 caffe::Solver<>::InitTestNets()
@ 0x499f2e caffe::Solver<>::Init()
@ 0x49a086 caffe::Solver<>::Solver()
@ 0x41a050 caffe::GetSolver<>()
@ 0x416474 train()
@ 0x410e61 main
@ 0x7f61f020fec5 (unknown)
@ 0x414fc7 (unknown)
@ (nil) (unknown)

I've tried both prebuilt package libhdf5-serial-dev and manually building latest HDFS 1.8.13 withut any luck.

Can you take a look?

[rodrigob]

#1: H5F.c line 1322 in H5F_open(): unable to truncate a file which is already open would point out to issues with read/write permissions, or an existing file.
Can you try changing the output filename ? And make sure you are trying to write in a folder you have permissions ?

Finally, there should be a unit test for the HDF5_OUTPUT layer. You might want to check that this unit test passes on your machine.

[starrynight]

It seems to be a permission issue. I'm writing to a folder of my own with -rw-rw-r-- permission.
The pred.h5 file was created by caffe but caffe still reported the above error. For the time being, I added a sticky bit to the folder and seems to be able to fix the issue. But it feels like a subprocess of caffe didn't inherit correct permission from its parent. Btw, the unit test was run and failed for the same reason.

[shelhamer]

Sounds like a weird libhdf5 issue. Try installing this dependency from another source.

Please ask usage and installation questions on the caffe-users mailing list. Thanks!

[prclibo]

@starrynight hi I met the same issue and chmod-ing the folder to be 777 doesn't fix it. Could you let me know you fix. Thank you!

[Tgaaly]

Also facing the same issue, any ideas on how to fix this?

[Tgaaly]

found a work around:
you have to create the .h5 file manually and then change line 19 of hdf5_output_layer.cpp to below. HDF5 exposes another function call, called H5Fopen(). After doing this do 'make all' for caffe and then it should work. Unfortunately if you want to re-run you have to delete the file and recreate it or else HDF5 throws another error. This is awkward but its a temporary fix.

file_id_ = H5Fopen(file_name_.c_str(), H5F_ACC_RDWR, H5P_DEFAULT);