"Completely empty record" warning when trying to update dandiset files

[mo-osman]

If I would like to update an existing dandiset that's currently online as a draft (i.e. this one), is that possible? I was expecting to be able to download the dataset with dandi download https://dandiarchive.org/dandiset/000251/draft, call dandi organize <source folder> in the downloaded directory to overwrite the current version of each file with an updated version, and then use dandi upload to update the online data.

The trouble seems to be that for each file in <source folder>, I get the following sort of warning when dandi organize is called:
2023-04-21 00:40:38,899 [ WARNING] Completely empty record for <source folder>/<file name>.nwb

I suspect the problem is that the file names I use in <source folder> don't match those in the dandiset. However, the names used in the dandiset were generated by calling dandi organize on <source folder>. So I'm a bit stuck, and any feedback would be greatly appreciated.

[yarikoptic]

I am afraid that it is the same issue as dandi/dandi-cli#1266 or dandi/dandi-cli#1270 some incarnation of inability to get metadata or finding external links in the file. Could you please try to rerun organize command as was asked in dandi/dandi-cli#1270 (comment) i.e. like DANDI_CACHE=ignore DANDI_DEVEL=1 dandi -l 1 organize --devel-debug ... and see what you find in the logs as a warning etc.

[mo-osman]

Thanks for the helpful suggestion, Yaroslav! I was able to resolve the issue by updating h5py, after getting the following error message in debug mode:
2023-04-21 10:19:55,780 [ INFO] Loading metadata from 1 files /opt/miniconda3/envs/snub/lib/python3.8/site-packages/dandi/pynwb_utils.py:530: H5pyDeprecationWarning: The default file mode will change to 'r' (read-only) in h5py 3.0. To suppress this warning, pass the mode you need to h5py.File(), or set the global default h5.get_config().default_file_mode, or set the environment variable H5PY_DEFAULT_READONLY=1. Available modes are: 'r', 'r+', 'w', 'w-'/'x', 'a'. See the docs for details. with open_readable(filepath) as f, h5py.File(f) as fp: 2023-04-21 10:19:56,163 [ DEBUG] Failed to get metadata for ../nwb_conversion/Ca_VS_MB/Ca_VS_MB_1.nwb: File not open for writing 2023-04-21 10:19:56,163 [ WARNING] Failed to load metadata for 1 out of 1 files 2023-04-21 10:19:56,164 [ WARNING] Completely empty record for ../nwb_conversion/Ca_VS_MB/Ca_VS_MB_1.nwb 2023-04-21 10:19:56,164 [ DEBUG] Caught exception 1 out of 1 files were found not containing all necessary metadata: ../nwb_conversion/Ca_VS_MB/Ca_VS_MB_1.nwb

[yarikoptic]

interesting -- I wonder why it wanted to have it opened for writing. Did you use --update-external-file-paths option?

meanwhile -- let's make it explicitly read-only where we expect it such: we did dandi/dandi-cli#1282 and released it.