NWB files failing dandi validate

[njmei]

Hi all,

I'm trying to troubleshoot why some NWB files containing intracellular ephys data are failing dandi-cli validation.

> pip freeze | grep -e pynwb -e hdmf -e h5py -e dandi
dandi==0.41.0
dandischema==0.7.1
h5py==3.7.0
hdmf==3.3.2
pynwb==2.0.1

> dandi validate /allen/programs/celltypes/production/humancelltypes/prod218/Ephys_Roi_Result_636991746/EPHYS_ATTACH_METADATA_QUEUE_1187479409/636991746_ephys.nwb
/home/nicholas.mei/miniconda3/envs/dandi-cli/lib/python3.7/site-packages/hdmf/utils.py:577: FutureWarning: DynamicTable.__init__: Using positional arguments for this method is discouraged and will be deprecated in a future major release. Please use keyword arguments to ensure future compatibility.
  warnings.warn(msg, FutureWarning)
2022-06-28 13:27:01,994 [   ERROR] /allen/programs/celltypes/production/humancelltypes/prod218/Ephys_Roi_Result_636991746/EPHYS_ATTACH_METADATA_QUEUE_1187479409/636991746_ephys.nwb: 2 error(s)
2022-06-28 13:27:01,994 [   ERROR]   VectorIndex/description (general/intracellular_ephys/sweep_table/series_index): argument missing
2022-06-28 13:27:01,995 [   ERROR]   VectorIndex (general/intracellular_ephys/sweep_table/series_index): incorrect type - expected 'uint8', got 'int32'
Summary: Validation errors in 1 out of 1 files
2022-06-28 13:27:01,996 [    INFO] Logs saved in /home/nicholas.mei/.cache/dandi-cli/log/20220628202650Z-82479.log

I would be happy to provide some examples of failing NWB files, though given file size limitations on github is there a preferred e-mail (or a slack, or other method) that DANDI maintainers would prefer I send the sample data?

• https://github.com/dandi/dandi-archive suggests [email protected]
• https://www.dandiarchive.org/handbook suggests [email protected]

Many thanks for the help!

[CodyCBakerPhD]

Hi there - can you try running the NWB Inspector, save the report file .txt, and upload that? Trying to distinguish if this is an io error or an invalidation, or something else entirely.

Great to hear you'd be willing to provide an example! Feel free to email/share one with me at [email protected]

Best,
Cody Baker
Sr. Neuro-data Scientist at CatalystNeuro

[CodyCBakerPhD]

Thanks for sharing the file and the report!

This is confirmed to be a 'technically' invalid NWB file - as for detailed reasoning of why this happens - validations are made one-to-one against the NWB schema itself, and as indicated the schema for the SweepTable it expects the description field to be specified. As for the int-typing issue, I'm not sure what the root of that might be.

Normally I would suggest raising an issue on PyNWB or MatNWB to try to get to the source of this issue with respect to your code used to run the conversion - however - this data type itself appears to be deprecated now. Thus I would recommend re-doing the conversion using the currently supported types shown in this nice tutorial, such as the IntracellularRecordingsTable.

Sorry for your trouble uploading to DANDI, hopefully we can resolve this quickly and easily!

Best,
Cody

[yarikoptic]

How (with pynwb or matnwb?) those files were produced?

I searched up seeing those validation errors in the past, e.g. NeurodataWithoutBorders/pynwb#1341 and I think it was due to those files being produced with some older versions of pynwb (or corresponding nwb extension).

[njmei]

Hi @yarikoptic the issue I encountered would be resolved by: dandi/dandi-cli#1036

These particular failing NWBs were initially generated by https://github.com/AllenInstitute/MIES.
I suspect the older schema versions are due to MIES using IPNWB (specifically the failing sweep_table is probably generated HERE)

Adding to the complicated picture, the NWB files generated by MIES are additionally modified by IPFX which adds spikes/metadata to the NWB files. It looks like IPFX (because it uses pynwb, whereas MIES does not) had newer versions of core and hdmf-common schemas. I put in a quickfix yesterday to not have IPFX cache its schema specs when modifying the original MIES NWB files.