Issues with dandi organize

[TheChymera]

I'm trying to follow this guide to upload some BIDS-NWB data to DANDI.
The NWB validation seems to find no critical errors:

It seems dandi organize edits the source dataset:

• removing the session intermediary directory (I guess this could make sense) but it seems a bit weird for DANDI to make decisions about the data hierarchy on its own
• removing the dataset_description.json

Demo:

[deco]~/.local/share/dandi/000546 ❱ dandi organize ~/.local/share/datalad/vStr_phase_stim/bids/
2023-05-30 03:08:31,168 [    INFO] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-05-30 03:08:31,168 [    INFO] NumExpr defaulting to 8 threads.
2023-05-30 03:08:32,560 [    INFO] Loading metadata from 1 files
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.
[Parallel(n_jobs=-1)]: Done   1 tasks      | elapsed:    1.9s
[Parallel(n_jobs=-1)]: Done   1 out of   1 | elapsed:    1.9s finished
2023-05-30 03:08:34,460 [    INFO] Symlink support autodetected; setting files_mode='symlink'
2023-05-30 03:08:34,467 [    INFO] Organized 1 paths. Visit /run/media/chymera/data2/dandi/000546/
2023-05-30 03:08:34,467 [    INFO] Logs saved in /home/chymera/.cache/dandi-cli/log/20230530070830Z-28068.log
[deco]~/.local/share/dandi/000546 ❱ tree
.
├── dandiset.yaml
└── sub-M322
    └── sub-M322_ecephys.nwb -> /home/chymera/.local/share/datalad/vStr_phase_stim/bids/sub-M322/ses-20220722/ieeg/sub-M322_ses-20220722_task-rest_run-01_ieeg.nwb

2 directories, 2 files
[deco]~/.local/share/dandi/000546 ❱ tree ~/.local/share/datalad/vStr_phase_stim/bids/
/home/chymera/.local/share/datalad/vStr_phase_stim/bids/
├── dataset_description.json
├── README.md
└── sub-M322
    └── ses-20220722
        └── ieeg
            └── sub-M322_ses-20220722_task-rest_run-01_ieeg.nwb -> ../../../../.git/annex/objects/KK/4V/MD5E-s7785862944--2851c1c5f6458d2d8431fa8a3eec6116.nwb/MD5E-s7785862944--2851c1c5f6458d2d8431fa8a3eec6116.nwb

4 directories, 3 files

Is this something we want to do? If so, why, if not, how can I prevent this from happening?

[TheChymera]

@yarikoptic if you have any ideas :)

[yarikoptic]

• removing the session intermediary directory (I guess this could make sense) but it seems a bit weird for DANDI to make decisions about the data hierarchy on its own

dandi organize never made promise to provide BIDS-compliant datasets. Before BIDS became more or less appropriate for the electrophys data at hands, we had to provide some consistency for the data to be uploaded. So dandi organize came about as a way to organize into BIDS-inspired, but not compliant (there is no session subfolders, no side car or dataset_description.json etc files) DANDI layout solely based on metadata in .nwb files. Work is still ongoing to formalize [BIDS Extension Proposal 032:Animal electrophysiology (ephys)] (https://bids.neuroimaging.io/bep032). I would invite you to join the effort:

• participate in that BIDS BEP (meetings are pretty regular)
  • there is currently ongoing effort to convert some of the sample datasets into BIDS, so your data could be another nice sample
  • if, similarly to how we did for DANDI, you could provide a branch with changes to BIDS schema to accommodate current BEP032 changes -- would be great, since we could then validate it.
• see how well (is it sufficient?) it represents your data
• dandi organize might as well get an option --bids [full,data-only,meta-only] to do the (re)organization into BIDS (well, BEP032) compliant way.

I guess we should also make it even more explicit and also extend documentation on that in e.g. https://www.dandiarchive.org/handbook/30_data_standards/ . Could you please propose a PR for https://github.com/dandi/handbook ?

[yarikoptic]

another possibly related to unification of DANDI layout and BIDS, on the topic of session folders (which might have little sense for electrophys data since would be one file per session folder, and many sessions) is a "mental exercise" I have done in bids-standard/bids-specification#751 (comment) to possibly make BIDS (2.0?) to be more flexible - as to allow and arbitrary (but clearly specified)) layering of folders - not just sub-/[ses-/]. (I filed bids-standard/bids-2-devel#54 on that regard now)

[TheChymera]

Ok so apparently dandi organize is not required. I think I should edit the guide.

[TheChymera]

Solved as of dandi/handbook#93