BIDS: does not upload anything BIDS if not explicitly specifying and --allow-any-path

[yarikoptic]

I thought that #1011 should have made it magically happened to use our dandi.utils.find_files (while excluding VCS) if bids dataset is detected, or may be just "valid bids" paths...

but nothing like that happens -- nothing gets uploaded unless I explicitly use --allow-any-paths in DANDI_DEVEL=1 mode:

(git)lena:~/proj/bids/bids-examples[master]ds210
$> dandi upload -i dandi-staging       
2022-07-08 19:52:39,208 [ WARNING] BIDSVersion 1.0.2 is less than the minimal working 1.7.0+012+dandi001. Falling back to 1.7.0+012+dandi001. To force the usage of earlier versions specify them explicitly when calling the validator.
2022-07-08 19:52:39,924 [   ERROR] The `.*?/README$` regex pattern file required by BIDS was not found.
2022-07-08 19:52:39,925 [   ERROR] The `.*?/CHANGES$` regex pattern file required by BIDS was not found.
2022-07-08 19:52:39,925 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-cuedSGT_echo-1_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:39,925 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-cuedSGT_echo-2_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:39,925 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-cuedSGT_echo-3_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:39,925 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-rest_echo-1_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:39,925 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-rest_echo-2_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:39,925 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-rest_echo-3_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:40,494 [    INFO] Found 1 files to consider
PATH                 SIZE    ERRORS UPLOAD STATUS    MESSAGE                  
dandiset.yaml        1.1 kB                skipped   should be edited online  
Summary:             1.1 kB                1 skipped 1 should be edited online
2022-07-08 19:52:40,603 [    INFO] Logs saved in /home/yoh/.cache/dandi-cli/log/20220708235237Z-501498.log
changes on filesystem: 226 changes                                                                                                                                                                                   
(dev3) 1 36354.....................................:Fri 08 Jul 2022 07:52:40 PM EDT:.
(git)lena:~/proj/bids/bids-examples[master]ds210
$> dandi upload -i dandi-staging sub-02
2022-07-08 19:52:46,570 [ WARNING] BIDSVersion 1.0.2 is less than the minimal working 1.7.0+012+dandi001. Falling back to 1.7.0+012+dandi001. To force the usage of earlier versions specify them explicitly when calling the validator.
2022-07-08 19:52:47,312 [   ERROR] The `.*?/README$` regex pattern file required by BIDS was not found.
2022-07-08 19:52:47,313 [   ERROR] The `.*?/CHANGES$` regex pattern file required by BIDS was not found.
2022-07-08 19:52:47,313 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-cuedSGT_echo-1_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:47,313 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-cuedSGT_echo-2_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:47,313 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-cuedSGT_echo-3_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:47,313 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-rest_echo-1_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:47,313 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-rest_echo-2_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:47,313 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-rest_echo-3_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:47,830 [    INFO] Found 0 files to consider
2022-07-08 19:52:47,834 [    INFO] Logs saved in /home/yoh/.cache/dandi-cli/log/20220708235244Z-501606.log
changes on filesystem: 226 changes                                                                                                                                                                                   
(dev3) 1 36355.....................................:Fri 08 Jul 2022 07:52:48 PM EDT:.
(git)lena:~/proj/bids/bids-examples[master]ds210
$> DANDI_DEVEL=1 dandi upload -i dandi-staging --allow-any-path sub-02
2022-07-08 19:52:58,673 [ WARNING] BIDSVersion 1.0.2 is less than the minimal working 1.7.0+012+dandi001. Falling back to 1.7.0+012+dandi001. To force the usage of earlier versions specify them explicitly when calling the validator.
2022-07-08 19:52:59,423 [   ERROR] The `.*?/README$` regex pattern file required by BIDS was not found.
2022-07-08 19:52:59,424 [   ERROR] The `.*?/CHANGES$` regex pattern file required by BIDS was not found.
2022-07-08 19:52:59,424 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-cuedSGT_echo-1_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:59,424 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-cuedSGT_echo-2_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:59,424 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-cuedSGT_echo-3_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:59,424 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-rest_echo-1_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:59,424 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-rest_echo-2_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:52:59,424 [ WARNING] The `/home/yoh/proj/bids/bids-examples/ds210/task-rest_echo-3_bold.json` file was not matched by any regex schema entry.
2022-07-08 19:53:00,099 [    INFO] Found 20 files to consider
PATH                                                      SIZE      ERRORS UPLOAD STATUS                MESSAGE
sub-02/func/sub-02_task-cuedSGT_run-01_echo-1_bold.nii.gz 60 Bytes    0           initiating upload            
sub-02/func/sub-02_task-cuedSGT_run-01_echo-2_bold.nii.gz 60 Bytes    0           initiating upload            
sub-02/func/sub-02_task-cuedSGT_run-01_echo-3_bold.nii.gz 60 Bytes    0           initiating upload            
sub-02/func/sub-02_task-cuedSGT_run-01_physio.tsv.gz      96.0 kB     0           initiating upload            
sub-02/func/sub-02_task-cuedSGT_run-02_echo-1_bold.nii.gz 60 Bytes    0           initiating upload            
sub-02/func/sub-02_task-cuedSGT_run-02_echo-2_bold.nii.gz                                                      

[TheChymera]

@yarikoptic

Ok, so I tracked this down. The issue I assume is the datasets lack the CHANGES file which BIDS insists is mandatory. The fact that the BIDS reference data is invalid as per the BIDS schema is presumably an artifact of the reference data being linked to the JS validator and not the new schema-synced and schema-loading validator.

The “error” (which is not an error, it's a proper validation failure) is caused by the following lines, where if validation is not set to “ignore” the dataset will not be detected as valid BIDS:

dandi-cli/dandi/upload.py

Lines 451 to 452 in 97fe900

	allow_missing_files=validation == "ignore",
	allow_invalid_filenames=validation == "ignore",

Additionally, the dataset you are trying to validate uses inheritance, for which support is not yet merged upstream (and we are not yet loading the upstream code anyway):
bids-standard/bids-specification#1120 (comment)

To confirm this, you can run:

chymera@decohost ~/src/bids-examples $ touch asl003/dandiset.yaml
chymera@decohost ~/src/bids-examples $ dandi upload --validation=ignore -i dandi-staging asl003/
chymera@decohost ~/src/bids-examples $ touch asl003/CHANGES
chymera@decohost ~/src/bids-examples $ dandi upload -i dandi-staging asl003/

This should work.

There are bids-examples datasets which have CHANGES, but by a funny coincidence, they all use inheritance, meaning that our validator would still fail.

We have the following options:

1. The best option: convince BIDS to drop the mandatory presence of CHANGES. There is no need for this to be mandatory.
2. The second-best option: create empty CHANGES files in all our archives.
3. The meh option: accept missing files by default (i.e. without validation="ignore").
4. The absolute worst: hard-code that CHANGES is not actually required.

[satra]

According to the spec https://bids-specification.readthedocs.io/en/stable/03-modality-agnostic-files.html#changes it does not appear to be mandatory.

…

On Tue, Jul 19, 2022, 3:06 AM Horea Christian ***@***.***> wrote: Ok, so I tracked this down. The issue I assume is the datasets lack the CHANGES file which BIDS insists is mandatory. The fact that the BIDS reference data is invalid as per the BIDS schema is presumably an artifact of the reference data being linked to the JS validator and not the new schema-synced and schema-loading validator. The “error” (which is not an error, it's a proper validation failure) is caused by the following lines, where if validation is not set to “ignore” the dataset will not be detected as valid BIDS: https://github.com/dandi/dandi-cli/blob/97fe900730c4d3723ae094402d704505c8677826/dandi/upload.py#L451-L452 Additionally, the dataset you are trying to validate, uses inheritance, for which support is not yet merged upstream (and we are not yet using the upstream code anyway): bids-standard/bids-specification#1120 (comment) <bids-standard/bids-specification#1120 (comment)> To confirm this, you can run: ***@***.*** ~/src/bids-examples $ touch asl003/dandiset.yaml ***@***.*** ~/src/bids-examples $ dandi upload --validation=ignore -i dandi-staging asl003/ ***@***.*** ~/src/bids-examples $ touch asl003/CHANGES ***@***.*** ~/src/bids-examples $ dandi upload -i dandi-staging asl003/ This should work. There are bids-examples datasets which have CHANGES, but by a funny coincidence, they all use inheritance, meaning that our validator would still fail. We have the following options: 1. The best option, convince BIDS to drop the mandatory presence of CHANGES. There is no need for this to be mandatory. 2. The second-best option, create empty CHANGES files on all our archives. 3. The worst, hard-code that CHANGES is not actually required. — Reply to this email directly, view it on GitHub <#1055 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABM574HOFTAOVXMPCLM54TVUZHWLANCNFSM53COAGXA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[yarikoptic]

nicely done wise @satra, indeed I do not think it is required -- I filed bids-standard/bids-specification#1151 .

@TheChymera the point then is to alert the user that "I have found BIDS dataset at location {path} but it is not valid ({len(errors)} errors detected) so it will not be considered as BIDS dataset for download" in case of --validation is not skip or ignore. If it is skip or ignore -- BIDS dataset, even if invalid, should get its files uploaded.

[TheChymera]

@yarikoptic I think this works pending #1164 , since I've been uploading datasets as part of the ls debugging all day. Do you want to test it locally as well, or can I close?

[TheChymera]

haven't had any issues with upload, and we have extensive tests. Given no reply to the message above I'll close this. Feel free to reopen and ping me if you encounter this again.