Tests for `ls` reinstated, underlying function fixed, support for ZARR-BIDS files added. #1164

TheChymera · 2022-11-23T18:25:34Z

Closes #1038.

Closes #1097.

codecov · 2022-11-23T18:40:51Z

Codecov Report

Base: 88.26% // Head: 88.43% // Increases project coverage by +0.16% 🎉

Coverage data is based on head (11c18f5) compared to base (a0b0015).
Patch coverage: 82.35% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1164      +/-   ##
==========================================
+ Coverage   88.26%   88.43%   +0.16%     
==========================================
  Files          73       73              
  Lines        8810     8878      +68     
==========================================
+ Hits         7776     7851      +75     
+ Misses       1034     1027       -7

Flag	Coverage Δ
unittests	`88.43% <82.35%> (+0.16%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
dandi/validate.py	`97.72% <ø> (ø)`
dandi/metadata.py	`87.46% <68.00%> (+0.44%)`	⬆️
dandi/cli/cmd_ls.py	`75.80% <83.33%> (+0.23%)`	⬆️
dandi/cli/tests/test_cmd_ls.py	`100.00% <100.00%> (ø)`
dandi/files/bids.py	`94.95% <100.00%> (+1.01%)`	⬆️
dandi/tests/test_upload.py	`100.00% <100.00%> (ø)`
dandi/validate_types.py	`100.00% <100.00%> (ø)`
dandi/support/threaded_walk.py	`93.10% <0.00%> (-1.73%)`	⬇️
dandi/download.py	`88.04% <0.00%> (-0.34%)`	⬇️
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheChymera · 2022-11-23T23:24:57Z

I'm a bit stuck on a cannot create weak reference to PosixPath object error which I get when trying to instantiate a BIDS file object via dandi.files.dandi_file()... I'm using the function in the same way it's used inside dandi.files.find_dandi_files(), and it works up to the return line of dandi_file().

(dev) chymera@decohost ~/src/bids-examples $ DANDI_CACHE=ignore dandi ls asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
jsfkfjskl
99999999999
00000000000000000
ſſſſſſſſſſſſſſſſſſſſ
asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
<class 'str'>
<class 'pathlib.PosixPath'>
<class 'pathlib.PosixPath'>
<class 'pathlib.PosixPath'>
hhhhhhhhhhhh
- errors: 1
  path: asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
  size: 21785
2022-11-23 18:17:33,886 [ WARNING] Failed to operate on some paths (empty records were listed):
 TypeError: 1 paths
 OSError: 1 paths
2022-11-23 18:17:33,887 [    INFO] Logs saved in /home/chymera/.cache/dandi-cli/log/20221123231731Z-16828.log
(dev) chymera@decohost ~/src/bids-examples $ cat $(ls /home/chymera/.cache/dandi-cli/log/20221123*.log | tail -1)
2022-11-23T18:17:31-0500 [INFO    ] dandi 16828:140549070866240 dandi v0.46.6+155.g5b5eac0.dirty, hdmf v3.4.6, pynwb v2.2.0, h5py v3.7.0
2022-11-23T18:17:31-0500 [INFO    ] dandi 16828:140549070866240 sys.argv = ['/home/chymera/src/dandi-cli/venvs/dev/bin/dandi', 'ls', 'asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz']
2022-11-23T18:17:31-0500 [INFO    ] dandi 16828:140549070866240 os.getcwd() = /home/chymera/src/bids-examples
2022-11-23T18:17:31-0500 [DEBUG   ] urllib3.connectionpool 16828:140549070866240 Starting new HTTPS connection (1): rig.mit.edu:443
2022-11-23T18:17:32-0500 [DEBUG   ] urllib3.connectionpool 16828:140549070866240 https://rig.mit.edu:443 "GET /et/projects/dandi/dandi-cli HTTP/1.1" 200 579
2022-11-23T18:17:32-0500 [DEBUG   ] dandi 16828:140549070866240 Running a newer version (0.46.6+155.g5b5eac0.dirty) of dandi/dandi-cli than available (0.46.6)
2022-11-23T18:17:32-0500 [DEBUG   ] h5py._conv 16828:140549070866240 Creating converter from 7 to 5
2022-11-23T18:17:32-0500 [DEBUG   ] h5py._conv 16828:140549070866240 Creating converter from 5 to 7
2022-11-23T18:17:32-0500 [DEBUG   ] h5py._conv 16828:140549070866240 Creating converter from 7 to 5
2022-11-23T18:17:32-0500 [DEBUG   ] h5py._conv 16828:140549070866240 Creating converter from 5 to 7
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'zlib'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'gzip'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'bz2'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'lzma'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'blosc'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'zstd'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'lz4'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'astype'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'delta'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'quantize'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'fixedscaleoffset'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'packbits'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'categorize'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'pickle'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'base64'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'shuffle'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'bitround'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'crc32'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'adler32'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'json2'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'vlen-utf8'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'vlen-bytes'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'vlen-array'
2022-11-23T18:17:32-0500 [DEBUG   ] numcodecs 16828:140549070866240 Registering codec 'n5_wrapper'
2022-11-23T18:17:33-0500 [DEBUG   ] dandi 16828:140549070866240 Problem obtaining metadata for asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz: cannot create weak reference to 'PosixPath' object
2022-11-23T18:17:33-0500 [DEBUG   ] dandi 16828:140549070866240 Problem obtaining metadata for asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz: Unable to open file (file signature not found)
2022-11-23T18:17:33-0500 [WARNING ] dandi 16828:140549070866240 Failed to operate on some paths (empty records were listed):
 TypeError: 1 paths
 OSError: 1 paths
2022-11-23T18:17:33-0500 [INFO    ] dandi 16828:140549070866240 Logs saved in /home/chymera/.cache/dandi-cli/log/20221123231731Z-16828.log
(dev) chymera@decohost ~/src/bids-examples $ pushd ~/src/dandi-cli/
~/src/dandi-cli ~/src/bids-examples
(dev) chymera@decohost ~/src/dandi-cli $ git rev-parse HEAD
cecb2929a09b0463602f089dd44bb94c80622d13

@yarikoptic @jwodder any ideas what I'm doing wrong? (don't mind the rest of the code too much, I'm just trying to figure out why the object won't work, after that it should be easy since we already have a metadata getting function... well, a method, but should do the job).

lgtm-com · 2022-11-28T13:42:32Z

This pull request introduces 1 alert when merging 18397b8 into 0439caf - view on LGTM.com

new alerts:

1 for Module is imported more than once

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

lgtm-com · 2022-11-28T14:02:27Z

This pull request introduces 1 alert when merging 3367aac into 0439caf - view on LGTM.com

new alerts:

1 for Module is imported more than once

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

TheChymera · 2022-11-28T14:15:05Z

Ok, I think I figured the previous issue out, as well as a few more. In any case we'll need to manually add README* to the list of files passed to validation for the time being (it will no longer be mandatory in future bids). The way this is set up is a bit awkward, since the code for some reason fails early if files are missing... this makes it difficult to point the validator at a single file. The fix we have for the other mandatory file BIDS_DATASET_DESCRIPTION = "dataset_description.json" is also hard-coded here and independent of the schema.... and also does not cover other files which might possibly become mandatory in the future. Ofc it would be a bad idea for BIDS to make more files mandatory, but we may or may not be able to make sure more mandatorization (particularly of the conditional kind) doesn't happen. I think we have 2 choices

query the schema manually to determine what's mandatory and add all those files analogously to "dataset_description.json". Imho this is a very bad idea and we'll end up re-creating a lot of logic particularly because some files might have variable extensions.
somehow (?) make sure that when validation is pointed at a single file, missing file errors get removed from the dictionary returned by the validator. I think this solution would be much superior, but what with the object structure I'm not at all sure where this would be done. Probably at some single-file object instantiation but not BIDS dataset asset instantiation 🤔

@jwodder @yarikoptic more pressingly however (feel free to ignore the above, more for my reference) I am getting a weird error when the BareAsset is instantiated. Basically at return BareAsset(**metadata) in dandi/files/bids.py. No idea what to to about it or what's wrong, really. It says digest and then that it needs to have an etag... How/where/why do I add one? Is getting to this point an indicator that I'm doing something else wrong? some auto-generation step I skipped?

(dev) chymera@decohost ~/src/dandi-cli $ git rev-parse HEAD
3367aac6c0755357affbdcdd6cf2c85047e0a9ba
(dev) chymera@decohost ~/src/dandi-cli $ pushd /tmp
/tmp ~/src/dandi-cli ~
(dev) chymera@decohost /tmp $ git clone --branch dandidev [email protected]:TheChymera/bids-examples.git &> /dev/null
(dev) chymera@decohost /tmp $ cd bids-examples/
(dev) chymera@decohost /tmp/bids-examples $ DANDI_CACHE=ignore dandi ls asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
Debugging ls for the following path:  asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
000000000000000000000000000000000
ăăăăăăăăăăăăăăăăă
2022-11-28 09:11:36,656 [ WARNING] BIDSVersion `1.5.0` is less than the minimal working `schema`. Falling back to `schema`. To force the usage of earlier versions specify them explicitly when calling the validator.
qqqqqqqqqqqqqqqqqqqqqq get_asset_metadata
defaultdict(<class 'dict'>, {'dataset_description.json': {'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': []}, 'sub-Sub1/anat/sub-Sub1_T1w.nii.gz': {'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': [{'schemaKey': 'Participant', 'identifier': 'Sub1'}]}, 'README': {'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': []}})
{'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': [{'schemaKey': 'Participant', 'identifier': 'Sub1'}]}
zzzzzzzzzzzzzzzzzzzzzzzzz get_metadata
{'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': [{'schemaKey': 'Participant', 'identifier': 'Sub1'}], 'digest': {}, 'dateModified': datetime.datetime(2022, 11, 28, 9, 11, 37, 678441, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400), 'EST')), 'blobDateModified': datetime.datetime(2022, 11, 28, 9, 10, 48, 919152, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400), 'EST')), 'contentSize': 21785, 'wasGeneratedBy': [Activity(id='urn:uuid:8f8566eb-95c1-46fa-92c7-c741cdfebd69', schemaKey='Activity', identifier=None, name='Metadata generation', description='Metadata generated by DANDI cli', startDate=None, endDate=None, wasAssociatedWith=[Software(id=None, schemaKey='Software', identifier='RRID:SCR_019009', name='DANDI Command Line Interface', version='0.46.6+160.g3367aac', url=HttpUrl('https://github.com/dandi/dandi-cli', ))], used=None)], 'encodingFormat': 'application/gzip', 'path': 'sub-Sub1/anat/sub-Sub1_T1w.nii.gz'}
- errors: 1
  path: asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
  size: 21785
2022-11-28 09:11:37,684 [ WARNING] Failed to operate on some paths (empty records were listed):
 ValidationError: 1 paths
 OSError: 1 paths
2022-11-28 09:11:37,684 [    INFO] Logs saved in /home/chymera/.cache/dandi-cli/log/20221128141134Z-17368.log
(dev) chymera@decohost /tmp/bids-examples $ cat $(ls ~/.cache/dandi-cli/log/202211*.log | tail -1)
2022-11-28T09:11:34-0500 [INFO    ] dandi 17368:140251460831040 dandi v0.46.6+160.g3367aac, hdmf v3.4.6, pynwb v2.2.0, h5py v3.7.0
2022-11-28T09:11:34-0500 [INFO    ] dandi 17368:140251460831040 sys.argv = ['/home/chymera/src/dandi-cli/venvs/dev/bin/dandi', 'ls', 'asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz']
2022-11-28T09:11:34-0500 [INFO    ] dandi 17368:140251460831040 os.getcwd() = /tmp/bids-examples
2022-11-28T09:11:34-0500 [DEBUG   ] urllib3.connectionpool 17368:140251460831040 Starting new HTTPS connection (1): rig.mit.edu:443
2022-11-28T09:11:34-0500 [DEBUG   ] urllib3.connectionpool 17368:140251460831040 https://rig.mit.edu:443 "GET /et/projects/dandi/dandi-cli HTTP/1.1" 200 579
2022-11-28T09:11:34-0500 [DEBUG   ] dandi 17368:140251460831040 Running a newer version (0.46.6+160.g3367aac) of dandi/dandi-cli than available (0.46.6)
2022-11-28T09:11:35-0500 [DEBUG   ] h5py._conv 17368:140251460831040 Creating converter from 7 to 5
2022-11-28T09:11:35-0500 [DEBUG   ] h5py._conv 17368:140251460831040 Creating converter from 5 to 7
2022-11-28T09:11:35-0500 [DEBUG   ] h5py._conv 17368:140251460831040 Creating converter from 7 to 5
2022-11-28T09:11:35-0500 [DEBUG   ] h5py._conv 17368:140251460831040 Creating converter from 5 to 7
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'zlib'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'gzip'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'bz2'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'lzma'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'blosc'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'zstd'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'lz4'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'astype'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'delta'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'quantize'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'fixedscaleoffset'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'packbits'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'categorize'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'pickle'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'base64'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'shuffle'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'bitround'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'crc32'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'adler32'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'json2'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'vlen-utf8'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'vlen-bytes'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'vlen-array'
2022-11-28T09:11:35-0500 [DEBUG   ] numcodecs 17368:140251460831040 Registering codec 'n5_wrapper'
2022-11-28T09:11:36-0500 [WARNING ] bids-schema 17368:140251460831040 BIDSVersion `1.5.0` is less than the minimal working `schema`. Falling back to `schema`. To force the usage of earlier versions specify them explicitly when calling the validator.
2022-11-28T09:11:37-0500 [DEBUG   ] dandi 17368:140251460831040 Problem obtaining metadata for asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz: 1 validation error for BareAsset
digest
  A non-zarr asset must have a dandi-etag. (type=value_error)
2022-11-28T09:11:37-0500 [DEBUG   ] dandi 17368:140251460831040 Problem obtaining metadata for asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz: Unable to open file (file signature not found)
2022-11-28T09:11:37-0500 [WARNING ] dandi 17368:140251460831040 Failed to operate on some paths (empty records were listed):
 ValidationError: 1 paths
 OSError: 1 paths
2022-11-28T09:11:37-0500 [INFO    ] dandi 17368:140251460831040 Logs saved in /home/chymera/.cache/dandi-cli/log/20221128141134Z-17368.log

jwodder · 2022-11-28T14:34:43Z

@TheChymera An etag is a kind of file digest. You may be interested in this code block:

dandi-cli/dandi/cli/cmd_ls.py

Lines 350 to 359 in 0439caf

    
           if use_fake_digest: 
        
               digest = "0" * 32 + "-1" 
        
           else: 
        
               lgr.info("Calculating digest for %s", path) 
        
               digest = get_digest(path, digest="dandi-etag") 
        
           rec = nwb2asset( 
        
               path, 
        
               schema_version=schema, 
        
               digest=Digest.dandi_etag(digest), 
        
           ).json_dict()

TheChymera · 2022-11-28T17:39:18Z

@jwodder thank you :) The GenericBIDSAsset which I create in dandi/metadata.py seems to inherit from a class which has a .get_digest() method. Any idea why that doesn't happen? I tried to call it after I create the object but that doesn't do anything either. DANDI_DEVEL=1 DANDI_CACHE=ignore dandi ls --use-fake-digest asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz also does nothing, because the code block is not accessed for non-nwb files.

I looked up how NWB validation handles it, but it seems to simply add the dummy digest after calling get_metadata in dandi/metadata.py, whereas for me that function will not finalize. It seems NWB metadata extraction has a separate function in pynwb_utils.py and does not rely on a get_metadata() method of the asset object, and can therefore delay instantiating a BareAsset model until it does so explicitly after calculating the digest... should I somehow try to integrate the digest creation in the GenericBIDSAsset object so that it's always there?

jwodder · 2022-11-28T17:49:34Z

@TheChymera

The GenericBIDSAsset which I create in dandi/metadata.py seems to inherit from a class which has a .get_digest() method. Any idea why that doesn't happen?

Why what doesn't happen? The method doesn't mutate anything; it returns the Digest object, and then you have to supply that when creating the BareAsset. You may be interested in this code, which GenericsBIDSAsset inherits:

dandi-cli/dandi/files/bids.py

Lines 166 to 175 in 0439caf

    
           def get_metadata( 
        
               self, 
        
               digest: Optional[Digest] = None, 
        
               ignore_errors: bool = True, 
        
           ) -> BareAsset: 
        
               metadata = self.bids_dataset_description.get_asset_metadata(self) 
        
               start_time = end_time = datetime.now().astimezone() 
        
               add_common_metadata(metadata, self.filepath, start_time, end_time, digest) 
        
               metadata["path"] = self.path 
        
               return BareAsset(**metadata)

lgtm-com · 2022-11-28T18:07:30Z

This pull request introduces 1 alert when merging 8462051 into 0439caf - view on LGTM.com

new alerts:

1 for Module is imported more than once

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

TheChymera · 2022-11-28T19:21:01Z

@jwodder thank you, yes, indeed, no need to mutate anything 💛 (yet :P).

It's still trying to open the file for some reason... will continue digging to find out where exactly it happens, but maybe you already know? I think it might actually be somewhere in dandi-schema

(dev) chymera@decohost /tmp/bids-examples $ git branch && git remote -v &&  git rev-parse HEAD
* dandidev
origin	[email protected]:TheChymera/bids-examples.git (fetch)
origin	[email protected]:TheChymera/bids-examples.git (push)
f74561309d369c44a7db59f0d94640cf8bb1ccce
(dev) chymera@decohost /tmp/bids-examples $ pushd ~/src/dandi-cli/
~/src/dandi-cli /tmp/bids-examples ~/bids-examples
(dev) chymera@decohost ~/src/dandi-cli $  git rev-parse HEAD
8462051407d1375f63039ac856c1380e794a0459
(dev) chymera@decohost ~/src/dandi-cli $ popd
/tmp/bids-examples ~/bids-examples
(dev) chymera@decohost /tmp/bids-examples $ DANDI_CACHE=ignore dandi ls asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
Debugging ls for the following path:  asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
('species', 'number_of_electrodes', 'number_of_units', 'nwb_version', 'genotype', 'description', 'bids_schema_version', 'nd_types', 'identifier', 'subject_id', 'experimenter', 'lab', 'keywords', 'url', 'related_publications', 'slice_id', 'tissue_sample_id', 'number_of_cells', 'number_of_tissue_samples', 'session_start_time', 'doi', 'session_description', 'age', 'institution', 'number_of_subjects', 'session_id', 'experiment_description', 'organism', 'date_of_birth', 'cell_id', 'name', 'version', 'variables_measured', 'license', 'probe_ids', 'sex')
2022-11-28 14:17:54,623 [    INFO] Calculating digest for asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
000000000000000000000000000000000
ăăăăăăăăăăăăăăăăă
GenericBIDSAsset(filepath=PosixPath('/tmp/bids-examples/asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz'), path='sub-Sub1/anat/sub-Sub1_T1w.nii.gz', bids_dataset_description_ref=<weakref at 0x7f863cb08950; to 'BIDSDatasetDescriptionAsset' at 0x7f863cfe3280>) <class 'dandi.files.bids.GenericBIDSAsset'>
2022-11-28 14:17:54,687 [ WARNING] BIDSVersion `1.5.0` is less than the minimal working `schema`. Falling back to `schema`. To force the usage of earlier versions specify them explicitly when calling the validator.
qqqqqqqqqqqqqqqqqqqqqq get_asset_metadata
defaultdict(<class 'dict'>, {'dataset_description.json': {'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': []}, 'sub-Sub1/anat/sub-Sub1_T1w.nii.gz': {'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': [{'schemaKey': 'Participant', 'identifier': 'Sub1'}]}, 'README': {'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': []}})
{'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': [{'schemaKey': 'Participant', 'identifier': 'Sub1'}]}
zzzzzzzzzzzzzzzzzzzzzzzzz get_metadata
{'schemaKey': 'Asset', 'schemaVersion': '0.6.3', 'access': [{'schemaKey': 'AccessRequirements', 'status': 'dandi:OpenAccess'}], 'wasAttributedTo': [{'schemaKey': 'Participant', 'identifier': 'Sub1'}], 'digest': {<DigestType.dandi_etag: 'dandi:dandi-etag'>: 'cbbcfc14a95b9645ea85142007bdaf01-1'}, 'dateModified': datetime.datetime(2022, 11, 28, 14, 17, 55, 859658, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400), 'EST')), 'blobDateModified': datetime.datetime(2022, 11, 28, 9, 10, 48, 919152, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400), 'EST')), 'contentSize': 21785, 'wasGeneratedBy': [Activity(id='urn:uuid:8b40a0cd-01f6-4e7b-8a78-bcbdda787d60', schemaKey='Activity', identifier=None, name='Metadata generation', description='Metadata generated by DANDI cli', startDate=None, endDate=None, wasAssociatedWith=[Software(id=None, schemaKey='Software', identifier='RRID:SCR_019009', name='DANDI Command Line Interface', version='0.46.6+161.g8462051', url=HttpUrl('https://github.com/dandi/dandi-cli', ))], used=None)], 'encodingFormat': 'application/gzip', 'path': 'sub-Sub1/anat/sub-Sub1_T1w.nii.gz'}
ßßßßßßßßßßßßßßßßßßßßßßßß
id=None schemaKey='Asset' schemaVersion='0.6.3' name=None description=None contributor=None about=None studyTarget=None license=None protocol=None ethicsApproval=None keywords=None acknowledgement=None access=[AccessRequirements(id=None, schemaKey='AccessRequirements', status=<AccessType.OpenAccess: 'dandi:OpenAccess'>, contactPoint=None, description=None, embargoedUntil=None)] url=None repository=None relatedResource=None wasGeneratedBy=[Activity(id='urn:uuid:8b40a0cd-01f6-4e7b-8a78-bcbdda787d60', schemaKey='Activity', identifier=None, name='Metadata generation', description='Metadata generated by DANDI cli', startDate=None, endDate=None, wasAssociatedWith=[Software(id=None, schemaKey='Software', identifier='RRID:SCR_019009', name='DANDI Command Line Interface', version='0.46.6+161.g8462051', url=HttpUrl('https://github.com/dandi/dandi-cli', ))], used=None)] contentSize=21785 encodingFormat='application/gzip' digest={<DigestType.dandi_etag: 'dandi:dandi-etag'>: 'cbbcfc14a95b9645ea85142007bdaf01-1'} path='sub-Sub1/anat/sub-Sub1_T1w.nii.gz' dateModified=datetime.datetime(2022, 11, 28, 14, 17, 55, 859658, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400), 'EST')) blobDateModified=datetime.datetime(2022, 11, 28, 9, 10, 48, 919152, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400), 'EST')) dataType=None sameAs=None approach=None measurementTechnique=None variableMeasured=None wasDerivedFrom=None wasAttributedTo=[Participant(id=None, schemaKey='Participant', identifier='Sub1', altName=None, strain=None, cellLine=None, vendor=None, age=None, sex=None, genotype=None, species=None, disorder=None, relatedParticipant=None, sameAs=None)]
- errors: 1
  path: asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
  size: 21785
2022-11-28 14:17:55,866 [ WARNING] Failed to operate on some paths (empty records were listed):
 OSError: 1 paths
2022-11-28 14:17:55,867 [    INFO] Logs saved in /home/chymera/.cache/dandi-cli/log/20221128191752Z-26936.log
(dev) chymera@decohost /tmp/bids-examples $ cat $(ls ~/.cache/dandi-cli/log/202211*.log | tail -1)
2022-11-28T14:17:52-0500 [INFO    ] dandi 26936:140215203645248 dandi v0.46.6+161.g8462051, hdmf v3.4.6, pynwb v2.2.0, h5py v3.7.0
2022-11-28T14:17:52-0500 [INFO    ] dandi 26936:140215203645248 sys.argv = ['/home/chymera/src/dandi-cli/venvs/dev/bin/dandi', 'ls', 'asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz']
2022-11-28T14:17:52-0500 [INFO    ] dandi 26936:140215203645248 os.getcwd() = /tmp/bids-examples
2022-11-28T14:17:52-0500 [DEBUG   ] urllib3.connectionpool 26936:140215203645248 Starting new HTTPS connection (1): rig.mit.edu:443
2022-11-28T14:17:52-0500 [DEBUG   ] urllib3.connectionpool 26936:140215203645248 https://rig.mit.edu:443 "GET /et/projects/dandi/dandi-cli HTTP/1.1" 200 579
2022-11-28T14:17:52-0500 [DEBUG   ] dandi 26936:140215203645248 Running a newer version (0.46.6+161.g8462051) of dandi/dandi-cli than available (0.46.6)
2022-11-28T14:17:53-0500 [DEBUG   ] h5py._conv 26936:140215203645248 Creating converter from 7 to 5
2022-11-28T14:17:53-0500 [DEBUG   ] h5py._conv 26936:140215203645248 Creating converter from 5 to 7
2022-11-28T14:17:53-0500 [DEBUG   ] h5py._conv 26936:140215203645248 Creating converter from 7 to 5
2022-11-28T14:17:53-0500 [DEBUG   ] h5py._conv 26936:140215203645248 Creating converter from 5 to 7
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'zlib'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'gzip'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'bz2'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'lzma'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'blosc'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'zstd'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'lz4'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'astype'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'delta'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'quantize'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'fixedscaleoffset'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'packbits'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'categorize'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'pickle'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'base64'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'shuffle'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'bitround'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'crc32'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'adler32'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'json2'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'vlen-utf8'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'vlen-bytes'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'vlen-array'
2022-11-28T14:17:53-0500 [DEBUG   ] numcodecs 26936:140215203645248 Registering codec 'n5_wrapper'
2022-11-28T14:17:54-0500 [INFO    ] dandi 26936:140215203645248 Calculating digest for asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz
2022-11-28T14:17:54-0500 [WARNING ] bids-schema 26936:140215203645248 BIDSVersion `1.5.0` is less than the minimal working `schema`. Falling back to `schema`. To force the usage of earlier versions specify them explicitly when calling the validator.
2022-11-28T14:17:55-0500 [DEBUG   ] dandi 26936:140215203645248 Problem obtaining metadata for asl004/sub-Sub1/anat/sub-Sub1_T1w.nii.gz: Unable to open file (file signature not found)
2022-11-28T14:17:55-0500 [WARNING ] dandi 26936:140215203645248 Failed to operate on some paths (empty records were listed):
 OSError: 1 paths
2022-11-28T14:17:55-0500 [INFO    ] dandi 26936:140215203645248 Logs saved in /home/chymera/.cache/dandi-cli/log/20221128191752Z-26936.log

lgtm-com · 2022-11-28T22:18:22Z

This pull request introduces 1 alert when merging 61a4ba9 into 0439caf - view on LGTM.com

new alerts:

1 for Module is imported more than once

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

lgtm-com · 2022-11-28T23:21:41Z

This pull request introduces 1 alert when merging f087e81 into 0439caf - view on LGTM.com

new alerts:

1 for Module is imported more than once

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

lgtm-com · 2022-11-29T18:10:00Z

This pull request introduces 1 alert when merging 28d6237 into 0439caf - view on LGTM.com

new alerts:

1 for Module is imported more than once

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

TheChymera · 2022-11-29T18:49:34Z

ok, the reading error appears to be not-critical. I think I figured it out.

TheChymera · 2022-11-29T19:00:57Z

Other than print calls interfering with tests and possibly typing issues, b76809e should have ls working with bids files, but it still complains about *.zarr “files” since they're actually directories.

lgtm-com · 2022-11-29T19:03:28Z

This pull request introduces 1 alert when merging b76809e into 0439caf - view on LGTM.com

new alerts:

1 for Module is imported more than once

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

lgtm-com · 2022-11-30T08:29:35Z

This pull request introduces 1 alert when merging 960eea9 into a0b0015 - view on LGTM.com

new alerts:

1 for Module is imported more than once

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

TheChymera · 2022-11-30T08:33:55Z

@jwodder hm, ok, judging by what you said on slack and the current code as best I understand it, this should just work(tm). The BIDSFileFactory in dandi/metadata/_private.py auto-detects the BIDS asset file type and sets it correctly... we have zarr checksums already... everything seems to be there. Yet I still get the error about not having the dandi-etag from return BareAsset(**metadata).

(dev) chymera@decohost /tmp/bids-examples $ DANDI_CACHE=ignore dandi ls micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
Debugging ls for the following path:  micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
('bids_schema_version', 'date_of_birth', 'description', 'genotype', 'lab', 'related_publications', 'slice_id', 'experimenter', 'nd_types', 'nwb_version', 'sex', 'keywords', 'url', 'species', 'session_id', 'number_of_subjects', 'institution', 'probe_ids', 'number_of_cells', 'session_start_time', 'age', 'number_of_tissue_samples', 'organism', 'version', 'session_description', 'experiment_description', 'variables_measured', 'tissue_sample_id', 'name', 'subject_id', 'identifier', 'cell_id', 'number_of_electrodes', 'number_of_units', 'doi', 'license')
2022-11-30 03:24:54,012 [    INFO] Calculating digest for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
qqqqqqqqqqqqqqqqqqqqqqqqqqqqq
<class 'dandi.files.bids.ZarrBIDSAsset'>
2022-11-30 03:24:54,067 [ WARNING] BIDSVersion `1.7.0` is less than the minimal working `schema`. Falling back to `schema`. To force the usage of earlier versions specify them explicitly when calling the validator.
- path: micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
2022-11-30 03:24:55,126 [ WARNING] Failed to operate on some paths (empty records were listed):
 ValidationError: 1 paths
2022-11-30 03:24:55,126 [    INFO] Logs saved in /home/chymera/.cache/dandi-cli/log/20221130082451Z-10611.log
(dev) chymera@decohost /tmp/bids-examples $ cat $(ls ~/.cache/dandi-cli/log/202211*.log | tail -1)
2022-11-30T03:24:51-0500 [INFO    ] dandi 10611:140608089950016 dandi v0.46.6+166.g960eea9, hdmf v3.4.6, pynwb v2.2.0, h5py v3.7.0
2022-11-30T03:24:51-0500 [INFO    ] dandi 10611:140608089950016 sys.argv = ['/home/chymera/src/dandi-cli/venvs/dev/bin/dandi', 'ls', 'micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr']
2022-11-30T03:24:51-0500 [INFO    ] dandi 10611:140608089950016 os.getcwd() = /tmp/bids-examples
2022-11-30T03:24:51-0500 [DEBUG   ] urllib3.connectionpool 10611:140608089950016 Starting new HTTPS connection (1): rig.mit.edu:443
2022-11-30T03:24:52-0500 [DEBUG   ] urllib3.connectionpool 10611:140608089950016 https://rig.mit.edu:443 "GET /et/projects/dandi/dandi-cli HTTP/1.1" 200 579
2022-11-30T03:24:52-0500 [DEBUG   ] dandi 10611:140608089950016 Running a newer version (0.46.6+166.g960eea9) of dandi/dandi-cli than available (0.46.6)
2022-11-30T03:24:52-0500 [DEBUG   ] h5py._conv 10611:140608089950016 Creating converter from 7 to 5
2022-11-30T03:24:52-0500 [DEBUG   ] h5py._conv 10611:140608089950016 Creating converter from 5 to 7
2022-11-30T03:24:52-0500 [DEBUG   ] h5py._conv 10611:140608089950016 Creating converter from 7 to 5
2022-11-30T03:24:52-0500 [DEBUG   ] h5py._conv 10611:140608089950016 Creating converter from 5 to 7
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'zlib'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'gzip'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'bz2'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'lzma'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'blosc'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'zstd'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'lz4'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'astype'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'delta'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'quantize'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'fixedscaleoffset'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'packbits'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'categorize'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'pickle'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'base64'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'shuffle'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'bitround'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'crc32'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'adler32'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'json2'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'vlen-utf8'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'vlen-bytes'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'vlen-array'
2022-11-30T03:24:52-0500 [DEBUG   ] numcodecs 10611:140608089950016 Registering codec 'n5_wrapper'
2022-11-30T03:24:54-0500 [INFO    ] dandi 10611:140608089950016 Calculating digest for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
2022-11-30T03:24:54-0500 [DEBUG   ] dandi.support.digests 10611:140606890227392 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/.zgroup
2022-11-30T03:24:54-0500 [DEBUG   ] dandi.support.digests 10611:140606890227392 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/.zattrs
2022-11-30T03:24:54-0500 [DEBUG   ] dandi.support.digests 10611:140606873441984 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/0/.zarray
2022-11-30T03:24:54-0500 [DEBUG   ] dandi.support.digests 10611:140606873441984 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/0/0/0
2022-11-30T03:24:54-0500 [WARNING ] bids-schema 10611:140608089950016 BIDSVersion `1.7.0` is less than the minimal working `schema`. Falling back to `schema`. To force the usage of earlier versions specify them explicitly when calling the validator.
2022-11-30T03:24:55-0500 [DEBUG   ] dandi 10611:140608089950016 Problem obtaining metadata for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr: 1 validation error for BareAsset
digest
  Digest must have an appropriate dandi-etag value. Got dcc41f4785db06a414a81d5d973727b3-4--1032 (type=value_error)
2022-11-30T03:24:55-0500 [WARNING ] dandi 10611:140608089950016 Failed to operate on some paths (empty records were listed):
 ValidationError: 1 paths
2022-11-30T03:24:55-0500 [INFO    ] dandi 10611:140608089950016 Logs saved in /home/chymera/.cache/dandi-cli/log/20221130082451Z-10611.log
(dev) chymera@decohost /tmp/bids-examples $ git branch && git remote -v &&  git rev-parse HEAD
* dandidev
origin	[email protected]:TheChymera/bids-examples.git (fetch)
origin	[email protected]:TheChymera/bids-examples.git (push)
f74561309d369c44a7db59f0d94640cf8bb1ccce
(dev) chymera@decohost /tmp/bids-examples $ pushd ~/src/dandi-cli/
~/src/dandi-cli /tmp/bids-examples ~/bids-examples
(dev) chymera@decohost ~/src/dandi-cli $ git rev-parse HEAD
960eea9541851266c99f12827dbc989e45858b11

Any ideas?

dandi/metadata.py

dandi/cli/cmd_ls.py

lgtm-com · 2022-12-01T00:37:16Z

This pull request introduces 2 alerts when merging a754bf1 into a0b0015 - view on LGTM.com

new alerts:

1 for Module is imported more than once
1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

dandi/cli/cmd_ls.py

lgtm-com · 2022-12-01T01:05:31Z

This pull request introduces 1 alert when merging adbf033 into a0b0015 - view on LGTM.com

new alerts:

1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

TheChymera · 2022-12-01T01:07:12Z

@jwodder thanks again, and sorry for pinging you so often on this :/ Still having issues, though:

(dev) chymera@decohost /tmp/bids-examples $ cat $(ls ~/.cache/dandi-cli/log/202211*.log | tail -1)
2022-11-30T03:32:21-0500 [INFO    ] dandi 11532:139778614024000 dandi v0.46.6+166.g960eea9.dirty, hdmf v3.4.6, pynwb v2.2.0, h5py v3.7.0
2022-11-30T03:32:21-0500 [INFO    ] dandi 11532:139778614024000 sys.argv = ['/home/chymera/src/dandi-cli/venvs/dev/bin/dandi', 'ls', 'micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr']
2022-11-30T03:32:21-0500 [INFO    ] dandi 11532:139778614024000 os.getcwd() = /tmp/bids-examples
2022-11-30T03:32:21-0500 [DEBUG   ] urllib3.connectionpool 11532:139778614024000 Starting new HTTPS connection (1): rig.mit.edu:443
2022-11-30T03:32:21-0500 [DEBUG   ] urllib3.connectionpool 11532:139778614024000 https://rig.mit.edu:443 "GET /et/projects/dandi/dandi-cli HTTP/1.1" 200 579
2022-11-30T03:32:22-0500 [DEBUG   ] dandi 11532:139778614024000 Running a newer version (0.46.6+166.g960eea9.dirty) of dandi/dandi-cli than available (0.46.6)
2022-11-30T03:32:22-0500 [DEBUG   ] h5py._conv 11532:139778614024000 Creating converter from 7 to 5
2022-11-30T03:32:22-0500 [DEBUG   ] h5py._conv 11532:139778614024000 Creating converter from 5 to 7
2022-11-30T03:32:22-0500 [DEBUG   ] h5py._conv 11532:139778614024000 Creating converter from 7 to 5
2022-11-30T03:32:22-0500 [DEBUG   ] h5py._conv 11532:139778614024000 Creating converter from 5 to 7
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'zlib'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'gzip'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'bz2'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'lzma'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'blosc'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'zstd'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'lz4'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'astype'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'delta'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'quantize'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'fixedscaleoffset'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'packbits'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'categorize'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'pickle'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'base64'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'shuffle'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'bitround'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'crc32'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'adler32'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'json2'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'vlen-utf8'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'vlen-bytes'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'vlen-array'
2022-11-30T03:32:22-0500 [DEBUG   ] numcodecs 11532:139778614024000 Registering codec 'n5_wrapper'
2022-11-30T03:32:23-0500 [INFO    ] dandi 11532:139778614024000 Calculating digest for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
2022-11-30T03:32:23-0500 [DEBUG   ] dandi.support.digests 11532:139777416279744 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/.zgroup
2022-11-30T03:32:23-0500 [DEBUG   ] dandi.support.digests 11532:139777416279744 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/.zattrs
2022-11-30T03:32:23-0500 [DEBUG   ] dandi.support.digests 11532:139777407887040 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/0/.zarray
2022-11-30T03:32:23-0500 [DEBUG   ] dandi.support.digests 11532:139777407887040 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/0/0/0
2022-11-30T03:32:23-0500 [WARNING ] bids-schema 11532:139778614024000 BIDSVersion `1.7.0` is less than the minimal working `schema`. Falling back to `schema`. To force the usage of earlier versions specify them explicitly when calling the validator.
2022-11-30T03:32:24-0500 [DEBUG   ] dandi 11532:139778614024000 Problem obtaining metadata for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr: 1 validation error for BareAsset
digest
  Digest must have an appropriate dandi-etag value. Got dcc41f4785db06a414a81d5d973727b3-4--1032 (type=value_error)
2022-11-30T03:32:24-0500 [WARNING ] dandi 11532:139778614024000 Failed to operate on some paths (empty records were listed):
 ValidationError: 1 paths
2022-11-30T03:32:24-0500 [INFO    ] dandi 11532:139778614024000 Logs saved in /home/chymera/.cache/dandi-cli/log/20221130083221Z-11532.log
(dev) chymera@decohost /tmp/bids-examples $ git branch && git remote -v &&  git rev-parse HEAD
* dandidev
origin	[email protected]:TheChymera/bids-examples.git (fetch)
origin	[email protected]:TheChymera/bids-examples.git (push)
f74561309d369c44a7db59f0d94640cf8bb1ccce
(dev) chymera@decohost /tmp/bids-examples $ pushd ~/src/dandi-cli/
~/src/dandi-cli /tmp/bids-examples ~/bids-examples
(dev) chymera@decohost ~/src/dandi-cli/dandi $ git rev-parse HEAD
adbf033128f73a3a7e1e918038c3ffe4793f5024

lgtm-com · 2022-12-01T01:17:35Z

This pull request introduces 1 alert when merging 72f3c02 into a0b0015 - view on LGTM.com

new alerts:

1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

TheChymera · 2022-12-01T10:32:13Z

@jwodder continued digging. Best I can tell is I need to explicitly specify the ZARR encoding as the get_metadata() method is inherited from the BIDS* classes and does not include the ZARR specifics. I think that's what I've done here f8d1423 — but now it complains about the size for some reason 🤔

jwodder · 2022-12-01T14:52:20Z

@TheChymera

but now it complains about the size for some reason

What is the complaint, exactly?

TheChymera · 2022-12-01T17:54:21Z

@jwodder

(dev) chymera@decohost /tmp/bids-examples $ DANDI_CACHE=ignore dandi ls micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
Debugging ls for the following path:  micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
('keywords', 'date_of_birth', 'session_description', 'experimenter', 'bids_schema_version', 'organism', 'number_of_tissue_samples', 'license', 'variables_measured', 'probe_ids', 'nd_types', 'tissue_sample_id', 'number_of_electrodes', 'sex', 'age', 'number_of_units', 'version', 'nwb_version', 'cell_id', 'related_publications', 'number_of_subjects', 'description', 'subject_id', 'lab', 'number_of_cells', 'genotype', 'session_start_time', 'experiment_description', 'institution', 'url', 'slice_id', 'name', 'doi', 'identifier', 'session_id', 'species')
2022-12-01 12:53:37,932 [    INFO] Calculating digest for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
qqqqqqqqqqqqqqqqqqqqqqqqqqqqq
<class 'dandi.files.bids.ZarrBIDSAsset'>
666666666666666666666666
2022-12-01 12:53:37,987 [ WARNING] BIDSVersion `1.7.0` is less than the minimal working `schema`. Falling back to `schema`. To force the usage of earlier versions specify them explicitly when calling the validator.
$$$$$$$$$$$$$$$$$$$$$
<class 'dandi.misctypes.Digest'>
èèèèèèèèèèèèèèèèèèèèè
u5u5u5u5u5u5u5u
- path: micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
2022-12-01 12:53:39,078 [ WARNING] Failed to operate on some paths (empty records were listed):
 ValidationError: 1 paths
2022-12-01 12:53:39,078 [    INFO] Logs saved in /home/chymera/.cache/dandi-cli/log/20221201175335Z-20751.log
(dev) chymera@decohost /tmp/bids-examples $ cat $(ls ~/.cache/dandi-cli/log/20221*.log | tail -1)
2022-12-01T12:53:35-0500 [INFO    ] dandi 20751:140023203546944 dandi v0.46.6+172.gf8d1423, hdmf v3.4.6, pynwb v2.2.0, h5py v3.7.0
2022-12-01T12:53:35-0500 [INFO    ] dandi 20751:140023203546944 sys.argv = ['/home/chymera/src/dandi-cli/venvs/dev/bin/dandi', 'ls', 'micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr']
2022-12-01T12:53:35-0500 [INFO    ] dandi 20751:140023203546944 os.getcwd() = /tmp/bids-examples
2022-12-01T12:53:36-0500 [DEBUG   ] urllib3.connectionpool 20751:140023203546944 Starting new HTTPS connection (1): rig.mit.edu:443
2022-12-01T12:53:36-0500 [DEBUG   ] urllib3.connectionpool 20751:140023203546944 https://rig.mit.edu:443 "GET /et/projects/dandi/dandi-cli HTTP/1.1" 200 579
2022-12-01T12:53:36-0500 [DEBUG   ] dandi 20751:140023203546944 Running a newer version (0.46.6+172.gf8d1423) of dandi/dandi-cli than available (0.46.6)
2022-12-01T12:53:36-0500 [DEBUG   ] h5py._conv 20751:140023203546944 Creating converter from 7 to 5
2022-12-01T12:53:36-0500 [DEBUG   ] h5py._conv 20751:140023203546944 Creating converter from 5 to 7
2022-12-01T12:53:36-0500 [DEBUG   ] h5py._conv 20751:140023203546944 Creating converter from 7 to 5
2022-12-01T12:53:36-0500 [DEBUG   ] h5py._conv 20751:140023203546944 Creating converter from 5 to 7
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'zlib'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'gzip'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'bz2'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'lzma'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'blosc'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'zstd'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'lz4'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'astype'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'delta'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'quantize'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'fixedscaleoffset'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'packbits'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'categorize'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'pickle'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'base64'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'shuffle'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'bitround'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'crc32'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'adler32'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'json2'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'vlen-utf8'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'vlen-bytes'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'vlen-array'
2022-12-01T12:53:36-0500 [DEBUG   ] numcodecs 20751:140023203546944 Registering codec 'n5_wrapper'
2022-12-01T12:53:37-0500 [INFO    ] dandi 20751:140023203546944 Calculating digest for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr
2022-12-01T12:53:37-0500 [DEBUG   ] dandi.support.digests 20751:140022005020352 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/.zgroup
2022-12-01T12:53:37-0500 [DEBUG   ] dandi.support.digests 20751:140022005020352 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/.zattrs
2022-12-01T12:53:37-0500 [DEBUG   ] dandi.support.digests 20751:140021996627648 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/0/.zarray
2022-12-01T12:53:37-0500 [DEBUG   ] dandi.support.digests 20751:140021996627648 Estimating digests for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr/0/0/0
2022-12-01T12:53:37-0500 [WARNING ] bids-schema 20751:140023203546944 BIDSVersion `1.7.0` is less than the minimal working `schema`. Falling back to `schema`. To force the usage of earlier versions specify them explicitly when calling the validator.
2022-12-01T12:53:39-0500 [DEBUG   ] dandi 20751:140023203546944 Problem obtaining metadata for micr_SEMzarr/sub-01/ses-01/micr/sub-01_ses-01_sample-A_SPIM.ome.zarr: 1 validation error for BareAsset
digest
  contentSize 4096 is not equal to the checksum size 1032. (type=value_error)
2022-12-01T12:53:39-0500 [WARNING ] dandi 20751:140023203546944 Failed to operate on some paths (empty records were listed):
 ValidationError: 1 paths
2022-12-01T12:53:39-0500 [INFO    ] dandi 20751:140023203546944 Logs saved in /home/chymera/.cache/dandi-cli/log/20221201175335Z-20751.log
(dev) chymera@decohost /tmp/bids-examples $ pushd ~/src/dandi-cli/
~/src/dandi-cli /tmp/bids-examples ~/bids-examples
(dev) chymera@decohost ~/src/dandi-cli $ git rev-parse HEAD
f8d14233c53d2c20a3d639b2ef96513c701029e8

Closes: #1097

Closes: #1038

jwodder

Please give this PR a descriptive title that is appropriate for listing in a CHANGELOG.

dandi/cli/cmd_ls.py

dandi/files/bids.py

dandi/metadata.py

Co-authored-by: John T. Wodder II <[email protected]>

TheChymera · 2022-12-07T20:07:04Z

Please give this PR a descriptive title that is appropriate for listing in a CHANGELOG.

@jwodder this is done. Anything else?

dandi/metadata.py

TheChymera · 2022-12-08T01:40:39Z

For reference, inclusion of NWB data examples upstream is pending: bids-standard/bids-examples#342

TheChymera · 2022-12-12T12:37:06Z

@jwodder what say you? the review comment you left has been resolved.

yarikoptic · 2022-12-12T17:28:31Z

dandi/metadata.py

+                pass
+            else:
+                meta[key] = value
+    elif path.endswith((".NWB", ".nwb")):


Suggested change

elif path.endswith((".NWB", ".nwb")):

if path.endswith((".NWB", ".nwb")):

in relation to prior comment #1164 (comment) -- the point is to validate .nwb regardless either they are part of BIDS or not. BIDS validation does not do any validation of .nwb files which could be within BIDS dataset.

yarikoptic · 2022-12-12T17:29:40Z

actually - I will proceed with the merge for now, but will make a dedicated issue for that nwb validation so we do not stay on this PR for another week.

TheChymera marked this pull request as draft November 23, 2022 18:28

github-advanced-security bot found potential problems Nov 30, 2022

View reviewed changes

dandi/metadata.py Fixed Show resolved Hide resolved

jwodder reviewed Nov 30, 2022

View reviewed changes

dandi/cli/cmd_ls.py Outdated Show resolved Hide resolved

jwodder reviewed Nov 30, 2022

View reviewed changes

dandi/cli/cmd_ls.py Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Dec 1, 2022

View reviewed changes

dandi/cli/cmd_ls.py Fixed Show fixed Hide fixed

TheChymera added 2 commits December 5, 2022 15:38

Checking ls output subject identifier

44a8ab4

Closes: #1097

Added ls command tests for zarrBIDS file

efab43e

Closes: #1038

jwodder added patch Increment the patch version when merged BIDS cmd-ls labels Dec 5, 2022

TheChymera marked this pull request as ready for review December 5, 2022 21:43

TheChymera mentioned this pull request Dec 5, 2022

BIDS: does not upload anything BIDS if not explicitly specifying and --allow-any-path #1055

Closed

TheChymera requested review from yarikoptic and jwodder December 5, 2022 22:35

jwodder requested changes Dec 6, 2022

View reviewed changes

dandi/cli/cmd_ls.py Outdated Show resolved Hide resolved

dandi/files/bids.py Outdated Show resolved Hide resolved

dandi/files/bids.py Outdated Show resolved Hide resolved

dandi/files/bids.py Outdated Show resolved Hide resolved

dandi/metadata.py Outdated Show resolved Hide resolved

TheChymera and others added 4 commits December 6, 2022 13:20

ZARR-appropriate fake checksum

1ed0f5f

Co-authored-by: John T. Wodder II <[email protected]>

More compact code

9900841

Co-authored-by: John T. Wodder II <[email protected]>

Dot in extension

3384ef5

Co-authored-by: John T. Wodder II <[email protected]>

Safer assert satatement

d9ccf8b

Co-authored-by: John T. Wodder II <[email protected]>

TheChymera force-pushed the ls branch from 4526b95 to d9ccf8b Compare December 6, 2022 19:23

Making sure the method which sets the attribute has been run

375947d

TheChymera force-pushed the ls branch from 1e2d6e7 to 375947d Compare December 6, 2022 20:56

TheChymera changed the title ~~Unravelling ls issues~~ Tests for ls reinstated, underlying function fixed, support for ZARR-BIDS files added. Dec 6, 2022

TheChymera requested a review from jwodder December 6, 2022 22:36

yarikoptic reviewed Dec 7, 2022

View reviewed changes

dandi/metadata.py Outdated Show resolved Hide resolved

Updated logic to better accommodate future BIDS NWB examples

11c18f5

jwodder approved these changes Dec 12, 2022

View reviewed changes

yarikoptic reviewed Dec 12, 2022

View reviewed changes

yarikoptic merged commit 923d7b0 into master Dec 12, 2022

yarikoptic deleted the ls branch December 12, 2022 17:29

TheChymera mentioned this pull request Dec 22, 2022

Merging validation functions #1154

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests for `ls` reinstated, underlying function fixed, support for ZARR-BIDS files added. #1164

Tests for `ls` reinstated, underlying function fixed, support for ZARR-BIDS files added. #1164

TheChymera commented Nov 23, 2022 •

edited

Loading

codecov bot commented Nov 23, 2022 •

edited

Loading

TheChymera commented Nov 23, 2022 •

edited

Loading

lgtm-com bot commented Nov 28, 2022

lgtm-com bot commented Nov 28, 2022

TheChymera commented Nov 28, 2022 •

edited

Loading

jwodder commented Nov 28, 2022

TheChymera commented Nov 28, 2022

jwodder commented Nov 28, 2022

lgtm-com bot commented Nov 28, 2022

TheChymera commented Nov 28, 2022 •

edited

Loading

lgtm-com bot commented Nov 28, 2022

lgtm-com bot commented Nov 28, 2022

lgtm-com bot commented Nov 29, 2022

TheChymera commented Nov 29, 2022

TheChymera commented Nov 29, 2022

lgtm-com bot commented Nov 29, 2022

lgtm-com bot commented Nov 30, 2022

TheChymera commented Nov 30, 2022

lgtm-com bot commented Dec 1, 2022

lgtm-com bot commented Dec 1, 2022

TheChymera commented Dec 1, 2022

lgtm-com bot commented Dec 1, 2022

TheChymera commented Dec 1, 2022

jwodder commented Dec 1, 2022

TheChymera commented Dec 1, 2022

jwodder left a comment

TheChymera commented Dec 7, 2022

TheChymera commented Dec 8, 2022

TheChymera commented Dec 12, 2022

yarikoptic Dec 12, 2022

yarikoptic commented Dec 12, 2022

	elif path.endswith((".NWB", ".nwb")):
	if path.endswith((".NWB", ".nwb")):

Tests for ls reinstated, underlying function fixed, support for ZARR-BIDS files added. #1164

Tests for ls reinstated, underlying function fixed, support for ZARR-BIDS files added. #1164

Conversation

TheChymera commented Nov 23, 2022 • edited Loading

codecov bot commented Nov 23, 2022 • edited Loading

Codecov Report

TheChymera commented Nov 23, 2022 • edited Loading

lgtm-com bot commented Nov 28, 2022

lgtm-com bot commented Nov 28, 2022

TheChymera commented Nov 28, 2022 • edited Loading

jwodder commented Nov 28, 2022

TheChymera commented Nov 28, 2022

jwodder commented Nov 28, 2022

lgtm-com bot commented Nov 28, 2022

TheChymera commented Nov 28, 2022 • edited Loading

lgtm-com bot commented Nov 28, 2022

lgtm-com bot commented Nov 28, 2022

lgtm-com bot commented Nov 29, 2022

TheChymera commented Nov 29, 2022

TheChymera commented Nov 29, 2022

lgtm-com bot commented Nov 29, 2022

lgtm-com bot commented Nov 30, 2022

TheChymera commented Nov 30, 2022

lgtm-com bot commented Dec 1, 2022

lgtm-com bot commented Dec 1, 2022

TheChymera commented Dec 1, 2022

lgtm-com bot commented Dec 1, 2022

TheChymera commented Dec 1, 2022

jwodder commented Dec 1, 2022

TheChymera commented Dec 1, 2022

jwodder left a comment

Choose a reason for hiding this comment

TheChymera commented Dec 7, 2022

TheChymera commented Dec 8, 2022

TheChymera commented Dec 12, 2022

yarikoptic Dec 12, 2022

Choose a reason for hiding this comment

yarikoptic commented Dec 12, 2022

Tests for `ls` reinstated, underlying function fixed, support for ZARR-BIDS files added. #1164

Tests for `ls` reinstated, underlying function fixed, support for ZARR-BIDS files added. #1164

TheChymera commented Nov 23, 2022 •

edited

Loading

codecov bot commented Nov 23, 2022 •

edited

Loading

TheChymera commented Nov 23, 2022 •

edited

Loading

TheChymera commented Nov 28, 2022 •

edited

Loading

TheChymera commented Nov 28, 2022 •

edited

Loading