Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merging validation functions #1154

Closed
wants to merge 95 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
e1809e3
Update pynwb_utils.py
CodyCBakerPhD Nov 1, 2022
921f67a
Update pynwb_utils.py
CodyCBakerPhD Nov 1, 2022
d7ab706
Update pynwb_utils.py
CodyCBakerPhD Nov 1, 2022
7261a91
Non-working draft
TheChymera Nov 9, 2022
242be00
Absolute instead of resolve to avoid symlink issues.
TheChymera Nov 11, 2022
26467d1
Update `test_validate_nwb_path_grouping` test
jwodder Nov 11, 2022
75da4bb
Merge pull request #1149 from catalystneuro/use_cached_namespace_vali…
yarikoptic Nov 11, 2022
46f7095
Fix typing error under mypy 0.990
jwodder Nov 11, 2022
588034c
Merge pull request #1157 from dandi/fix-grouping-test
yarikoptic Nov 12, 2022
15ea79f
Merge pull request #1156 from dandi/fix-typing
yarikoptic Nov 12, 2022
77a380e
No longer mark `test_rename_type_mismatch` as xfailing
jwodder Nov 16, 2022
b11835a
Merge pull request #1161 from dandi/rm-xfail
yarikoptic Nov 16, 2022
b97689e
Renamed failing test, added prospective use case for NWBI warning.
TheChymera Nov 21, 2022
a50c4fe
importance_threshold and debugging print calls
TheChymera Nov 21, 2022
ede4a20
Created and using new fixture for non-critical NWBI error
TheChymera Nov 22, 2022
b6ee44a
Indentation fix
TheChymera Nov 22, 2022
d0ef688
Imports at top of module
TheChymera Nov 22, 2022
0439caf
Merge pull request #1162 from dandi/error_fix
TheChymera Nov 24, 2022
1d59b33
Add CodeQL workflow for GitHub code scanning
lgtm-migrator Nov 25, 2022
61e9b80
Make CodeQL ignore the files that LGTM ignored
jwodder Nov 28, 2022
25b9d98
Address some CodeQL alerts
jwodder Nov 28, 2022
1137dbd
Eliminate use of tempfile.mktemp()
jwodder Nov 28, 2022
d2d6836
Corrected reporting function logic to complete group message variable
TheChymera Nov 28, 2022
0ad1346
Merge pull request #1166 from dandi/reporting
yarikoptic Nov 29, 2022
a0b0015
Merge pull request #1165 from lgtm-migrator/codeql
yarikoptic Nov 29, 2022
3d3bb9a
dandi-cli readme edit
melster1010 Dec 1, 2022
d3ec5f1
Unravelling `ls` issues
TheChymera Nov 23, 2022
dc34ab0
strange `Cannot creat weak reference to PosixPath object`
TheChymera Nov 23, 2022
8aef541
Fixed weakref issue
TheChymera Nov 28, 2022
b3a4b9a
Added readme listing and removed some diagnostic print calls
TheChymera Nov 28, 2022
3272c86
Removed more debugging print calls
TheChymera Nov 28, 2022
ba99146
Removed more debugging print calls
TheChymera Nov 28, 2022
64677c1
Creating and passing digests
TheChymera Nov 28, 2022
f8a0da8
Added validation bids schema version to result object metadata
TheChymera Nov 28, 2022
430fb06
Assigning bids schema version for validation to metadata dict
TheChymera Nov 28, 2022
279ecf9
Trying to insert bids_schema_version ... somewhere.
TheChymera Nov 29, 2022
60b2408
Added (get_validation)_bids_version to BIDS assets
TheChymera Nov 29, 2022
4a4530e
Creating ZARR digest for ZARR files
TheChymera Nov 30, 2022
9df52ad
Using correct ZARR digest class (still not working)
TheChymera Dec 1, 2022
0e0ff8e
More debugging print calls trying to track down digest error
TheChymera Dec 1, 2022
cc28fbb
Removed unneeded import
TheChymera Dec 1, 2022
5d59ae8
Using zarr extensions variable from constants module
TheChymera Dec 1, 2022
5d62fcc
Reinstated fake digest after introducing ZARR conditional
TheChymera Dec 1, 2022
f0de103
Attempting to fix digest type conflict (still not working)
TheChymera Dec 1, 2022
86ae9c0
Use Zarr checksum to set metadata contentSize
jwodder Dec 1, 2022
9866d0c
Removed debugging print calls
TheChymera Dec 2, 2022
785dec0
Not requiring bids_version for ValidatorOrigin
TheChymera Dec 2, 2022
2059dd7
Typing fixes and improved variable name
TheChymera Dec 2, 2022
a2119ec
Attemting type check fix for missing attribute
TheChymera Dec 2, 2022
722e804
Removed commented imports
TheChymera Dec 2, 2022
3649b5e
Actually fix typing
jwodder Dec 2, 2022
32d0379
Debugging upload tests
TheChymera Dec 2, 2022
ba7d60c
Reinstating test
TheChymera Dec 5, 2022
e8a64e1
Associate BIDS asset errors with the actual assets
jwodder Dec 5, 2022
ccda9ec
Fixed novel README validation error
TheChymera Dec 5, 2022
44a8ab4
Checking ls output subject identifier
TheChymera Dec 5, 2022
efab43e
Added ls command tests for zarrBIDS file
TheChymera Dec 5, 2022
79c829c
fix spelling
melster1010 Dec 6, 2022
1ed0f5f
ZARR-appropriate fake checksum
TheChymera Dec 6, 2022
9900841
More compact code
TheChymera Dec 6, 2022
3384ef5
Dot in extension
TheChymera Dec 6, 2022
d9ccf8b
Safer assert satatement
TheChymera Dec 6, 2022
375947d
Making sure the method which sets the attribute has been run
TheChymera Dec 6, 2022
87c37b0
Incorporate review feedback
melster1010 Dec 7, 2022
11c18f5
Updated logic to better accommodate future BIDS NWB examples
TheChymera Dec 7, 2022
97555c7
Update dandi and ls --help output
melster1010 Dec 9, 2022
923d7b0
Merge pull request #1164 from dandi/ls
yarikoptic Dec 12, 2022
4cbadb1
Merge pull request #1170 from dandi/readme-cli
yarikoptic Dec 12, 2022
03854a7
Add validation of filepaths for non-BIDS NWB assets
jwodder Dec 13, 2022
55df999
Adjust organized filename regex
jwodder Dec 14, 2022
101d6ed
Merge pull request #1173 from dandi/gh-1163
yarikoptic Dec 14, 2022
72258b1
BF: convert str errors from checking nwb version into proper Validati…
yarikoptic Dec 14, 2022
8e67482
Merge pull request #1174 from dandi/bf-errors
yarikoptic Dec 14, 2022
8847ee8
Starter to allow user to specify mandatory if not empty fields
yarikoptic Dec 8, 2022
899c2ca
Add `dandi organize` option and change "mandatory" to "required"
jwodder Dec 16, 2022
0a5f20c
Update docs
jwodder Dec 16, 2022
d593992
Fix typing
jwodder Dec 16, 2022
5686277
Add test
jwodder Dec 16, 2022
be00d45
Move `potential_fields` to consts.py and rename to `dandi_layout_fields`
jwodder Dec 19, 2022
b704bca
Merge pull request #1171 from dandi/enh--mandatory
yarikoptic Dec 19, 2022
48e8ccd
Update CHANGELOG.md [skip ci]
Dec 19, 2022
47c524c
Update client for change in Zarr entries API
jwodder Dec 15, 2022
bfb6362
Remove deleted classes from docs
jwodder Dec 15, 2022
1a06a17
Content URLs for Zarrs are no longer followable redirects.
jwodder Dec 19, 2022
c0f018d
Fix uploading of Zarr file entries over pre-existing directories
jwodder Dec 19, 2022
54e2a12
Merge pull request #1175 from dandi/archive-1394
yarikoptic Dec 19, 2022
0eaec43
Update CHANGELOG.md [skip ci]
Dec 19, 2022
1ad8c23
Upload test to check for metadata
TheChymera Dec 20, 2022
ddfe305
Retrieving metadata
TheChymera Dec 20, 2022
cc01263
Getting all relevant assets for testing
TheChymera Dec 20, 2022
288715e
Improving check to not pass if the identifier changes
TheChymera Dec 21, 2022
9aae3bf
Merge pull request #1180 from dandi/metadata
yarikoptic Dec 21, 2022
40d55f1
Non-working draft
TheChymera Nov 9, 2022
2befd3a
Absolute instead of resolve to avoid symlink issues.
TheChymera Nov 11, 2022
b4e2e59
Merge branch 'merge_validation' of github.com:dandi/dandi-cli into me…
TheChymera Dec 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 13 additions & 40 deletions dandi/cli/cmd_validate.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,7 @@

@click.command()
@click.option(
"--schema", help="Validate against new BIDS schema version.", metavar="VERSION"
)
@click.option(
"--report-path",
help="Write report under path, this option implies `--report/-r`.",
"--bids-schema", help="Validate against new BIDS schema version.", metavar="VERSION"
)
@click.option(
"--report",
Expand All @@ -24,42 +20,9 @@
help="Whether to write a report under a unique path in the DANDI log directory.",
)
@click.option(
"--grouping",
"-g",
help="How to group error/warning reporting.",
type=click.Choice(["none", "path"], case_sensitive=False),
default="none",
"--report-path",
help="Write report under path, this option implies `--report/-r`.",
)
@click.argument("paths", nargs=-1, type=click.Path(exists=True, dir_okay=True))
@map_to_click_exceptions
def validate_bids(
paths,
schema,
report,
report_path,
grouping="none",
):
"""Validate BIDS paths.

Notes
-----
Used from bash, eg:
dandi validate-bids /my/path
"""

from ..validate import validate_bids as validate_bids_

validator_result = validate_bids_( # Controller
*paths,
report=report,
report_path=report_path,
schema_version=schema,
)

_process_issues(validator_result, grouping)


@click.command()
@devel_option("--schema", help="Validate against new schema version", metavar="VERSION")
@devel_option(
"--allow-any-path",
Expand All @@ -79,6 +42,9 @@ def validate_bids(
@map_to_click_exceptions
def validate(
paths,
bids_schema,
report,
report_path,
schema=None,
devel_debug=False,
allow_any_path=False,
Expand Down Expand Up @@ -108,9 +74,16 @@ def validate(
validator_result = validate_(
*paths,
schema_version=schema,
bids_schema_version=bids_schema,
devel_debug=devel_debug,
allow_any_path=allow_any_path,
)
from collections import Counter

# for some reason we get the same error repeated over and over again, probably related
# to different behaviour of yield and return
print(Counter([i.id for i in validator_result]))
raise

_process_issues(validator_result, grouping)

Expand Down
3 changes: 1 addition & 2 deletions dandi/cli/command.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ def main(ctx, log_level, pdb=False):
from .cmd_service_scripts import service_scripts # noqa: E402
from .cmd_shell_completion import shell_completion # noqa: E402
from .cmd_upload import upload # noqa: E402
from .cmd_validate import validate, validate_bids # noqa: E402
from .cmd_validate import validate # noqa: E402

__all_commands__ = (
delete,
Expand All @@ -162,7 +162,6 @@ def main(ctx, log_level, pdb=False):
shell_completion,
upload,
validate,
validate_bids,
)

for cmd in __all_commands__:
Expand Down
21 changes: 19 additions & 2 deletions dandi/files/bids.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def bids_root(self) -> Path:
"""
The directory on the filesystem in which the BIDS dataset is located
"""
return self.filepath.parent
return self.filepath.parent.resolve()

def _validate(self) -> None:
with self._lock:
Expand All @@ -71,19 +71,36 @@ def _validate(self) -> None:
list
)
self._asset_metadata = defaultdict(dict)
print("..")
for result in results:
print(result)
if result.id in BIDS_ASSET_ERRORS:
print("jj")
print(result)
print("jj")
assert result.path
self._asset_errors[str(result.path)].append(result)
elif result.id in BIDS_DATASET_ERRORS:
print("ii")
print(result)
self._dataset_errors.append(result)
print("ii")
elif result.id == "BIDS.MATCH":
print("kk")
print(result)
print("kk")
assert result.path
print(result.path)
print("kk1")
bids_path = result.path.relative_to(self.bids_root).as_posix()
print("kk2")
assert result.metadata is not None
print("kk3")
self._asset_metadata[bids_path] = prepare_metadata(
result.metadata
)
print("kk4")
print("..")

def get_asset_errors(self, asset: BIDSAsset) -> list[ValidationResult]:
""":meta private:"""
Expand Down Expand Up @@ -154,7 +171,7 @@ def bids_path(self) -> str:
"""
``/``-separated path to the asset from the root of the BIDS dataset
"""
return self.filepath.relative_to(self.bids_root).as_posix()
return self.filepath.resolve().relative_to(self.bids_root).as_posix()
TheChymera marked this conversation as resolved.
Show resolved Hide resolved

def get_validation_errors(
self,
Expand Down
1 change: 1 addition & 0 deletions dandi/validate.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ def validate_bids(

def validate(
*paths: str,
bids_schema_version: Optional[str] = None,
schema_version: Optional[str] = None,
devel_debug: bool = False,
allow_any_path: bool = False,
Expand Down