Merging validation functions

.github/workflows/codeql.yml

@@ -0,0 +1,42 @@
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ "master" ]
+  pull_request:
+    branches: [ "master" ]
+  schedule:
+    - cron: "31 5 * * 5"
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ python ]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@v2
+        with:
+          languages: ${{ matrix.language }}
+          queries: +security-and-quality
+          config-file: codeql.yml
+
+      - name: Autobuild
+        uses: github/codeql-action/autobuild@v2
+
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/analyze@v2
+        with:
+          category: "/language:${{ matrix.language }}"

CHANGELOG.md

@@ -1,3 +1,63 @@
+# 0.48.0 (Mon Dec 19 2022)
+
+#### 🚀 Enhancement
+
+- Update client for change in Zarr entries API [#1175](https://github.com/dandi/dandi-cli/pull/1175) ([@jwodder](https://github.com/jwodder))
+
+#### Authors: 1
+
+- John T. Wodder II ([@jwodder](https://github.com/jwodder))
+
+---
+
+# 0.47.0 (Mon Dec 19 2022)
+
+#### 🚀 Enhancement
+
+- Add validation of filepaths for non-BIDS NWB assets [#1173](https://github.com/dandi/dandi-cli/pull/1173) ([@jwodder](https://github.com/jwodder))
+- Exclude special dotfiles from Zarrs [#1147](https://github.com/dandi/dandi-cli/pull/1147) ([@jwodder](https://github.com/jwodder))
+- Structured validation results [#1104](https://github.com/dandi/dandi-cli/pull/1104) ([@jwodder](https://github.com/jwodder) [@yarikoptic](https://github.com/yarikoptic) [@TheChymera](https://github.com/TheChymera))
+
+#### 🐛 Bug Fix
+
+- Allow user to specify mandatory (if not empty) fields in organize [#1171](https://github.com/dandi/dandi-cli/pull/1171) ([@yarikoptic](https://github.com/yarikoptic) [@jwodder](https://github.com/jwodder))
+- BF: convert str errors from checking nwb version into proper ValidationResult [#1174](https://github.com/dandi/dandi-cli/pull/1174) ([@yarikoptic](https://github.com/yarikoptic))
+- Tests for `ls` reinstated, underlying function fixed, support for ZARR-BIDS files added. [#1164](https://github.com/dandi/dandi-cli/pull/1164) ([@TheChymera](https://github.com/TheChymera) [@jwodder](https://github.com/jwodder))
+- Add CodeQL workflow for GitHub code scanning and fix few bugs it detected [#1165](https://github.com/dandi/dandi-cli/pull/1165) ([@lgtm-migrator](https://github.com/lgtm-migrator) [@jwodder](https://github.com/jwodder) [@lgtm-com[bot]](https://github.com/lgtm-com[bot]))
+- Corrected reporting function logic to complete group message variable [#1166](https://github.com/dandi/dandi-cli/pull/1166) ([@TheChymera](https://github.com/TheChymera))
+- Renamed failing test, added prospective use case for NWBI warning. [#1162](https://github.com/dandi/dandi-cli/pull/1162) ([@TheChymera](https://github.com/TheChymera))
+- Use cached namespace validation [#1149](https://github.com/dandi/dandi-cli/pull/1149) ([@CodyCBakerPhD](https://github.com/CodyCBakerPhD))
+
+#### 🏠 Internal
+
+- Make `list_paths()` include dotfiles [#1142](https://github.com/dandi/dandi-cli/pull/1142) ([@jwodder](https://github.com/jwodder))
+
+#### 📝 Documentation
+
+- dandi-cli readme edit [#1170](https://github.com/dandi/dandi-cli/pull/1170) ([@melster1010](https://github.com/melster1010))
+- Remove Parameters description from docstring used by click [#1150](https://github.com/dandi/dandi-cli/pull/1150) ([@yarikoptic](https://github.com/yarikoptic))
+
+#### 🧪 Tests
+
+- No longer mark `test_rename_type_mismatch` as xfailing [#1161](https://github.com/dandi/dandi-cli/pull/1161) ([@jwodder](https://github.com/jwodder))
+- Fix typing error under mypy 0.990 [#1156](https://github.com/dandi/dandi-cli/pull/1156) ([@jwodder](https://github.com/jwodder))
+- Update `test_validate_nwb_path_grouping` test [#1157](https://github.com/dandi/dandi-cli/pull/1157) ([@jwodder](https://github.com/jwodder))
+- Fixed logic and commented a temporarily broken BIDS (lacking README) dataset [#1148](https://github.com/dandi/dandi-cli/pull/1148) ([@TheChymera](https://github.com/TheChymera))
+- Installing hdf5 for Python 3.10 is no longer needed [#1145](https://github.com/dandi/dandi-cli/pull/1145) ([@jwodder](https://github.com/jwodder))
+- Fix a typing issue involving the outdated `tmpdir_factory` [#1144](https://github.com/dandi/dandi-cli/pull/1144) ([@jwodder](https://github.com/jwodder))
+
+#### Authors: 7
+
+- [@lgtm-com[bot]](https://github.com/lgtm-com[bot])
+- Cody Baker ([@CodyCBakerPhD](https://github.com/CodyCBakerPhD))
+- Horea Christian ([@TheChymera](https://github.com/TheChymera))
+- John T. Wodder II ([@jwodder](https://github.com/jwodder))
+- LGTM Migrator ([@lgtm-migrator](https://github.com/lgtm-migrator))
+- Mary Elise Dedicke ([@melster1010](https://github.com/melster1010))
+- Yaroslav Halchenko ([@yarikoptic](https://github.com/yarikoptic))
+
+---
+
 # 0.46.6 (Fri Oct 21 2022)
 
 #### 🐛 Bug Fix

README.md

@@ -6,29 +6,36 @@
 [![PyPI version fury.io](https://badge.fury.io/py/dandi.svg)](https://pypi.python.org/pypi/dandi/)
 [![Documentation Status](https://readthedocs.org/projects/dandi/badge/?version=latest)](https://dandi.readthedocs.io/en/latest/?badge=latest)
 
-This project is under heavy development.  Beware of [hidden](I-wish-we-knew) and
-[disclosed](https://github.com/dandi/dandi-cli/issues) issues, or
+The [DANDI Python client](https://pypi.org/project/dandi/) allows you to:
+
+* Download `Dandisets` and individual subject folders or files
+* Validate data to locally conform to standards
+* Organize your data locally before upload
+* Upload `Dandisets`
+* Interact with the DANDI archive's web API from Python
+* Delete data in the DANDI archive
+* Perform other auxiliary operations with data or the DANDI archive
+
+**Note**: This project is under heavy development. See [the issues log](https://github.com/dandi/dandi-cli/issues) or
 [Work-in-Progress (WiP)](https://github.com/dandi/dandi-cli/pulls).
 
 ## Installation
 
-At the moment DANDI client releases are [available from PyPI](https://pypi.org/project/dandi)
-and [conda-forge](https://anaconda.org/conda-forge/dandi).  You could
-install them in your Python (native, virtualenv, or conda) environment via
+DANDI Client releases are [available from PyPI](https://pypi.org/project/dandi)
+and [conda-forge](https://anaconda.org/conda-forge/dandi).  Install them in your Python (native, virtualenv, or 
+conda) environment via
 
     pip install dandi
 
 or
 
     conda install -c conda-forge dandi
 
-if you are in a conda environment.
 
-## dandi tool
+## CLI Tool
 
-This package provides a `dandi` command line utility with a basic interface
-which should assist you in preparing and uploading your data to and/or obtaining
-data from the http://dandiarchive.org:
+This package provides a command line utility with a basic interface
+to help you prepare and upload your data to, or obtain data from, the [DANDI archive](http://dandiarchive.org):
 
 ```bash
 $> dandi
@@ -45,51 +52,85 @@ Usage: dandi [OPTIONS] COMMAND [ARGS]...
 
 Options:
   --version
-  -l, --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
-                                  Log level name  [default: INFO]
+ -l, --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
+                                  Log level (case insensitive).  May be
+                                  specified as an integer.  [default: INFO]
   --pdb                           Fall into pdb if errors out
   --help                          Show this message and exit.
 
 Commands:
-  download  Download a file or entire folder from DANDI
-  ls        List .nwb files and dandisets metadata.
-  organize  (Re)organize files according to the metadata.
-  register  Register a new dandiset in the DANDI archive
-  upload    Upload dandiset (files) to DANDI archive.
-  validate  Validate files for NWB (and DANDI) compliance.
+  delete            Delete dandisets and assets from the server.
+  digest            Calculate file digests
+  download          Download a file or entire folder from DANDI.
+  instances         List known Dandi Archive instances that the CLI can...
+  ls                List .nwb files and dandisets metadata.
+  move              Move or rename assets in a local Dandiset and/or on...
+  organize          (Re)organize files according to the metadata.
+  service-scripts   Various utility operations
+  shell-completion  Emit shell script for enabling command completion.
+  upload            Upload Dandiset files to DANDI Archive.
+  validate          Validate files for NWB and DANDI compliance.
+  validate-bids     Validate BIDS paths.
 ```
 
-Each of the commands has a set of options to alter their behavior.  Please run
-`dandi COMMAND --help` to get more information, e.g.
+Each of the commands has a set of options to alter its behavior.  Run
+`dandi COMMAND --help` to get more information:
 
 ```
 $> dandi ls --help
-Usage: dandi ls [OPTIONS] [PATHS]...
-
-  List .nwb files metadata
+Usage: dandi ls [OPTIONS] PATH|URL
+
+  List .nwb files and dandisets metadata.
+
+  The arguments may be either resource identifiers or paths to local
+  files/directories.
+
+  Accepted resource identifier patterns:
+   - DANDI:<dandiset id>[/<version>]
+   - https://dandiarchive.org/...
+   - https://identifiers.org/DANDI:<dandiset id>[/<version id>] (<version id> cannot be 'draft')
+   - https://<server>[/api]/[#/]dandiset/<dandiset id>[/<version>][/files[?location=<path>]]
+   - https://*dandiarchive-org.netflify.app/...
+   - https://<server>[/api]/dandisets/<dandiset id>[/versions[/<version>]]
+   - https://<server>[/api]/assets/<asset id>[/download]
+   - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/<asset id>[/download]
+   - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?path=<path>
+   - dandi://<instance name>/<dandiset id>[@<version>][/<path>]
+   - https://<server>/...
 
 Options:
   -F, --fields TEXT               Comma-separated list of fields to display.
                                   An empty value to trigger a list of
                                   available fields to be printed out
-  -f, --format [auto|pyout|json|json_pp|yaml]
+  -f, --format [auto|pyout|json|json_pp|json_lines|yaml]
                                   Choose the format/frontend for output. If
                                   'auto', 'pyout' will be used in case of
                                   multiple files, and 'yaml' for a single
                                   file.
+  -r, --recursive                 Recurse into content of
+                                  dandisets/directories. Only .nwb files will
+                                  be considered.
+  -J, --jobs INTEGER              Number of parallel download jobs.  [default:
+                                  6]
+  --metadata [api|all|assets]
+  --schema VERSION                Convert metadata to new schema version
   --help                          Show this message and exit.
 ```
 
-See [DANDI Handbook](https://www.dandiarchive.org/handbook/10_using_dandi/)
-for examples on how to use this client in various use cases.
 
-## Development/contributing
+## Third-party Components
 
-Please see [DEVELOPMENT.md](./DEVELOPMENT.md) file.
+**dandi/tests/skip.py** -- from https://github.com/ReproNim/reproman, as of v0.2.1-40-gf4f026d
+Copyright (c) 2016-2020  ReproMan Team
 
-## 3rd party components included
+## Resources
 
-### dandi/tests/skip.py
+* To learn how to interact with the DANDI archive and for examples on how to use the DANDI Client in various use cases,
+see [the handbook](https://www.dandiarchive.org/handbook/).
 
-From https://github.com/ReproNim/reproman, as of v0.2.1-40-gf4f026d
-Copyright (c) 2016-2020  ReproMan Team
+* To get help:
+  - ask a question: https://github.com/dandi/helpdesk/discussions
+  - file a feature request or bug report: https://github.com/dandi/helpdesk/issues/new/choose
+  - contact the DANDI team: [email protected]
+
+* To understand how to contribute to the dandi-cli repository, see the [DEVELOPMENT.md](./DEVELOPMENT.md) file.

codeql.yml

@@ -0,0 +1,4 @@
+paths-ignore:
+  - dandi/_version.py
+  - dandi/due.py
+  - versioneer.py

dandi/cli/cmd_ls.py

@@ -5,6 +5,7 @@
 import click
 
 from .base import devel_option, lgr, map_to_click_exceptions
+from ..consts import ZARR_EXTENSIONS, metadata_all_fields
 from ..dandiarchive import DandisetURL, _dandi_url_parser, parse_dandi_url
 from ..misctypes import Digest
 from ..utils import is_url
@@ -92,7 +93,6 @@ def ls(
         PYOUTFormatter,
         YAMLFormatter,
     )
-    from ..consts import metadata_all_fields
 
     # TODO: avoid
     from ..support.pyout import PYOUT_SHORT_NAMES_rev
@@ -358,7 +358,20 @@ def fn():
                             digest=Digest.dandi_etag(digest),
                         ).json_dict()
                 else:
-                    rec = get_metadata(path)
+                    if path.endswith(tuple(ZARR_EXTENSIONS)):
+                        if use_fake_digest:
+                            digest = "0" * 32 + "-0--0"
+                        else:
+                            lgr.info("Calculating digest for %s", path)
+                            digest = get_digest(path, digest="zarr-checksum")
+                        rec = get_metadata(path, Digest.dandi_zarr(digest))
+                    else:
+                        if use_fake_digest:
+                            digest = "0" * 32 + "-1"
+                        else:
+                            lgr.info("Calculating digest for %s", path)
+                            digest = get_digest(path, digest="dandi-etag")
+                        rec = get_metadata(path, Digest.dandi_etag(digest))
             except Exception as exc:
                 _add_exc_error(path, rec, errors, exc)
             if flatten:

dandi/cli/cmd_organize.py

@@ -1,7 +1,9 @@
+from __future__ import annotations
+
 import click
 
 from .base import dandiset_path_option, devel_debug_option, map_to_click_exceptions
-from ..consts import file_operation_modes
+from ..consts import dandi_layout_fields, file_operation_modes
 
 
 @click.command()
@@ -46,11 +48,22 @@
     default=None,
     help="How to relocate video files referenced by NWB files",
 )
+@click.option(
+    "--required-field",
+    "required_fields",
+    type=click.Choice(list(dandi_layout_fields)),
+    multiple=True,
+    help=(
+        "Force a given field to be included in the organized filename of any"
+        " file for which it is nonempty.  Can be specified multiple times."
+    ),
+)
 @click.argument("paths", nargs=-1, type=click.Path(exists=True))
 @devel_debug_option()
 @map_to_click_exceptions
 def organize(
     paths,
+    required_fields,
     dandiset_path=None,
     invalid="fail",
     files_mode="auto",
@@ -101,4 +114,5 @@ def organize(
         devel_debug=devel_debug,
         update_external_file_paths=update_external_file_paths,
         media_files_mode=media_files_mode,
+        required_fields=required_fields,
     )