From 56490989b877ff2d14e6d7ca295a9edcd62de597 Mon Sep 17 00:00:00 2001 From: bendichter Date: Thu, 2 Feb 2023 14:42:56 -0300 Subject: [PATCH 01/15] add best practices from https://github.com/NeurodataWithoutBorders/nwb-overview/pull/72 --- docs/best_practices/extensions.rst | 158 ++++++++++++++++++++++++++++- 1 file changed, 155 insertions(+), 3 deletions(-) diff --git a/docs/best_practices/extensions.rst b/docs/best_practices/extensions.rst index c6f7b43c..64ceda4e 100644 --- a/docs/best_practices/extensions.rst +++ b/docs/best_practices/extensions.rst @@ -16,15 +16,103 @@ If an extension is required, tutorials for the process may be found through the It is also encouraged for extensions to contain their own check functions for their own best practices. See the` :ref:`adding_custom_checks` section of the Developer Guide for how to do this. +Define new ``neurodata_types`` at the top-level (a.k.a., do not nest type definitions) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Rather than nesting definitions of ``neurodata_types``, all new types should be defined +at the top-level of the schema. To include a specific ``neurodata_type`` in another type +use the ``neurodata_type_inc`` key instead. For example: -Use Existing Neurodata Types -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. tabs:: + + .. tab:: Do This + + .. tabs:: + + .. code-tab:: py Python + + from pynwb.spec import NWBGroupSpec + + # Define the first type + type1_ext = NWBGroupSpec( + name='custom_type1', + doc='Example extension type 1', + neurodata_type_def='MyNewType1', + neurodata_type_inc='LabMetaData', + ) + + # Then define the second type and include the first type + type2_ext = NWBGroupSpec( + name='custom_type2', + doc='Example extension type 2', + neurodata_type_def='MyNewType2', + groups=[NWBGroupSpec(neurodata_type_inc='MyNewType1', + doc='Included group of ype MyNewType1')] + ) + + .. code-tab:: yaml YAML + + groups: + - neurodata_type_def: MyNewType1 + neurodata_type_inc: LabMetaData + name: custom_type1 + doc: Example extension type 1 + - neurodata_type_def: MyNewType2 + neurodata_type_inc: NWBContainer + name: custom_type2 + doc: Example extension type 2 + groups: + - neurodata_type_inc: MyNewType1 + doc: Included group of ype MyNewType1 + + .. tab:: DON'T do this + + .. tabs:: + + .. code-tab:: py Python + + from pynwb.spec import NWBGroupSpec + + # Do NOT define a new type via ``neurodata_type_def`` within the + # definition of another type. Always define the types separately + # and use ``neurodata_type_inc`` to include the type + type2_ext = NWBGroupSpec( + name='custom_type2', + doc='Example extension type 2', + neurodata_type_def='MyNewType2', + groups=[NWBGroupSpec( + name='custom_type1', + doc='Example extension type 1', + neurodata_type_def='MyNewType1', + neurodata_type_inc='LabMetaData', + )] + ) + + .. code-tab:: yaml YAML + + groups: + - neurodata_type_def: MyNewType2 + neurodata_type_inc: NWBContainer + name: custom_type2 + doc: Example extension type 2 + groups: + - neurodata_type_def: MyNewType1 + neurodata_type_inc: LabMetaData + name: custom_type1 + doc: Example extension type 1 + +Build on and Reuse Existing Neurodata Types +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When possible, use existing types when creating extensions either by creating new ``neurodata_types`` that inherit from existing ones, or by creating ``neurodata_types`` that contain existing ones. Building on existing types facilitates the reuse of existing functionality and interpretation of the data. If a community extension already exists that has a -similar scope, it is preferable to use that extension rather than creating a new one. +similar scope, it is preferable to use that extension rather than creating a new one. For example: +* Extend ``TimeSeries`` for storing timeseries data. NWB provides main types of ``TimeSeries`` + and you should identify the most specific type of ``TimeSeries`` relevant for your use case + (e.g., extend ``ElectricalSeries`` to define a new kind of electrical recording). +* Extend ``DynamicTable`` to store tabular data/ +* Extend ``TimeIntervals`` to store specific annotations of intervals in time. Provide Documentation @@ -45,3 +133,67 @@ anybody who receives the data also receives the necessary data to interpret it. .. note:: In :pynwb-docs:`PyNWB <>`, the extension is cached automatically. This can be specified explicitly with ``io.write(filepath, cache_spec=True)`` + + +Use Attributes for small metadata related to a particular data object (Group or Dataset) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Attributes should be used mainly to store small metadata (usually less than 64 Kbytes) that +is associated with a particular Group or Dataset. Typical uses of attributes are, e.g., to +define the ``unit`` of measurement of a dataset or to store a short ``description`` of +a group or dataset. For larger data, datasets should be used instead. + +In practice, the main difference is that in PyNWB and MatNWB all Attributes are read into memory when reading the +NWB file. If you would like to allow users to read a file without reading all of these particular data values, use a +Dataset. + + +Link data to relevant metadata +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Often metadata relevant to a particular type of data is stored elsewhere, e.g., information +about the ``Device`` used. To ensure relevant metadata can be uniquely identified, the data +should include links to the relevant metadata. NWB provides a few key mechanisms for linking: + +* Use ``links`` (defined via ``NWBLinkSpec``) to link to a particular dataset or group +* Use ``DynamicTableRegion`` to link to a set of rows in a ``DynamicTable`` +* Use a ``dataset`` with an object reference data type to store collections of links + to other objects, e.g., the following dtype to define a dataset of links to ``TimeSeries`` + + .. code-block:: yaml + dtype: + target_type: TimeSeries + reftype: object + + +Best practices for object names +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Names for groups, datasets, attributes, or links should typically: + +* **Use lowercase letters only** +* **Use ``_`` instead of `` `` to separate parts in names**. E.g., use the name + ``starting_time`` instead of ``starting time`` +* **Explicit**. E.g., avoid the use of ambiguous abbreviation in names. + + +Best practices for naming ``neurodata_types`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For defining new types via ``neurodata_type_def`` use: + +* **Use camelcase:** notation, i.e., names of types should NOT include spaces, + always start with an uppercase letter, and use a single capitalized letter to + separate parts of the name. E.g,. ``neurodata_type_def: LaserMeasurement`` +* **Use the postfix ``Series`` when extending a ``TimeSeries`` type.** E.g., when + creating a new ``TimeSeries`` for laser measurements then add ``Series`` to + the type name, e.g,. ``neurodata_type_def: LaserMeasurementSeries`` +* **Use the postfix ``Table`` when extending a ``DynamicTable`` type.** e.g., + ``neurodata_type_def: LaserSettingsTable`` +* **Explicit**. E.g., avoid the use of ambiguous abbreviation in names. + +Use the ``ndx-template`` to create new extensions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +By using the :nwb_extension_git:`ndx-template` to create new extensions helps ensure +that extensions can be easily shared and reused and published via the :ndx-catalog:`NDX Catalog <>`. \ No newline at end of file From a750c3f91ef7238712a0b6c497509ee70d946bd7 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 2 Feb 2023 17:43:49 +0000 Subject: [PATCH 02/15] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/best_practices/extensions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/best_practices/extensions.rst b/docs/best_practices/extensions.rst index 64ceda4e..1116af69 100644 --- a/docs/best_practices/extensions.rst +++ b/docs/best_practices/extensions.rst @@ -196,4 +196,4 @@ Use the ``ndx-template`` to create new extensions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By using the :nwb_extension_git:`ndx-template` to create new extensions helps ensure -that extensions can be easily shared and reused and published via the :ndx-catalog:`NDX Catalog <>`. \ No newline at end of file +that extensions can be easily shared and reused and published via the :ndx-catalog:`NDX Catalog <>`. From 99d248211faf07451e55f940e61e2c6400fb6ca6 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 2 Feb 2023 12:50:37 -0500 Subject: [PATCH 03/15] Update docs/best_practices/extensions.rst --- docs/best_practices/extensions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/best_practices/extensions.rst b/docs/best_practices/extensions.rst index 1116af69..6c7fb7a4 100644 --- a/docs/best_practices/extensions.rst +++ b/docs/best_practices/extensions.rst @@ -182,7 +182,7 @@ Best practices for naming ``neurodata_types`` For defining new types via ``neurodata_type_def`` use: -* **Use camelcase:** notation, i.e., names of types should NOT include spaces, +* **Use CamelCase:** notation, i.e., names of types should NOT include spaces, always start with an uppercase letter, and use a single capitalized letter to separate parts of the name. E.g,. ``neurodata_type_def: LaserMeasurement`` * **Use the postfix ``Series`` when extending a ``TimeSeries`` type.** E.g., when From 4076d7061a3022cd857ea2773754fcbc129f1f04 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 2 Feb 2023 13:12:52 -0500 Subject: [PATCH 04/15] Update conf.py --- docs/conf.py | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/conf.py b/docs/conf.py index ebd86c73..0c3898b8 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -18,6 +18,7 @@ "sphinx.ext.intersphinx", "sphinx.ext.extlinks", "sphinx_copybutton", + 'sphinx_tabs.tabs', ] templates_path = ["_templates"] master_doc = "index" From ef20e9d85ec2e77f34b84ef9b024e3c2bbee7478 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 2 Feb 2023 13:41:28 -0500 Subject: [PATCH 05/15] Update requirements-rtd.txt --- requirements-rtd.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/requirements-rtd.txt b/requirements-rtd.txt index 71bd1014..e426f6b2 100644 --- a/requirements-rtd.txt +++ b/requirements-rtd.txt @@ -5,3 +5,4 @@ sphinx==5.1.1 sphinx_rtd_theme==0.5.1 readthedocs-sphinx-search==0.1.0rc3 sphinx-copybutton==0.5.0 +sphinx-tabs From 6af0fe7ff2d35709229207ae9be15b252c9b6c7f Mon Sep 17 00:00:00 2001 From: bendichter Date: Fri, 3 Feb 2023 11:08:17 -0300 Subject: [PATCH 06/15] fix underline length --- docs/best_practices/extensions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/best_practices/extensions.rst b/docs/best_practices/extensions.rst index 64ceda4e..877d1f0b 100644 --- a/docs/best_practices/extensions.rst +++ b/docs/best_practices/extensions.rst @@ -167,7 +167,7 @@ should include links to the relevant metadata. NWB provides a few key mechanisms Best practices for object names -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Names for groups, datasets, attributes, or links should typically: From 21c5e8ab25f3f7beb894739a0194ac90dbb581db Mon Sep 17 00:00:00 2001 From: bendichter Date: Fri, 3 Feb 2023 11:11:03 -0300 Subject: [PATCH 07/15] fix bullet list --- docs/best_practices/extensions.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/best_practices/extensions.rst b/docs/best_practices/extensions.rst index 26a276c6..49f49ac9 100644 --- a/docs/best_practices/extensions.rst +++ b/docs/best_practices/extensions.rst @@ -108,6 +108,7 @@ When possible, use existing types when creating extensions either by creating ne existing ones, or by creating ``neurodata_types`` that contain existing ones. Building on existing types facilitates the reuse of existing functionality and interpretation of the data. If a community extension already exists that has a similar scope, it is preferable to use that extension rather than creating a new one. For example: + * Extend ``TimeSeries`` for storing timeseries data. NWB provides main types of ``TimeSeries`` and you should identify the most specific type of ``TimeSeries`` relevant for your use case (e.g., extend ``ElectricalSeries`` to define a new kind of electrical recording). @@ -160,10 +161,11 @@ should include links to the relevant metadata. NWB provides a few key mechanisms * Use a ``dataset`` with an object reference data type to store collections of links to other objects, e.g., the following dtype to define a dataset of links to ``TimeSeries`` - .. code-block:: yaml - dtype: - target_type: TimeSeries - reftype: object +.. code-block:: yaml + + dtype: + target_type: TimeSeries + reftype: object Best practices for object names @@ -182,7 +184,7 @@ Best practices for naming ``neurodata_types`` For defining new types via ``neurodata_type_def`` use: -* **Use CamelCase:** notation, i.e., names of types should NOT include spaces, +* **Use camelcase:** notation, i.e., names of types should NOT include spaces, always start with an uppercase letter, and use a single capitalized letter to separate parts of the name. E.g,. ``neurodata_type_def: LaserMeasurement`` * **Use the postfix ``Series`` when extending a ``TimeSeries`` type.** E.g., when @@ -196,4 +198,4 @@ Use the ``ndx-template`` to create new extensions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By using the :nwb_extension_git:`ndx-template` to create new extensions helps ensure -that extensions can be easily shared and reused and published via the :ndx-catalog:`NDX Catalog <>`. +that extensions can be easily shared and reused and published via the :ndx-catalog:`NDX Catalog <>`. \ No newline at end of file From 2e00f7faad49cd1beef40febd96a2b7496eb8811 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 3 Feb 2023 14:11:23 +0000 Subject: [PATCH 08/15] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/best_practices/extensions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/best_practices/extensions.rst b/docs/best_practices/extensions.rst index 49f49ac9..23d60eaa 100644 --- a/docs/best_practices/extensions.rst +++ b/docs/best_practices/extensions.rst @@ -198,4 +198,4 @@ Use the ``ndx-template`` to create new extensions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By using the :nwb_extension_git:`ndx-template` to create new extensions helps ensure -that extensions can be easily shared and reused and published via the :ndx-catalog:`NDX Catalog <>`. \ No newline at end of file +that extensions can be easily shared and reused and published via the :ndx-catalog:`NDX Catalog <>`. From f17e1b9667e7b6ebd68564c57396fe247e73b857 Mon Sep 17 00:00:00 2001 From: bendichter Date: Fri, 3 Feb 2023 11:11:41 -0300 Subject: [PATCH 09/15] adjust language --- docs/best_practices/extensions.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/best_practices/extensions.rst b/docs/best_practices/extensions.rst index 49f49ac9..b7d7cb33 100644 --- a/docs/best_practices/extensions.rst +++ b/docs/best_practices/extensions.rst @@ -16,8 +16,8 @@ If an extension is required, tutorials for the process may be found through the It is also encouraged for extensions to contain their own check functions for their own best practices. See the` :ref:`adding_custom_checks` section of the Developer Guide for how to do this. -Define new ``neurodata_types`` at the top-level (a.k.a., do not nest type definitions) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Define new ``neurodata_types`` at the top-level (do not nest type definitions) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Rather than nesting definitions of ``neurodata_types``, all new types should be defined at the top-level of the schema. To include a specific ``neurodata_type`` in another type From 9ffe41a8d0052bd0d7522859a179b776a1c9ada7 Mon Sep 17 00:00:00 2001 From: bendichter Date: Fri, 3 Feb 2023 11:59:43 -0300 Subject: [PATCH 10/15] fix external links --- docs/conf_extlinks.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/conf_extlinks.py b/docs/conf_extlinks.py index 812c7dff..fa1e7780 100644 --- a/docs/conf_extlinks.py +++ b/docs/conf_extlinks.py @@ -33,6 +33,8 @@ "black-coding-style": ("https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html%s", None), "ncbi": ("https://www.ncbi.nlm.nih.gov/taxonomy%s", None), "ontobee": ("https://ontobee.org/%s", None), + "ndx-catalog": ("https://nwb-extensions.github.io/", None), + "nwb_extension_git": ("https://github.com/nwb-extensions/%s", None), } # Use this for mapping for links to commonly used documentation From 73d5b9bd34ca9a4512b06484eff3ce5745223388 Mon Sep 17 00:00:00 2001 From: bendichter Date: Fri, 3 Feb 2023 14:10:33 -0300 Subject: [PATCH 11/15] fix warning --- docs/conf_extlinks.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conf_extlinks.py b/docs/conf_extlinks.py index fa1e7780..3ddd6c31 100644 --- a/docs/conf_extlinks.py +++ b/docs/conf_extlinks.py @@ -34,7 +34,7 @@ "ncbi": ("https://www.ncbi.nlm.nih.gov/taxonomy%s", None), "ontobee": ("https://ontobee.org/%s", None), "ndx-catalog": ("https://nwb-extensions.github.io/", None), - "nwb_extension_git": ("https://github.com/nwb-extensions/%s", None), + "nwb_extension_git": ("https://github.com/nwb-extensions/", None), } # Use this for mapping for links to commonly used documentation From aa57eae353991169f40c60bda3201340fd12b343 Mon Sep 17 00:00:00 2001 From: bendichter Date: Fri, 3 Feb 2023 14:10:59 -0300 Subject: [PATCH 12/15] fix warning --- docs/conf_extlinks.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/conf_extlinks.py b/docs/conf_extlinks.py index 3ddd6c31..f3461de8 100644 --- a/docs/conf_extlinks.py +++ b/docs/conf_extlinks.py @@ -33,8 +33,8 @@ "black-coding-style": ("https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html%s", None), "ncbi": ("https://www.ncbi.nlm.nih.gov/taxonomy%s", None), "ontobee": ("https://ontobee.org/%s", None), - "ndx-catalog": ("https://nwb-extensions.github.io/", None), - "nwb_extension_git": ("https://github.com/nwb-extensions/", None), + "ndx-catalog": ("https://nwb-extensions.github.io/%s", None), + "nwb_extension_git": ("https://github.com/nwb-extensions/%s", None), } # Use this for mapping for links to commonly used documentation From ba3268286cf06af0ded974bf2a61504787738a88 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Fri, 3 Feb 2023 17:13:45 -0500 Subject: [PATCH 13/15] Update docs/best_practices/extensions.rst Co-authored-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com> --- docs/best_practices/extensions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/best_practices/extensions.rst b/docs/best_practices/extensions.rst index 073f6a59..32235fe3 100644 --- a/docs/best_practices/extensions.rst +++ b/docs/best_practices/extensions.rst @@ -112,7 +112,7 @@ similar scope, it is preferable to use that extension rather than creating a new * Extend ``TimeSeries`` for storing timeseries data. NWB provides main types of ``TimeSeries`` and you should identify the most specific type of ``TimeSeries`` relevant for your use case (e.g., extend ``ElectricalSeries`` to define a new kind of electrical recording). -* Extend ``DynamicTable`` to store tabular data/ +* Extend ``DynamicTable`` to store tabular data. * Extend ``TimeIntervals`` to store specific annotations of intervals in time. From efd03d89bc7015e88d1622b55907f970b12cd608 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Fri, 3 Feb 2023 17:16:17 -0500 Subject: [PATCH 14/15] Update docs/best_practices/extensions.rst Co-authored-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com> --- docs/best_practices/extensions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/best_practices/extensions.rst b/docs/best_practices/extensions.rst index 32235fe3..5f5d0298 100644 --- a/docs/best_practices/extensions.rst +++ b/docs/best_practices/extensions.rst @@ -144,7 +144,7 @@ is associated with a particular Group or Dataset. Typical uses of attributes are define the ``unit`` of measurement of a dataset or to store a short ``description`` of a group or dataset. For larger data, datasets should be used instead. -In practice, the main difference is that in PyNWB and MatNWB all Attributes are read into memory when reading the +In practice, the main difference is that in PyNWB and MatNWB all attributes are read into memory when reading the NWB file. If you would like to allow users to read a file without reading all of these particular data values, use a Dataset. From f38013c45711050a6fd17e9fe2b1909100602708 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 18 Jul 2023 09:23:43 +0000 Subject: [PATCH 15/15] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 7c258808..65b87412 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -19,7 +19,7 @@ "sphinx.ext.intersphinx", "sphinx.ext.extlinks", "sphinx_copybutton", - 'sphinx_tabs.tabs', + "sphinx_tabs.tabs", ] templates_path = ["_templates"] master_doc = "index"