Docs Overhaul. Ignore pre-commit

CITATION.cff

@@ -2,7 +2,7 @@
 # Visit https://bit.ly/cffinit to generate yours today!
 
 cff-version: 1.2.0
-title: spyglass
+title: 'Spyglass: a data analysis framework for reproducible and shareable neuroscience research'
 message: 'If you use this software, please cite it as below.'
 type: software
 authors:
@@ -45,8 +45,8 @@ authors:
     affiliation: 'University of California, San Francisco'
     orcid: 'https://orcid.org/0000-0003-4013-214X'
   - given-names: Rhino
-    family-names: Nevers
-    email: rhino.nevers@ucsf.edu
+    family-names: Never
+    email: rhino.never@ucsf.edu
     affiliation: 'University of California, San Francisco'
   - given-names: Philip
     family-names: Adenekan
@@ -84,18 +84,18 @@ authors:
     email: [email protected]
     affiliation: 'University of California, San Francisco'
     orcid: 'https://orcid.org/0000-0001-5559-2910'
-  - given-names: Shin
-    family-names: Donghoon
+  - given-names: Donghoon
+    family-names: Shin
     email: [email protected]
     affiliation: 'University of California, San Francisco'
     orcid: 'https://orcid.org/0009-0000-8916-7314'
-  - given-names: Chiang
-    family-names: Sharon
+  - given-names: Sharon
+    family-names: Chiang
     email: [email protected]
     affiliation: 'University of California, San Francisco'
     orcid: 'https://orcid.org/0000-0002-4548-4550'
-  - given-names: Holobetz
-    family-names: Cristofer
+  - given-names: Cristofer
+    family-names: Holobetz
     email: [email protected]
     affiliation: 'University College London'
     orcid: 'https://orcid.org/0009-0009-8567-3290'

README.md

@@ -5,14 +5,19 @@
 
 ![Spyglass Figure](docs/src/images/fig1.png)
 
-[Demo](https://spyglass.hhmi.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FLorenFrankLab%2Fspyglass-demo&urlpath=lab%2Ftree%2Fspyglass-demo%2Fnotebooks%2F01_Insert_Data.ipynb&branch=main) | [Installation](https://lorenfranklab.github.io/spyglass/latest/installation/) | [Docs](https://lorenfranklab.github.io/spyglass/) | [Tutorials](https://github.com/LorenFrankLab/spyglass/tree/master/notebooks) | [Citation](#citation)
+[Demo](https://spyglass.hhmi.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FLorenFrankLab%2Fspyglass-demo&urlpath=lab%2Ftree%2Fspyglass-demo%2Fnotebooks%2F02_Insert_Data.ipynb&branch=main)
+| [Installation](https://lorenfranklab.github.io/spyglass/latest/installation/)
+| [Docs](https://lorenfranklab.github.io/spyglass/) |
+[Tutorials](https://github.com/LorenFrankLab/spyglass/tree/master/notebooks) |
+[Citation](#citation)
 
 `spyglass` is a data analysis framework that facilitates the storage, analysis,
 visualization, and sharing of neuroscience data to support reproducible
 research. It is designed to be interoperable with the NWB format and integrates
 open-source tools into a coherent framework.
 
-Try out a demo [here](https://spyglass.hhmi.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FLorenFrankLab%2Fspyglass-demo&urlpath=lab%2Ftree%2Fspyglass-demo%2Fnotebooks%2F01_Insert_Data.ipynb&branch=main)!
+Try out a demo
+[here](https://spyglass.hhmi.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FLorenFrankLab%2Fspyglass-demo&urlpath=lab%2Ftree%2Fspyglass-demo%2Fnotebooks%2F02_Insert_Data.ipynb&branch=main)!
 
 Features of Spyglass include:
 
@@ -60,16 +65,16 @@ Documentation can be found at -
 ## Installation
 
 For installation instructions see -
-[https://lorenfranklab.github.io/spyglass/latest/installation/](https://lorenfranklab.github.io/spyglass/latest/installation/)
+[https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/](https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/)
 
 Typical installation time is: 5-10 minutes
 
 ## Tutorials
 
-The tutorials for `spyglass` is currently in the form of Jupyter Notebooks and
+The tutorials for `spyglass` are currently in the form of Jupyter Notebooks and
 can be found in the
 [notebooks](https://github.com/LorenFrankLab/spyglass/tree/master/notebooks)
-directory. We strongly recommend opening them in the context of `jupyterlab`.
+directory. We strongly recommend running the notebooks yourself.
 
 ## Contributing
 
@@ -85,17 +90,31 @@ License and Copyright notice can be found at
 
 ## System requirements
 
-Spyglass has been tested on Linux Ubuntu 20.04 and MacOS 10.15. It has not been tested on Windows and likely will not work.
+Spyglass has been tested on Linux Ubuntu 20.04 and MacOS 10.15. It has not been
+tested on Windows and likely will not work.
 
-No specific hardware requirements are needed to run spyglass. However, the amount of data that can be stored and analyzed is limited by the available disk space and memory. GPUs are required for some of the analysis tools, such as DeepLabCut.
+No specific hardware requirements are needed to run spyglass. However, the
+amount of data that can be stored and analyzed is limited by the available disk
+space and memory. GPUs are required for some of the analysis tools, such as
+DeepLabCut.
 
-See [pyproject.toml](pyproject.toml), [environment.yml](environment.yml), or [environment_dlc.yml](environment_dlc.yml) for software dependencies.
+See [pyproject.toml](pyproject.toml), [environment.yml](environment.yml), or
+[environment_dlc.yml](environment_dlc.yml) for software dependencies.
 
-See [spec-file.txt](https://github.com/LorenFrankLab/spyglass-demo/blob/main/spec-file/spec-file.txt) for the conda environment used in the demo.
+See
+[spec-file.txt](https://github.com/LorenFrankLab/spyglass-demo/blob/main/spec-file/spec-file.txt)
+for the conda environment used in the demo.
 
 ## Citation
 
-> Lee, K.H.\*, Denovellis, E.L.\*, Ly, R., Magland, J., Soules, J., Comrie, A.E., Gramling, D.P., Guidera, J.A., Nevers, R., Adenekan, P., Brozdowski, C., Bray, S., Monroe, E., Bak, J.H., Coulter, M.E., Sun, X., Broyles, E., Shin, D., Chiang, S., Holobetz, C., Tritt, A., Rübel, O., Nguyen, T., Yatsenko, D., Chu, J., Kemere, C., Garcia, S., Buccino, A., Frank, L.M., 2024. Spyglass: a data analysis framework for reproducible and shareable neuroscience research. bioRxiv. [10.1101/2024.01.25.577295](https://doi.org/10.1101/2024.01.25.577295).
+> Lee, K.H.\*, Denovellis, E.L.\*, Ly, R., Magland, J., Soules, J., Comrie,
+> A.E., Gramling, D.P., Guidera, J.A., Nevers, R., Adenekan, P., Brozdowski, C.,
+> Bray, S., Monroe, E., Bak, J.H., Coulter, M.E., Sun, X., Broyles, E., Shin,
+> D., Chiang, S., Holobetz, C., Tritt, A., Rübel, O., Nguyen, T., Yatsenko, D.,
+> Chu, J., Kemere, C., Garcia, S., Buccino, A., Frank, L.M., 2024. Spyglass: a
+> data analysis framework for reproducible and shareable neuroscience research.
+> bioRxiv.
+> [10.1101/2024.01.25.577295](https://doi.org/10.1101/2024.01.25.577295).
 
 *\* Equal contribution*
 

docs/README.md

@@ -16,11 +16,9 @@ The remainder of `mkdocs.yml` specifies the site's
 ## GitHub
 
 Whenever a new tag is pushed, GitHub actions will run
-`.github/workflows/publish-docs.yml`. Progress can be monitored in the 'Actions'
-tab within the repo.
-
-Releases should be tagged with `X.Y.Z`. A tag to redeploy docs should use the
-current version, with an alpha release suffix, e.g. `X.Y.Za1`.
+`.github/workflows/publish-docs.yml`. From the repository, select the Actions
+tab, and then the 'Publish Docs' workflow on the left to monitor progress. The
+process can also be manually triggered by selecting 'Run workflow' on the right.
 
 To deploy on your own fork without a tag, follow turn on github pages in
 settings, following a `documentation` branch, and then push to `test_branch`.
@@ -47,7 +45,7 @@ the root notebooks directory may not be reflected when rebuilding.
 Use a browser to navigate to `localhost:8000/` to inspect the site. For
 auto-reload of markdown files during development, use
 `mkdocs serve -f ./docs/mkdosc.yaml`. The `mike` package used in the build
-script manages versioning, but does not support dynamic versioning.
+script manages versioning, but does not support dynamic reloading.
 
 The following items can be commented out in `mkdocs.yml` to reduce build time:
 

docs/mkdocs.yml

@@ -42,15 +42,14 @@ theme:
 
 nav:
   - Home: index.md
-  - Installation: installation.md
   - Tutorials:
       - Overview: notebooks/index.md
       - Intro:
           - Setup: notebooks/00_Setup.ipynb
-          - Insert Data: notebooks/01_Insert_Data.ipynb
-          - Data Sync: notebooks/02_Data_Sync.ipynb
-          - Merge Tables: notebooks/03_Merge_Tables.ipynb
-          - Config Populate: notebooks/04_PopulateConfigFile.ipynb
+          - Concepts: notebooks/01_Concepts.ipynb
+          - Insert Data: notebooks/02_Insert_Data.ipynb
+          - Data Sync: notebooks/03_Data_Sync.ipynb
+          - Merge Tables: notebooks/04_Merge_Tables.ipynb
           - Export: notebooks/05_Export.ipynb
       - Spikes:
           - Spike Sorting V0: notebooks/10_Spike_SortingV0.ipynb
@@ -69,18 +68,22 @@ nav:
           - Decoding Clusterless: notebooks/41_Decoding_Clusterless.ipynb
           - Decoding Sorted Spikes: notebooks/42_Decoding_SortedSpikes.ipynb
       - MUA Detection: notebooks/50_MUA_Detection.ipynb
-  - Miscellaneous:
-      - Overview: misc/index.md
-      - Common Errors: misc/common_errs.md
-      - Database Management: misc/database_management.md
-      - Export: misc/export.md
-      - FigURL: misc/figurl_views.md
-      - Insert Data: misc/insert_data.md
-      - Merge Tables: misc/merge_tables.md
-      - Mixin: misc/mixin.md
-      - Session Groups: misc/session_groups.md
+  - Features:
+    - Overview: Features/index.md
+    - FigURL: Features/FigURL.md
+    - Merge Tables: Features/Merge.md
+    - Export: Features/Export.md
+    - Session Groups: Features/SessionGroups.md
+    - Centralized Code: Features/Mixin.md
+  - For Developers:
+    - Overview: ForDevelopers/index.md
+    - How to Contribute: ForDevelopers/Contribute.md
+    - Database Management: ForDevelopers/Management.md
+    - Code Reuse: ForDevelopers/Reuse.md
+    - Table Types: ForDevelopers/TableTypes.md
+    - Understanding a Schema: ForDevelopers/Schema.md
+    - Using NWB: ForDevelopers/UsingNWB.md
   - API Reference: api/ # defer to gen-files + literate-nav
-  - How to Contribute: contribute.md
   - Change Log: CHANGELOG.md
   - Copyright: LICENSE.md
 

docs/src/misc/mixin.md → docs/src/Features/Mixin.md

@@ -6,7 +6,7 @@ functionalities that have been added to DataJoint tables. This includes...
 - Fetching NWB files
 - Long-distance restrictions.
 - Delete functionality, including permission checks and part/master pairs
-- Export logging. See [export doc](export.md) for more information.
+- Export logging. See [export doc](./Export.md) for more information.
 
 To add this functionality to your own tables, simply inherit from the mixin:
 
@@ -53,8 +53,8 @@ to a `_nwb_table` attribute.
 In complicated pipelines like Spyglass, there are often tables that 'bury' their
 foreign keys as secondary keys. This is done to avoid having to pass a long list
 of foreign keys through the pipeline, potentially hitting SQL limits (see also
-[Merge Tables](./merge_tables.md)). This burrying makes it difficult to restrict
-a given table by familiar attributes.
+[Merge Tables](./Merge.md)). This burrying makes it difficult to restrict a
+given table by familiar attributes.
 
 Spyglass provides a function, `restrict_by`, to handle this. The function takes
 your restriction and checks parents/children until the restriction can be
@@ -122,7 +122,7 @@ If the user shares a lab team with the session experimenter, the deletion is
 permitted.
 
 This is not secure system and is not a replacement for database backups (see
-[database management](./database_management.md)). A user could readily
+[database management](../ForDevelopers/Management.md)). A user could readily
 curcumvent the default permission checks by adding themselves to the relevant
 team or removing the mixin from the class declaration. However, it provides a
 reasonable level of security for the average user.
@@ -134,11 +134,11 @@ entry without deleting the corresponding master. This is useful for enforcing
 the custom of adding/removing all parts of a master at once and avoids orphaned
 masters, or null entry masters without matching data.
 
-For [Merge tables](./merge_tables.md), this is a significant problem. If a user
-wants to delete all entries associated with a given session, she must find all
-part table entries, including Merge tables, and delete them in the correct
-order. The mixin provides a function, `delete_downstream_parts`, to handle this,
-which is run by default when calling `delete`.
+For [Merge tables](./Merge.md), this is a significant problem. If a user wants
+to delete all entries associated with a given session, she must find all part
+table entries, including Merge tables, and delete them in the correct order. The
+mixin provides a function, `delete_downstream_parts`, to handle this, which is
+run by default when calling `delete`.
 
 `delete_downstream_parts`, also aliased as `ddp`, identifies all part tables
 with foreign key references downstream of where it is called. If `dry_run=True`,

docs/src/Features/index.md

@@ -0,0 +1,12 @@
+# Features
+
+This directory contains a series of explainers on tools that have been added to
+Spyglass.
+
+- [Export](./Export.md) - How to export an analysis.
+- [FigURL](./FigURL.md) - How to use FigURL to share figures.
+- [Merge Tables](./Merge.md) - Tables for pipeline versioning.
+- [Mixin](./Mixin.md) - Spyglass-specific functionalities to DataJoint tables,
+    including fetching NWB files, long-distance restrictions, and permission
+    checks on delete operations.
+- [Session Groups](./SessionGroups.md) - How to operate on sets of sessions.

docs/src/ForDevelopers/Contribute.md

@@ -0,0 +1,57 @@
+# Contributing to Spyglass
+
+This document provides an overview of the Spyglass development, and provides
+guidance for folks looking to contribute to the project itself. For information
+on setting up custom tables, skip to Code Organization.
+
+## Development workflow
+
+New contributors should follow the
+[Fork-and-Branch workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/forking-workflow).
+See GitHub instructions
+[here](https://docs.github.com/en/get-started/quickstart/contributing-to-projects).
+
+Regular contributors may choose to follow the
+[Feature Branch Workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow)
+for features that will involve multiple contributors.
+
+## Code organization
+
+- Tables are grouped into schemas by topic (e.g., `common_metrics`)
+- Schemas
+    - Are defined in a `py` pile.
+    - Correspond to MySQL 'databases'.
+    - Are organized into modules (e.g., `common`) by folders.
+- The _common_ module
+    - In principle, contains schema that are shared across all projects.
+    - In practice, contains shared tables (e.g., Session) and the first draft of
+        schemas that have since been split into their own
+        modality-specific\
+        modules (e.g., `lfp`)
+    - Should not be added to without discussion.
+- A pipeline
+    - Refers to a set of tables used for processing data of a particular modality
+        (e.g., LFP, spike sorting, position tracking).
+    - May span multiple schema.
+- For analysis that will be only useful to you, create your own schema.
+
+## Misc
+
+- During development, we suggest using a Docker container. See
+    [example](../notebooks/00_Setup.ipynb).
+- `numpy` style docstrings will be interpreted by API docs. To check for
+    compliance, monitor the output when building docs (see `docs/README.md`)
+
+## Making a release
+
+Spyglass follows [Semantic Versioning](https://semver.org/) with versioning of
+the form `X.Y.Z` (e.g., `0.4.2`).
+
+1. In `CITATION.cff`, update the `version` key.
+2. Make a pull request with changes.
+3. After the pull request is merged, pull this merge commit and tag it with
+    `git tag {version}`
+4. Publish the new release tag. Run `git push origin {version}`. This will
+    rebuild docs and push updates to PyPI.
+5. Make a new
+    [release on GitHub](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository).