Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] State of the schema sprint #1075

Merged
merged 139 commits into from
Apr 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
c5f89c6
SCHEMA: Encode first two MR metadata tables as "sidecars.yml" (#1016)
tsalo Feb 24, 2022
3af5b98
Add rules for the rest of MRI common fields.
tsalo Feb 24, 2022
0422a16
Add anatomical MRI rules.
tsalo Feb 24, 2022
f2d9b8d
Draft entity-metadata rules too.
tsalo Feb 24, 2022
1bf3866
Fix formatting.
tsalo Feb 24, 2022
2ef6f1a
Update based on discussion today.
tsalo Mar 3, 2022
576e931
Address review.
tsalo Mar 17, 2022
226c2a0
Merge pull request #1017 from tsalo/tsalo-tables
effigies Mar 17, 2022
4c6571c
[SCHEMA] Define YAML tables for DWI/ASL/(i)EEG/micr/beh metadata (#1029)
tsalo Apr 7, 2022
a9db664
Merge remote-tracking branch 'upstream/master' into schema/sidecar_va…
effigies Apr 7, 2022
e332798
Merge gh-1027
bendhouseart Apr 7, 2022
30d8250
[SCHEMA] Add MEG, func, and fmap metadata tables (#1054)
tsalo Apr 8, 2022
341a73b
Draft ASL validation file.
tsalo Apr 12, 2022
c571358
Fix level.
tsalo Apr 12, 2022
a6b22c4
Reorder based on code.
tsalo Apr 12, 2022
60d1572
Add 196 and reorder selectors.
tsalo Apr 12, 2022
f8214ba
Add a few DWI and events rules.
tsalo Apr 12, 2022
b5dc39f
started adding special_errors corresponding to columne 'Reason' Valid…
bendhouseart Apr 12, 2022
a295fc2
Fix linter issues.
tsalo Apr 12, 2022
5d6f7cd
Add a few more rules.
tsalo Apr 12, 2022
f541ca0
moved codes to sub-entities
bendhouseart Apr 12, 2022
66ffd5d
added existing validator issues/codes to required if's where relevant
bendhouseart Apr 13, 2022
e9ac46b
Merge remote-tracking branch 'upstream/master' into schema-sprint
effigies Apr 13, 2022
ab3b818
SCHEMA: Bump SCHEMA_VERSION to indicate incompatibility with 0.1.0
effigies Apr 13, 2022
2abd5cb
fixed formatting issues
bendhouseart Apr 13, 2022
d2d262b
add message and name field to code key
bendhouseart Apr 13, 2022
a3e55ba
missed 3 of em
bendhouseart Apr 13, 2022
d171b99
Replace addendum with description_addendum.
tsalo Apr 13, 2022
43f6b58
Fix line length.
tsalo Apr 13, 2022
47a6457
replaced code: with issue:
bendhouseart Apr 13, 2022
41f44ca
SCHEMA: Initial shot at EEG channels and electrodes
effigies Apr 12, 2022
fe1cf97
SCHEMA: Add aslcontext column validation
effigies Apr 12, 2022
2e83332
SCHEMA add MEG channels.tsv
effigies Apr 12, 2022
0955ed4
SCHEMA: eeg addtional_columnns
effigies Apr 12, 2022
7f77952
SCHEMA: Add iEEG channels/electrodes TSV
effigies Apr 12, 2022
042ed90
Define task columns
effigies Apr 13, 2022
885b509
SCHEMA: Define physio columns
effigies Apr 13, 2022
232f41d
Update src/schema/rules/tabular_data/eeg.yaml
effigies Apr 13, 2022
3bec3f8
Add a few more rules.
tsalo Apr 13, 2022
069abd6
SCHEMA: Separate out channel types for *EG
effigies Apr 13, 2022
c2bbd19
Merge pull request #1061 from effigies/schema/tsv-task
effigies Apr 13, 2022
d6b1375
first pass at blodd tsv for pet
rwblair Apr 12, 2022
1517379
RF: Break blood TSV rules into conditioned rules
effigies Apr 12, 2022
269dc00
lint fixes for pet.yaml
rwblair Apr 13, 2022
df339be
common derivatives schema start with no good very bad yaml formatting
rwblair Apr 13, 2022
e2ba28c
update common derivatives sidecar yaml to look more like other sideca…
rwblair Apr 14, 2022
a307300
Merge pull request #1066 from tsalo/schema-sprint-taylor-2
effigies Apr 14, 2022
4cc1c37
split derivative checks into own files outside sidecars
rwblair Apr 14, 2022
acb2d92
remove nifti_header schemam
rwblair Apr 14, 2022
252c135
Merge pull request #1070 from rwblair/schema/pet_tsv
effigies Apr 14, 2022
c900dea
create derivatives datatypes for use in sidecar rule application
rwblair Apr 14, 2022
a5ecdc8
Update src/schema/rules/sidecars/common_derivative.yaml
rwblair Apr 14, 2022
8f4eec7
Update src/schema/rules/sidecars/asl_validation.yaml
tsalo Apr 14, 2022
c040e09
Switch to new context terms.
tsalo Apr 14, 2022
3b40559
More.
tsalo Apr 14, 2022
b3dbb44
[SCHEMA] Add datatypes field to datatype rule files (#1012)
tsalo Apr 14, 2022
6530002
Use YAML schema types.
tsalo Apr 14, 2022
516eb72
Merge branch 'schema-sprint' into schema-sprint
bendhouseart Apr 14, 2022
57f8a21
Merge branch 'schema-sprint' of github.com:bids-standard/bids-specifi…
rwblair Apr 14, 2022
14f389e
Update mri.yaml
bendhouseart Apr 14, 2022
9001945
Update func.yaml
bendhouseart Apr 14, 2022
c054ae5
Update asl.yaml
bendhouseart Apr 14, 2022
f04b47f
SCHEMA: Convert datatype lists to objects
effigies Apr 14, 2022
24c0aa4
SCHEMACODE: Update loop logic to use dict instead of list
effigies Apr 14, 2022
a022462
Merge pull request #1063 from bendhouseart/schema-sprint
effigies Apr 14, 2022
5fcde7b
Merge pull request #1074 from effigies/name-tables
effigies Apr 14, 2022
fa3d0cd
Merge branch 'schema-sprint' of github.com:bids-standard/bids-specifi…
rwblair Apr 14, 2022
6a32925
Hostile takeover
effigies Apr 14, 2022
4a0a124
Merge branch 'schema/common_derivatives' of github.com:rwblair/bids-s…
rwblair Apr 14, 2022
0dbfaa0
Quiet latin check, clean up some message text
effigies Apr 15, 2022
cdb0524
Merge remote-tracking branch 'upstream/master' into schema-sprint
effigies Apr 15, 2022
e73a60c
first pass at a description of what format nifti headers will be shap…
rwblair Apr 12, 2022
220f531
lint nifti_header.yaml
rwblair Apr 12, 2022
9b8d2c7
lint fixes for nifti_header.yaml
rwblair Apr 13, 2022
3dec62a
SCHEMA: Create context schema in meta/
effigies Apr 14, 2022
844518a
Update nifti_header based on more recent work from Ross
effigies Apr 14, 2022
4798fa3
Flesh out context object
effigies Apr 14, 2022
c7fd45c
Specify that columns have type array
effigies Apr 14, 2022
e48aa03
Add dataset.modalities
effigies Apr 14, 2022
3d0be5a
Add json object
effigies Apr 14, 2022
007af7f
Define type of entire context
effigies Apr 14, 2022
bc8418a
Back to JSON schema
effigies Apr 14, 2022
749c6af
Add aslcontext and events
effigies Apr 14, 2022
035b67e
Add suffix and extension to context
effigies Apr 14, 2022
bde127b
Move all association fields inside the associations context
effigies Apr 14, 2022
d468d7b
Add m0scan, magnitude images and bval/bvec associated files
effigies Apr 15, 2022
ae52408
Comment context
effigies Apr 15, 2022
1fd48a0
STY: Fix YAML, shorten descriptions
effigies Apr 15, 2022
9820ae7
Apply suggestions from code review
tsalo Apr 15, 2022
ebf3c23
associated --> associations
tsalo Apr 15, 2022
fd046c4
seq --> array
tsalo Apr 15, 2022
a2852a9
Merge branch 'schema-sprint-taylor' of https://github.com/tsalo/bids-…
tsalo Apr 15, 2022
4e9b980
Revert extra change.
tsalo Apr 15, 2022
14e8491
Merge pull request #1069 from effigies/schema/context
effigies Apr 15, 2022
493bd9d
Merge pull request #1060 from tsalo/schema-sprint-taylor
effigies Apr 15, 2022
eb58a91
generate common_derivatives datatypes from all other existing datatyp…
rwblair Apr 15, 2022
75153bd
yamllint fix for sidecars common_derivatives
rwblair Apr 15, 2022
08b0467
Latin
effigies Apr 15, 2022
318e5ad
Update src/schema/rules/datatypes/derivatives.yaml
rwblair Apr 15, 2022
1f6e4d2
STY: fix YAML lint
effigies Apr 15, 2022
2eae61c
SCHEMA: Encode requirement levels for dataset_description and genetic…
effigies Apr 13, 2022
a79d5db
WIP: Example of nested structures
effigies Apr 13, 2022
9684bbf
Make path a selector
rwblair Apr 13, 2022
d8facfe
Update selectors to fit context schema
effigies Apr 14, 2022
93ebdbe
Update selectors to match current context object
effigies Apr 14, 2022
49ee3fd
Implement gh-1068
effigies Apr 14, 2022
d96fed4
STY: indent 2
effigies Apr 15, 2022
1461c52
Tag issues 113 and 128
effigies Apr 15, 2022
0b4158a
Use curly-bracket syntax
effigies Apr 15, 2022
e834851
Merge branch 'schema-sprint' of github.com:bids-standard/bids-specifi…
rwblair Apr 15, 2022
8714e1d
---
effigies Apr 15, 2022
4c2af1c
YAML: syntax error
effigies Apr 15, 2022
5e116de
Conform to issue syntax
effigies Apr 15, 2022
109eee9
Conform schema/rules for sidecars and tabular_data
ericearl Apr 15, 2022
e230fd8
continue to bring derivatives schema in line with spec.
rwblair Apr 15, 2022
425fee6
fix datatypes/derivatives/common_derivatives.yaml header
rwblair Apr 15, 2022
f563219
Merge branch 'schema-sprint' into fix/conform-schema
effigies Apr 15, 2022
ea25b43
Apply suggestions from code review
effigies Apr 15, 2022
721729b
Merge pull request #1078 from ericearl/fix/conform-schema
effigies Apr 15, 2022
e3d460c
Merge branch 'schema-sprint' of github.com:bids-standard/bids-specifi…
rwblair Apr 15, 2022
79f7946
STY: pacify yaml linter
effigies Apr 15, 2022
32c2955
Move validation yaml files to checks subdir
effigies Apr 15, 2022
2db6b95
STY: Conform checks to 2-space convention
effigies Apr 15, 2022
0976eda
add refs that contain datatype to common imaging derivatives
rwblair Apr 15, 2022
f29eabf
add missing periods and use full entity name in dataype common imagin…
rwblair Apr 15, 2022
ddab685
DOC: Fix iEEG channel object name
effigies Apr 16, 2022
d9e38fa
change to _ref in schema
rwblair Apr 18, 2022
713469e
Merge pull request #1064 from effigies/schema/dataset_description
effigies Apr 18, 2022
dca1564
remove excessive commas and add missing underscore (#1083)
rwblair Apr 19, 2022
f51869b
Revert "change to _ref in schema"
rwblair Apr 20, 2022
bb2285c
remove 'schema' at root of references in derivative schema
rwblair Apr 20, 2022
43a4288
correct for derivative files found in test_render.py datatypes count …
rwblair Apr 20, 2022
b16a746
yamllint asl_validation
rwblair Apr 20, 2022
833074a
Merge branch 'schema-sprint' of github.com:bids-standard/bids-specifi…
rwblair Apr 20, 2022
f8cc27e
Merge pull request #1072 from rwblair/schema/common_derivatives
rwblair Apr 21, 2022
4beb22d
convert exisiting schema selectors and checks to be more js like
rwblair Apr 25, 2022
2c6fbf5
yamllint fixes and rerun sed on overlooked cases/files
rwblair Apr 25, 2022
2d81171
update isArray and typof to use a 'type()' function
rwblair Apr 28, 2022
fd9ab3a
Merge pull request #1094 from rwblair/schema/convert-checks-to-js
rwblair Apr 28, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@ and a guide for using macros can be found at
{{ MACROS___make_columns_table(
{
"name__channels": "REQUIRED",
"type__channels": "REQUIRED",
"type__meg_channels": "REQUIRED",
"units": "REQUIRED",
}
) }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ and a guide for using macros can be found at
{{ MACROS___make_columns_table(
{
"name__channels": "REQUIRED",
"type__channels": "REQUIRED",
"type__eeg_channels": "REQUIRED",
"units": "REQUIRED",
}
) }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ and a guide for using macros can be found at
"name__channels": ("REQUIRED", "When a corresponding electrode is specified in `_electrodes.tsv`, "
"the name of that electrode MAY be specified here and the reference electrode "
"name MAY be provided in the `reference` column."),
"type__channels": "REQUIRED",
"type__ieeg_channels": "REQUIRED",
"units": "REQUIRED",
"low_cutoff": "REQUIRED",
"high_cutoff": "REQUIRED",
Expand Down
6 changes: 3 additions & 3 deletions src/schema/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# BIDS-schema

Portions of the BIDS specification are defined using YAML files, in order to
Portions of the BIDS specification are defined using YAML files in order to
make the specification machine-readable.

Currently, the portions of the specification that rely on this schema are
Currently the portions of the specification that rely on this schema are
the entity tables, entity definitions, filename templates, and metadata tables.
Any changes to the specification should be mirrored in the schema.

Expand Down Expand Up @@ -73,7 +73,7 @@ For objects with `snake_case` names, two underscores must be used.
There should also be a comment near the object definition in the YAML file describing the nature of the different objects.

For example, the TSV column `"reference"` means different things when used for EEG data, as compared to iEEG data.
As such, there are two definitions in `columns.yaml` for the `"reference"` column: `"reference__eeg"` and `"reference_ieeg"`.
As such, there are two definitions in `columns.yaml` for the `"reference"` column: `"reference__eeg"` and `"reference__ieeg"`.

```yaml
# reference column for channels.tsv files for EEG data
Expand Down
2 changes: 1 addition & 1 deletion src/schema/SCHEMA_VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.1.0
0.2.0
260 changes: 260 additions & 0 deletions src/schema/meta/context.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,260 @@
#
# The context provides the namespaces available to rules.
# These namespaces are used by selectors to define the scope of application
# for a rule, as well as assertions, to determine whether the rule is satisfied.
#
# Not all components of the context will be defined; for example a NIfTI header
# will only be defined when examining a NIfTI file.
#
# The dataset namespace is constructed once and is available when visiting all
# files.
#
# The subject namespace is constructed once per subject, and is available when
# visiting all files within that subject.
#
# All other (current) namespaces are defined on individual files.
# Sidecar metadata and file associations are built according to the inheritance
# principle.
#
---
context:
type: object
properties:
dataset:
description: "Properties and contents of the entire dataset"
type: object
properties:
dataset_description:
description: "Contents of /dataset_description.json"
type: object
files:
description: "List of all files in dataset"
type: array
tree:
description: "Tree view of all files in dataset"
type: object
ignored:
description: "Set of ignored files"
type: array
modalities:
description: "Modalities present in the dataset"
type: array
subjects:
description: "Collections of subjects in dataset"
type: object
properties:
sub_dirs:
description: "Subjects as determined by sub-*/ directories"
type: array
items:
type: string
participant_id:
description: "The participant_id column of participants.tsv"
type: array
items:
type: string
phenotype:
description: "The union of participant_id columns in phenotype files"
type: array
items:
type: string
subject:
description: "Properties and contents of the current subject"
type: object
properties:
sessions:
description: "Collections of sessions in subject"
type: object
properties:
ses_dirs:
description: "Sessions as determined by ses-*/ directories"
type: array
items:
type: string
session_id:
description: "The session_id column of sessions.tsv"
type: array
items:
type: string
phenotype:
description: "The union of session_id columns in phenotype files"
type: array
items:
type: string

# Properties of the current file
path:
description: "Path of the current file"
type: string
entities:
description: "Entities parsed from the current filename"
type: object
datatype:
description: "Datatype of current file, for examples, anat"
type: string
suffix:
description: "Suffix of current file"
type: string
extension:
description: "Extension of current file including initial dot"
type: string
modality:
description: "Modality of current file, for examples, MRI"
type: string

sidecar:
description: "Sidecar metadata constructed via the inheritance principle"
type: object
associations:
# Note that this is not intended to be an exhaustive list of associated files
# or to expose every attribute of those files. It is specifically those files
# and attributes for which a rule needs to be applied from an originating file.
description: |
Associated files, indexed by suffix, selected according to the inheritance principle
type: object
properties:
events:
description: "Events file"
type: object
properties:
path:
description: "Path to associated events file"
type: string
onset:
description: "Contents of the onset column"
type: array
items:
type: string
aslcontext:
description: "ASL context file"
type: object
properties:
path:
description: "Path to associated aslcontext file"
type: string
n_rows:
description: "Number of rows in aslcontext.tsv"
type: integer
volume_type:
description: "Contents of the volume_type column"
type: array
items:
type: string
m0scan:
description: "M0 scan file"
type: object
properties:
path:
description: "Path to associated M0 scan file"
type: string
magnitude:
description: "Magnitude image file"
type: object
properties:
path:
description: "Path to associated magnitude file"
type: string
magnitude1:
description: "Magnitude1 image file"
type: object
properties:
path:
description: "Path to associated magnitude1 file"
type: string
bval:
description: "B value file"
type: object
properties:
path:
description: "Path to associated bval file"
type: string
n_cols:
description: "Number of columns in bval file"
type: integer
bvec:
description: "B vector file"
type: object
properties:
path:
description: "Path to associated bvec file"
type: string
n_cols:
description: "Number of columns in bvec file"
type: integer

# The following properties are populated if the current file is of an appropriate type
columns:
description: "TSV columns, indexed by column header, values are arrays with column contents"
type: object
additionalProperties:
type: array
json:
description: "Contents of the current JSON file"
type: object
nifti_header:
name: "NIfTI Header"
description: "Parsed contents of NIfTI header referenced elsewhere in schema."
type: object
properties:
dim_info:
name: "Dimension Information"
description: "Metadata about dimensions data."
type: object
properties:
freq:
name: "Frequency"
description: "These fields encode which spatial dimension (1, 2, or 3)."
type: integer
phase:
name: "Phase"
description: "Corresponds to which acquisition dimension for MRI data."
type: integer
slice:
name: "Slice"
description: "Slice dimensions."
type: integer
dim:
name: "Data Dimensions"
description: "Data seq dimensions."
type: array
minItems: 8
maxItems: 8
items:
type: integer
pixdim:
name: "Pixel Dimension"
description: "Grid spacings (unit per dimension)."
type: array
minItems: 8
maxItems: 8
items:
type: number
xyzt_units:
name: "XYZT Units"
description: "Units of pixdim[1..4]"
xyz:
name: "XYZ Units"
description: "String representing the unit of voxel spacing."
type: string
enum:
- "unknown"
- "meter"
- "mm"
- "um"
t:
name: "Time Unit"
description: "String representing the unit of inter-volume intervals."
type: string
enum:
- "unknown"
- "sec"
- "msec"
- "usec"
qform_code:
name: "qform code"
description: "Use of the quaternion fields."
type: integer
sform_code:
name: "sform code"
description: "Use of the affine fields."
type: integer
53 changes: 52 additions & 1 deletion src/schema/objects/columns.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -466,7 +466,31 @@ trigger:
continuous measurement of the scanner trigger signal
type: number
# type column in channels.tsv files
type__channels:
type__eeg_channels:
name: type
description: |
Type of channel; MUST use the channel types listed below.
Note that the type MUST be in upper-case.
type: string
enum:
- AUDIO
- EEG
- EOG
- ECG
- EMG
- EYEGAZE
- GSR
- HEOG
- MISC
- PPG
- PUPIL
- REF
- RESP
- SYSCLOCK
- TEMP
- TRIG
- VEOG
type__meg_channels:
name: type
description: |
Type of channel; MUST use the channel types listed below.
Expand Down Expand Up @@ -501,6 +525,33 @@ type__channels:
- HLU
- FITERR
- OTHER
type__ieeg_channels:
name: type
description: |
Type of channel; MUST use the channel types listed below.
Note that the type MUST be in upper-case.
type: string
enum:
- EEG
- ECOG
- SEEG
- DBS
- VEOG
- HEOG
- EOG
- ECG
- EMG
- TRIG
- AUDIO
- PD
- EYEGAZE
- PUPIL
- MISC
- SYSCLOCK
- ADC
- DAC
- REF
- OTHER
# type column for electrodes.tsv files
type__electrodes:
name: type
Expand Down
Loading