False positive (?) in legacy validator: SUSPICIOUSLY_LONG_EVENT_DESIGN

[yarikoptic]

We get this WARNing reported

(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ head /tmp/bids-validator.log
...
[email protected]
(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ grep -A12 'SUSPICIOUSLY_LONG_EVENT_DESIGN)' /tmp/bids-validator.log
        2: [WARN] The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long.  (code: 85 - SUSPICIOUSLY_LONG_EVENT_DESIGN)
                ./sub-0002/ses-03/func/sub-0002_ses-03_task-shortvideo_acq-mb8_run-01_events.tsv
                ./sub-0004/ses-02/func/sub-0004_ses-02_task-faces_acq-mb8_run-01_events.tsv
                ./sub-0005/ses-04/func/sub-0005_ses-04_task-fractional_acq-mb8_run-01_events.tsv
                ./sub-0009/ses-01/func/sub-0009_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0010/ses-01/func/sub-0010_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0011/ses-01/func/sub-0011_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0013/ses-01/func/sub-0013_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0014/ses-01/func/sub-0014_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0015/ses-01/func/sub-0015_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0016/ses-01/func/sub-0016_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ... and 57 more files having this issue (Use --verbose to see them all).

whenever all numbers tell otherwise:

(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ tail ./sub-0002/ses-03/func/sub-0002_ses-03_task-shortvideo_acq-mb8_run-01_events.tsv
708.127 3.124   cue     n/a     n/a     similarity      n/a     n/a
711.251 5.508   video   n/a     n/a     similarity      n/a     task-shortvideos/ses-03_run-01_order-15_content-giving.mp4
716.813 n/a     rating  n/a     Moderate        similarity      n/a     n/a
n/a     n/a     rating_mouse_trajectory n/a     Moderate        similarity      n/a     n/a
721.922 5.012   video   n/a     n/a     similarity      n/a     task-shortvideos/ses-03_run-01_order-21_content-iceblock.mp4
727.0   3.518   rating  24.21   Moderate        similarity      n/a     n/a
728.234 2.285   rating_mouse_trajectory 24.21   Moderate        similarity      n/a     n/a
732.104 6.122   video   n/a     n/a     similarity      n/a     task-shortvideos/ses-03_run-01_order-14_content-alwaysafamily.mp4
738.288 4.983   rating  8.56    Weak    similarity      n/a     n/a
743.055 0.217   rating_mouse_trajectory 8.56    Weak    similarity      n/a     n/a

(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ jq .RepetitionTime ./sub-0002/ses-03/func/sub-0002_ses-03_task-shortvideo_acq-mb8_run-01_bold.json
0.46

(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ nib-ls ./sub-0002/ses-03/func/sub-0002_ses-03_task-shortvideo_acq-mb8_run-01_bold.nii.gz
./sub-0002/ses-03/func/sub-0002_ses-03_task-shortvideo_acq-mb8_run-01_bold.nii.gz int16 [ 82,  82,  56, 1616] 2.68x2.68x2.70x0.46   sform

(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$  python -c 'print(1616*0.46, 743.055+0.217)'
743.36 743.2719999999999

So the total duration is less than even the end of the last event and thus this error makes no sense. I thought that may be it is because of those n/a entries even for onset (trials when there were no mouse trajectory rating given thus no specific time), so I

(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ sed -i -e '/^n/d' ./sub-*/ses-*/func/sub-*_events.tsv

which edited 316 files (so many more than those original reported) and reran validator on such modified dataset and getting the same'ish (more files reported!?)

(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ grep -A12 'SUSPICIOUSLY_LONG_EVENT_DESIGN)' /tmp/bids-validator-naremoved.log
        2: [WARN] The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long.  (code: 85 - SUSPICIOUSLY_LONG_EVENT_DESIGN)
                ./sub-0002/ses-03/func/sub-0002_ses-03_task-shortvideo_acq-mb8_run-01_events.tsv
                ./sub-0004/ses-02/func/sub-0004_ses-02_task-faces_acq-mb8_run-01_events.tsv
                ./sub-0005/ses-04/func/sub-0005_ses-04_task-fractional_acq-mb8_run-01_events.tsv
                ./sub-0009/ses-01/func/sub-0009_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0010/ses-01/func/sub-0010_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0011/ses-01/func/sub-0011_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0011/ses-02/func/sub-0011_ses-02_task-narratives_acq-mb8_run-02_events.tsv
                ./sub-0013/ses-01/func/sub-0013_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0014/ses-01/func/sub-0014_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ./sub-0015/ses-01/func/sub-0015_ses-01_task-alignvideo_acq-mb8_run-01_events.tsv
                ... and 61 more files having this issue (Use --verbose to see them all).

any way to troubleshoot this, or how does it get "last event" and "total duration"?

With deno version, we also do get hits but less of them!

        [WARNING] SUSPICIOUSLY_LONG_EVENT_DESIGN The onset of the last event is after the total duration of the corresponding scan.
This design is suspiciously long.

                /sub-0004/ses-02/func/sub-0004_ses-02_task-faces_acq-mb8_run-01_bold.nii.gz
                /sub-0005/ses-04/func/sub-0005_ses-04_task-fractional_acq-mb8_run-01_bold.nii.gz

                12 more files with the same issue

and in that first sample it is legitimate (attn @jungheejung) -- that first bold seems to be truncated in comparison to events file:

(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ tail sub-0004/ses-02/func/sub-0004_ses-02_task-faces_acq-mb8_run-01_events.tsv
405.077 0.968   rating_mouse_trajectory 0.7553  age     anger   male    African old     n/a
407.934 2.045   face    n/a     age     pain    male    WC      young   task-faces/7Pain_Model3_2Male_1WC_1Young.mp4
409.979 1.085   rating  0.288   age     pain    male    WC      young   n/a
410.545 0.518   rating_mouse_trajectory 0.288   age     pain    male    WC      young   n/a
411.884 1.313   face    n/a     age     disgust male    WC      young   task-faces/4Disgust_Model1_2Male_1WC_1Young.mp4
413.197 1.752   rating  0.553   age     disgust male    WC      young   n/a
413.88  1.068   rating_mouse_trajectory 0.553   age     disgust male    WC      young   n/a
415.103 1.312   face    n/a     age     disgust female  WC      old     task-faces/4Disgust_Model3_1Female_1WC_2Old.mp4
416.415 1.869   rating  0.7006  age     disgust female  WC      old     n/a
416.931 1.352   rating_mouse_trajectory 0.7006  age     disgust female  WC      old     n/a
(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ nib-ls sub-0004/ses-02/func/sub-0004_ses-02_task-faces_acq-mb8_run-01_bold.nii.gz
sub-0004/ses-02/func/sub-0004_ses-02_task-faces_acq-mb8_run-01_bold.nii.gz int16 [ 82,  82,  56, 194] 2.68x2.68x2.70x0.46   sform

(deno) yoh@typhon:/mnt/DATA/data/yoh/1076_spacetop$ python -c 'print(194*0.46, 416.931+1.352)'
89.24000000000001 418.28299999999996

so may be the issue is only with legacy one somehow... may be there is some collision somewhere so it assigns some file paths where they do not really belong or alike?

FWIW -- this is on not yet published a https://openneuro.org/datasets/ds005256 but I could not push fresh changes due to need for

• Allow for .bids-validator-config.json filename OpenNeuroOrg/openneuro#3141

[effigies]

The check is here:

bids-validator/bids-validator/validators/events/events.js

Lines 37 to 112 in 35306a2

	const checkDesignLength = function (events, headers, jsonContents) {
	const issues = []
	// get all headers associated with task data
	const taskHeaders = headers.filter((header) => {
	const file = header[0]
	return file.relativePath.includes('_task-')
	})
	// loop through headers with files that are tasks
	taskHeaders.forEach((taskHeader) => {
	// extract the fourth element of 'dim' field of header - this is the
	// number of volumes that were obtained during scan (numVols)
	const file = taskHeader[0]
	const header = taskHeader[1]
	const dim = header.dim
	const numVols = dim[4]
	// get the json sidecar dictionary associated with that nifti scan
	const potentialSidecars = utils.files.potentialLocations(
	file.relativePath.replace('.gz', '').replace('.nii', '.json'),
	)
	const mergedDictionary = utils.files.generateMergedSidecarDict(
	potentialSidecars,
	jsonContents,
	)
	// extract the 'RepetitionTime' field from said sidecar (TR)
	const TR = mergedDictionary.RepetitionTime
	// calculate max reasonable scan time (TR * numVols = longDurationThreshold)
	const longDurationThreshold = Math.floor(TR * numVols)
	// calculate min reasonable scan time (.5 * TR * numVols = shortDurationThreshold)
	const shortDurationThreshold = Math.floor(0.5 * longDurationThreshold)
	// get the _events.tsv associated with this task scan
	const potentialEvents = utils.files.potentialLocations(
	file.relativePath.replace('.gz', '').replace('bold.nii', 'events.tsv'),
	)
	const associatedEvents = events.filter(
	(event) => potentialEvents.indexOf(event.path) > -1,
	)
	// loop through all events associated with this task scan
	for (let event of associatedEvents) {
	// get all non-empty rows
	const rows = event.contents
	.split('\n')
	.filter((row) => !(!row || /^\s*$/.test(row)))
	// get the 'onset' field of the last event (lastEventOnset)
	const lastEventOnset = rows[rows.length - 1].trim().split('\t')[0]
	// check if lastEventOnset > longDurationThreshold - append issue if so
	if (lastEventOnset > longDurationThreshold) {
	issues.push(
	new Issue({
	file: event.file,
	code: 85,
	}),
	)
	}
	// check if the lastEventOnset < shortDurationThreshold - append issue if so
	if (lastEventOnset < shortDurationThreshold) {
	issues.push(
	new Issue({
	file: event.file,
	code: 86,
	}),
	)
	}
	}
	})
	return issues
	}