Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use md5sum to check download integrity with vdb-validate as fallback #5024

Merged
merged 21 commits into from
Apr 7, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
738077b
use md5sum to check download integrity with vdb-validate as fallback
suhrig Mar 1, 2024
62d8176
Merge branch 'master' into md5sum_replaces_vdb-validate
suhrig Mar 25, 2024
2248b3d
use md5sum to check download integrity with vdb-validate as fallback
suhrig Mar 1, 2024
5182880
Merge branch 'md5sum_replaces_vdb-validate' of github.com:suhrig/modu…
suhrig Mar 26, 2024
e2ad351
Merge branch 'master' into md5sum_replaces_vdb-validate
suhrig Mar 26, 2024
5c5239a
use vdb-validate when reliable and curl as fallback
suhrig Apr 1, 2024
49ce8ce
Merge branch 'md5sum_replaces_vdb-validate' of github.com:suhrig/modu…
suhrig Apr 1, 2024
248ebd6
long options
suhrig Apr 4, 2024
cb6ce70
no debug logging
suhrig Apr 4, 2024
7ff2f70
print error message when MD5 sum check fails
suhrig Apr 4, 2024
15b6c29
use md5sum to check download integrity with vdb-validate as fallback
suhrig Mar 1, 2024
429db7a
use vdb-validate when reliable and curl as fallback
suhrig Apr 1, 2024
4e95d48
long options
suhrig Apr 4, 2024
b709dca
no debug logging
suhrig Apr 4, 2024
dd832eb
print error message when MD5 sum check fails
suhrig Apr 4, 2024
1e2126d
Merge branch 'md5sum_replaces_vdb-validate' of github.com:suhrig/modu…
suhrig Apr 5, 2024
5d9f451
indent as blanks instead of tabs
suhrig Apr 5, 2024
4e66f9d
Merge branch 'master' into md5sum_replaces_vdb-validate
suhrig Apr 5, 2024
b59df94
update versions.yml MD5 sums for nf-test
suhrig Apr 6, 2024
5b2444e
Merge branch 'md5sum_replaces_vdb-validate' of github.com:suhrig/modu…
suhrig Apr 6, 2024
1451779
use short options for grep for compatibility
suhrig Apr 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion modules/nf-core/sratools/prefetch/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ channels:
- bioconda
- defaults
dependencies:
- bioconda::sra-tools=3.0.8
- bioconda::sra-tools=3.1.0
- conda-forge::curl=8.5.0
suhrig marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 2 additions & 2 deletions modules/nf-core/sratools/prefetch/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ process SRATOOLS_PREFETCH {

conda "${moduleDir}/environment.yml"
container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'https://depot.galaxyproject.org/singularity/sra-tools:3.0.8--h9f5acd7_0' :
'biocontainers/sra-tools:3.0.8--h9f5acd7_0' }"
'https://depot.galaxyproject.org/singularity/sra-tools:3.1.0--h9f5acd7_0' :
'biocontainers/sra-tools:3.1.0--h9f5acd7_0' }"

input:
tuple val(meta), val(id)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env bash

set -u
set -u -x
suhrig marked this conversation as resolved.
Show resolved Hide resolved

retry_with_backoff() {
local max_attempts=${1}
Expand Down Expand Up @@ -47,9 +47,16 @@ retry_with_backoff !{args2} \
!{args} \
!{id}

[ -f !{id}.sralite ] && vdb-validate !{id}.sralite || vdb-validate !{id}
# check file integrity using vdb-validate or (when archive contains no checksums) md5sum
vdb-validate !{id} > vdb-validate_result.txt 2>&1 || exit 1
if grep -q "checksums missing" vdb-validate_result.txt; then
VALID_MD5SUMS=$(curl -s -f -L --retry 3 --retry-delay 60 'https://locate.ncbi.nlm.nih.gov/sdl/2/retrieve?filetype=run&acc=!{id}')
LOCAL_MD5SUMS=$(md5sum !{id}/* | cut -f1 -d' ')
grep -q -F -f <(echo "$LOCAL_MD5SUMS") <(echo "$VALID_MD5SUMS") || exit 1
suhrig marked this conversation as resolved.
Show resolved Hide resolved
fi

cat <<-END_VERSIONS > versions.yml
"!{task.process}":
sratools: $(prefetch --version 2>&1 | grep -Eo '[0-9.]+')
curl: $(curl --version | head -n 1 | sed 's/^curl //; s/ .*$//')
END_VERSIONS