-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(235) explicit output uri (#237)
#235 and friends --------- Co-authored-by: Dr. Ernie Prabhakar <[email protected]>
- Loading branch information
Showing
20 changed files
with
647 additions
and
240 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,9 +6,22 @@ Nextflow plugin for reading and writing Quilt packages as a FileSystem | |
developed by [Quilt Data](https://quiltdata.com/) that enables you read and write directly | ||
to Quilt packages using `quilt+s3` URIs wherever your Nextflow pipeline currently use `s3` URIs. | ||
|
||
In v0.8+, the plugin can even be used with "native" URIs, and it will automatically register a Quilt package at the root of the bucket. | ||
## NEW: Use nf-quilt plugin with existing S3 URIs | ||
|
||
Inspired by the original [`nf-quilt`](https://github.com/nextflow-io/nf-quilt) plugin (v0.2.0) developed by Seqera labs. | ||
In v0.8+, the plugin can even be used with "native" S3 URIs. You can continue using your exising S3 URIs,and Nextflow will write the data out as usual. However, simply by adding the `nf-quilt` plugin, you can also "overlay" that data with a Quilt package containing all the metadata from that run. | ||
|
||
For example: | ||
|
||
```shell | ||
nextflow run nf-core/rnaseq -plugins nf-quilt --outdir "s3://quilt-example-bucket/test/nf_quilt_rnaseq" | ||
# other parameters omitted for brevity | ||
``` | ||
|
||
will automatically create the package: | ||
|
||
```url | ||
quilt+s3://quilt-example-bucket#package=test/nf_quilt_rnaseq | ||
``` | ||
|
||
## I. Using the nf-quilt plugin in Production | ||
|
||
|
@@ -80,8 +93,8 @@ From the command-line, do, e.g.: | |
```bash | ||
# export NXF_VER=23.04.3 | ||
export LOG4J_DEBUG=true # for verbose logging | ||
export NXF_PLUGINS_TEST_REPOSITORY=https://github.com/quiltdata/nf-quilt/releases/download/0.8.0/nf-quilt-0.8.0-meta.json | ||
nextflow run main.nf -plugins [email protected].0 | ||
export NXF_PLUGINS_TEST_REPOSITORY=https://github.com/quiltdata/nf-quilt/releases/download/0.8.6/nf-quilt-0.8.6-meta.json | ||
nextflow run main.nf -plugins [email protected].6 | ||
``` | ||
|
||
For Tower, you can use the "Pre-run script" to set the environment variables. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -125,7 +125,7 @@ jacocoTestCoverageVerification { | |
violationRules { | ||
rule { | ||
limit { | ||
minimum = 0.65 | ||
minimum = 0.7 | ||
} | ||
} | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.