Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GenomicsDB on azure #8438

Merged
merged 1 commit into from
Dec 9, 2023
Merged

GenomicsDB on azure #8438

merged 1 commit into from
Dec 9, 2023

Conversation

lbergelson
Copy link
Member

@lbergelson lbergelson commented Jul 26, 2023

adding two new parameters which work together to allow passing through files from azure too genomicsDB
--header <vcf> which lets you specify a vcf file to use the header from as your merged header. Do not mess this up or you will likely be doomed.
--avoid-nio which disables GATK sanity checks that involve reading the files since this would require opening them on azure.

This needs tests but I wanted to put it here for @meganshand to try.

@meganshand meganshand self-assigned this Aug 2, 2023
@meganshand
Copy link
Contributor

I see I didn't respond here previously, but this branch has been working for me and is what I'm currently using. Would be great to get it merged along with #8470

@gatk-bot
Copy link

gatk-bot commented Nov 16, 2023

Github actions tests reported job failures from actions build 6895762275
Failures in the following jobs:

Test Type JDK Job ID Logs
integration 17.0.6+10 6895762275.11 logs
integration 17.0.6+10 6895762275.0 logs

@gatk-bot
Copy link

gatk-bot commented Nov 16, 2023

Github actions tests reported job failures from actions build 6896352136
Failures in the following jobs:

Test Type JDK Job ID Logs
integration 17.0.6+10 6896352136.11 logs
integration 17.0.6+10 6896352136.0 logs

@gatk-bot
Copy link

gatk-bot commented Dec 8, 2023

Github actions tests reported job failures from actions build 7143483544
Failures in the following jobs:

Test Type JDK Job ID Logs
cloud 17.0.6+10 7143483544.10 logs

@gatk-bot
Copy link

gatk-bot commented Dec 8, 2023

Github actions tests reported job failures from actions build 7144734512
Failures in the following jobs:

Test Type JDK Job ID Logs
cloud 17.0.6+10 7144734512.10 logs

* GATK's lack of support for az:// uri means that although genomicsdb can
  natively read them, parts of the java code crash when they're passed through
* Adding --avoid-nio and --header parameters
  these allow disabling all of the java codes interaction with the az:// links
  and simply pass them through to genomicsdb
  This disables some safeguards but allows operating on files in azure
* Move GenomicsDB version to released 1.5.1 for azure improved support

* There are no direct tests on azure since we do not yet have any infrastructure
  to generate the necessary tokens

---------

Co-authored-by: Nalini Ganapati <[email protected]>
Co-authored-by: Nalini Ganapati <[email protected]>
@lbergelson lbergelson marked this pull request as ready for review December 9, 2023 01:00
Copy link
Contributor

@droazen droazen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lbergelson lbergelson merged commit 2ad4a3e into master Dec 9, 2023
20 checks passed
@lbergelson lbergelson deleted the lb_genomicsdb_on_azure branch December 9, 2023 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants