Simplified genotype likelihood calculation (no change in output) #6351

davidbenjamin · 2020-01-07T16:30:53Z

@jamesemery Could you review this? I think you may appreciate it. It took several tries, but I was finally able to write a stripped-down version of the code that actually slightly outperforms the old version. What I realized after a lot of profiling the old code and various failed rewrites was that cache-friendliness is the critical thing here. It turns out that this can be achieved without too many buffers, without precomputing the log frequencies, and without storing 2D and 3D arrays as flattened 1D arrays.

jamesemery

Do you have numbers for the performance here? How big is this code in the profiler before or after? I'm curious.

jamesemery · 2020-01-30T15:45:54Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeAlleleCounts.java

@@ -661,7 +661,9 @@ GenotypeAlleleCounts copy() {


    public void forEachAlleleIndexAndCount(final IntBiConsumer action) {
-        new IndexRange(0, distinctAlleleCount).forEach(n -> action.accept(sortedAlleleCounts[2*n], sortedAlleleCounts[2*n+1]));
+        for (int n = 0; n < distinctAlleleCount; n++) {


Did this show up in the profiler? I'm generally for optimizing away Java Stream calls wherever it makes a difference.

This change was a shot in the dark from before I realized the cache-cleverness of the original code, but it doesn't show up in the profiler. I'm sure it would if the code was an IntStream, but IndexRange is just a simple object with two integers -- there's nothing lurking behind it's constructor there way there is for Java streams.

davidbenjamin · 2020-01-30T16:28:48Z

Do you have numbers for the performance here? How big is this code in the profiler before or after? I'm curious.

The unit tests are about 5% faster. In practice, this won't affect HaplotyeCaller because the overwhelming majority of the CPU cost of those tests comes from the ploidy = 20, allele count = 6 cases. For anything else the genotyping likelihoods calculation is not only not a bottleneck, it's completely negligible.

samuelklee · 2022-01-21T16:11:02Z

Do we want to try to get this in at some point? @davidbenjamin if you'd like to push a quick rebase, I can try to take a stab at a review, although I'd guess @jamesemery would be better qualified.

davidbenjamin · 2022-01-21T20:34:55Z

@samuelklee you are my hero. Let me see how hard the rebase is.

davidbenjamin · 2022-01-27T21:18:29Z

@samuelklee It wasn't just a rebase, it was a complete rewrite because the old code had since become completely entangled with DRAGEN code. But I did it! Everything is passing, the code is dramatically simpler, and it's even a bit faster.

I have done my best to make a coherent commit history. I would recommend reviewing one commit at a time in side-by-side diff mode. Note that some commits rip out old code and replace it with pseudocode, deferring the new code to a later commit. Other commits tell a story of what all the different caches meant in order to motivate the simpification of later commits.

The baroqueness of the old code was motivated by three considerations:

cache-friendliness -- traversing all arrays by incrementing the innermost index, reads. This is absolutely essential.
flattening 3D arrays into 1D arrays. This was a premature optimization.
Precomputing addition operations -- this was misguided.

The DRAGEN code relied on these caches in a rather complex way, which fortunately turned out not to be necessary and which could be dramatically simplified. My notes on tracking all the variables from the parent genotype calculator down to the DRAGEN calculator are in this google doc: https://docs.google.com/document/d/1v6s57mUAwfj38nL3VdktjA059kYBkJfokq18IDy79E8/edit?usp=sharing

Good luck and don't hesitate to ask me to explain anything.

samuelklee · 2022-02-04T16:45:27Z

src/main/java/org/broadinstitute/hellbender/utils/MathUtils.java

+     * @param k number of successes
+     * @return the binomial coefficient
+     */
+    public static long exactBinomialCoefficient(final int n, final int k) {


Perhaps we should document the reasons we need our own implementation and beef up correctness and tests, or just consider using an available implementation?

Just iterating up over n and k, it appears that this implementation begins to disagree with CombinatoricsUtils.binomialCoefficient and Wolfram Alpha around n = 48:

n=48, k=21, MathUtils.exactBinomialCoefficient=22314239266529, CombinatoricsUtils.binomialCoefficient=22314239266528 n=48, k=22, MathUtils.exactBinomialCoefficient=27385657281649, CombinatoricsUtils.binomialCoefficient=27385657281648 n=48, k=26, MathUtils.exactBinomialCoefficient=27385657281649, CombinatoricsUtils.binomialCoefficient=27385657281648 n=48, k=27, MathUtils.exactBinomialCoefficient=22314239266529, CombinatoricsUtils.binomialCoefficient=22314239266528 n=49, k=18, MathUtils.exactBinomialCoefficient=11554258485617, CombinatoricsUtils.binomialCoefficient=11554258485616 n=49, k=19, MathUtils.exactBinomialCoefficient=18851684897585, CombinatoricsUtils.binomialCoefficient=18851684897584 n=49, k=20, MathUtils.exactBinomialCoefficient=28277527346377, CombinatoricsUtils.binomialCoefficient=28277527346376 n=49, k=21, MathUtils.exactBinomialCoefficient=39049918716426, CombinatoricsUtils.binomialCoefficient=39049918716424 n=49, k=22, MathUtils.exactBinomialCoefficient=49699896548179, CombinatoricsUtils.binomialCoefficient=49699896548176 n=49, k=23, MathUtils.exactBinomialCoefficient=58343356817427, CombinatoricsUtils.binomialCoefficient=58343356817424 n=49, k=24, MathUtils.exactBinomialCoefficient=63205303218877, CombinatoricsUtils.binomialCoefficient=63205303218876 n=49, k=25, MathUtils.exactBinomialCoefficient=63205303218877, CombinatoricsUtils.binomialCoefficient=63205303218876 ...

I haven't carefully tracked down the sources of the discrepancies, but just from skimming the CombinatoricsUtils implementation, it looks like there's additional logic to account for possible overflow in intermediate quantities.

Aside from these discrepancies, there are differences in how the implementations handle overflow in the final, returned value. It appears this implementation can silently return incorrect values (e.g., n = 100, k =40 returns -1---and one might naively assume from the bounds in line 650 that we can safely handle this regime), whereas the CombinatoricsUtils implementation throws a MathArithmeticException.

Just skimming the rest of the code in this PR, it looks like this code is only intended to calculate indices into an array? So that might naturally delimit the reasonable use cases. Perhaps we should roll an encapsulated implementation that only accounts for these use cases here, if the CombinatoricsUtils implementation does not suffice? (Note that we already trust CombinatoricsUtils to generate the cached exact factorials.)

How about for other use cases in the GATK that might want to make use of a general MathUtils implementation? In which regions of input space do we really need an exact/cheap/etc. calculation? Is it ever desired behavior to throw an exception, or do we prefer returning an approximate value? And so on.

I'm going to submit this review comment separately so you can go ahead and start thinking about it, if you like---I expect that it will take me a bit of time to become acquainted with both old/new versions of the rest of the code, anyway!

OK, now that I'm resuming my review, let's just use CombinatoricsUtils unless we care about possible performance differences due to caching of the factorials. It seems like we probably don't, based on the conversation on the main thread.

More generally, it might be nice to quickly run some tests that give us a sanity check on performance differences. Seems like the unit tests aren't too sensitive, but are there existing integration tests that might be better for this? Or maybe we have some high ploidy test data we could run?

samuelklee

Thanks for this work, @davidbenjamin! Kudos for significantly simplifying the main computation.

Many of the comments are notes to myself while I was working through the PR or are aimed at possible future work. (Also, sorry if some of them seem out of order---might be because I started off going through commit-by-commit, as you suggested.)

I think we might want to add some additional documentation or shuffle around a few more methods while we're here, though---primarily to better delineate the functionality of the GenotypeIndexCalculator/GenotypeLikelihoodCalculator(s) classes, which might better enable that future work!

There are a few comments regarding runtime/memory. More generally it might be nice to run and document a few quick sanity checks on realistic data, if you haven't already. Given the complexity of the old code, it's hard for me to say whether one might obviously expect regressions/improvments. But maintainability has definitely improved!

BTW, I only briefly skimmed the DRAGEN changes, but I did run a few of the exact match integration tests and checked coverage. Hopefully those tests do indeed cover changes, but maybe @jamesemery can take a quick look.

samuelklee · 2022-02-22T16:32:15Z

src/main/java/org/broadinstitute/hellbender/utils/MathUtils.java

+     * @param k number of successes
+     * @return the binomial coefficient
+     */
+    public static long exactBinomialCoefficient(final int n, final int k) {


OK, now that I'm resuming my review, let's just use CombinatoricsUtils unless we care about possible performance differences due to caching of the factorials. It seems like we probably don't, based on the conversation on the main thread.

More generally, it might be nice to quickly run some tests that give us a sanity check on performance differences. Seems like the unit tests aren't too sensitive, but are there existing integration tests that might be better for this? Or maybe we have some high ploidy test data we could run?

...ava/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculators.java

src/main/java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeAlleleCounts.java

...java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculator.java

davidbenjamin · 2022-03-14T05:35:52Z

Things left for later:

GenotypeIndexCalculator sometimes interacts with primitive arrays, sometimes with GenotypeAlleleCounts
GenotypeLikelihoodCalculator has extraneous responsibilities and doesn't interact with GenotypeAlleleCounts as well as it should.
alleleCountsToIndex(final GenotypeAlleleCounts newGAC, final int[] newToOldAlleleMap) in GenotypeIndexCalculator needs refactoring.
GenotypeLikelihoodCalculators is really just a cache of GenotypeAlleleCounts.
GenotypeAlleleCounts has some unused and barely-used methods, and it precomputes a lot of quantities that are not often needed and could be computed on-the-fly without difficulty or expense.

davidbenjamin · 2022-03-14T14:32:30Z

@samuelklee I wrote some benchmarks for the exact combinatorics and you were right, my optimization was pointless. Although the CombinatoricsUtils method does explicitly multiply out instead of using cached factorials 1) the number of multiplications is only min(ploidy, (allele count - 1)), and 2) it actually takes quite a while (much larger than reasonable ploidy and allele count) for multiplication to take longer than the memory access of stored factorials.

I have removed this error in judgment.

gatk-bot · 2022-03-14T15:10:42Z

Travis reported job failures from build 38091
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
unit	openjdk11	38091.13	logs
integration	openjdk11	38091.12	logs
unit	openjdk8	38091.3	logs

gatk-bot · 2022-03-15T07:34:18Z

Travis reported job failures from build 38110
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
cloud	openjdk11	38110.14	logs
cloud	openjdk8	38110.1	logs
unit	openjdk11	38110.13	logs
integration	openjdk11	38110.12	logs
unit	openjdk8	38110.3	logs
variantcalling	openjdk8	38110.4	logs
integration	openjdk8	38110.2	logs

davidbenjamin · 2022-03-15T13:06:38Z

You know, I think I will clean up all the entangled genotype allele count caching and iterating.

I have benchmarked pretty thoroughly and discovered that caching GenotypeAlleleCounts for the sake of iterating in sequence is totally pointless. The GenotypeAlleleCounts::increase method is already so efficient that it makes no difference. In fact, caching is slower than using increase when the allele count and ploidy yield more than a few hundred genotypes. Caching is a bit faster for the commonest cases of 2 or 3 alleles in a diploid genotype, but the savings is less than a tenth of a second over an entire WGS run.

davidbenjamin · 2022-03-16T03:32:19Z

@samuelklee Now it's back to you. I agreed with and implemented all of your suggestions. GenotypeIndexCalculator, GenotypeAlleleCounts, GenotypeLikelihoodsCalculator and GenotypeLikelihoodsCalculator (renamed to GenotypesCache) now have clearly-defined roles. A lot of premature optimization is gone.

samuelklee · 2022-03-17T14:12:49Z

Thanks @davidbenjamin for essentially refactoring this code three times now!

Looking forward to reviewing your latest changes, but it may have to wait until early next week. Apologies for the delay. In the meantime, I see that there is a minor rebase conflict—up to you if you want to address it now, no biggie if you want to wait until after review.

samuelklee · 2022-04-12T20:27:50Z

@davidbenjamin just letting you know---I will try my best to get this back to you before I head out on vacation next week! Hopefully still well before the other delaying factor.

samuelklee · 2022-04-28T14:43:39Z

@davidbenjamin as we just discussed, back from vacation now and modulo other circumstances (which we also just discussed) should be able to get this back to you in a week. If you could squash the commits after the last review, that might actually be a little cleaner for me to review---perhaps comment a copy of the squashed commit messages here if you'd like to keep them for posterity. Thanks!

samuelklee · 2022-05-13T14:35:36Z

...in/java/org/broadinstitute/hellbender/tools/copynumber/models/AlleleFractionLikelihoods.java

@@ -87,8 +86,7 @@ static double hetLogLikelihood(final AlleleFractionGlobalParameters parameters,
                - n * log(majorFraction + minorFraction * lambda0RefMinor);
        final double refMinorLogLikelihood = logNotPi + logcRefMinor + Gamma.logGamma(rhoRefMinor) - rhoRefMinor * log(tauRefMinor);

-        final double outlierLogLikelihood = logPi + log10ToLog(log10Factorial(a) + log10Factorial(r) - log10Factorial(a + r + 1));
-
+        final double outlierLogLikelihood = logPi - FastMath.log(a + r + 1) - CombinatoricsUtils.binomialCoefficientLog(a+r,a);


In principle, I support the removal of log10Factorial, but also note #7652. Although this expression is mathematically equivalent, numerically, it will result in a slight change in the output of ModelSegments---which has been stable for 3 years now.

Not a problem, but @tmelman should be aware of this potential change and plan accordingly, depending on what order the PRs are merged. And perhaps @droazen can make sure that minor numerical changes are mentioned in the release notes when appropriate.

Minor nitpick: can you fix the whitespace in the arguments of binomialCoefficientLog?

Whitespace has been fixed.

@cmnbroad may also want to be aware of these numerical changes, since I believe he is also seeing changes in his Java 17 branch.

samuelklee

@davidbenjamin sorry for the delays. I tried to take a stab at re-review---just went ahead and pushed my own rebased branch to db_glcalc_rebase, but I think it's worth doing pushing the rebase here to re-enable tests on GitHub Actions and make any necessary fixes before I go further. Looks like there are some failing tests due to the numerical changes in MathUtils methods. And while I'm in favor of such changes, they might be tricky, so we should make them carefully---see comment about the upcoming PR #7652 as well.

But in general, the further refactorings post-review look much cleaner! So thanks again for revisiting this code so many times.

src/main/java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeAlleleCounts.java

.../org/broadinstitute/hellbender/tools/walkers/genotyper/afcalc/AlleleFrequencyCalculator.java

src/main/java/org/broadinstitute/hellbender/tools/walkers/mutect/Mutect2Engine.java

src/main/java/org/broadinstitute/hellbender/utils/recalibration/RecalDatum.java

samuelklee · 2022-06-27T15:43:52Z

@davidbenjamin any bandwidth for keeping up momentum on this one? Maybe we can try to get a rebase and activate GitHub Actions so we can see where we're at?

gatk-bot · 2022-08-04T22:19:24Z

Github actions tests reported job failures from actions build 2799863346
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
cloud	11	2799863346.11	logs
cloud	8	2799863346.10	logs
unit	11	2799863346.13	logs
integration	11	2799863346.12	logs
unit	8	2799863346.1	logs
integration	8	2799863346.0	logs

gatk-bot · 2022-08-07T03:46:35Z

Github actions tests reported job failures from actions build 2811164603
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
unit	11	2811164603.13	logs
integration	11	2811164603.12	logs
unit	8	2811164603.1	logs
integration	8	2811164603.0	logs

codecov · 2022-08-07T13:05:29Z

Codecov Report

Merging #6351 (28549c9) into master (b370c06) will increase coverage by 0.361%.
The diff coverage is 93.182%.

❗ Current head 28549c9 differs from pull request most recent head 440130f. Consider uploading reports for the commit 440130f to get more accurate results

Additional details and impacted files

@@               Coverage Diff               @@
##              master     #6351       +/-   ##
===============================================
+ Coverage     86.297%   86.658%   +0.361%     
+ Complexity     39445     38662      -783     
===============================================
  Files           2351      2329       -22     
  Lines         185118    181309     -3809     
  Branches       20397     19882      -515     
===============================================
- Hits          159752    157119     -2633     
+ Misses         18185     17210      -975     
+ Partials        7181      6980      -201

Impacted Files	Coverage Δ
...tools/copynumber/ModelSegmentsIntegrationTest.java	`96.078% <ø> (ø)`
...dinstitute/hellbender/utils/MathUtilsUnitTest.java	`92.606% <ø> (+0.842%)`	⬆️
...s/help/DocumentationGenerationIntegrationTest.java	`23.529% <66.667%> (+9.244%)`	⬆️
.../broadinstitute/hellbender/utils/MannWhitneyU.java	`94.118% <75.000%> (-0.026%)`	⬇️
...ender/utils/genotyper/GenotypePriorCalculator.java	`88.462% <80.000%> (+0.962%)`	⬆️
...bender/utils/recalibration/RecalDatumUnitTest.java	`95.238% <80.000%> (ø)`
...ols/walkers/genotyper/GenotypeIndexCalculator.java	`83.019% <83.019%> (ø)`
...bender/tools/walkers/genotyper/GenotypesCache.java	`86.957% <86.957%> (ø)`
...rs/genotyper/afcalc/AlleleFrequencyCalculator.java	`84.328% <86.957%> (+2.106%)`	⬆️
...ute/hellbender/utils/recalibration/RecalDatum.java	`87.500% <90.909%> (-0.735%)`	⬇️
... and 25 more

... and 287 files with indirect coverage changes

davidbenjamin · 2023-05-12T15:08:52Z

#carrot(HaplotypeCaller CARROT Regression Tests, VariantCallingCarrotOrchestrated.gatk_docker, )

davidbenjamin · 2023-05-12T15:09:32Z

Does it work when I type the magical CARROT incantation?

CarrotBroadBot · 2023-05-12T15:09:44Z

🥕CARROT🥕 run started

Test: HaplotypeCaller CARROT Regression Tests | Status: building

Run: HaplotypeCaller CARROT Regression Tests_run_2023-05-12 15:09:42.984220519 UTC

Full details

 
 {
  "run_id": "2891f1b5-e415-4d62-ab4e-49fba755b096",
  "test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
  "run_group_ids": [],
  "name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-12 15:09:42.984220519 UTC",
  "status": "building",
  "test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
  "test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
  "test_wdl_dependencies": null,
  "test_wdl_dependencies_hash": null,
  "eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
  "eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
  "eval_wdl_dependencies": null,
  "eval_wdl_dependencies_hash": null,
  "test_input": {
    "VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
    "VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
    "VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
    "VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
    "VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
    "VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
    "VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
    "VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
    "VariantCallingCarrotOrchestrated.contamination": 0.0,
    "VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
    "VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
    "VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
    "VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
    "VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:2022-03-04-4.2.5.0-9-gb097f75c5-NIGHTLY-SNAPSHOT",
    "VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|d791874a0490fec31b2587c06ad6c2f2ad397000",
    "VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
    "VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
    "VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
  },
  "test_options": {
    "read_from_cache": false
  },
  "eval_input": {
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
    "BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
    "BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
    ],
    "BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
      "HCR",
      "LCR"
    ]
  },
  "eval_options": {
    "read_from_cache": false
  },
  "test_cromwell_job_id": null,
  "eval_cromwell_job_id": null,
  "created_at": "2023-05-12T15:09:42.984289",
  "created_by": null,
  "finished_at": null,
  "results": null,
  "errors": null
}

davidbenjamin · 2023-05-12T15:22:56Z

It does!

CarrotBroadBot · 2023-05-13T01:07:35Z

🥕CARROT🥕 run finished

Test: HaplotypeCaller CARROT Regression Tests | Status: succeeded

Run: HaplotypeCaller CARROT Regression Tests_run_2023-05-12 15:09:42.984220519 UTC

Results

Results
CHM controlHCprocesshours	79.42513333333335
CHM controlHCsystemhours	0.15543611111111108
CHM controlHCwallclockhours	56.046666666666674
CHM controlHCwallclockmax	3.0881333333333334
CHM controlMonitoringLogs	View in the GCS Console
CHM controlindelF1Score	0.8724
CHM controlindelPrecision	0.8814
CHM controlsnpF1Score	0.9784
CHM controlsnpPrecision	0.9706
CHM controlsnpRecall	0.9863
CHM controlsummary	View in the GCS Console
CHM evalHCprocesshours	84.94748611111112
CHM evalHCsystemhours	0.19002777777777768
CHM evalHCwallclockhours	61.06326111111111
CHM evalHCwallclockmax	3.2047833333333333
CHM evalMonitoringLogs	View in the GCS Console
CHM evalindelF1Score	0.8724
CHM evalindelPrecision	0.8814
CHM evalsnpF1Score	0.9784
CHM evalsnpPrecision	0.9706
CHM evalsnpRecall	0.9863
CHM evalsummary	View in the GCS Console
EXOME1 controlindelF1Score	0.727
EXOME1 controlindelPrecision	0.632
EXOME1 controlsnpF1Score	0.9878
EXOME1 controlsnpPrecision	0.9815
EXOME1 controlsnpRecall	0.9941
EXOME1 controlsummary	View in the GCS Console
EXOME1 evalindelF1Score	0.727
EXOME1 evalindelPrecision	0.632
EXOME1 evalsnpF1Score	0.9878
EXOME1 evalsnpPrecision	0.9815
EXOME1 evalsnpRecall	0.9941
EXOME1 evalsummary	View in the GCS Console
NIST controlHCprocesshours	96.55857222222222
NIST controlHCsystemhours	0.1707444444444444
NIST controlHCwallclockhours	69.28645
NIST controlHCwallclockmax	3.8631972222222224
NIST controlMonitoringLogs	View in the GCS Console
NIST controlindelF1Score	0.9902
NIST controlindelPrecision	0.9903
NIST controlsnpF1Score	0.9899
NIST controlsnpPrecision	0.9887
NIST controlsnpRecall	0.9911
NIST controlsummary	View in the GCS Console
NIST evalHCprocesshours	103.49634722222224
NIST evalHCsystemhours	0.20633611111111116
NIST evalHCwallclockhours	75.91255833333332
NIST evalHCwallclockmax	3.76305
NIST evalMonitoringLogs	View in the GCS Console
NIST evalindelF1Score	0.9902
NIST evalindelPrecision	0.9903
NIST evalsnpF1Score	0.9899
NIST evalsnpPrecision	0.9887
NIST evalsnpRecall	0.9911
NIST evalsummary	View in the GCS Console
ROC_Plots_Reported	View in the GCS Console

Full details

 
 {
  "run_id": "2891f1b5-e415-4d62-ab4e-49fba755b096",
  "test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
  "run_group_ids": [],
  "name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-12 15:09:42.984220519 UTC",
  "status": "succeeded",
  "test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
  "test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
  "test_wdl_dependencies": null,
  "test_wdl_dependencies_hash": null,
  "eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
  "eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
  "eval_wdl_dependencies": null,
  "eval_wdl_dependencies_hash": null,
  "test_input": {
    "VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
    "VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
    "VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
    "VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
    "VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
    "VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
    "VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
    "VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
    "VariantCallingCarrotOrchestrated.contamination": 0.0,
    "VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
    "VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
    "VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
    "VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
    "VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:2022-03-04-4.2.5.0-9-gb097f75c5-NIGHTLY-SNAPSHOT",
    "VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|d791874a0490fec31b2587c06ad6c2f2ad397000",
    "VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
    "VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
    "VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
  },
  "test_options": {
    "read_from_cache": false
  },
  "eval_input": {
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
    "BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
    "BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
    ],
    "BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
      "HCR",
      "LCR"
    ]
  },
  "eval_options": {
    "read_from_cache": false
  },
  "test_cromwell_job_id": "b7d06271-38b7-47d2-9d7c-af5543460de9",
  "eval_cromwell_job_id": "beb77715-227e-4dbd-803f-4458c83607c8",
  "created_at": "2023-05-12T15:09:42.984289",
  "created_by": null,
  "finished_at": "2023-05-13T01:07:11.594",
  "results": {
    "CHM controlHCprocesshours": "79.42513333333335",
    "CHM controlHCsystemhours": "0.15543611111111108",
    "CHM controlHCwallclockhours": "56.046666666666674",
    "CHM controlHCwallclockmax": "3.0881333333333334",
    "CHM controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CHMSampleHeadToHead/BenchmarkComparison/f1b0b4cf-1a3f-47b3-84fa-529f118419ce/call-CONTROLRuntimeTask/monitoring.pdf",
    "CHM controlindelF1Score": "0.8724",
    "CHM controlindelPrecision": "0.8814",
    "CHM controlsnpF1Score": "0.9784",
    "CHM controlsnpPrecision": "0.9706",
    "CHM controlsnpRecall": "0.9863",
    "CHM controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CHMSampleHeadToHead/BenchmarkComparison/f1b0b4cf-1a3f-47b3-84fa-529f118419ce/call-BenchmarkVCFControlSample/Benchmark/fb68536c-eb99-4d0d-a5c3-4f5accf94546/call-CombineSummaries/summary.csv",
    "CHM evalHCprocesshours": "84.94748611111112",
    "CHM evalHCsystemhours": "0.19002777777777768",
    "CHM evalHCwallclockhours": "61.06326111111111",
    "CHM evalHCwallclockmax": "3.2047833333333333",
    "CHM evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CHMSampleHeadToHead/BenchmarkComparison/f1b0b4cf-1a3f-47b3-84fa-529f118419ce/call-EVALRuntimeTask/monitoring.pdf",
    "CHM evalindelF1Score": "0.8724",
    "CHM evalindelPrecision": "0.8814",
    "CHM evalsnpF1Score": "0.9784",
    "CHM evalsnpPrecision": "0.9706",
    "CHM evalsnpRecall": "0.9863",
    "CHM evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CHMSampleHeadToHead/BenchmarkComparison/f1b0b4cf-1a3f-47b3-84fa-529f118419ce/call-BenchmarkVCFTestSample/Benchmark/4353eabb-b85f-4cce-a275-4dba68f9d644/call-CombineSummaries/summary.csv",
    "EXOME1 controlindelF1Score": "0.727",
    "EXOME1 controlindelPrecision": "0.632",
    "EXOME1 controlsnpF1Score": "0.9878",
    "EXOME1 controlsnpPrecision": "0.9815",
    "EXOME1 controlsnpRecall": "0.9941",
    "EXOME1 controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-EXOME1SampleHeadToHead/BenchmarkComparison/bec4eb1e-a99f-4e27-a91c-a56274c3824a/call-BenchmarkVCFControlSample/Benchmark/4d248a53-86e2-46f5-9d28-3364c82b9d0c/call-CombineSummaries/summary.csv",
    "EXOME1 evalindelF1Score": "0.727",
    "EXOME1 evalindelPrecision": "0.632",
    "EXOME1 evalsnpF1Score": "0.9878",
    "EXOME1 evalsnpPrecision": "0.9815",
    "EXOME1 evalsnpRecall": "0.9941",
    "EXOME1 evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-EXOME1SampleHeadToHead/BenchmarkComparison/bec4eb1e-a99f-4e27-a91c-a56274c3824a/call-BenchmarkVCFTestSample/Benchmark/fcd023fe-e278-475d-8fce-613b57518972/call-CombineSummaries/summary.csv",
    "NIST controlHCprocesshours": "96.55857222222222",
    "NIST controlHCsystemhours": "0.1707444444444444",
    "NIST controlHCwallclockhours": "69.28645",
    "NIST controlHCwallclockmax": "3.8631972222222224",
    "NIST controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-NISTSampleHeadToHead/BenchmarkComparison/243c7bf2-b0d7-48ed-acd0-e2ebd74b9fd3/call-CONTROLRuntimeTask/monitoring.pdf",
    "NIST controlindelF1Score": "0.9902",
    "NIST controlindelPrecision": "0.9903",
    "NIST controlsnpF1Score": "0.9899",
    "NIST controlsnpPrecision": "0.9887",
    "NIST controlsnpRecall": "0.9911",
    "NIST controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-NISTSampleHeadToHead/BenchmarkComparison/243c7bf2-b0d7-48ed-acd0-e2ebd74b9fd3/call-BenchmarkVCFControlSample/Benchmark/135b02c2-d7c5-4fd2-9cc5-cdeeed953bbc/call-CombineSummaries/summary.csv",
    "NIST evalHCprocesshours": "103.49634722222224",
    "NIST evalHCsystemhours": "0.20633611111111116",
    "NIST evalHCwallclockhours": "75.91255833333332",
    "NIST evalHCwallclockmax": "3.76305",
    "NIST evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-NISTSampleHeadToHead/BenchmarkComparison/243c7bf2-b0d7-48ed-acd0-e2ebd74b9fd3/call-EVALRuntimeTask/monitoring.pdf",
    "NIST evalindelF1Score": "0.9902",
    "NIST evalindelPrecision": "0.9903",
    "NIST evalsnpF1Score": "0.9899",
    "NIST evalsnpPrecision": "0.9887",
    "NIST evalsnpRecall": "0.9911",
    "NIST evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-NISTSampleHeadToHead/BenchmarkComparison/243c7bf2-b0d7-48ed-acd0-e2ebd74b9fd3/call-BenchmarkVCFTestSample/Benchmark/ad8885d7-137d-4645-b37d-f54f8362713d/call-CombineSummaries/summary.csv",
    "ROC_Plots_Reported": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CreateHTMLReport/report.html"
  },
  "errors": null
}

CarrotBroadBot · 2023-05-13T01:15:34Z

🥕CARROT🥕 report map stub finished

for test HaplotypeCaller CARROT Regression Tests (run: 2891f1b5-e415-4d62-ab4e-49fba755b096)

File	URI
empty_notebook	View in the GCS Console
html_report	View in the GCS Console
populated_notebook	View in the GCS Console
run_csv_zip	View in the GCS Console

davidbenjamin · 2023-05-15T22:13:49Z

Rebased on top of latest master. Going to run carrot again.

gatk-bot · 2023-05-15T22:35:05Z

Github actions tests reported job failures from actions build 4985658896
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
cloud	17.0.6+10	4985658896.10	logs
cloud	17.0.6+10	4985658896.10	logs
cloud	17.0.6+10	4985658896.10	logs
cloud	17.0.6+10	4985658896.10	logs

davidbenjamin · 2023-05-15T22:38:01Z

#carrot(HaplotypeCaller CARROT Regression Tests, VariantCallingCarrotOrchestrated.gatk_docker, )

CarrotBroadBot · 2023-05-15T22:38:44Z

🥕CARROT🥕 run started

Test: HaplotypeCaller CARROT Regression Tests | Status: building

Run: HaplotypeCaller CARROT Regression Tests_run_2023-05-15 22:38:43.733246263 UTC

Full details

 
 {
  "run_id": "1bf5817a-d896-41de-b193-4a53a1264470",
  "test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
  "run_group_ids": [],
  "name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-15 22:38:43.733246263 UTC",
  "status": "building",
  "test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
  "test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
  "test_wdl_dependencies": null,
  "test_wdl_dependencies_hash": null,
  "eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
  "eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
  "eval_wdl_dependencies": null,
  "eval_wdl_dependencies_hash": null,
  "test_input": {
    "VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
    "VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
    "VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
    "VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
    "VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
    "VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
    "VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
    "VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
    "VariantCallingCarrotOrchestrated.contamination": 0.0,
    "VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
    "VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
    "VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
    "VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
    "VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:2022-03-04-4.2.5.0-9-gb097f75c5-NIGHTLY-SNAPSHOT",
    "VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|fde1064710b236c7cdab83248817093046494660",
    "VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
    "VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
    "VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
  },
  "test_options": {
    "read_from_cache": false
  },
  "eval_input": {
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
    "BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
    "BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
    ],
    "BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
      "HCR",
      "LCR"
    ]
  },
  "eval_options": {
    "read_from_cache": false
  },
  "test_cromwell_job_id": null,
  "eval_cromwell_job_id": null,
  "created_at": "2023-05-15T22:38:43.733338",
  "created_by": null,
  "finished_at": null,
  "results": null,
  "errors": null
}

CarrotBroadBot · 2023-05-16T08:31:35Z

🥕CARROT🥕 run finished

Test: HaplotypeCaller CARROT Regression Tests | Status: succeeded

Run: HaplotypeCaller CARROT Regression Tests_run_2023-05-15 22:38:43.733246263 UTC

Results

Results
CHM controlHCprocesshours	78.81892777777776
CHM controlHCsystemhours	0.15627777777777782
CHM controlHCwallclockhours	55.94185833333335
CHM controlHCwallclockmax	3.053286111111111
CHM controlMonitoringLogs	View in the GCS Console
CHM controlindelF1Score	0.8724
CHM controlindelPrecision	0.8814
CHM controlsnpF1Score	0.9784
CHM controlsnpPrecision	0.9706
CHM controlsnpRecall	0.9863
CHM controlsummary	View in the GCS Console
CHM evalHCprocesshours	84.33091111111112
CHM evalHCsystemhours	0.18621944444444444
CHM evalHCwallclockhours	61.43
CHM evalHCwallclockmax	3.073069444444444
CHM evalMonitoringLogs	View in the GCS Console
CHM evalindelF1Score	0.8724
CHM evalindelPrecision	0.8814
CHM evalsnpF1Score	0.9784
CHM evalsnpPrecision	0.9706
CHM evalsnpRecall	0.9863
CHM evalsummary	View in the GCS Console
EXOME1 controlindelF1Score	0.727
EXOME1 controlindelPrecision	0.632
EXOME1 controlsnpF1Score	0.9878
EXOME1 controlsnpPrecision	0.9815
EXOME1 controlsnpRecall	0.9941
EXOME1 controlsummary	View in the GCS Console
EXOME1 evalindelF1Score	0.727
EXOME1 evalindelPrecision	0.632
EXOME1 evalsnpF1Score	0.9878
EXOME1 evalsnpPrecision	0.9815
EXOME1 evalsnpRecall	0.9941
EXOME1 evalsummary	View in the GCS Console
NIST controlHCprocesshours	99.85891111111113
NIST controlHCsystemhours	0.17817777777777768
NIST controlHCwallclockhours	70.22329166666665
NIST controlHCwallclockmax	3.8036305555555554
NIST controlMonitoringLogs	View in the GCS Console
NIST controlindelF1Score	0.9902
NIST controlindelPrecision	0.9903
NIST controlsnpF1Score	0.9899
NIST controlsnpPrecision	0.9887
NIST controlsnpRecall	0.9911
NIST controlsummary	View in the GCS Console
NIST evalHCprocesshours	104.20126111111112
NIST evalHCsystemhours	0.20587777777777783
NIST evalHCwallclockhours	76.10080000000004
NIST evalHCwallclockmax	3.949438888888889
NIST evalMonitoringLogs	View in the GCS Console
NIST evalindelF1Score	0.9902
NIST evalindelPrecision	0.9903
NIST evalsnpF1Score	0.9899
NIST evalsnpPrecision	0.9887
NIST evalsnpRecall	0.9911
NIST evalsummary	View in the GCS Console
ROC_Plots_Reported	View in the GCS Console

Full details

 
 {
  "run_id": "1bf5817a-d896-41de-b193-4a53a1264470",
  "test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
  "run_group_ids": [],
  "name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-15 22:38:43.733246263 UTC",
  "status": "succeeded",
  "test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
  "test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
  "test_wdl_dependencies": null,
  "test_wdl_dependencies_hash": null,
  "eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
  "eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
  "eval_wdl_dependencies": null,
  "eval_wdl_dependencies_hash": null,
  "test_input": {
    "VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
    "VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
    "VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
    "VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
    "VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
    "VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
    "VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
    "VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
    "VariantCallingCarrotOrchestrated.contamination": 0.0,
    "VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
    "VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
    "VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
    "VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
    "VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:2022-03-04-4.2.5.0-9-gb097f75c5-NIGHTLY-SNAPSHOT",
    "VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|fde1064710b236c7cdab83248817093046494660",
    "VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
    "VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
    "VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
  },
  "test_options": {
    "read_from_cache": false
  },
  "eval_input": {
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
    "BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
    "BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
    ],
    "BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
      "HCR",
      "LCR"
    ]
  },
  "eval_options": {
    "read_from_cache": false
  },
  "test_cromwell_job_id": "c18d0f5f-52c9-4e60-92ed-e4b33c6553c7",
  "eval_cromwell_job_id": "9c49383b-01a9-4bc0-90fa-cde7e1090a47",
  "created_at": "2023-05-15T22:38:43.733338",
  "created_by": null,
  "finished_at": "2023-05-16T08:30:08.614",
  "results": {
    "CHM controlHCprocesshours": "78.81892777777776",
    "CHM controlHCsystemhours": "0.15627777777777782",
    "CHM controlHCwallclockhours": "55.94185833333335",
    "CHM controlHCwallclockmax": "3.053286111111111",
    "CHM controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CHMSampleHeadToHead/BenchmarkComparison/deb85607-d693-4232-a4da-0fb88dd29cad/call-CONTROLRuntimeTask/monitoring.pdf",
    "CHM controlindelF1Score": "0.8724",
    "CHM controlindelPrecision": "0.8814",
    "CHM controlsnpF1Score": "0.9784",
    "CHM controlsnpPrecision": "0.9706",
    "CHM controlsnpRecall": "0.9863",
    "CHM controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CHMSampleHeadToHead/BenchmarkComparison/deb85607-d693-4232-a4da-0fb88dd29cad/call-BenchmarkVCFControlSample/Benchmark/c0877490-fd2d-4f42-bb92-f06210e94d95/call-CombineSummaries/summary.csv",
    "CHM evalHCprocesshours": "84.33091111111112",
    "CHM evalHCsystemhours": "0.18621944444444444",
    "CHM evalHCwallclockhours": "61.43",
    "CHM evalHCwallclockmax": "3.073069444444444",
    "CHM evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CHMSampleHeadToHead/BenchmarkComparison/deb85607-d693-4232-a4da-0fb88dd29cad/call-EVALRuntimeTask/monitoring.pdf",
    "CHM evalindelF1Score": "0.8724",
    "CHM evalindelPrecision": "0.8814",
    "CHM evalsnpF1Score": "0.9784",
    "CHM evalsnpPrecision": "0.9706",
    "CHM evalsnpRecall": "0.9863",
    "CHM evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CHMSampleHeadToHead/BenchmarkComparison/deb85607-d693-4232-a4da-0fb88dd29cad/call-BenchmarkVCFTestSample/Benchmark/a15fdeb6-16e8-48d7-82cb-168726f4dc18/call-CombineSummaries/summary.csv",
    "EXOME1 controlindelF1Score": "0.727",
    "EXOME1 controlindelPrecision": "0.632",
    "EXOME1 controlsnpF1Score": "0.9878",
    "EXOME1 controlsnpPrecision": "0.9815",
    "EXOME1 controlsnpRecall": "0.9941",
    "EXOME1 controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-EXOME1SampleHeadToHead/BenchmarkComparison/75ea4d64-414f-43aa-a8d6-9c34870b1491/call-BenchmarkVCFControlSample/Benchmark/0f001ca8-d7af-4d01-b9ef-d6ddbe35317d/call-CombineSummaries/summary.csv",
    "EXOME1 evalindelF1Score": "0.727",
    "EXOME1 evalindelPrecision": "0.632",
    "EXOME1 evalsnpF1Score": "0.9878",
    "EXOME1 evalsnpPrecision": "0.9815",
    "EXOME1 evalsnpRecall": "0.9941",
    "EXOME1 evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-EXOME1SampleHeadToHead/BenchmarkComparison/75ea4d64-414f-43aa-a8d6-9c34870b1491/call-BenchmarkVCFTestSample/Benchmark/540fbadc-ba57-4012-8ff1-76461ecb7bb3/call-CombineSummaries/summary.csv",
    "NIST controlHCprocesshours": "99.85891111111113",
    "NIST controlHCsystemhours": "0.17817777777777768",
    "NIST controlHCwallclockhours": "70.22329166666665",
    "NIST controlHCwallclockmax": "3.8036305555555554",
    "NIST controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-NISTSampleHeadToHead/BenchmarkComparison/75625b9d-e48b-4859-803e-58989e3ccf62/call-CONTROLRuntimeTask/monitoring.pdf",
    "NIST controlindelF1Score": "0.9902",
    "NIST controlindelPrecision": "0.9903",
    "NIST controlsnpF1Score": "0.9899",
    "NIST controlsnpPrecision": "0.9887",
    "NIST controlsnpRecall": "0.9911",
    "NIST controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-NISTSampleHeadToHead/BenchmarkComparison/75625b9d-e48b-4859-803e-58989e3ccf62/call-BenchmarkVCFControlSample/Benchmark/21373bda-c620-4200-ad29-1e3886ea52ad/call-CombineSummaries/summary.csv",
    "NIST evalHCprocesshours": "104.20126111111112",
    "NIST evalHCsystemhours": "0.20587777777777783",
    "NIST evalHCwallclockhours": "76.10080000000004",
    "NIST evalHCwallclockmax": "3.949438888888889",
    "NIST evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-NISTSampleHeadToHead/BenchmarkComparison/75625b9d-e48b-4859-803e-58989e3ccf62/call-EVALRuntimeTask/monitoring.pdf",
    "NIST evalindelF1Score": "0.9902",
    "NIST evalindelPrecision": "0.9903",
    "NIST evalsnpF1Score": "0.9899",
    "NIST evalsnpPrecision": "0.9887",
    "NIST evalsnpRecall": "0.9911",
    "NIST evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-NISTSampleHeadToHead/BenchmarkComparison/75625b9d-e48b-4859-803e-58989e3ccf62/call-BenchmarkVCFTestSample/Benchmark/b91bffd4-8057-453f-a8e2-4767648da91a/call-CombineSummaries/summary.csv",
    "ROC_Plots_Reported": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CreateHTMLReport/report.html"
  },
  "errors": null
}

CarrotBroadBot · 2023-05-16T08:39:34Z

🥕CARROT🥕 report map stub finished

for test HaplotypeCaller CARROT Regression Tests (run: 1bf5817a-d896-41de-b193-4a53a1264470)

File	URI
empty_notebook	View in the GCS Console
html_report	View in the GCS Console
populated_notebook	View in the GCS Console
run_csv_zip	View in the GCS Console

jamesemery · 2023-05-16T17:14:55Z

#carrot(HaplotypeCaller CARROT Regression Tests, VariantCallingCarrotOrchestrated.gatk_docker, )

CarrotBroadBot · 2023-05-16T17:15:44Z

🥕CARROT🥕 run started

Test: HaplotypeCaller CARROT Regression Tests | Status: building

Run: HaplotypeCaller CARROT Regression Tests_run_2023-05-16 17:15:43.799630906 UTC

Full details

 
 {
  "run_id": "00d212a9-8eac-4575-9cde-b0a911b3c2f2",
  "test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
  "run_group_ids": [],
  "name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-16 17:15:43.799630906 UTC",
  "status": "building",
  "test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
  "test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
  "test_wdl_dependencies": null,
  "test_wdl_dependencies_hash": null,
  "eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
  "eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
  "eval_wdl_dependencies": null,
  "eval_wdl_dependencies_hash": null,
  "test_input": {
    "VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
    "VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
    "VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
    "VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
    "VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
    "VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
    "VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
    "VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
    "VariantCallingCarrotOrchestrated.contamination": 0.0,
    "VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
    "VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
    "VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
    "VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
    "VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:latest",
    "VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|fde1064710b236c7cdab83248817093046494660",
    "VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
    "VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
    "VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
  },
  "test_options": {
    "read_from_cache": false
  },
  "eval_input": {
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
    "BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
    "BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
    ],
    "BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
      "HCR",
      "LCR"
    ]
  },
  "eval_options": {
    "read_from_cache": false
  },
  "test_cromwell_job_id": null,
  "eval_cromwell_job_id": null,
  "created_at": "2023-05-16T17:15:43.799702",
  "created_by": null,
  "finished_at": null,
  "results": null,
  "errors": null
}

CarrotBroadBot · 2023-05-17T02:35:35Z

🥕CARROT🥕 run finished

Test: HaplotypeCaller CARROT Regression Tests | Status: succeeded

Run: HaplotypeCaller CARROT Regression Tests_run_2023-05-16 17:15:43.799630906 UTC

Results

Results
CHM controlHCprocesshours	84.8981027777778
CHM controlHCsystemhours	0.19177500000000003
CHM controlHCwallclockhours	60.16600277777776
CHM controlHCwallclockmax	3.0439777777777777
CHM controlMonitoringLogs	View in the GCS Console
CHM controlindelF1Score	0.8724
CHM controlindelPrecision	0.8814
CHM controlsnpF1Score	0.9784
CHM controlsnpPrecision	0.9706
CHM controlsnpRecall	0.9863
CHM controlsummary	View in the GCS Console
CHM evalHCprocesshours	83.2423166666667
CHM evalHCsystemhours	0.18843333333333337
CHM evalHCwallclockhours	61.06540555555557
CHM evalHCwallclockmax	3.1854916666666666
CHM evalMonitoringLogs	View in the GCS Console
CHM evalindelF1Score	0.8724
CHM evalindelPrecision	0.8814
CHM evalsnpF1Score	0.9784
CHM evalsnpPrecision	0.9706
CHM evalsnpRecall	0.9863
CHM evalsummary	View in the GCS Console
EXOME1 controlindelF1Score	0.727
EXOME1 controlindelPrecision	0.632
EXOME1 controlsnpF1Score	0.9878
EXOME1 controlsnpPrecision	0.9815
EXOME1 controlsnpRecall	0.9941
EXOME1 controlsummary	View in the GCS Console
EXOME1 evalindelF1Score	0.727
EXOME1 evalindelPrecision	0.632
EXOME1 evalsnpF1Score	0.9878
EXOME1 evalsnpPrecision	0.9815
EXOME1 evalsnpRecall	0.9941
EXOME1 evalsummary	View in the GCS Console
NIST controlHCprocesshours	103.49216944444444
NIST controlHCsystemhours	0.21042500000000003
NIST controlHCwallclockhours	74.8884888888889
NIST controlHCwallclockmax	3.995058333333333
NIST controlMonitoringLogs	View in the GCS Console
NIST controlindelF1Score	0.9902
NIST controlindelPrecision	0.9903
NIST controlsnpF1Score	0.9899
NIST controlsnpPrecision	0.9887
NIST controlsnpRecall	0.9911
NIST controlsummary	View in the GCS Console
NIST evalHCprocesshours	103.23083611111107
NIST evalHCsystemhours	0.2083694444444444
NIST evalHCwallclockhours	76.16374166666664
NIST evalHCwallclockmax	3.743883333333333
NIST evalMonitoringLogs	View in the GCS Console
NIST evalindelF1Score	0.9902
NIST evalindelPrecision	0.9903
NIST evalsnpF1Score	0.9899
NIST evalsnpPrecision	0.9887
NIST evalsnpRecall	0.9911
NIST evalsummary	View in the GCS Console
ROC_Plots_Reported	View in the GCS Console

Full details

 
 {
  "run_id": "00d212a9-8eac-4575-9cde-b0a911b3c2f2",
  "test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
  "run_group_ids": [],
  "name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-16 17:15:43.799630906 UTC",
  "status": "succeeded",
  "test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
  "test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
  "test_wdl_dependencies": null,
  "test_wdl_dependencies_hash": null,
  "eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
  "eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
  "eval_wdl_dependencies": null,
  "eval_wdl_dependencies_hash": null,
  "test_input": {
    "VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
    "VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
    "VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
    "VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
    "VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
    "VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
    "VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
    "VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
    "VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
    "VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
    "VariantCallingCarrotOrchestrated.contamination": 0.0,
    "VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
    "VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
    "VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
    "VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
    "VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:latest",
    "VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|fde1064710b236c7cdab83248817093046494660",
    "VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
    "VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
    "VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
  },
  "test_options": {
    "read_from_cache": false
  },
  "eval_input": {
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
    "BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
    "BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
    "BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
    "BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
    "BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
    "BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
    "BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
      "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
    ],
    "BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
      "HCR",
      "LCR"
    ]
  },
  "eval_options": {
    "read_from_cache": false
  },
  "test_cromwell_job_id": "b9fadac2-4e94-424f-a397-004684d1e51e",
  "eval_cromwell_job_id": "acc9e2ac-b10a-4d6a-b586-cd3e47f04e41",
  "created_at": "2023-05-16T17:15:43.799702",
  "created_by": null,
  "finished_at": "2023-05-17T02:34:53.616",
  "results": {
    "CHM controlHCprocesshours": "84.8981027777778",
    "CHM controlHCsystemhours": "0.19177500000000003",
    "CHM controlHCwallclockhours": "60.16600277777776",
    "CHM controlHCwallclockmax": "3.0439777777777777",
    "CHM controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CHMSampleHeadToHead/BenchmarkComparison/1731c546-7466-4adf-9790-3f99d07df05b/call-CONTROLRuntimeTask/monitoring.pdf",
    "CHM controlindelF1Score": "0.8724",
    "CHM controlindelPrecision": "0.8814",
    "CHM controlsnpF1Score": "0.9784",
    "CHM controlsnpPrecision": "0.9706",
    "CHM controlsnpRecall": "0.9863",
    "CHM controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CHMSampleHeadToHead/BenchmarkComparison/1731c546-7466-4adf-9790-3f99d07df05b/call-BenchmarkVCFControlSample/Benchmark/669edf6c-76a1-4d82-8cf7-5cd104df2496/call-CombineSummaries/summary.csv",
    "CHM evalHCprocesshours": "83.2423166666667",
    "CHM evalHCsystemhours": "0.18843333333333337",
    "CHM evalHCwallclockhours": "61.06540555555557",
    "CHM evalHCwallclockmax": "3.1854916666666666",
    "CHM evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CHMSampleHeadToHead/BenchmarkComparison/1731c546-7466-4adf-9790-3f99d07df05b/call-EVALRuntimeTask/monitoring.pdf",
    "CHM evalindelF1Score": "0.8724",
    "CHM evalindelPrecision": "0.8814",
    "CHM evalsnpF1Score": "0.9784",
    "CHM evalsnpPrecision": "0.9706",
    "CHM evalsnpRecall": "0.9863",
    "CHM evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CHMSampleHeadToHead/BenchmarkComparison/1731c546-7466-4adf-9790-3f99d07df05b/call-BenchmarkVCFTestSample/Benchmark/8e83736f-3023-4bee-9c42-36c836b75297/call-CombineSummaries/summary.csv",
    "EXOME1 controlindelF1Score": "0.727",
    "EXOME1 controlindelPrecision": "0.632",
    "EXOME1 controlsnpF1Score": "0.9878",
    "EXOME1 controlsnpPrecision": "0.9815",
    "EXOME1 controlsnpRecall": "0.9941",
    "EXOME1 controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-EXOME1SampleHeadToHead/BenchmarkComparison/a98aa003-bddd-492a-8691-dfa50191e2c6/call-BenchmarkVCFControlSample/Benchmark/a332ee8c-3de3-4a6f-b1de-7b273c094e84/call-CombineSummaries/summary.csv",
    "EXOME1 evalindelF1Score": "0.727",
    "EXOME1 evalindelPrecision": "0.632",
    "EXOME1 evalsnpF1Score": "0.9878",
    "EXOME1 evalsnpPrecision": "0.9815",
    "EXOME1 evalsnpRecall": "0.9941",
    "EXOME1 evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-EXOME1SampleHeadToHead/BenchmarkComparison/a98aa003-bddd-492a-8691-dfa50191e2c6/call-BenchmarkVCFTestSample/Benchmark/269b49fd-36aa-4381-a08a-a3f2a4586967/call-CombineSummaries/summary.csv",
    "NIST controlHCprocesshours": "103.49216944444444",
    "NIST controlHCsystemhours": "0.21042500000000003",
    "NIST controlHCwallclockhours": "74.8884888888889",
    "NIST controlHCwallclockmax": "3.995058333333333",
    "NIST controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-NISTSampleHeadToHead/BenchmarkComparison/56974c24-19c8-4f87-b7b1-b71028109732/call-CONTROLRuntimeTask/monitoring.pdf",
    "NIST controlindelF1Score": "0.9902",
    "NIST controlindelPrecision": "0.9903",
    "NIST controlsnpF1Score": "0.9899",
    "NIST controlsnpPrecision": "0.9887",
    "NIST controlsnpRecall": "0.9911",
    "NIST controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-NISTSampleHeadToHead/BenchmarkComparison/56974c24-19c8-4f87-b7b1-b71028109732/call-BenchmarkVCFControlSample/Benchmark/670f9cb4-5bb0-48e2-95c9-15a2e1ae7dee/call-CombineSummaries/summary.csv",
    "NIST evalHCprocesshours": "103.23083611111107",
    "NIST evalHCsystemhours": "0.2083694444444444",
    "NIST evalHCwallclockhours": "76.16374166666664",
    "NIST evalHCwallclockmax": "3.743883333333333",
    "NIST evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-NISTSampleHeadToHead/BenchmarkComparison/56974c24-19c8-4f87-b7b1-b71028109732/call-EVALRuntimeTask/monitoring.pdf",
    "NIST evalindelF1Score": "0.9902",
    "NIST evalindelPrecision": "0.9903",
    "NIST evalsnpF1Score": "0.9899",
    "NIST evalsnpPrecision": "0.9887",
    "NIST evalsnpRecall": "0.9911",
    "NIST evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-NISTSampleHeadToHead/BenchmarkComparison/56974c24-19c8-4f87-b7b1-b71028109732/call-BenchmarkVCFTestSample/Benchmark/b84fd1b7-a21e-4098-aeaf-05de3b35b2df/call-CombineSummaries/summary.csv",
    "ROC_Plots_Reported": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CreateHTMLReport/report.html"
  },
  "errors": null
}

CarrotBroadBot · 2023-05-17T02:43:34Z

🥕CARROT🥕 report map stub finished

for test HaplotypeCaller CARROT Regression Tests (run: 00d212a9-8eac-4575-9cde-b0a911b3c2f2)

File	URI
empty_notebook	View in the GCS Console
html_report	View in the GCS Console
populated_notebook	View in the GCS Console
run_csv_zip	View in the GCS Console

davidbenjamin · 2023-05-17T03:06:47Z

Woohoo, eval was faster than control!!!!!!!!!!!!!!!!

jamesemery

With that fixed, my objections have dropped. 👍

davidbenjamin · 2023-05-17T14:49:02Z

Terrific! I'll merge once the bucket migration error gets resolved and that last failing test passes.

samuelklee · 2023-05-17T15:07:40Z

Just curious, what resolved the memory/runtime issues? Apologies if I missed it somewhere above!

jamesemery · 2023-05-17T17:47:16Z

@samuelklee The performance issues were a mirage. It turned out that there was a misconfiguration in the Carrot tests so it was pegging the "control" version to an older GATK release from a year ago. There was a slight regression in the past year the we haven't caught but that is a separate discussion.

… in output

gatk-bot · 2023-05-18T04:57:25Z

Github actions tests reported job failures from actions build 5010565729
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
cloud	17.0.6+10	5010565729.10	logs

davidbenjamin · 2023-05-18T15:13:00Z

The failing test is unrelated to this branch, and I have secured permission to merge. Here we go. . .

Approval given

davidbenjamin requested a review from jamesemery January 7, 2020 16:30

davidbenjamin assigned jamesemery Jan 7, 2020

jamesemery reviewed Jan 30, 2020

View reviewed changes

davidbenjamin force-pushed the db_glcalc branch from 06ff1fa to e9d52f1 Compare January 27, 2022 21:10

samuelklee reviewed Feb 4, 2022

View reviewed changes

samuelklee requested changes Feb 24, 2022

View reviewed changes

davidbenjamin assigned samuelklee and unassigned jamesemery Mar 16, 2022

samuelklee reviewed May 13, 2022

View reviewed changes

samuelklee previously requested changes May 13, 2022

View reviewed changes

samuelklee mentioned this pull request May 14, 2022

Added numerical-stability tests and updated test data for all ModelSegments single-sample and multiple-sample modes. #7652

Merged

davidbenjamin force-pushed the db_glcalc branch 2 times, most recently from 615489c to ee3b494 Compare August 4, 2022 21:11

davidbenjamin force-pushed the db_glcalc branch from d791874 to fde1064 Compare May 15, 2023 22:12

jamesemery approved these changes May 17, 2023

View reviewed changes

jamesemery mentioned this pull request May 17, 2023

Find the source of the 5% HaplotypeCaller performance regression in the past year #8327

Open

Huge simplication of genotyping likelihoods calculations -- no change…

fdff6db

… in output

davidbenjamin force-pushed the db_glcalc branch from fde1064 to fdff6db Compare May 18, 2023 04:37

davidbenjamin merged commit 7415882 into master May 18, 2023

davidbenjamin deleted the db_glcalc branch May 18, 2023 15:20

Simplified genotype likelihood calculation (no change in output) #6351

Simplified genotype likelihood calculation (no change in output) #6351

Conversation

davidbenjamin commented Jan 7, 2020

jamesemery left a comment

Choose a reason for hiding this comment

jamesemery Jan 30, 2020

Choose a reason for hiding this comment

davidbenjamin Jan 30, 2020

Choose a reason for hiding this comment

davidbenjamin commented Jan 30, 2020

samuelklee commented Jan 21, 2022

davidbenjamin commented Jan 21, 2022

davidbenjamin commented Jan 27, 2022

samuelklee Feb 4, 2022 • edited Loading

Choose a reason for hiding this comment

samuelklee Feb 22, 2022

Choose a reason for hiding this comment

samuelklee left a comment • edited Loading

Choose a reason for hiding this comment

samuelklee Feb 22, 2022

Choose a reason for hiding this comment

davidbenjamin commented Mar 14, 2022 • edited Loading

davidbenjamin commented Mar 14, 2022

gatk-bot commented Mar 14, 2022 • edited Loading

gatk-bot commented Mar 15, 2022 • edited Loading

davidbenjamin commented Mar 15, 2022

davidbenjamin commented Mar 16, 2022

samuelklee commented Mar 17, 2022

samuelklee commented Apr 12, 2022

samuelklee commented Apr 28, 2022 • edited Loading

samuelklee May 13, 2022

Choose a reason for hiding this comment

davidbenjamin Aug 4, 2022

Choose a reason for hiding this comment

samuelklee Aug 18, 2022

Choose a reason for hiding this comment

samuelklee left a comment

Choose a reason for hiding this comment

samuelklee commented Jun 27, 2022

gatk-bot commented Aug 4, 2022 • edited Loading

gatk-bot commented Aug 7, 2022 • edited Loading

codecov bot commented Aug 7, 2022 • edited Loading

Codecov Report

davidbenjamin commented May 12, 2023

davidbenjamin commented May 12, 2023

CarrotBroadBot commented May 12, 2023

🥕CARROT🥕 run started

Test: HaplotypeCaller CARROT Regression Tests | Status: building

davidbenjamin commented May 12, 2023

CarrotBroadBot commented May 13, 2023

🥕CARROT🥕 run finished

Test: HaplotypeCaller CARROT Regression Tests | Status: succeeded

CarrotBroadBot commented May 13, 2023

🥕CARROT🥕 report map stub finished

davidbenjamin commented May 15, 2023

gatk-bot commented May 15, 2023 • edited Loading

davidbenjamin commented May 15, 2023

CarrotBroadBot commented May 15, 2023

🥕CARROT🥕 run started

Test: HaplotypeCaller CARROT Regression Tests | Status: building

CarrotBroadBot commented May 16, 2023

🥕CARROT🥕 run finished

Test: HaplotypeCaller CARROT Regression Tests | Status: succeeded

CarrotBroadBot commented May 16, 2023

🥕CARROT🥕 report map stub finished

jamesemery commented May 16, 2023

CarrotBroadBot commented May 16, 2023

🥕CARROT🥕 run started

Test: HaplotypeCaller CARROT Regression Tests | Status: building

CarrotBroadBot commented May 17, 2023

🥕CARROT🥕 run finished

Test: HaplotypeCaller CARROT Regression Tests | Status: succeeded

CarrotBroadBot commented May 17, 2023

🥕CARROT🥕 report map stub finished

davidbenjamin commented May 17, 2023

jamesemery left a comment

Choose a reason for hiding this comment

davidbenjamin commented May 17, 2023

samuelklee commented May 17, 2023 • edited Loading

samuelklee Feb 4, 2022 •

edited

Loading

samuelklee left a comment •

edited

Loading

davidbenjamin commented Mar 14, 2022 •

edited

Loading

gatk-bot commented Mar 14, 2022 •

edited

Loading

gatk-bot commented Mar 15, 2022 •

edited

Loading

samuelklee commented Apr 28, 2022 •

edited

Loading

gatk-bot commented Aug 4, 2022 •

edited

Loading

gatk-bot commented Aug 7, 2022 •

edited

Loading

codecov bot commented Aug 7, 2022 •

edited

Loading

gatk-bot commented May 15, 2023 •

edited

Loading

samuelklee commented May 17, 2023 •

edited

Loading