CNV TODOs before release.

[samuelklee]

Legacy pipeline (note, the following should only be done after final ModelSegments PR is in):

• Delete prototype tools. (Delete old CNV tools. #3887) (SL, PR issued by 12/1)
• Add deprecated/legacy tag to legacy pipeline tools. (SL, PR issued by 12/1 EDIT: need further input from @vdauwera )
• Update docs/arguments (w/ Comms, see All hands on deck: tool doc updates #3853). This will follow deletion of prototype tools. (all, PR issued by 12/15)
• (Reach) Collect all legacy code in a new package.
• Delete old pipelines. (SL, Deleted legacy CNV pipeline. #3935 awaiting review)

ModelSegments pipeline:

• Review and merge denoising PR ( Added denoising tools for ModelSegments CNV pipeline. #3820).
• Add WDL changes from @LeeTL1220, @meganshand, and @jsotobroad to dev branch. (Note that we exposed PreprocessIntervals.bin_length in these WDLs; I'm assuming that Pass inputs to subworkflow in inputs.json cromwell#2912 will allow this to be specified via the json, so I reverted this change.)
• Make simple improvements to ReCapSeg caller (Make and evaluate simple changes to ReCapSegCaller. #3825).
• Review and merge modeling/WDL PR. (Added code and WDL to complete ModelSegments CNV pipeline. #3913 awaiting review. Note that this PR also deletes the old germline WDL.)
  • Write MultidimensionalKernelSegmenterUnitTest. (SL, punting, filed Add a few more tests to ModelSegments pipeline. #3916)
  • Write ModelSegmentsIntegrationTest. (SL, punting, filed Add a few more tests to ModelSegments pipeline. #3916)
• Preliminary PCAWG or HCC1143 purity evaluation. (@LeeTL1220) (LL, should be done in time for @vdauwera to present at Broad retreat)
• Update docs/arguments (w/ Comms, see All hands on deck: tool doc updates #3853). This will follow deletion of prototype tools. (PR Added documentation and kebab-case updates for ModelSegments pipeline (excluding PreprocessIntervals), along with some WDL updates. #4010 awaiting review.)
• Add SM tag and sequence dictionary headers to all appropriate files and sort accordingly. (SL, Added SAM headers to ModelSegments CNV pipeline output. #3914 awaiting review)
• Update tutorial data. (@MartonKN)
• (Reach) Add VCF output.
• (Reach) Add PG tags to all files.
• (Reach) Replace ReCapSeg caller with improved version. (@MartonKN)

gCNV pipeline:

• Review and merge Python code (gCNV python computational kernel #3838). (MB and SL, PR Added CLIs and WDL for python gCNV pipeline. #3925 awaiting review.)
• CLI for ploidy determination (cohort). (@samuelklee)
• CLI for ploidy determination (case). (@samuelklee)
• CLI for calling (cohort). (@samuelklee)
• CLI for calling (case). (@samuelklee)
• CLI for post-processing calls. (@asmirnov239) (AS, PR issued by 12/4)
• Python environment. (Update: I've verified that gCNV works on the gsa server with a manual setup of conda (python=3.6) + @mbabadi's pip install---although I do get an "install mkl" warning from theano. We can discuss autoloading of this environment after release, but should at least have some clear documentation.)
• WDL and Cromwell tests. (SL, PR issued by 12/1)
• Preliminary evaluation. (MB, should be done in time for @vdauwera to present at Broad retreat)
• Update docs/arguments (w/ Comms, see All hands on deck: tool doc updates #3853). This will follow deletion of prototype tools. (all, PE Added CLIs and WDL for python gCNV pipeline. #3925 awaiting review.)

Miscellaneous:

• Update PreprocessIntervals behavior for WES. (Issue Restore PadTargets padding/merging behavior to PreprocessIntervals. #3981, PR Changed PreprocessIntervals behavior to not merge intervals that overlap after padding and updated docs. #4027 awaiting review.)

[samuelklee]

@vdauwera We will try to have some preliminary slides/evaluations for you for both of the new pipelines, if you'd like to include them in your retreat talk---nothing too technical or detailed, mostly for marketing purposes. Let us know when you'd need things finalized by.

[samuelklee]

@asmirnov239 @MartonKN @mbabadi Note the remaining TODOs. Let's be in a position where we can get all gCNV-related branches merged shortly after @mbabadi returns on 1/4 but with a margin before release on 1/9. I will spend some time over break to get my review in.

I will also finish up docs for ModelSegments + update PreprocessIntervals this week. Review should be quick.

I think we can consider preliminary evaluations complete; we'll run more and shore up evaluation infrastructure after break.

@LeeTL1220 Following our discussion about evaluation tools, I added VCF output for the somatic pipeline as a TODO. Hopefully I can have some basic output by release, if not shortly thereafter.

[LeeTL1220]

VCF output can be a reach for the release.

…

On Wed, Dec 20, 2017 at 1:44 PM, samuelklee ***@***.***> wrote: @asmirnov239 <https://github.com/asmirnov239> @MartonKN <https://github.com/martonkn> @mbabadi <https://github.com/mbabadi> Note the remaining TODOs. Let's be in a position where we can get all gCNV-related branches merged shortly after @mbabadi <https://github.com/mbabadi> returns on 1/4 but with a margin before release on 1/9. I will spend some time over break to get my review in. I will also finish up docs for ModelSegments + update PreprocessIntervals this week. Review should be quick. I think we can consider preliminary evaluations complete; we'll run more and shore up evaluation infrastructure after break. @LeeTL1220 <https://github.com/leetl1220> Following our discussion about evaluation tools, I added VCF output for the somatic pipeline as a TODO. Hopefully I can have some basic output by release, if not shortly thereafter. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3826 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACDXk02sFz6dsTAaLaJ3gzcMLWzbFAJGks5tCVWKgaJpZM4Qcfhw> .

-- Lee Lichtenstein Broad Institute 75 Ames Street, Room 8011A Cambridge, MA 02142 617 714 8632

[MartonKN]

Hi Sam, updating ReCapSegCaller with the new version and updating the tutorial data is assigned to me as a task. But as far as I understand those tasks will not need to be ready by the release. Could you confirm it? Happy holidays, Marton

…

On Wed, Nov 29, 2017 at 4:08 PM, samuelklee ***@***.***> wrote: Assigned #3826 <#3826> to @MartonKN <https://github.com/martonkn>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#3826 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGPkHrgsag4Wro95TToAuawWDdtzSxp4ks5s7cfKgaJpZM4Qcfhw> .

-- Marton Kanasz-Nagy Physics Department, Harvard University 17 Oxford Street, Cambridge, MA 02138 [email protected]

[samuelklee]

@MartonKN I've labeled the update of the caller as a "reach", so I'm not expecting that it gets done before release.

However, I expect that the tutorial data should be updated well before release. The tutorial data runs quickly (~1 hr for coverage collection, which is mostly limited by the slowest samples or cloud preemptions, and then ~minutes once collection has been call cached), so we should have plenty of time.

Whether or not the actual tutorial itself will be ready depends on whether @sooheelee has available bandwidth and if it is a high priority for comms.

[samuelklee]

Going to close this out and spin off new issues.