HaplotypeCallerSpark doesn't write g.vcf.gz files

[lbergelson]

If you ask HaplotypeCallerSpark for a gvcf.gz it outputs a base pair resolution GVCF with no blocking. This is due to confusion in hadoop-bam / VariantSparkSink.

It works fine if you write an uncompressed g.vcf.

This is due to a conditional statement in KeyIgnoringVCFOutputFormat.getRecordWriter(askAttemptContext ctx)

		if (!isCompressed) {
			return getRecordWriter(ctx, file);
		} else {
			FileSystem fs = file.getFileSystem(conf);
			return getRecordWriter(ctx, codec.createOutputStream(fs.create(file)));
		}

The two branches call two different overloads of getRecordWriter

getRecordWriter(TaskAttemptContext ctx, Path out)

getRecordWriter(TaskAttemptContext ctx, OutputStream outputStream)

The first is public, and overriden to provide GVCF writers in our code, the second is private and doesn't know about our GVCF writer. We could override getRecordWriter(ctx) but we need access to a constructor for VCFRecordWriter that takes a stream and propagates the ctx which doesn't exist.

[lbergelson]

see #4275 for the temporary workaround

[lbergelson]

This will likely require a fix in hadoop-bam unless we either copy most of the the hadoop-bam code into gatk or someone comes up with a more clever solution than I.

[lbergelson]

Similarly, see #4303 for the inability to write g.bcf files. A much lower priority problem...

[droazen]

@tomwhite Should be a fairly easy fix in Hadoop-BAM, we think.