UMI FASTQ file

[adamrtalbot]

UMI FASTQ file composed of random 9bp synthetic oligos, all with uniform quality.

Generated by stripping the UMI sequence from the existing FASTQ and turning it into a separate file. This will be a valid reference format for sequencing kits where the UMI is embedded in the index.

[lescai]

Hi :)
could you please describe a little more this file?
if this is the use case where UMIs are present in a third FASTQ, then the test dataset should include 3 files: forward and reverse (without UMIs in the sequence), and a UMIs file.

[lescai]

also, UMIs structure is needed in order to process the sequences

[adamrtalbot]

Yep no problem.

The entire read is the UMI sequence, it matches the existing FASTQs that are in the repository. Here is the existing FASTQ files:

# test-datasets/data/genomics/homo_sapiens/illumina/fastq$ zcat test.umi_1.fastq.gz | head -8
@922332/1
ATTTCAGAGAGAGGATCTCGTGTAGAAATTGCTTTGAGCTGTTCTTTGTCATTTTCCCTTAATTCATTGTCTCTAGCTAGTCTGTTACTCTGTAAAATAAAATAATAAGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTTAAGGTCAGTG
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE<EEEEEEEEEEEAEAEE6AAEEEEE/EAAAA<AEEEEAAEEAAAA<EEE/
@928177/1
ACATAAACAAAAGTATATAAGTAATACATATTTATAAATCTATTAAGAAAGCAAGTAATATGTACCTTAAGAATTTAATGGGAAAATAATTAGACTTACTTTAAATGCCAAAAGAAAAAGTGCCCAATCCTTTGATTAGTCAATGCTTTCT
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEEAEEEEEEEEAEE<EE<EEEEEEEEEEEEEEEEEEEEEEEEEAEEE<EEEEEEEAEAAEEEE<EAEAAAAE<<AAEEAEEAEEE

# test-datasets/data/genomics/homo_sapiens/illumina/fastq$ zcat test.umi_2.fastq.gz | head -8
@922332/2
TATTATTTTATTTTACAGAGTAACAGACTAGCTAGAGACAATGAATTAAGGGAAAATGACAAAGAACAGCTCAAAGCAATTTCTACACGAGATCCTCTCTCTGAAATAGATCGGAAGAGCACACGTCTGAACTCCAGTCACGAACCGCGAT
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAEA<AEEE<<<<AAEEEEEEEEEEA<EEEAEAE//A<AAE<6
@928177/2
TGAGATTTTTACTGAAGAAAGCATTGACTAATCAAAGGATTGGGCACTTTTTCTTTTGGCATTTAAAGTAAGTCTAATTATTTTCCCATTAAATTCTTAAGGTACATATTACTTGCTTTCTTAATAGATTTATAAATATGTATTACTTATA
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEE6EEEEEEEEEEEEEEEAEEEEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEEAEEEE/EEEE<EEEEEEEEEEEEE<AAEEEAEAEEEEAEE<AAAEA/AEEEEAEAEEEEEAEEAE/

and here is the new one:

# test-datasets/data/genomics/homo_sapiens/illumina/fastq$ zcat test.umi_umi.fastq.gz | head -8
@922332
TGACCATTT
+
FFFFFFFFF
@928177
TTTGAACAG
+
FFFFFFFFF

As you can see, the UMI FASTQ file matches the existing FASTQ files, saving us some storage. I generated the FASTQs by:

• Aligning to the human genome
• Grouping reads by position
• Randomly assigning them to a UMI family (poisson distribution, lambda 2)
• Creating a FASTQ file based on those families.

I'll upload the script later today and update here. I've checked the method and it seems to work fine in our pipeline.

The bases mask is +T +T +M where input is test.umi_1.fastq.gz test.umi_2.fastq.gz test.umi_umi.fastq.gz. You could be more explicit with 150T 150T 9M, or use the bases mask to cut out the existing UMIs from those files.

[adamrtalbot]

I've just checked your development branch, and I think the syntax would be:
ext.args = "--read-structures +T 23S+T +M"

[adamrtalbot]

Slight change - I've extracted those first 12bp and put them in that FASTQ file. This now should have exactly the same UMI sequences as the existing FASTQ and should create almost identical consensus reads.

test-datasets/data/genomics/homo_sapiens/illumina/fastq$ zcat test.umi_umi.fastq.gz | head -8
@922332
TATTATTTTATT
+
AAAAAEEEEEEE
@928177
TGAGATTTTTAC
+
AAAAAEEEEEEE

@lescai I've checked your subworkflow in development and it already works with three FASTQ files nicely! We just have to add an additional test.

[adamrtalbot]

@lescai did you have a chance to check this?