FastQC gives KeyError if report generated from an empty FastQ file

[elderberry-smells]

Description of bug:
running MultiQC in a snakemake pipeline, it should be joing up the log data for the fastqc reports, and trimmomatic reports (this is working). Snakemake sees the inputs fine, fastqc runs and generates an html and zip file for each fastq file, but the multiqc doesn't like the data fastqc is generating, throwing a "key error" as though the fastqc data is missing a column?

MultiQC Error log:

[INFO   ]         multiqc : This is MultiQC v1.8
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching   : /home/bioinf/gbs_data/sample3/log
[INFO   ]     trimmomatic : Found 179 logs
[INFO   ]          fastqc : Found 368 reports
[ERROR  ]         multiqc : Oops! The 'fastqc' MultiQC module broke... 
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues 
  If possible, please include a log file that triggers the error - the last file found was:
    log/fastqc/SK-GBD-000435.1_fastqc.zip
============================================================
Module fastqc raised an exception: Traceback (most recent call last):
  File "/home/bioinf/anaconda3/envs/gbs/lib/python3.6/site-packages/multiqc/multiqc.py", line 546, in run
    output = mod()
  File "/home/bioinf/anaconda3/envs/gbs/lib/python3.6/site-packages/multiqc/modules/fastqc/fastqc.py", line 94, in __init__
    self.fastqc_general_stats()
  File "/home/bioinf/anaconda3/envs/gbs/lib/python3.6/site-packages/multiqc/modules/fastqc/fastqc.py", line 204, in fastqc_general_stats
    'avg_sequence_length': bs['avg_sequence_length'],
KeyError: 'avg_sequence_length'

File that triggers the error:

rule all:
    input:
        "log/multiqc_report.html"

rule fastqc:
    input:
        snps = snp_file,
        qc = "demultiplex/{sample}.{read}.fastq"
    output:
        "log/fastqc/{sample}.{read}_fastqc.html",
        "log/fastqc/{sample}.{read}_fastqc.zip"
    threads: 4
    shell:
        "fastqc -o log/fastqc/ -t {threads} {input.qc}"

rule multiqc:
    input:
        expand(["log/fastqc/{sample}.{read}_fastqc.html"], sample=samples, read = [1, 2])
    output:
        "log/multiqc_report.html"
    shell:
        "multiqc log -o log/"

MultiQC run details (please complete the following):

• Command used to run MultiQC: multiqc log -o log/
• MultiQC Version: 1.8
• Operating System: Ubuntu
• Python Version: 3.6.10
• Method of MultiQC installation: conda

Additional context
Add any other context about the problem here.

[VGalata]

I had the same error and found out that some of my FASTQ files were empty (for whatever reason). Therefore, the FastQC reports did not contain any data which most likely triggered the key error. I fixed the issue with the FASTQ files and now everything runs through.

[maurya-anand]

I have the same error. Please suggest some workaround. Thanks.

[INFO   ]         multiqc : This is MultiQC v1.8
[INFO   ]         multiqc : Template    : default

Module fastqc raised an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/multiqc/multiqc.py", line 546, in run
    output = mod()
  File "/usr/local/lib/python3.6/dist-packages/multiqc/modules/fastqc/fastqc.py", line 94, in __init__
    self.fastqc_general_stats()
  File "/usr/local/lib/python3.6/dist-packages/multiqc/modules/fastqc/fastqc.py", line 204, in fastqc_general_stats
    'avg_sequence_length': bs['avg_sequence_length'],
KeyError: 'avg_sequence_length'

[elderberry-smells]

I had the same error and found out that some of my FASTQ files were empty (for whatever reason). Therefore, the FastQC reports did not contain any data which most likely triggered the key error. I fixed the issue with the FASTQ files and now everything runs through.

Thanks so much for this reply. That was the issue, I had a fastq file that was empty as a result of a demultiplex script and the sample not being in the library.

I removed the empty fastq and the program runs as expected.

[ewels]

Nice, thanks all! If someone has one of these FastQC reports (zipped), please could they attach it here? We could presumably add some code to MultiQC to check for this and give a nice warning message and skip the file. Would be better than an ugly KeyError exception and stopping for all further FastQC reports.

Phil

[elderberry-smells]

empty_fastqc.zip

That is what the file looks like when it produces a fastqc report from an empty fastq file.

[ewels]

Ok, thanks all - this is now fixed. Instead of falling over with a KeyError, MultiQC now throws a warning to the console but troops on and does the best it can with the report.

If you run on only empty FastQC reports then the report looks kind of strange 😅 But when mixed with normal samples it should behave more like you'd expect.

Amusingly, most of the FastQC modules give a passing status for an empty file..

Thanks all for your input on this - let me know if you find any problems 👍

Phil