Elasticsearch issue : no shards available

[bnouyou]

Hello,

I have installed Seqr in a local instance.
I would like to know how to add more shards in Elasticsearch? I am facing an issue when loading more VCF files, because it seems there is a limit of 1000 shards (I suppose this is the maximum). Should I add a new Elasticsearch service in the docker-compose to solve this problem?

This is the error in the logs:

elasticsearch.exceptions.RequestError: RequestError(400, 'validation_exception', 'Validation Failed: 1: this action would add [1] shards, but this cluster currently has [1000]/[1000] maximum normal shards open;')

Thank you in advance for your help.

[hanars]

So the default configuration in docker compose for elasticsearch includes only a single node. We do not have specific support for running multiple nodes, but you could probably adapt elasticsearch's documentation to work with seqr:
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/docker.html#docker-compose-file

The alternative would be to ensure that you actually need all 1000 shards you currently have created, that sounds kind of high to me (although I do not know how much data you have). It may be instead worth cleaning up your data to lower the number of shards you have. Reasons you may have more shards than you need:

• You have old indices no longer being used by seqr that were never deleted
• Instead of reloading entire projects with all their data you have lots of little indices associated with a single project. Reloading all the data in a project into a single index would improve seqr seqr performance in addition to reducing your shard count
• You have old indices run on an older version of the pipeline that set an unnecessarily high default number of shards. The current script for loading data sets --es-index-min-num-shards to 1 but older versions did not necessarily do that. The recommended number of shards per index can be computed as follows, so you should do a spot check on some of your larger indices to see if they are following this rule or if they in fact have more shards than recommended:

  (num_rows*num_columns)/1.4*10**9
  

[bnouyou]

Thank you for your answer.

I have about 500 samples in my seqr instance, and each one (or trios) have been loaded one by one (one VCF file for one family). And yes, I have old samples that have been imported with an old seqr version (and I have notived that these ones have more shards than the new ones).

Do you suggest that I reload some data ? But, will I loose information (for example if some variants have been taggued) ?

[hanars]

I would recommend you reload your data. This is not really how seqr is designed to be used, as you should be reloading a full seqr project every time you load data to it with all the samples in the project joint called together. You will not lose any information like tagged variants

[bnouyou]

Do you mean that, every time I need to add samples in a project, I need to joint call all previous samples with new ones in an uniq VCF file. And then load this VCF in seqr?

[hanars]

That is the recommended way to use seqr, although what you have been doing works if thats not feasible with your workflow. But a one-time reload would greatly reduce your number of shards even if you do not have the bandwidth to recall your data every time you add samples

[bnouyou]

Thank you for your answer.

So, I have combined my VCFs to one using this command (gatk 3.7):

gatk -T CombineVariants \
     -R $REF \
     -V $VCF1 \
     -V $VCF2 \
     -V $VCF3 \
     -o $OUTPUT \
     --genotypemergeoption UNSORTED

I have loaded the final VCF in seqr, but I have this error:

hail.utils.java.FatalError: HailException: RVD error! Keys found out of order:
  Current key:  { locus: { contig: 1, position: 36203701 }, alleles: [2; G, GAAA] }
  Previous key: { locus: { contig: 1, position: 36203701 }, alleles: [2; G, T] }
This error can occur after a split_multi if the dataset
contains both multiallelic variants and duplicated loci. 

[hanars]

So combining VCFs is not the same as joint calling. You can find more information about joint calling here: https://storage.googleapis.com/seqr-reference-data/seqr-vcf-info.pdf