Issue in SeqrMTToESTask #2001

bnouyou · 2021-07-28T06:47:22Z

bnouyou
Jul 28, 2021

Hello,

I have ran the command for annotate and load on-prem (python3 -m seqr_loading SeqrMTToESTask ... with GRCh37).
It was working until stage 7.
I have this error message:

[Stage 7:=>                                                       (9 + 2) / 500]ERROR: [pid 137] Worker Worker(salt=419393300, workers=1, host=273dd4bcb41b, usernam                              e=root, pid=137) failed    SeqrMTToESTask(source_path=<luigi.parameter.Parameter object at 0x7efdafbb3278>, use_temp_loading_nodes=True, es_host=elasticsearch, es_p                              ort=9200, es_index=exome-11_21-28566-a-04-00_270720210936, es_username=pipeline, es_password=None, es_index_min_num_shards=3, source_paths=/input_vcfs/EXOME-11/IOP_                              21-28566-A-04-00.combined.snp_indel.annovar.hg19_multianno.vcf.gz, dest_path=/output/EXOME-11/21-28566-A-04-00/, genome_version=37, vep_runner=VEP, reference_ht_pat                              h=/seqr-reference-data/GRCh37/all_reference_data/combined_reference_data_grch37.ht, clinvar_ht_path=/seqr-reference-data/GRCh37/clinvar/clinvar.GRCh37.2020-06-15.ht                              , hgmd_ht_path=None, sample_type=WES, dont_validate=True, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep-GRCh37.json)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/hail-elasticsearch-pipelines/luigi_pipeline/seqr_loading.py", line 201, in run
    self.export_table_to_elasticsearch(row_table, self._mt_num_shards(mt))
  File "/hail-elasticsearch-pipelines/luigi_pipeline/lib/hail_tasks.py", line 236, in export_table_to_elasticsearch
    write_null_values=True)
  File "/hail-elasticsearch-pipelines/hail_scripts/v02/utils/elasticsearch_client.py", line 192, in export_table_to_elasticsearch
    table, self._host, int(self._port), index_name, index_type_name, block_size, elasticsearch_config, verbose
  File "<decorator-gen-1318>", line 2, in export_elasticsearch
  File "/usr/local/lib/python3.7/site-packages/hail/typecheck/check.py", line 614, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/usr/local/lib/python3.7/site-packages/hail/methods/impex.py", line 2464, in export_elasticsearch
    Env.hail().io.ElasticsearchConnector.export(jdf, host, port, index, index_type, block_size, config, verbose)
  File "/usr/local/lib/python3.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 32, in deco
    'Error summary: %s' % (deepest, full, hail.__version__, deepest), error_id) from None
hail.utils.java.FatalError: TaskCompletionListenerException: is.hail.relocated.org.elasticsearch.hadoop.rest.EsHadoopRemoteException: circuit_breaking_exception: [p                              arent] Data too large, data for [<http_request>] would be [1036517280/988.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1036517280/988.                              4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, accounting=60864224/58mb]
null

Java stack trace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 7.0 failed 1 times, most recent failure: Lost task 8.0 in stage 7.0 (TID 3508, lo                              calhost, executor driver): org.apache.spark.util.TaskCompletionListenerException: is.hail.relocated.org.elasticsearch.hadoop.rest.EsHadoopRemoteException: circuit_b                              reaking_exception: [parent] Data too large, data for [<http_request>] would be [1036517280/988.4mb], which is larger than the limit of [1020054732/972.7mb], real us                              age: [1036517280/988.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, accounting=60864224/58mb]
null
        at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:138)
        at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
        at org.apache.spark.scheduler.Task.run(Task.scala:137)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:403)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2114)
        at is.hail.relocated.org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:101)
        at is.hail.relocated.org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:83)
        at is.hail.relocated.org.elasticsearch.spark.sql.package$SparkDataFrameFunctions.saveToEs(package.scala:49)
        at is.hail.io.ElasticsearchConnector$.export(ElasticsearchConnector.scala:44)
        at is.hail.io.ElasticsearchConnector$.export(ElasticsearchConnector.scala:20)
        at is.hail.io.ElasticsearchConnector.export(ElasticsearchConnector.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)

org.apache.spark.util.TaskCompletionListenerException: is.hail.relocated.org.elasticsearch.hadoop.rest.EsHadoopRemoteException: circuit_breaking_exception: [parent]                               Data too large, data for [<http_request>] would be [1036517280/988.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1036517280/988.4mb],                               new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, accounting=60864224/58mb]
null
        at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:138)
        at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
        at org.apache.spark.scheduler.Task.run(Task.scala:137)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:403)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)




Hail version: 0.2.61-3c86d3ba497a
Error summary: TaskCompletionListenerException: is.hail.relocated.org.elasticsearch.hadoop.rest.EsHadoopRemoteException: circuit_breaking_exception: [parent] Data t                              oo large, data for [<http_request>] would be [1036517280/988.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1036517280/988.4mb], new byt                              es reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, accounting=60864224/58mb]
null
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   SeqrMTToESTask__seqr_reference__VARIANTS__output_EXOME_11_7e6a7d8d23   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 1 pending tasks possibly being run by other workers
DEBUG: There are 1 pending tasks unique to this worker
DEBUG: There are 1 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=419393300, workers=1, host=273dd4bcb41b, username=root, pid=137) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====

Scheduled 3 tasks of which:
* 1 complete ones were encountered:
    - 1 VcfFile(filename=/input_vcfs/EXOME-11/IOP_21-28566-A-04-00.combined.snp_indel.annovar.hg19_multianno.vcf.gz)
* 1 ran successfully:
    - 1 SeqrVCFToMTTask(...)
* 1 failed:
    - 1 SeqrMTToESTask(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

Answered by hanars

Jul 28, 2021

This is an issue with your ES cluster being too small. I would suggest you add additional, potentially larger, data nodes

View full answer

hanars · 2021-07-28T14:15:30Z

hanars
Jul 28, 2021
Maintainer

This is an issue with your ES cluster being too small. I would suggest you add additional, potentially larger, data nodes

5 replies

bnouyou Jul 29, 2021
Author

Hello,
I see. I have done that. And then I have done again the command for this sample. This time, it ends very quickly, and in the output it's written that there is no fail. Is it normal that it hasn't been re-processed completely ?

I have uploaded the sample in seqr and I see that there is not a lot of variants. I am wondering if it is complete. How can I do to re-process this sample ? Launch again the command and change the --es-indexand --dest-path parameters ?

Thank you for your help.

hanars Jul 29, 2021
Maintainer

So you want to reuse the VcfFile and SeqrVCFToMTTask as they completed successfully, but you are correct you don;t want to reuse the SeqrMTToESTask. Seeing the actual log output from your latest run would make it much easier for me to advise further, but as a guess you should delete the index that got partially created yesterday and failed

bnouyou Jul 29, 2021
Author

Problem solved ! I have seen in Data Management Pages > Elasticsearch Status that I had 2 rows for this sample (I suppose one is for the "failed" sample and one for the "passed" sample). The "failed" one has size at 3 Mb whereas the other has a size at 117 Mb. I think I have loaded the wrong es index. So I just tried the other es index, and now it seems good.

One question : since I have update my seqr version, I can se that when I search variants without criteria, I have now 10.000 variants, for all samples. Is it a limit for the results?

hanars Jul 29, 2021
Maintainer

yes, seqr now has a hard limit of 10,000 results per query

bnouyou Jul 30, 2021
Author

Thank you for your answer. That's what I suspected. So it's not a bug on my side, that's good.
Thank you for your help !

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue in SeqrMTToESTask #2001

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Issue in SeqrMTToESTask #2001

bnouyou Jul 28, 2021

Replies: 1 comment · 5 replies

hanars Jul 28, 2021 Maintainer

bnouyou Jul 29, 2021 Author

hanars Jul 29, 2021 Maintainer

bnouyou Jul 29, 2021 Author

hanars Jul 29, 2021 Maintainer

bnouyou Jul 30, 2021 Author

bnouyou
Jul 28, 2021

Replies: 1 comment 5 replies

hanars
Jul 28, 2021
Maintainer

bnouyou Jul 29, 2021
Author

hanars Jul 29, 2021
Maintainer

bnouyou Jul 29, 2021
Author

hanars Jul 29, 2021
Maintainer

bnouyou Jul 30, 2021
Author