Template installation should waiting or reconnecting ES clusters when failing

[omallo]

Problem

I would like to use the new index template feature introduced in #194 but once I configure it, my fluentd setup stops working.

Steps to replicate

Here is the configuration that I'm using:

<match *.**>
  type elasticsearch
  host elasticsearch
  port 9200
  include_tag_key true
  tag_key @tag
  logstash_format true
  template_name logstash
  template_file /fluentd/etc/logstash-index-template.json
  flush_interval 10s
</match>

The error I'm getting is the following:

2016-09-29 15:35:57 +0000 [error]: fluent/supervisor.rb:378:rescue in main_process: unexpected error error="Can not reach Elasticsearch cluster ({:host=>\"elasticsearch\", :port=>9200, :scheme=>\"http\"})! connect_write timeout reached"
  2016-09-29 15:35:57 +0000 [error]: fluent/supervisor.rb:338:fork: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:148:in `rescue in client'
  2016-09-29 15:35:57 +0000 [error]: fluent/supervisor.rb:338:fork: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:145:in `client'
  2016-09-29 15:35:57 +0000 [error]: fluent/supervisor.rb:338:fork: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/elasticsearch_index_template.rb:12:in `template_exists?'
  2016-09-29 15:35:57 +0000 [error]: fluent/supervisor.rb:338:fork: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/elasticsearch_index_template.rb:23:in `template_install'
  2016-09-29 15:35:57 +0000 [error]: fluent/supervisor.rb:338:fork: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:88:in `configure'

The problem seems to be that I'm using a "special" hostname elasticsearch which is actually the name of my Docker service for which Docker defines this hostname. After a first analysis, I can say the following:

• If I remove the template_name and template_file settings, everything works fine, i.e. the plugin is able to connect to ElasticSearch without problems so the hostname resolution seems to be OK in general.
• If I replace the hostname with the corresponding IP address, everything works well also.

[kmotrebski]

@pitr @omallo I am encountering similar case and wondering what is the problem.

Isn't it about the fact that when you start containers (probably with Docker Compose) the elasticsearch cluster is not available yet as it takes some time (sometimes even 90-120 seconds) for Elasticsearch to fully get up and running? Fluentd is up much faster, after few seconds or less, and then it instantly crashes as Elasticsearch is not responding when Fluentd tries to create index template.

(1) Is it possible the above is the case?

(2) If yes, does anybody know any solution to workaround it so that Fluentd will gracefully wait till Elasticsearch cluster is ready and will create template and index all logs afterwards?

(3) If yes, can we expect such feature of waiting for the cluster to be implemented in the plugin? 🌞

TIA!

[cosmo0920]

Hi, thanks for drilled down into root cause of this issue.

(1) Is it possible the above is the case?

Possible. Because ES plugin gives up to connect and install Elasticsearch templates when connecting failure.

(2) If yes, does anybody know any solution to workaround it so that Fluentd will gracefully wait till Elasticsearch cluster is ready and will create template and index all logs afterwards?

No. There is no solution on ES plugin side.

(3) If yes, can we expect such feature of waiting for the cluster to be implemented in the plugin?

Ideally, ES plugin should wait ES cluster booting. But it is not implemented yet.

[kmotrebski]

@cosmo0920 Thanks for a super fast response!

Is there sth that you could suggest me to do with this? Is there some trick to be used with other plugin or whatever, buffer so that Fluentd will wait for Elasticsearch in this case?

[cosmo0920]

We are not familiar with docker's DHCP mechanism.
I'm not sure how to deal with on Docker side (and, currently, there is no error handling on ES plugin about this).

We can only handle this issue with provisional avoidance in operation....

1. Confirm to be booted ES cluster containers
2. Boot Fluentd container