Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fluentd aggregator does not start #73

Closed
ricsanfre opened this issue Sep 5, 2022 · 1 comment · Fixed by #74
Closed

fluentd aggregator does not start #73

ricsanfre opened this issue Sep 5, 2022 · 1 comment · Fixed by #74
Labels
bug Something isn't working

Comments

@ricsanfre
Copy link
Owner

ricsanfre commented Sep 5, 2022 •
edited
Loading

Bug description

fluentd pod does not start with the following error:

unexpected error error_class=Faraday::Error error=":excon is not registered on Faraday::Adapter" (see full error below)

fluentd pod log file:

2022-09-04 10:34:07 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/../../../etc/fluent/fluent.conf"
2022-09-04 10:34:07 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.3'
2022-09-04 10:34:07 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2022-09-04 10:34:07 +0000 [info]: gem 'fluentd' version '1.15.2'
2022-09-04 10:34:07 +0000 [info]: gem 'fluentd' version '1.14.6'
2022-09-04 10:34:08 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified
2022-09-04 10:34:08 +0000 [info]: using configuration file: <ROOT>
  <label @FLUENT_LOG>
    <match **>
      @type null
      @id ignore_fluent_logs
    </match>
  </label>
  <source>
    @type forward
    @label @FORWARD
    bind "0.0.0.0"
    port 24224
    <transport tls>
      cert_path "/etc/fluent/certs/tls.crt"
      private_key_path "/etc/fluent/certs/tls.key"
    </transport>
    <security>
      self_hostname "fluentd-aggregator"
      shared_key xxxxxx
    </security>
  </source>
  <source>
    @type prometheus
    @id in_prometheus
    bind "0.0.0.0"
    port 24231
    metrics_path "/metrics"
  </source>
  <source>
    @type prometheus_monitor
    @id in_prometheus_monitor
  </source>
  <source>
    @type prometheus_output_monitor
    @id in_prometheus_output_monitor
  </source>
  <label @FORWARD>
    <match kube.var.log.containers.fluentd**>
      @type relabel
      @label @FLUENT_LOG
    </match>
    <filter kube.**>
      @type record_transformer
      enable_ruby true
      remove_keys log_processed
      <record>
        json_message.${record["k8s.container.name"]} ${(record.has_key?('log_processed'))? record['log_processed'] : nil}
      </record>
    </filter>
    <match **>
      @type relabel
      @label @DISPATCH
    </match>
  </label>
  <label @DISPATCH>
    <filter **>
      @type prometheus
      <metric>
        name fluentd_input_status_num_records_total
        type counter
        desc The total number of incoming records
        <labels>
          tag ${tag}
          hostname ${hostname}
        </labels>
      </metric>
    </filter>
    <match **>
      @type relabel
      @label @OUTPUT
    </match>
  </label>
  <label @OUTPUT>
    <match **>
      @type elasticsearch
      @id out_es
      @log_level "info"
      include_tag_key true
      host "efk-es-http"
      port 9200
      path ""
      scheme http
      ssl_verify true
      ssl_version TLSv1_2
      user "elastic"
      password xxxxxx
      reload_connections false
      reconnect_on_error true
      reload_on_failure true
      log_es_400_reason true
      logstash_prefix "fluentd"
      logstash_dateformat "%Y.%m.%d"
      logstash_format true
      index_name "logstash"
      target_index_key 
      type_name "fluentd"
      include_timestamp false
      template_name 
      template_file 
      template_overwrite false
      sniffer_class_name "Fluent::Plugin::ElasticsearchSimpleSniffer"
      request_timeout 5s
      application_name default
      suppress_type_name true
      enable_ilm false
      ilm_policy_id logstash-policy
      ilm_policy {}
      ilm_policy_overwrite false
      <buffer>
        flush_thread_count 8
        flush_interval 5s
        chunk_limit_size 2M
        queue_limit_length 32
        retry_max_interval 30
        retry_forever true
      </buffer>
    </match>
  </label>
</ROOT>
2022-09-04 10:34:08 +0000 [info]: starting fluentd-1.15.2 pid=7 ruby="2.7.6"
2022-09-04 10:34:08 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "--config", "/fluentd/etc/../../../etc/fluent/fluent.conf", "--plugin", "/fluentd/plugins", "-r", "/usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2022-09-04 10:34:11 +0000 [info]: adding match in @FLUENT_LOG pattern="**" type="null"
2022-09-04 10:34:12 +0000 [info]: adding match in @FORWARD pattern="kube.var.log.containers.fluentd**" type="relabel"
2022-09-04 10:34:12 +0000 [info]: adding filter in @FORWARD pattern="kube.**" type="record_transformer"
2022-09-04 10:34:12 +0000 [info]: adding match in @FORWARD pattern="**" type="relabel"
2022-09-04 10:34:12 +0000 [info]: adding filter in @DISPATCH pattern="**" type="prometheus"
2022-09-04 10:34:12 +0000 [info]: adding match in @DISPATCH pattern="**" type="relabel"
2022-09-04 10:34:12 +0000 [info]: adding match in @OUTPUT pattern="**" type="elasticsearch"
2022-09-04 10:34:12 +0000 [error]: #0 unexpected error error_class=Faraday::Error error=":excon is not registered on Faraday::Adapter"
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/middleware_registry.rb:57:in `lookup_middleware'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/rack_builder.rb:113:in `adapter'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/forwardable.rb:235:in `adapter'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:600:in `block in client'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:91:in `initialize'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday.rb:98:in `new'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday.rb:98:in `new'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/http/faraday.rb:65:in `__build_connection'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:169:in `block in __connections_from_host'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:161:in `map'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:161:in `__connections_from_host'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:149:in `__build_connections'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:60:in `initialize'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:617:in `new'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:617:in `client'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_index_template.rb:39:in `host_unreachable_exceptions'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_index_template.rb:57:in `rescue in retry_operate'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_index_template.rb:55:in `retry_operate'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:484:in `handle_last_seen_es_major_version'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:336:in `configure'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin.rb:187:in `configure'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/agent.rb:132:in `add_match'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/agent.rb:74:in `block in configure'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/agent.rb:64:in `each'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/agent.rb:64:in `configure'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/label.rb:31:in `configure'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:146:in `block in configure'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:146:in `each'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:146:in `configure'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/engine.rb:105:in `configure'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/engine.rb:80:in `run_configure'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:763:in `block in run_worker'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:1037:in `main_process'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:755:in `run_worker'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/command/fluentd.rb:381:in `<top (required)>'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/bin/fluentd:15:in `<top (required)>'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `load'
  2022-09-04 10:34:12 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `<main>'
2022-09-04 10:34:12 +0000 [error]: Worker 0 finished unexpectedly with status 1

There is an issue with the latest built fluentd customized docker image. See issue ricsanfre/fluentd-aggregator#1.
@ricsanfre ricsanfre added the bug Something isn't working label Sep 5, 2022
@ricsanfre
Copy link
Owner Author

The workaround proposed in ricsanfre/fluentd-aggregator#1 (comment) solves the issue

ricsanfre added a commit that referenced this issue Sep 6, 2022
@ricsanfre
Upgrading to multi-architectture fluentd 1.15 base image. Fix #73
1370319
@ricsanfre ricsanfre mentioned this issue Sep 7, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Memory Optimization activities ricsanfre/pi-cluster
1 participant
@ricsanfre