Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fatal error: concurrent map writes #2102

Closed
zrice57 opened this issue Nov 29, 2016 · 2 comments · Fixed by #2185
Closed

fatal error: concurrent map writes #2102

zrice57 opened this issue Nov 29, 2016 · 2 comments · Fixed by #2185
Assignees
@sparrc
Labels
bug unexpected problem or unintended behavior panic issue that results in panics from Telegraf
Milestone
1.2.0

Comments

@zrice57
Copy link

zrice57 commented Nov 29, 2016

Bug report

Telegraf crashes with error "fatal error: concurrent map writes"

Relevant telegraf.conf:

See gist.

Relevant Input

I have two statsd inputs configured and a few other inputs. I've only included the statsd input config because it seems the most relevent.

This has been happening on multiple servers regularly for at least a week.

System info:

Steps to reproduce:

  1. Run telegraf
  2. telegraf will run without issues for many hours
  3. kapacitor will tell me one of my servers has stopped sending data
  4. kapacitor will tell me "JK it's all good" about 30 seconds later
  5. Check telegraf logs (in /varr/log/messages)
  6. Learn that telegraf has crashed and there is a huge crash log

Expected behavior:

Telegraf will run until the heat death of the universe.

Actual behavior:

Telegraf runs for a few hours.

Additional info:

Here is the link to the crash log and configs: Link

I will provide anything else you need.
Thank you for your time!
@sparrc
Copy link
Contributor

sparrc commented Nov 29, 2016

thanks for the crash log, might be related to this: #1975

@sparrc sparrc added this to the 1.2.0 milestone Nov 29, 2016
@sparrc sparrc added bug unexpected problem or unintended behavior panic issue that results in panics from Telegraf labels Nov 29, 2016
@zrice57
Copy link
Author

zrice57 commented Dec 2, 2016

I've disabled one of the statsd inputs and have not seen a crash since.

sparrc added a commit that referenced this issue Dec 20, 2016
@sparrc
Do not create a global statsd "previous instance"
200237a 
this basically reverts #887

at some point we might want to do some special handling of reloading
plugins and keeping their state intact, but that will need to be done at
a higher level, and in a way that is thread-safe for multiple input
plugins of the same type.

Unfortunately this is a rather large feature that will not have a quick
fix available for it.

fixes #1975
fixes #2102
@sparrc sparrc mentioned this issue Dec 20, 2016
Merged
3 tasks
njwhite pushed a commit to njwhite/telegraf that referenced this issue Jan 31, 2017
@sparrc
Do not create a global statsd "previous instance"
505b183 
this basically reverts influxdata#887

at some point we might want to do some special handling of reloading
plugins and keeping their state intact, but that will need to be done at
a higher level, and in a way that is thread-safe for multiple input
plugins of the same type.

Unfortunately this is a rather large feature that will not have a quick
fix available for it.

fixes influxdata#1975
fixes influxdata#2102
maxunt pushed a commit that referenced this issue Jun 26, 2018
@sparrc
Do not create a global statsd "previous instance"
73acd11 
this basically reverts #887

at some point we might want to do some special handling of reloading
plugins and keeping their state intact, but that will need to be done at
a higher level, and in a way that is thread-safe for multiple input
plugins of the same type.

Unfortunately this is a rather large feature that will not have a quick
fix available for it.

fixes #1975
fixes #2102
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sparrc sparrc

Labels
bug unexpected problem or unintended behavior panic issue that results in panics from Telegraf
Projects
None yet
Milestone
1.2.0
Development

Successfully merging a pull request may close this issue.

Do not create a global statsd "previous instance" influxdata/telegraf
3 participants
@sparrc @121watts @zrice57