-
Notifications
You must be signed in to change notification settings - Fork 107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
relay is sending data randomly while kill -HUP happens #204
Comments
some questions: |
We have not aggregations and we have only single target forward clusters. IMHO this seems something related to this other issue (#199) |
Could be, perhaps, yet I'm searching for how this could happen, since you see misdirected traffic. |
Hi, I'm working with @toni-moreno on the same project.
They worked perfectly until the upgrade. To add more information, as you can see, the metrics had fallen from 31K to 2K after the kill -HUP described by @toni-moreno. Thanks for all! |
Hmmm, that's absolutely no good! |
FYI: I'm working on #199, but my time is limited at the moment. In the meanwhile I continue to look for clues how misdirection can happen. |
Hi, grobian. Sorry, I couldn't retrieve the info earlier. Following lines are the direct output of Version FROM: carbon-c-relay v0.40 (2015-05-18) Thanks for all! |
thanks |
hmmm, metricsBlackholed only appeared in v0.45, so are you sure you went from 0.40? |
Sorry @grobian, couldn't answer you earlier. Yes, we upgradded from 0.40: · We upgraded from v0.40 (2015-05-18) to carbon-c-relay v2.1 (b8e663) at 7:32, so the new metric: · At 7:57, we sent the · After that, we had seen the strange behaivour explained on this issue. Thanks for all, |
I see, so the reload (SIGHUP) causes the problem |
We have the same problem with SIGHUP on recent 2.2 version. It starts sending data randomly during the reload. |
In addition to @Civil comment: we was upgraded from 2.1 with some patches to 2.2-2e76c18 one day before this happened. So 2.2-2e76c18 is definitely affected. |
That actually was 2.1 + cherry-picked patches to fix previous issues with SIGHUP (#188) |
An embarassing logic error caused queues from totally unrelated servers to be swapped, causing all kinds of random erroneously delivered metrics.
@toni-moreno: I found a logic error which explains your observed mis-behaviour. I'm confident that one's going to help lot. For the other people on this issue, I'm not sure it's the same problem, it feels not, so I'll keep the issue open for now. |
Civil seems to be talking about the same issue, @sbengo seems to hint at a problem with match rules no longer working after the HUP. |
@sbengo @toni-moreno is this problem still happening, or another problem now? I believe the original problem for this bug was fixed in v2.4. If you still have this problem, please reopen. If you see another problem, please file a new issue. |
Hi @grobian , I've updated to carbon-c-relay v2.1 (b8e663), last week, and I 've found a unpleasant surprise when I have added a new match line and sent kill -HUP to the process.
We are currently processing 940K input metrics with 425 matching rules over 5 clusters, all matching rules follows the same format.
And all cluster are forward clusters like that
After my kill -HUP , and only during one little moment ( in the exact second on witch I did the -HUP signal) carbon-c-relay sended randomly .
By example:
On cluster04 we have received metrics from 10 match rules configured to send to cluster03, ( no all metrics only a 886 metrics , and only for this moment).
On cluster03 we have received metrics from 3 match rules configured to sent to cluster04 (191 metrics)
On cluster05 we have received metrics from 7 match rules configured to send to cluster02 (823 metrics)
although very few metrics erroneously sent. There is enough to avoid kill -HUP in the future. And I will need this feature to allow dinamic configuration for new systems.
Thank you very much for your patience.
The text was updated successfully, but these errors were encountered: