Negative totalCallTime, totalExclusiveTime and very big sumOfSquares received
#1891
Comments
|
Hi @KGOH thanks for letting us know about this issue. A few clarifying questions: Have you noticed this occurring more than once? If yes, approximately how often does it occur? Any repeatable pattern that you can discern? |
|
@kford-newrelic this happens constantly, approx. every 15 minutes. But only for a specific transaction on the specific java app. It contains many broken metrics within this transaction, such as accessing db etc. I suspect that the transaction may be asynchronous as I see the |
|
Any chance you can create a repro app from that transaction that you can share with us? If we can get a repro, we might stand a chance of figuring this out... |
|
@kford-newrelic unfortunately, I don't know how to reproduce this :( |
|
Is there anything unique with regards to the problem transaction? Does it use make external calls or use the Process API in anyway? Does it use any sort of async framework? Would it be possible to get some source code or pseudo-code? We're obviously looking for any pointers in what the underlying cause is since we've been unable to repro this ourselves. |
|
@jtduffy here are transaction metrics I managed to extract. I erased sensitive information, but this should still convey the overall idea of technologies used |
|
For a counter example, here's another transaction, that also uses Redis and lettuce, but never gives negative values: |
|
@jtduffy Is this info helpful? |
|
It's more data, but unfortunately I don't think it gets us closer to a solid repro app. This has been a sporadic issue for years and we've never been able to track it down. It just seems like there's no common denominator across applications when it does pop up. |
|
@KGOH This is probably a long shot, but if the same transaction is still regularly generating these negative/large metric values every 15 minutes, can you go into the Metric Explorer and check if there are any metrics prefixed with If there are, try adding this common: &default_settings
legacy_async_api_skip_suspend: trueI believe that the agent should detect this yaml change and be able to apply it dynamically without requiring a JVM restart, which usually only takes a few minutes. After 15-30 minutes passes do you still see the negative/large metric values associated with the problematic transaction? Again, this is a long shot, but based on some of the metrics for the problematic transaction I'm wondering if the async servlet/Jetty continuations APIs are being used under the hood and if this might be related to our legacy async API that traces such activity. |
|
@KGOH apologies but we don't have enough information to investigate this further. If you later identify a pattern or other details that might be helpful for us to reproduce this issue, do let us know. |
Description
I receive stats with bad data from a single java app within the same transaction and the whole system metrics as a result. I haven't encountered such behavior on other java apps with the same agent.
callCountand min/max call time may be incorrect too though they don't look like such in this example:Expected Behavior
I expect:
totalCallTimeandtotalExclusiveTime.sumOfSquares <= maxCallTime*maxCallTime*callCountSteps to Reproduce
I have no idea how to reproduce this. It's just a random behavior from a single app .
Your Environment
java agent version 8.11.0, OpenJDK 17.0.8
Additional context
Looks like my problem is related to this issue
#709
And this PR
https://github.com/newrelic/newrelic-java-agent/pull/1685/files
The text was updated successfully, but these errors were encountered: