[Spike] Investigate memory use of HttpUrlConnection instrumentation

[meiao]

Description

A customer noticed some unusual memory behavior in his application. The behavior was consistent with a memory leak, but there was no OOM error reported. This happens in agents 8.7.0 or newer.

The customer investigated and noticed that most of the memory is in a weak keyed map inside HttpUrlConnection_1016704860_nr_ext. This is a class created during runtime by the agent that holds the fields introduced by the @NewField annotation. He/she also pointed out that this may be related to the changes introduced by PR #1537 . The suspicion is that the instrumentation is not reaching MetricState.reportExternalCall which would null out its fields dtTracer and externalTracer.

[workato-integration]

https://new-relic.atlassian.net/browse/NR-230647

[meiao]

While trying to reproduce, a simple app that makes requests was created. The app simply creates a transaction and makes an HTTP request to localhost. There is a server running locally that responds to that request.

public class Repro {
    public static void main(String[] args) throws Exception {
        for (int i = 0; i < 20; i++) {
            new Thread(() -> {
                while (true) {
                    try {
                        doCall();
                    } catch (Exception ex) {
                        // keep on
                    }
                }

            }).start();
        }
    }

    @Trace(dispatcher = true)
    public static void doCall() throws Exception {
        URL url = new URL("http://localhost:8080");
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        connection.connect();
    }
}

After running this application for a few minutes it was noticed that SlowTransactionService was holding over 100k transactions and the application was experiencing the symptoms of a memory leak.
This may or may be related to the behavior the customer reported, or it could also be something different.

[eNorby1]

Hi, we have also been experiencing a memory leak - our application's G1 Old Gen heap memory usage constantly kept rising, until eventually running into OOM after a couple weeks. Using MAT we traced the leak suspect to newrelic HttpUrlConnections (see screenshot below). Our application is running on java 21, so we cannot try versions older than 8.7.0, so not sure when the issue got first introduced. For now we had to disable the newrelic agent completely, to prevent the memory issues.

If there's any info I can provide to help get this fixed, let me know.

[meiao]

@eNorby1 do you have slow transaction detection on?
If so, can you try disabling it by?

common:
  slow_transactions:
    enabled: false

[eNorby1]

@eNorby1 do you have slow transaction detection on? If so, can you try disabling it by?

common:
  slow_transactions:
    enabled: false

I'll give it a try and report back

[eNorby1]

Hi @meiao problem seems the same with disabled slow transaction detection - would take a couple weeks to run OOM, but you can already see it consistently going up after deploying with the newrelic agent reenabled:

[meiao]

@eNorby1 Can you provide an example of how you are using the HttpUrlConnection object?
For example:

var connection = (HttpURLConnection) new URL("http://example.org").openConnection();
int responseCode = connection.getResponseCode();
...

[eNorby1]

@eNorby1 Can you provide an example of how you are using the HttpUrlConnection object? For example:

var connection = (HttpURLConnection) new URL("http://example.org").openConnection();
int responseCode = connection.getResponseCode();
...

I'm not using it directly, they are via Spring Cloud OpenFeign.

[meiao]

That's helpful, we'll try to reproduce using that. Thanks.

[WTThomas1]

In case it's helpful, I believe we may be seeing the same issue and/or something related:

This started after updating from 8.6 -> 8.10 (then downgrading to 8.9.1 due to discovering the deadlock issue the hard way).

The vast majority of the codebase is HCL DX / IBM Portal based and not utilizing HttpURLConnection directly. However, I do see one direct usage of HttpURLConnection as follows:

final URL url = new URL( apiUrl );

...

final HttpURLConnection conn = (HttpURLConnection) url.openConnection();

conn.setRequestMethod( "POST" );
conn.setRequestProperty( "Content-Type", "application/x-www-form-urlencoded" );
conn.setRequestProperty( "Content-Length", String.valueOf( postDataBytes.length ) );
conn.addRequestProperty( "Authorization", basicAuthPayload );
conn.setDoOutput( true );

conn.getOutputStream().write( postDataBytes );

final Reader in = new BufferedReader( new InputStreamReader( conn.getInputStream(), "UTF-8" ) );

Based on the location in the code, this would not be frequently triggered. Something on the order of 20 / week when the above is ~2 days.

[jbedell-newrelic]

@eNorby1 and @WTThomas1 We believe we may have identified the source of the leak. Would you be able to take this custom JAR (from this build) and see if that resolves your issue? I would advise doing this in a lower environment, since it's a 1-off build of the agent.

[WTThomas1]

Good Afternoon,

I do apologize. We don't really have enough traffic in lower environments to be able to effectively test this issue. For multiple reasons, it really isn't possible to test in production at this point. We're frozen at 8.6 until this issue is resolved (and the associated release has been available / in use without issues for a while).

Given our other projects and deadlines, I don't know that I can really be much help at this point (at least until late June). If testing is still needed once we get to a point where I do have some time, I can see if I can put something together (and I'll update if that is the case).

WTT

[eNorby1]

Hi @jbedell-newrelic, pretty much the same for me too -- we could never generate enough traffic on our staging environments to reproduce the issue, it needed to run for a few days on production for the memory leak to become apparent.

[jbedell-newrelic]

Okay, I've merged my changes in and they should be in the next release. If that doesn't resolve it for you, let us know.