Application unable to start intermittently after upgrading from 5.8.0 to 7.4.2

[twcrone]

Customer upgraded Java agent to 7.4.2 from 5.8.0 to address the log4j security issue.

After upgrade, customer could not start their application intermittently. They get the following error in newrelic.log when that happens. This happens very randomly. According to customer, usually this happens 1 out of 4 application restarts.

java.lang.NullPointerException: null
at com.newrelic.agent.stats.StatsServiceImpl.doStatsWork(StatsServiceImpl.java:61) ~[newrelic.jar:7.4.2]
at com.newrelic.agent.service.ServiceManagerImpl.replayStartupStatsWork(ServiceManagerImpl.java:585) ~[newrelic.jar:7.4.2]
at com.newrelic.agent.service.ServiceManagerImpl.doStart(ServiceManagerImpl.java:195) ~[newrelic.jar:7.4.2]
at com.newrelic.agent.service.AbstractService.start(AbstractService.java:63) ~[newrelic.jar:7.4.2]
at com.newrelic.agent.Agent.continuePremain(Agent.java:162) [newrelic.jar:7.4.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]```

[twcrone]

https://newrelic.atlassian.net/browse/GTSE-12882

[meiao]

Fix should leverage:

• guarding the specific code to prevent NPE;
• preventing a null StatWork from being added to the collection, and logging when such thing happens with some context on what was supposed to be the null StatWork.

[jasonjkeller]

Copying a comment from Jeff:

I really don't think it's a multithreaded issue; the "real" premain isn't even complete.
But given that it appears that the class transformer service does start an executor prior to the startup stats happening, it's possible there are multithreaded things.
I'm not clear on Lists -- we're changing the list, not the field, so I'm not clear that making the field volatile passes semantics onto the full object.
Nonetheless, it seems reasonable that we could get unguarded nulls in the list. The API doesn't prevent it. So I think it's reasonable to both a) synchronize access to the list and b) avoid nulls when pulling stuff out of the list.