Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI has gateway timeouts #1488

Closed
BridgeAR opened this issue Sep 6, 2018 · 12 comments
Closed

CI has gateway timeouts #1488

BridgeAR opened this issue Sep 6, 2018 · 12 comments
Labels
incident

Comments

@BridgeAR
Copy link
Member

BridgeAR commented Sep 6, 2018

The webpage reacts super slow today and I got a gateway timeout as well.
@rvagg
Copy link
Member

rvagg commented Sep 6, 2018

So FYI gateway timeouts are because the nginx instance fronting jenkins has a shorter timeout than jenkins is able to respond in. Obviously it means Jenkins is overloaded for some reason, perhaps just because it's a huge mess of memory hogging Java but in the past it's come from having large numbers of jobs and massive amounts of objects to keep in memory (or serialize and deserialize to/from XML which is super slow). It could be just a need for a restart of Jenkins, but I'll check to make sure we're still doing proper historical build pruning which we've used to solve this kind of problem in the past.

@rvagg
Copy link
Member

rvagg commented Sep 6, 2018

I think part of our problem is these long running jobs that get stuck for some reason, e.g.

If you see these kinds of jobs and they don't seem to have any obvious activity on the console, feel free to try and stop them. If they don't stop then it may require a Jenkins restart. I'm going to queue up a restart when there's a little less action than there is now.

@rvagg
Copy link
Member

rvagg commented Sep 7, 2018

OK, updated the system and restarted everything on CI.

It looks like we're cleaning things up OK, but there are some biggies in there which are costly. Here's the top ones in terms of size:

3.3G    citgm-smoker
20G     node-compile-windows
6.2G    node-cross-compile
1.8G    node-test-binary-arm
55G     node-test-binary-windows
3.2G    node-test-commit-aix
7.6G    node-test-commit-arm
2.0G    node-test-commit-custom-suites-freestyle
4.2G    node-test-commit-freebsd
4.9G    node-test-commit-linux
14G     node-test-commit-linux-containered
3.7G    node-test-commit-linuxone
1.9G    node-test-commit-osx
2.6G    node-test-commit-plinux
1.1G    node-test-commit-v8-linux

The jobs with sub-jobs or multiple configurations have sub-jobs within them. For instance node-test-linux-containered which is 14G has 7 axis-nodes configurations in it. But even those are quite big and I'm not entirely sure why yet.

3.7G    ubuntu1604_sharedlibs_debug_x64
179M    ubuntu1604_sharedlibs_fips20_x64
138M    ubuntu1604_sharedlibs_openssl102_x64
2.0G    ubuntu1604_sharedlibs_openssl110_x64
2.5G    ubuntu1604_sharedlibs_shared_x64
2.5G    ubuntu1604_sharedlibs_withoutintl_x64
2.7G    ubuntu1604_sharedlibs_zlib_x64

One theory off the top of my head is that I think we have log timestamps turned on for all of these containered jobs so maybe it's simply log size.

@rvagg
Copy link
Member

rvagg commented Sep 7, 2018

hm, it's just the log file size

root@infra-digitalocean-ubuntu14-x64-1:/var/lib/jenkins/jobs/node-test-commit-linux-containered/configurations/axis-nodes/ubuntu1604_sharedlibs_zlib_x64/builds/6901# ls -alR
.:
total 3944
drwxr-xr-x   3 jenkins jenkins    4096 Sep  6 16:22 .
drwxr-xr-x 462 jenkins jenkins   12288 Sep  6 16:46 ..
-rw-r--r--   1 jenkins jenkins   14742 Sep  6 16:22 build.xml
-rw-r--r--   1 jenkins jenkins    1206 Sep  6 16:21 changelog.xml
-rw-r--r--   1 jenkins jenkins    2999 Sep  6 16:22 injectedEnvVars.txt
-rw-r--r--   1 jenkins jenkins     155 Sep  6 16:22 junitResult.xml
-rw-r--r--   1 jenkins jenkins 3986204 Sep  6 16:22 log
drwxr-xr-x   2 jenkins jenkins    4096 Sep  6 16:22 timestamper

./timestamper:
total 16
drwxr-xr-x 2 jenkins jenkins 4096 Sep  6 16:22 .
drwxr-xr-x 3 jenkins jenkins 4096 Sep  6 16:22 ..
-rw-r--r-- 1 jenkins jenkins 3736 Sep  6 16:22 timestamps
-rw-r--r-- 1 jenkins jenkins   41 Sep  6 16:22 timestamps.SHA-1

And I don't believe the log, as a separate file, is automatically loaded unless someone goes looking at it and afterward I would hope that it's cleared from cache. Maybe the timestamper (no timestamps in the log file btw, just pure stdout/stderr) complicates matters?

@jasnell
Copy link
Member

jasnell commented Sep 8, 2018

Still getting consistent CI gateway timeouts this afternoon.

@kfarnung
Copy link

kfarnung commented Sep 8, 2018

Currently unable to load Jenkins, just getting the 504 timeout response.

@refack refack added the incident label Sep 8, 2018
@targos
Copy link
Member

targos commented Sep 9, 2018

It seems related to Jenkins being unable to contact GitHub for the login?
When I don't get a 504, Jenkins shows a stacktrace:

java.net.SocketException: Socket closed
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
	at sun.security.ssl.InputRecord.read(InputRecord.java:532)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at okio.Okio$2.read(Okio.java:139)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:211)
Caused: java.net.SocketTimeoutException: timeout
	at okio.Okio$3.newTimeoutException(Okio.java:207)
	at okio.AsyncTimeout.exit(AsyncTimeout.java:261)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:215)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196)
	at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186)
	at com.squareup.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127)
	at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:737)
	at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:609)
	at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:460)
	at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:399)
	at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:527)
	at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105)
	at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25)
	at org.kohsuke.github.Requester.parse(Requester.java:607)
Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/user
	at org.kohsuke.github.Requester.parse(Requester.java:638)
	at org.kohsuke.github.Requester.parse(Requester.java:636)
	at org.kohsuke.github.Requester.parse(Requester.java:636)
	at org.kohsuke.github.Requester.parse(Requester.java:599)
	at org.kohsuke.github.Requester._to(Requester.java:277)
	at org.kohsuke.github.Requester.to(Requester.java:239)
	at org.kohsuke.github.GitHub.getMyself(GitHub.java:383)
	at org.kohsuke.github.GitHub.<init>(GitHub.java:157)
	at org.kohsuke.github.GitHubBuilder.build(GitHubBuilder.java:207)
	at org.jenkinsci.plugins.GithubAuthenticationToken.getGitHub(GithubAuthenticationToken.java:211)
	at org.jenkinsci.plugins.GithubAuthenticationToken.<init>(GithubAuthenticationToken.java:126)
	at org.jenkinsci.plugins.GithubSecurityRealm.doFinishLogin(GithubSecurityRealm.java:374)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
	at org.kohsuke.stapler.MetaClass$3.doDispatch(MetaClass.java:209)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:860)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at org.jenkinsci.plugins.modernstatus.ModernStatusFilter.doFilter(ModernStatusFilter.java:52)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:92)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
	at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:530)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

@BridgeAR
Copy link
Member Author

BridgeAR commented Sep 9, 2018

I got weird login errors since round about two weeks ago. They are likely the same as the ones reported by @targos.

@targos
Copy link
Member

targos commented Sep 10, 2018

I'm starting to believe that this issue does not affect everyone. I have been completely unable to access Jenkins since last Friday while other collaborators could start CI jobs and landed PR (assuming the were able to check the results)

@targos
Copy link
Member

targos commented Sep 10, 2018

I see this error now:

Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/user
at org.kohsuke.github.Requester.parse(Requester.java:638)
at

@refack
Copy link
Contributor

refack commented Sep 11, 2018

Some requests I still see consume a huge chunk of CPU:
/job/node-test-binary-windows/19921/

This is the stack trace (250 frames deep)

Handling GET /job/node-test-binary-windows/19921/ from 65.96.162.86 : RequestHandlerThread[#77] AbstractBuild/index.jelly MatrixBuild/main.jelly MatrixBuild/ajaxMatrix.jelly
com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:498)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:56)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
hudson.util.RobustMapConverter.read(RobustMapConverter.java:58)
hudson.util.RobustMapConverter.putCurrentEntryIntoMap(RobustMapConverter.java:49)
com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:98)
com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:92)
com.thoughtworks.xstream.converters.collections.MapConverter.unmarshal(MapConverter.java:87)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
hudson.util.XStream2.unmarshal(XStream2.java:161)
hudson.util.XStream2.unmarshal(XStream2.java:132)
com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
hudson.XmlFile.unmarshal(XmlFile.java:178)
hudson.XmlFile.unmarshal(XmlFile.java:161)
hudson.model.Run.reload(Run.java:338)
hudson.model.Run.<init>(Run.java:326)
hudson.model.AbstractBuild.<init>(AbstractBuild.java:173)
hudson.matrix.MatrixBuild.<init>(MatrixBuild.java:87)
sun.reflect.GeneratedConstructorAccessor96.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:165)
jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:142)
hudson.model.RunMap.retrieve(RunMap.java:224)
hudson.model.RunMap.retrieve(RunMap.java:57)
jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:344)
jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:371)
hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:194)
hudson.matrix.MatrixBuild.getBaseBuild(MatrixBuild.java:198)
hudson.matrix.MatrixBuild.getRunForConfiguration(MatrixBuild.java:235)
hudson.matrix.MatrixBuild.getRun(MatrixBuild.java:208)
hudson.matrix.MatrixBuild$RunPtr.getRun(MatrixBuild.java:130)
sun.reflect.GeneratedMethodAccessor359.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:46)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)
org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
org.kohsuke.stapler.Stapler.service(Stapler.java:238)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:860)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:92)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.server.Server.handle(Server.java:530)
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

@refack refack mentioned this issue Sep 13, 2018
Closed
@refack
Copy link
Contributor

refack commented Sep 22, 2018

Resolved (for now) via #1495

@refack refack closed this as completed Sep 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
incident
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@refack @jasnell @rvagg @kfarnung @targos @BridgeAR