Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHubAppCredentials.writeReplace to avoid recalculating tokens #302

Merged
merged 3 commits into from
May 27, 2020

Conversation

jglick
Copy link
Member

@jglick jglick commented May 18, 2020

Despite #291, using app authentication made checkouts on agents really slow. On a server I tested with using the kubernetes plugin, this reduces the time taken by checkout scm on a tiny github.com repo from 46s to 26s. Otherwise agent stack traces during checkout show lots of delays like

"pool-1-thread-11 for …
	at java.lang.Object.wait(Native Method)
	-  waiting on hudson.remoting.RemoteInvocationHandler$RPCRequest@7745dfbe
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
	at com.sun.proxy.$Proxy5.fetch3(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:211)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	-  locked hudson.remoting.RemoteClassLoader@26c7de78
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at io.jsonwebtoken.lang.Classes.<clinit>(Classes.java:32)
	at io.jsonwebtoken.Jwts.builder(Jwts.java:115)
	at org.jenkinsci.plugins.github_branch_source.JwtHelper.createJWT(JwtHelper.java:45)
	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:106)
	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:151)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createPasswordFile(CliGitAPIImpl.java:2122)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1943)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:80)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:563)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:787)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at …
"pool-1-thread-11 for …
	at java.lang.Object.wait(Native Method)
	-  waiting on hudson.remoting.RemoteInvocationHandler$RPCRequest@284ec61d
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
	at com.sun.proxy.$Proxy5.fetch3(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:211)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	-  locked hudson.remoting.RemoteClassLoader@6a818a9e
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.<clinit>(BasicSerializerFactory.java:80)
	at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:641)
	at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:540)
	at io.jsonwebtoken.io.JacksonSerializer.<clinit>(JacksonSerializer.java:12)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at io.jsonwebtoken.lang.Classes.newInstance(Classes.java:156)
	at io.jsonwebtoken.lang.Classes.newInstance(Classes.java:136)
	at io.jsonwebtoken.impl.io.RuntimeClasspathSerializerLocator.locate(RuntimeClasspathSerializerLocator.java:34)
	at io.jsonwebtoken.impl.io.RuntimeClasspathSerializerLocator.getInstance(RuntimeClasspathSerializerLocator.java:21)
	at io.jsonwebtoken.impl.io.RuntimeClasspathSerializerLocator.getInstance(RuntimeClasspathSerializerLocator.java:12)
	at io.jsonwebtoken.impl.DefaultJwtBuilder.compact(DefaultJwtBuilder.java:301)
	at org.jenkinsci.plugins.github_branch_source.JwtHelper.createJWT(JwtHelper.java:53)
	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:106)
	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:151)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createPasswordFile(CliGitAPIImpl.java:2122)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1943)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:80)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:563)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:787)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at …
"pool-1-thread-11 for …
	at java.lang.Object.wait(Native Method)
	-  waiting on hudson.remoting.RemoteInvocationHandler$RPCRequest@169fca66
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
	at com.sun.proxy.$Proxy5.fetch3(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:211)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	-  locked hudson.remoting.RemoteClassLoader@6da2dee4
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at org.bouncycastle.jcajce.provider.symmetric.util.ClassUtil.loadClass(Unknown Source)
	at org.bouncycastle.jce.provider.BouncyCastleProvider.loadAlgorithms(Unknown Source)
	at org.bouncycastle.jce.provider.BouncyCastleProvider.setup(Unknown Source)
	at org.bouncycastle.jce.provider.BouncyCastleProvider.access$000(Unknown Source)
	at org.bouncycastle.jce.provider.BouncyCastleProvider$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.bouncycastle.jce.provider.BouncyCastleProvider.<init>(Unknown Source)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at io.jsonwebtoken.lang.Classes.newInstance(Classes.java:156)
	at io.jsonwebtoken.lang.RuntimeEnvironment.enableBouncyCastleIfPossible(RuntimeEnvironment.java:53)
	at io.jsonwebtoken.lang.RuntimeEnvironment.<clinit>(RuntimeEnvironment.java:62)
	at io.jsonwebtoken.impl.crypto.RsaProvider.<clinit>(RsaProvider.java:59)
	at io.jsonwebtoken.impl.crypto.DefaultSignerFactory.createSigner(DefaultSignerFactory.java:43)
	at io.jsonwebtoken.impl.crypto.DefaultJwtSigner.<init>(DefaultJwtSigner.java:51)
	at io.jsonwebtoken.impl.crypto.DefaultJwtSigner.<init>(DefaultJwtSigner.java:39)
	at io.jsonwebtoken.impl.DefaultJwtBuilder.createSigner(DefaultJwtBuilder.java:370)
	at io.jsonwebtoken.impl.DefaultJwtBuilder.compact(DefaultJwtBuilder.java:352)
	at org.jenkinsci.plugins.github_branch_source.JwtHelper.createJWT(JwtHelper.java:53)
	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:106)
	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:151)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createPasswordFile(CliGitAPIImpl.java:2122)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1943)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:80)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:563)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:787)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at …
"pool-1-thread-11 for …
	at java.lang.Object.wait(Native Method)
	-  waiting on hudson.remoting.RemoteInvocationHandler$RPCRequest@4ae587eb
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
	at com.sun.proxy.$Proxy5.fetch3(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:211)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	-  locked hudson.remoting.RemoteClassLoader@45d5a06d
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.getDeclaredMethods(Class.java:1975)
	at com.fasterxml.jackson.databind.util.ClassUtil.getDeclaredMethods(ClassUtil.java:1120)
	at com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1143)
	at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:110)
	at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:42)
	at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collectMethods(AnnotatedMethodCollector.java:33)
	at com.fasterxml.jackson.databind.introspect.AnnotatedClass._methods(AnnotatedClass.java:365)
	at com.fasterxml.jackson.databind.introspect.AnnotatedClass.memberMethods(AnnotatedClass.java:305)
	at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:525)
	at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:309)
	at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getPropertyMap(POJOPropertiesCollector.java:287)
	at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getProperties(POJOPropertiesCollector.java:170)
	at com.fasterxml.jackson.databind.introspect.BasicBeanDescription._properties(BasicBeanDescription.java:164)
	at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findProperties(BasicBeanDescription.java:239)
	at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._findCreatorsFromProperties(BasicDeserializerFactory.java:292)
	at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:276)
	at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:224)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:220)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143)
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:414)
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
	-  locked java.util.HashMap@c84e754
	at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
	at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:446)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:183)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:27)
	at com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:653)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:484)
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
	-  locked java.util.HashMap@c84e754
	at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
	at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:479)
	at com.fasterxml.jackson.databind.ObjectReader._prefetchRootDeserializer(ObjectReader.java:2094)
	at com.fasterxml.jackson.databind.ObjectReader.forType(ObjectReader.java:681)
	at com.fasterxml.jackson.databind.ObjectReader.forType(ObjectReader.java:701)
	at org.kohsuke.github.GitHubResponse.parseBody(GitHubResponse.java:85)
	at org.kohsuke.github.Requester.lambda$fetch$1(Requester.java:71)
	at org.kohsuke.github.Requester$$Lambda$33/440271353.apply(Unknown Source)
	at org.kohsuke.github.GitHubClient.createResponse(GitHubClient.java:406)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:360)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:312)
	at org.kohsuke.github.Requester.fetch(Requester.java:71)
	at org.kohsuke.github.GHAppCreateTokenBuilder.create(GHAppCreateTokenBuilder.java:87)
	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:127)
	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:151)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createPasswordFile(CliGitAPIImpl.java:2122)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1943)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:80)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:563)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:787)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at …

because not only is the token generated on the master not being reused, the agent is loading a ton of additional classes.

Note that the poor design of git-client-api still results in plenty of overhead; you would expect the master to just issue some /usr/bin/git commands to the agent, but instead a lot of classes are transferred to the agent and computation done there. See JENKINS-30600.

Note that https://github.com/jenkinsci/credentials-plugin/blob/6f710a7e30156e94aff8a3c6c6799cef04ad35df/docs/implementation.adoc#additional-concerns suggests using CredentialsSnapshotTaker. After jenkinsci/ssh-credentials-plugin@18b3121 there seems to be no implementation in a widely used credentials type. I did not see any particular benefit to using this extension point since you need to implement your own writeReplace anyway—may as well inline the snapshotting (no reason for it to be extensible).

@jglick jglick requested review from timja and bitwiseman May 18, 2020 22:53
@jglick jglick marked this pull request as ready for review May 18, 2020 22:53
@bitwiseman
Copy link
Contributor

There are code paths in this plugin that depend on the type of credentials used being GitHubAppCredentials or not. I think those paths are only run on the Jenkins master, but I that might not always be the case - what happens if it is not? Would it be possible to return a subtype of GitHubAppCredentials instead?

@jglick
Copy link
Member Author

jglick commented May 19, 2020

I think those paths are only run on the Jenkins master

Checked, and they are. On the agent side, you are just interested in a StandardUsernamePasswordCredentials.

An alternative fix would be to create a GitHubAppCredentials which would copy the otherwise transient cache fields (cachedToken + tokenCacheTime), so the agent could refresh the token if it somehow was using the same instance several minutes later. I do not think that could happen, but may be safer.

@jglick jglick marked this pull request as draft May 19, 2020 14:58
@jglick jglick changed the title GitHubAppCredentials.writeReplace → UsernamePasswordCredentialsImpl GitHubAppCredentials.writeReplace to avoid recalculating tokens May 19, 2020
@jglick jglick marked this pull request as ready for review May 19, 2020 15:49
Copy link
Contributor

@bitwiseman bitwiseman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This definitely seems better to me. Thanks!

Now we just need a test for it.

Copy link
Member

@timja timja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok, I've never seen writeReplace before this PR though

@jglick
Copy link
Member Author

jglick commented May 19, 2020

I've never seen writeReplace before this PR

See the before state in jenkinsci/ssh-credentials-plugin@18b3121.

@jglick
Copy link
Member Author

jglick commented May 19, 2020

Now we just need a test for it.

I suppose you are talking about some AbstractGitHubWireMockTest subclass which does a smoke test of GitHubAppCredentials incl. a remote build with a checkout scm? Not sure how that would work; you cannot easily mock out the Git clone.

@psalaberria002

This comment has been minimized.

@jglick

This comment has been minimized.

@bitwiseman
Copy link
Contributor

@jglick
We need something to verify this is working.
Or at very least we need something in the code base that will stop someone from accidentally regressing this behavior.

Even comments in the code about why it is needed would be start.

@jglick
Copy link
Member Author

jglick commented May 21, 2020

Added comments. As previously mentioned, while I believe it would be straightforward to add test coverage for GitHubAppCredentials generally, I struggle to see how (without a lot of effort) we would simulate a Git clone operation, which is the only thing I can think of which the agent would be using Git credentials for in a realistic build. Even something like

node('remote') {
  withCredentials([usernameColonPassword(credentialsId: 'ghapp', variable: 'AUTH')]) {
    sh 'curl -f -u $AUTH $ENDPOINT/api/v3/app'
  }
}

will not cut it, because getPassword will be called on the master side and the AUTH value remoted. We could write a faux build step that packages the StandardUsernamePasswordCredentials in a MasterToSlaveCallable I suppose.

Copy link
Contributor

@bitwiseman bitwiseman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, this sounds reasonable to me.

@bitwiseman bitwiseman requested a review from dwnusbaum May 21, 2020 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants