Add an Option to Bypass Exception for Invalid Kubernetes Config #37939

cuichenli · 2023-12-26T09:14:43Z

Description

Context: Our application uses the Kubernetes extension, leveraging SSO for cluster authentication.

Issue: When executing unit tests locally, should the SSO session be outdated, the unit tests fail, generating the following errors:

Caused by: java.lang.IllegalArgumentException: Cannot construct instance of `io.fabric8.kubernetes.client.Config$ExecCredential` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('The SSO session associated with this profile has expired or is otherwise invalid. To refresh this SSO session run aws sso login with the corresponding profile.')
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4544)
	at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4485)
	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.parseYaml(KubernetesSerialization.java:276)
	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:252)
	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:342)
	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:327)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:185)
	at io.fabric8.kubernetes.client.Config.getExecCredentialFromExecConfig(Config.java:817)
	at io.fabric8.kubernetes.client.Config.mergeKubeConfigExecCredential(Config.java:784)
	at io.fabric8.kubernetes.client.Config.mergeKubeConfigAuthInfo(Config.java:763)
	at io.fabric8.kubernetes.client.Config.mergeKubeConfigContents(Config.java:733)
	at io.fabric8.kubernetes.client.Config.loadFromKubeconfig(Config.java:707)
	at io.fabric8.kubernetes.client.Config.tryKubeConfig(Config.java:669)
	at io.fabric8.kubernetes.client.Config.autoConfigure(Config.java:290)
	at io.fabric8.kubernetes.client.Config.autoConfigure(Config.java:286)
	at io.quarkus.kubernetes.client.runtime.KubernetesClientUtils.createConfig(KubernetesClientUtils.java:21)
	at io.quarkus.kubernetes.client.runtime.KubernetesClientUtils.createClient(KubernetesClientUtils.java:54)
	at io.quarkus.kubernetes.config.runtime.KubernetesConfigSourceFactoryBuilder$KubernetesConfigFactory.getConfigSources(KubernetesConfigSourceFactoryBuilder.java:31)
	at io.quarkus.kubernetes.config.runtime.KubernetesConfigSourceFactoryBuilder$KubernetesConfigFactory.getConfigSources(KubernetesConfigSourceFactoryBuilder.java:22)
	at io.smallrye.config.ConfigSourceFactory$ConfigurableConfigSourceFactory.getConfigSources(ConfigSourceFactory.java:58)
	at io.smallrye.config.ConfigurableConfigSource.getConfigSources(ConfigurableConfigSource.java:50)
	at io.smallrye.config.SmallRyeConfig$ConfigSources.mapLateSources(SmallRyeConfig.java:687)
	at io.smallrye.config.SmallRyeConfig$ConfigSources.<init>(SmallRyeConfig.java:577)
	at io.smallrye.config.SmallRyeConfig.<init>(SmallRyeConfig.java:68)
	at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:698)
	at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
	at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
	at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
	... 8 more

Though the issue resolves once we log in through SSO, it poses an inconvenience as unit tests ideally should not necessitate a Kubernetes setup.

Proposal: Could we possibly introduce an option that prevents this error from being thrown during unit testing?

Additional Note: I've attempted using the '@WithKubernetesServer' annotation, and it made the unit test working as expected. But it heavily consumes memory and leads to OOM errors.

Implementation ideas

Possible Solution: We could consider modifying this line: https://github.com/quarkusio/quarkus/blob/main/extensions/kubernetes-client/runtime-internal/src/main/java/io/quarkus/kubernetes/client/runtime/KubernetesClientUtils.java#L21

In case any errors are thrown and an option such as allow-empty-config is specified, we could then resort to using:

Config base = Config.empty();

This approach would enable us to bypass the error during unit tests when the configuration is invalid.

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2023-12-26T09:14:47Z

/cc @geoand (kubernetes), @iocanel (kubernetes)

geoand · 2024-01-03T13:33:50Z

Is there some reason that quarkus.kubernetes-config.enabled=false does not fit your needs?

cuichenli · 2024-01-08T02:48:11Z

Is there some reason that quarkus.kubernetes-config.enabled=false does not fit your needs?

What's strange is that, even after disabling the configuration, it continues to throw the same error.

geoand · 2024-01-08T06:55:00Z

Do you have a sample application I can look at?

cuichenli · 2024-01-08T08:53:32Z

Here is a sample application available at this GitHub link: https://github.com/cuichenli/quarkus-k8s-client-bug

To reproduce the issue, please follow these steps:

Corrupt your AWS SSO configuration files. In my case, I added some random texts in the ~/.aws/cli/cache/*.json file.
Run the command aws sts get-caller-identity. You should see the following output:

An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.

After that, when you run ./gradlew test, you will see the following error:

Starting a Gradle Daemon, 1 incompatible Daemon could not be reused, use --status for details
Unrecognized configuration key "%s" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
could not get token: SSOProviderInvalidToken: the SSO session has expired or is invalid
caused by: open /Users/cuichli/.aws/sso/cache/d87df4d0fabf6e036e28f685d28ab8714a3a6092.json: no such file or directory

HTTP Request to %s failed, error id: %s
java.lang.RuntimeException: Error injecting io.fabric8.kubernetes.client.Config me.cuichen.GreetingResource.config
        at me.cuichen.GreetingResource_Bean.doCreate(Unknown Source)
        at me.cuichen.GreetingResource_Bean.create(Unknown Source)
        at me.cuichen.GreetingResource_Bean.create(Unknown Source)
        at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:117)
        at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:41)
        at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
        at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
        at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
        at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:18)
        at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:38)
        at me.cuichen.GreetingResource_Bean.get(Unknown Source)
        at me.cuichen.GreetingResource_Bean.get(Unknown Source)
        at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:553)
        at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:533)
        at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:566)
        at io.quarkus.arc.impl.ArcContainerImpl$3.get(ArcContainerImpl.java:330)
        at io.quarkus.arc.impl.ArcContainerImpl$3.get(ArcContainerImpl.java:327)
        at io.quarkus.arc.runtime.BeanContainerImpl$1.create(BeanContainerImpl.java:58)
        at io.quarkus.resteasy.reactive.common.runtime.ArcBeanFactory.createInstance(ArcBeanFactory.java:27)
        at org.jboss.resteasy.reactive.server.handlers.InstanceHandler.handle(InstanceHandler.java:26)
        at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:139)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: mapping values are not allowed here
 in reader, line 1, column 45:
     ... t token: SSOProviderInvalidToken: the SSO session has expired or ...
                                         ^

        at org.snakeyaml.engine.v2.scanner.ScannerImpl.fetchValue(ScannerImpl.java:839)
        at org.snakeyaml.engine.v2.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:362)
        at org.snakeyaml.engine.v2.scanner.ScannerImpl.checkToken(ScannerImpl.java:192)
        at org.snakeyaml.engine.v2.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:711)
        at org.snakeyaml.engine.v2.parser.ParserImpl.lambda$produce$1(ParserImpl.java:232)
        at java.base@16/java.util.Optional.ifPresent(Optional.java:178)
        at org.snakeyaml.engine.v2.parser.ParserImpl.produce(ParserImpl.java:232)
        at org.snakeyaml.engine.v2.parser.ParserImpl.peekEvent(ParserImpl.java:206)
        at org.snakeyaml.engine.v2.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:57)
        at org.snakeyaml.engine.v2.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:43)
        at org.snakeyaml.engine.v2.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:135)
        at org.snakeyaml.engine.v2.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:115)
        at org.snakeyaml.engine.v2.composer.Composer.composeScalarNode(Composer.java:244)
        at org.snakeyaml.engine.v2.composer.Composer.composeNode(Composer.java:206)
        at org.snakeyaml.engine.v2.composer.Composer.composeValueNode(Composer.java:364)
        at org.snakeyaml.engine.v2.composer.Composer.composeMappingChildren(Composer.java:343)
        at org.snakeyaml.engine.v2.composer.Composer.composeMappingNode(Composer.java:321)
        at org.snakeyaml.engine.v2.composer.Composer.composeNode(Composer.java:210)
        at org.snakeyaml.engine.v2.composer.Composer.next(Composer.java:162)
        at org.snakeyaml.engine.v2.api.Load$YamlIterator.next(Load.java:234)
        at io.fabric8.kubernetes.client.utils.KubernetesSerialization.parseYaml(KubernetesSerialization.java:271)
        at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:252)
        at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:342)
        at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:327)
        at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:185)
        at io.fabric8.kubernetes.client.Config.getExecCredentialFromExecConfig(Config.java:820)
        at io.fabric8.kubernetes.client.Config.mergeKubeConfigExecCredential(Config.java:787)
        at io.fabric8.kubernetes.client.Config.mergeKubeConfigAuthInfo(Config.java:766)
        at io.fabric8.kubernetes.client.Config.mergeKubeConfigContents(Config.java:736)
        at io.fabric8.kubernetes.client.Config.loadFromKubeconfig(Config.java:710)
        at io.fabric8.kubernetes.client.Config.tryKubeConfig(Config.java:672)
        at io.fabric8.kubernetes.client.Config.autoConfigure(Config.java:292)
        at io.fabric8.kubernetes.client.Config.autoConfigure(Config.java:288)
        at io.quarkus.kubernetes.client.runtime.KubernetesClientUtils.createConfig(KubernetesClientUtils.java:21)
        at io.quarkus.kubernetes.client.runtime.KubernetesConfigProducer.config(KubernetesConfigProducer.java:23)
        at io.quarkus.kubernetes.client.runtime.KubernetesConfigProducer_ProducerMethod_config_ewn22jBb2T5xafKtDuqV02U4h7s_Bean.doCreate(Unknown Source)
        at io.quarkus.kubernetes.client.runtime.KubernetesConfigProducer_ProducerMethod_config_ewn22jBb2T5xafKtDuqV02U4h7s_Bean.create(Unknown Source)
        at io.quarkus.kubernetes.client.runtime.KubernetesConfigProducer_ProducerMethod_config_ewn22jBb2T5xafKtDuqV02U4h7s_Bean.create(Unknown Source)
        at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:117)
        at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:41)
        at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
        at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
        at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
        at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:18)
        at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:38)
        at io.quarkus.kubernetes.client.runtime.KubernetesConfigProducer_ProducerMethod_config_ewn22jBb2T5xafKtDuqV02U4h7s_Bean.get(Unknown Source)
        at io.quarkus.kubernetes.client.runtime.KubernetesConfigProducer_ProducerMethod_config_ewn22jBb2T5xafKtDuqV02U4h7s_Bean.get(Unknown Source)
        ... 29 more

geoand · 2024-01-08T13:28:49Z

Oh, I see what's going here...

This actually doesn't have to do with the Kubernetes Config extension per-se, but with the Kubernetes Client itself.

@manusa does the Client have any way to configure it to be lenient with the configuration it finds on disk?

manusa · 2024-01-08T14:15:20Z

I'm not sure I'm following the problem.

However, I recall we did fix similar problems not long ago:

My assumption is that the ExecCredential can't be deserialized because despite the command failure we're still attempting to read it.

Regarding unit testing and disabling the default Kubernetes Client behavior, a possible workaround would be to set the KUBERNETES_DISABLE_AUTOCONFIG environment variable or (even better) the kubernetes.disable.autoConfig system property to false. In this case, though, I really don't get how are the unit tests instantiating the Kubernetes Client, and how this instance is valid for the testing scenarios.

geoand · 2024-01-08T14:32:19Z

I really don't get how are the unit tests instantiating the Kubernetes Client, and how this instance is valid for the testing scenarios.

It's getting created because KubernetesClient is a bean.

geoand · 2024-01-08T14:33:30Z

My assumption is that the ExecCredential can't be deserialized because despite the command failure we're still attempting to read it.

Is this something worth looking into on the Client side?

manusa · 2024-01-08T14:38:02Z

My assumption is that the ExecCredential can't be deserialized because despite the command failure we're still attempting to read it.

Is this something worth looking into on the Client side?

I was already on it :)

geoand · 2024-01-08T14:38:17Z

🥇

cuichenli · 2024-01-09T03:30:54Z

i have tested with the snapshot release build and it is working as expected. awesome!

may i know when will 6.10 be released @manusa ? thanks!

manusa · 2024-01-09T05:53:02Z

may i know when will 6.10 be released @manusa ? thanks!

If everything works as expected, we should cut a minor release (6.10.0) before the end of the week.

geoand · 2024-01-22T07:25:27Z

Closing as done

cuichenli added the kind/enhancement New feature or request label Dec 26, 2023

quarkus-bot bot added the area/kubernetes label Dec 26, 2023

geoand added the triage/needs-feedback We are waiting for feedback. label Jan 4, 2024

manusa mentioned this issue Jan 8, 2024

fix: ExecCredential doesn't fail with unsupported command output fabric8io/kubernetes-client#5694

Merged

11 tasks

geoand closed this as completed Jan 22, 2024

geoand added this to the 3.7.0.CR1 milestone Jan 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an Option to Bypass Exception for Invalid Kubernetes Config #37939

Add an Option to Bypass Exception for Invalid Kubernetes Config #37939

cuichenli commented Dec 26, 2023

quarkus-bot bot commented Dec 26, 2023

geoand commented Jan 3, 2024

cuichenli commented Jan 8, 2024

geoand commented Jan 8, 2024

cuichenli commented Jan 8, 2024 •

edited

Loading

geoand commented Jan 8, 2024

manusa commented Jan 8, 2024 •

edited

Loading

geoand commented Jan 8, 2024

geoand commented Jan 8, 2024

manusa commented Jan 8, 2024

geoand commented Jan 8, 2024

cuichenli commented Jan 9, 2024

manusa commented Jan 9, 2024

geoand commented Jan 22, 2024

Add an Option to Bypass Exception for Invalid Kubernetes Config #37939

Add an Option to Bypass Exception for Invalid Kubernetes Config #37939

Comments

cuichenli commented Dec 26, 2023

Description

Implementation ideas

quarkus-bot bot commented Dec 26, 2023

geoand commented Jan 3, 2024

cuichenli commented Jan 8, 2024

geoand commented Jan 8, 2024

cuichenli commented Jan 8, 2024 • edited Loading

geoand commented Jan 8, 2024

manusa commented Jan 8, 2024 • edited Loading

geoand commented Jan 8, 2024

geoand commented Jan 8, 2024

manusa commented Jan 8, 2024

geoand commented Jan 8, 2024

cuichenli commented Jan 9, 2024

manusa commented Jan 9, 2024

geoand commented Jan 22, 2024

cuichenli commented Jan 8, 2024 •

edited

Loading

manusa commented Jan 8, 2024 •

edited

Loading