Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

io.fabric8.kubernetes.client.informers.impl.cache.Reflector - listSyncAndWatch failed #6059

Closed
vicozohili opened this issue Jun 17, 2024 · 6 comments · Fixed by #6063
Closed

io.fabric8.kubernetes.client.informers.impl.cache.Reflector - listSyncAndWatch failed #6059

vicozohili opened this issue Jun 17, 2024 · 6 comments · Fixed by #6063
Assignees
@shawkins
Milestone
7.0.0

Comments

@vicozohili
Copy link

vicozohili commented Jun 17, 2024 •
edited
Loading

Suddenly getting this error:

io.fabric8.kubernetes.client.informers.impl.cache.Reflector - listSyncAndWatch failed for v1/namespaces//pods, will stop
java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.util.concurrent.RejectedExecutionException
at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:47)
at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.execute(SharedProcessor.java:179)
at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$null$4(Reflector.java:136)
at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorStore.retainAll(ProcessorStore.java:116)
at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$listSyncAndWatch$6(Reflector.java:130)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
... 16 more

Fabric8 Kubernetes Client version

6.13.0

Steps to reproduce

Pod list

Expected behavior

Pod list available

Runtime

OpenShift

Kubernetes API Server version

1.25.3@latest

Environment

Linux

Fabric8 Kubernetes Client Logs

No response

Additional context

No response
@maeryo
Copy link

maeryo commented Jun 18, 2024

I am also encountering a similar error in version 6.13.0.
When I call watchLog() to view the logs of a container in a pod or deployment, the following error occurs:

2024-06-18 15:13:49.131 ERROR 45818 --- [Http Dispatcher] i.f.k.c.informers.impl.cache.Reflector   : listSyncAndWatch failed for v1/namespaces/ns-1/pods, will stop


java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.RejectedExecutionException: null
	at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:47)
	at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.execute(SharedProcessor.java:179)
	at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$null$4(Reflector.java:136)
	at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorStore.retainAll(ProcessorStore.java:116)
	at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$listSyncAndWatch$6(Reflector.java:130)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
	... 16 common frames omitted

@manusa
Copy link
Member

manusa commented Jun 18, 2024

Could you provide a reproducer project?

/cc @shawkins

@shawkins
Copy link
Contributor

RejectedExecutionException - would indicate that the client has been closed.

@olevitt
Copy link

olevitt commented Jun 19, 2024 •
edited
Loading

We are facing the same issue since 6.13.0, I managed to reproduce it with the simplest code (created a POC repo here : https://github.com/olevitt/bug-fabric8) :

try (KubernetesClient kubernetesClient = new KubernetesClientBuilder().build()) {
            String result = kubernetesClient.pods()
                    .inNamespace(
                            "namespacename")
                    .withName("podname")
                    .getLog();
        }

This code works and retrieves the logs on all versions but produces the following error on 6.13.0 (and not on earlier versions) :

2024-06-19T09:21:21.748+02:00 ERROR 29629 --- [demo] [Http Dispatcher] i.f.k.c.informers.impl.cache.Reflector   : listSyncAndWatch failed for v1/namespaces/namespacename/podname, will stop

java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[na:na]
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142) ~[kubernetes-client-api-6.13.0.jar:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[na:na]
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51) ~[kubernetes-client-api-6.13.0.jar:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[na:na]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136) ~[kubernetes-httpclient-okhttp-6.13.0.jar:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Caused by: java.util.concurrent.RejectedExecutionException: null
	at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:47) ~[kubernetes-client-6.13.0.jar:na]
	at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.execute(SharedProcessor.java:179) ~[kubernetes-client-6.13.0.jar:na]
	at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$null$4(Reflector.java:136) ~[kubernetes-client-6.13.0.jar:na]
	at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorStore.retainAll(ProcessorStore.java:116) ~[kubernetes-client-6.13.0.jar:na]
	at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$listSyncAndWatch$6(Reflector.java:130) ~[kubernetes-client-6.13.0.jar:na]
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[na:na]
	... 16 common frames omitted

@olevitt olevitt mentioned this issue Jun 19, 2024
Closed
shawkins added a commit to shawkins/kubernetes-client that referenced this issue Jun 19, 2024
@shawkins
fix: swallow rejected execution after shutdown
8c57460 
closes: fabric8io#6059
@shawkins shawkins mentioned this issue Jun 19, 2024
Merged
11 tasks
@shawkins
Copy link
Contributor

Should be addressed by #6063

@manusa a backport is optional as this creates an erroneous log, but otherwise doen't cause a behavioral issue. The log however is at the error level.

@manusa
Copy link
Member

manusa commented Jun 19, 2024

a backport is optional as this creates an erroneous log, but otherwise doen't cause a behavioral issue. The log however is at the error level.

I will probably backport this and release 6.13.1 (will wait a few days or a week in case there is more patching required) since we want to have a last v6 minor as stable as possible.

@shawkins shawkins self-assigned this Jun 24, 2024
@shawkins shawkins mentioned this issue Jun 25, 2024
Closed
@manusa manusa added this to the 7.0.0 milestone Jun 25, 2024 — with automated-tasks
@manusa manusa closed this as completed in 1ac6b9a Jun 25, 2024
manusa pushed a commit to manusa/kubernetes-client that referenced this issue Jul 2, 2024
@shawkins @manusa
fix: swallow rejected execution after shutdown
d264884 
closes: fabric8io#6059

(cherry picked from commit 1ac6b9a)
Signed-off-by: Marc Nuri <[email protected]>
@panbingkun panbingkun mentioned this issue Jul 4, 2024
Closed
dongjoon-hyun pushed a commit to apache/spark that referenced this issue Jul 9, 2024
@panbingkun @dongjoon-hyun
[SPARK-48801][BUILD][K8S] Upgrade kubernetes-client to 6.13.1
5b00786 
### What changes were proposed in this pull request?
The pr aims to upgrade `kubernetes-client` from `6.13.0` to `6.13.1`.

### Why are the changes needed?
- The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.1
- The newest version fixed some bug, eg:
  Fix fabric8io/kubernetes-client#6059: Swallow rejected execution from internal usage of the informer executor
  Fix fabric8io/kubernetes-client#6068: KubernetesMockServer provides incomplete Configuration while creating test Config for KubernetesClient
  Fix fabric8io/kubernetes-client#6085: model getters have same annotations as fields (breaks native)

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #47206 from panbingkun/SPARK-48801.

Authored-by: panbingkun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
ericm-db pushed a commit to ericm-db/spark that referenced this issue Jul 10, 2024
@panbingkun @ericm-db
[SPARK-48801][BUILD][K8S] Upgrade kubernetes-client to 6.13.1
126e809 
### What changes were proposed in this pull request?
The pr aims to upgrade `kubernetes-client` from `6.13.0` to `6.13.1`.

### Why are the changes needed?
- The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.1
- The newest version fixed some bug, eg:
  Fix fabric8io/kubernetes-client#6059: Swallow rejected execution from internal usage of the informer executor
  Fix fabric8io/kubernetes-client#6068: KubernetesMockServer provides incomplete Configuration while creating test Config for KubernetesClient
  Fix fabric8io/kubernetes-client#6085: model getters have same annotations as fields (breaks native)

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#47206 from panbingkun/SPARK-48801.

Authored-by: panbingkun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
jingz-db pushed a commit to jingz-db/spark that referenced this issue Jul 22, 2024
@panbingkun @jingz-db
[SPARK-48801][BUILD][K8S] Upgrade kubernetes-client to 6.13.1
ec6fadb 
### What changes were proposed in this pull request?
The pr aims to upgrade `kubernetes-client` from `6.13.0` to `6.13.1`.

### Why are the changes needed?
- The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.1
- The newest version fixed some bug, eg:
  Fix fabric8io/kubernetes-client#6059: Swallow rejected execution from internal usage of the informer executor
  Fix fabric8io/kubernetes-client#6068: KubernetesMockServer provides incomplete Configuration while creating test Config for KubernetesClient
  Fix fabric8io/kubernetes-client#6085: model getters have same annotations as fields (breaks native)

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#47206 from panbingkun/SPARK-48801.

Authored-by: panbingkun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
szehon-ho pushed a commit to szehon-ho/spark that referenced this issue Aug 7, 2024
@panbingkun @dongjoon-hyun
[SPARK-48801][BUILD][K8S] Upgrade kubernetes-client to 6.13.1
75a0db3 
### What changes were proposed in this pull request?
The pr aims to upgrade `kubernetes-client` from `6.13.0` to `6.13.1`.

### Why are the changes needed?
- The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.1
- The newest version fixed some bug, eg:
  Fix fabric8io/kubernetes-client#6059: Swallow rejected execution from internal usage of the informer executor
  Fix fabric8io/kubernetes-client#6068: KubernetesMockServer provides incomplete Configuration while creating test Config for KubernetesClient
  Fix fabric8io/kubernetes-client#6085: model getters have same annotations as fields (breaks native)

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#47206 from panbingkun/SPARK-48801.

Authored-by: panbingkun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shawkins shawkins

Labels
None yet
Projects
None yet
Milestone
7.0.0
5 participants
@manusa @shawkins @olevitt @maeryo @vicozohili