Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

should resync be executed in resyncExecutor.scheduleWithFixedDelay instead of resyncExecutor.scheduleAtFixedRate ? #3016

Closed
akram opened this issue Apr 20, 2021 · 3 comments · Fixed by #3031
Closed

should resync be executed in resyncExecutor.scheduleWithFixedDelay instead of resyncExecutor.scheduleAtFixedRate ? #3016

akram opened this issue Apr 20, 2021 · 3 comments · Fixed by #3031
Assignees
@rohanKanojia

Comments

@akram
Copy link
Contributor

akram commented Apr 20, 2021 •
edited
Loading

The resync task in kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/cache/Controller.java is currently submitted into the resyncExecutor with a scheduleAtFixedRate .

The scheduleAtFixedRate of ServiceExecutor does not wait for a previous task to be terminated to run the next one.
If the resync tasks takes too long, that may make the Controller to restart the resync quickly. In recent kubernetes versions the requests can then be throttled which may cascade other issues.

By replacing it with scheduleWithFixedDelay, we can be sure too have a reasonnable delay between syncs.
@rohanKanojia
Copy link
Member

Is this related to #2977?

@rohanKanojia
Copy link
Member

Thanks a lot for investigating this issue. As you pointed out using scheduleWithFixedDelay seems to be producing correct results.

behavior with scheduleAtFixedRate. For a resync period of 30 seconds, I see resync happening sometimes after 30 seconds and sometimes after 60 seconds

kubernetes-examples : $ mvn exec:java -Dexec.mainClass="io.fabric8.kubernetes.examples.SharedInformerExample"ample"

[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO] 
[INFO] -------------------< io.fabric8:kubernetes-examples >-------------------
[INFO] Building Fabric8 :: Kubernetes :: Client Examples 5.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ kubernetes-examples ---
[io.fabric8.kubernetes.examples.SharedInformerExample.main()] INFO :: - Informer factory initialized.
[io.fabric8.kubernetes.examples.SharedInformerExample.main()] INFO :: - Starting all registered informers
[informer-controller-Pod] INFO io.fabric8.kubernetes.client.informers.cache.Controller - informer#Controller: ready to run resync and reflector runnable
[informer-controller-Pod] INFO io.fabric8.kubernetes.client.informers.cache.Reflector - Started ReflectorRunnable watch for class io.fabric8.kubernetes.api.model.Pod
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.233613 postgres-cf987cb96-hhdgc pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.234716 random-generator-c4c5ffd8-fnftf pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.235252 todo-backend-6dccfd658d-vd7jw pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.235674 coredns-558bd4d5db-r469v pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.23608 etcd-minikube pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.236432 kube-apiserver-minikube pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.23693 kube-controller-manager-minikube pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.23727 kube-proxy-c8fw8 pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.2376 kube-scheduler-minikube pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:04.237957 storage-provisioner pod added



[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.692739 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.693176 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.693459 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.693818 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.69426 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.694719 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.695217 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.69572 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.696137 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:25:33.696591 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.69142 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.692159 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.692624 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.693038 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.693456 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.693901 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.694283 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.694645 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.695048 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:26:33.695475 coredns-558bd4d5db-r469v pod updated



[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.691741 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.692703 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.693575 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.694147 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.694746 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.695477 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.696204 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.697039 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.698703 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:03.699557 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.691853 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.693048 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.69414 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.695091 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.696026 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.697652 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.699128 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.700595 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.701937 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:27:33.702993 coredns-558bd4d5db-r469v pod updated



[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.691824 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.693113 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.694187 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.695165 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.696144 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.6971 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.698009 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.698936 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.700013 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:28:33.70101 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.691539 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.692156 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.692547 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.692872 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.693241 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.693636 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.693977 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.694282 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.694616 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:29:33.694922 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.692219 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.69314 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.694076 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.694807 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.695936 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.697193 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.698591 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.699672 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.700591 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:30:03.70153 coredns-558bd4d5db-r469v pod updated

behavior with scheduleWithFixedDelay. For a resync period of 30 seconds, I'm able to see consistent resync after every 30 seconds:

kubernetes-examples : $ mvn exec:java -Dexec.mainClass="io.fabric8.kubernetes.examples.SharedInformerExample"
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO] 
[INFO] -------------------< io.fabric8:kubernetes-examples >-------------------
[INFO] Building Fabric8 :: Kubernetes :: Client Examples 5.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ kubernetes-examples ---
[io.fabric8.kubernetes.examples.SharedInformerExample.main()] INFO :: - Informer factory initialized.
[io.fabric8.kubernetes.examples.SharedInformerExample.main()] INFO :: - Starting all registered informers
[informer-controller-Pod] INFO io.fabric8.kubernetes.client.informers.cache.Controller - informer#Controller: ready to run resync and reflector runnable
[informer-controller-Pod] INFO io.fabric8.kubernetes.client.informers.cache.Reflector - Started ReflectorRunnable watch for class io.fabric8.kubernetes.api.model.Pod
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.902997 postgres-cf987cb96-hhdgc pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.903889 random-generator-c4c5ffd8-fnftf pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.904372 todo-backend-6dccfd658d-vd7jw pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.904769 coredns-558bd4d5db-r469v pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.905169 etcd-minikube pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.905502 kube-apiserver-minikube pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.905856 kube-controller-manager-minikube pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.906254 kube-proxy-c8fw8 pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.9066 kube-scheduler-minikube pod added
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:29.906978 storage-provisioner pod added


[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.28592 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.288316 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.289954 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.291253 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.292834 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.294062 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.295278 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.296604 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.297522 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:34:59.298583 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.286733 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.287879 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.288972 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.28997 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.291047 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.292228 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.29323 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.294168 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.295106 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:29.295934 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.287948 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.289232 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.291193 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.29249 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.293652 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.294801 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.296364 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.29771 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.298714 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:35:59.299592 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.288008 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.288595 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.28909 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.289496 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.290003 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.290555 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.291158 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.291473 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.29179 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:29.292168 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.288982 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.29003 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.291088 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.291995 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.293245 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.294637 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.295799 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.29693 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.297841 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:36:59.298924 coredns-558bd4d5db-r469v pod updated



[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.289596 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.291332 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.292354 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.293159 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.294096 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.295108 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.296102 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.297246 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.298383 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:29.299474 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.290331 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.291242 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.292228 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.29304 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.293938 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.295037 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.296081 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.297019 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.298039 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:37:59.298857 coredns-558bd4d5db-r469v pod updated


[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.291213 kube-apiserver-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.292073 kube-controller-manager-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.292797 kube-proxy-c8fw8 pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.293523 postgres-cf987cb96-hhdgc pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.294482 random-generator-c4c5ffd8-fnftf pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.295525 todo-backend-6dccfd658d-vd7jw pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.29652 kube-scheduler-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.297509 storage-provisioner pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.298609 etcd-minikube pod updated
[pool-1-thread-1] INFO :: - 2021-04-21 22:38:29.299545 coredns-558bd4d5db-r469v pod updated

@rohanKanojia rohanKanojia self-assigned this Apr 21, 2021
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Apr 21, 2021
@rohanKanojia
Fix fabric8io#3016: Use scheduleWithFixedDelay for resync task execution
94709d3 
+ When using `resyncExecutor.scheduleAtFixedRate` we're seeing inconsistent
  resync intervals since it doesn't wait for previous task to be
  terminated. Hence using `resyncExecutor.scheduleWithFixedDelay` instead.
+ Remove unused reflectExecutor from Controller and Reflector since we
  don't use scheduled reflector relist since fabric8io#2964
This was referenced Apr 21, 2021
Closed
Merged
@rohanKanojia
Copy link
Member

I also checked client-go docs and seems like scheduleWithFixedDelay is the right choice

The actual time
// between any two resyncs may be longer than the nominal period
// because the implementation takes time to do work and there may
// be competing load and scheduling noise.

https://github.com/kubernetes/client-go/blob/master/tools/cache/shared_informer.go#L148

rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Apr 22, 2021
@rohanKanojia
Fix fabric8io#3016: Use scheduleWithFixedDelay for resync task execution
b106bfb 
+ When using `resyncExecutor.scheduleAtFixedRate` we're seeing inconsistent
  resync intervals since it doesn't wait for previous task to be
  terminated. Hence using `resyncExecutor.scheduleWithFixedDelay` instead.
+ Remove unused reflectExecutor from Controller and Reflector since we
  don't use scheduled reflector relist since fabric8io#2964
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Apr 30, 2021
@rohanKanojia
Fix fabric8io#3016: Use scheduleWithFixedDelay for resync task execution
51badc4 
+ When using `resyncExecutor.scheduleAtFixedRate` we're seeing inconsistent
  resync intervals since it doesn't wait for previous task to be
  terminated. Hence using `resyncExecutor.scheduleWithFixedDelay` instead.
+ Remove unused reflectExecutor from Controller and Reflector since we
  don't use scheduled reflector relist since fabric8io#2964
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Apr 30, 2021
@rohanKanojia
Fix fabric8io#3016: Use scheduleWithFixedDelay for resync task execution
655f827 
+ When using `resyncExecutor.scheduleAtFixedRate` we're seeing inconsistent
  resync intervals since it doesn't wait for previous task to be
  terminated. Hence using `resyncExecutor.scheduleWithFixedDelay` instead.
+ Minor changes in ControllerTest regarding resync expectations/wait
  time
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Apr 30, 2021
@rohanKanojia
Fix fabric8io#3016: Use scheduleWithFixedDelay for resync task execution
5f4f256 
+ When using `resyncExecutor.scheduleAtFixedRate` we're seeing inconsistent
  resync intervals since it doesn't wait for previous task to be
  terminated. Hence using `resyncExecutor.scheduleWithFixedDelay` instead.
+ Minor changes in ControllerTest regarding resync expectations/wait
  time
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Apr 30, 2021
@rohanKanojia
Fix fabric8io#3016: Use scheduleWithFixedDelay for resync task execution
e4da6aa 
+ When using `resyncExecutor.scheduleAtFixedRate` we're seeing inconsistent
  resync intervals since it doesn't wait for previous task to be
  terminated. Hence using `resyncExecutor.scheduleWithFixedDelay` instead.
+ Minor changes in ControllerTest regarding resync expectations/wait
  time
manusa pushed a commit that referenced this issue Apr 30, 2021
@rohanKanojia @manusa
Fix #3016: Use scheduleWithFixedDelay for resync task execution
bfc3170 
+ When using `resyncExecutor.scheduleAtFixedRate` we're seeing inconsistent
  resync intervals since it doesn't wait for previous task to be
  terminated. Hence using `resyncExecutor.scheduleWithFixedDelay` instead.
+ Minor changes in ControllerTest regarding resync expectations/wait
  time
@manusa manusa mentioned this issue Jul 19, 2021
Merged
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rohanKanojia rohanKanojia

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix #3016: Use scheduleWithFixedDelay for resync task execution rohanKanojia/kubernetes-client
2 participants
@akram @rohanKanojia