Nginx reloaded every 3 seconds

[jerryjxj]

Tested in commit 7844415, I found the following log. Nginx is reloaded constantly.

E0830 04:21:12.490421 11 template.go:615] expected an Ingress
I0830 04:21:12.649545 11 controller.go:478] ingress backend successfully reloaded...
I0830 04:21:15.812189 11 controller.go:469] backend reload required
E0830 04:21:15.814736 11 template.go:615] expected an Ingress
I0830 04:21:15.975626 11 controller.go:478] ingress backend successfully reloaded...
I0830 04:21:19.146257 11 controller.go:469] backend reload required
E0830 04:21:19.149182 11 template.go:615] expected an Ingress
I0830 04:21:19.317383 11 controller.go:478] ingress backend successfully reloaded...
I0830 04:21:22.478650 11 controller.go:469] backend reload required
E0830 04:21:22.481419 11 template.go:615] expected an Ingress
I0830 04:21:22.643070 11 controller.go:478] ingress backend successfully reloaded...
I0830 04:21:25.811394 11 controller.go:469] backend reload required
E0830 04:21:25.813810 11 template.go:615] expected an Ingress
I0830 04:21:25.968366 11 controller.go:478] ingress backend successfully reloaded...
I0830 04:21:29.146397 11 controller.go:469] backend reload required
E0830 04:21:29.149227 11 template.go:615] expected an Ingress
I0830 04:21:29.309935 11 controller.go:478] ingress backend successfully reloaded...
I0830 04:21:32.479700 11 controller.go:469] backend reload required
E0830 04:21:32.483580 11 template.go:615] expected an Ingress
I0830 04:21:32.644088 11 controller.go:478] ingress backend successfully reloaded...
I0830 04:21:35.812015 11 controller.go:469] backend reload required
E0830 04:21:35.814576 11 template.go:615] expected an Ingress
I0830 04:21:35.976488 11 controller.go:478] ingress backend successfully reloaded...
I0830 04:21:39.144720 11 controller.go:469] backend reload required
E0830 04:21:39.147110 11 template.go:615] expected an Ingress
I0830 04:21:39.321595 11 controller.go:478] ingress backend successfully reloaded...
I0830 04:21:42.480317 11 controller.go:469] backend reload required
E0830 04:21:42.484820 11 template.go:615] expected an Ingress
I0830 04:21:42.658050 11 controller.go:478] ingress backend successfully reloaded...

[aledbf]

@jerryjxj please use the image quay.io/aledbf/nginx-ingress-controller:0.202
Also post the ingress rules you are using

Edit: also add the flag --v=2 to see what's changing in the configuration

[aledbf]

Closing. Please reopen when you can provide the logs after the update (if the issue continues)

[jerryjxj]

Hi @aledbf ,
I used gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.12. This issue still exists. Except that the message level.

0906 05:49:10.270157 11 controller.go:469] backend reload required
I0906 05:49:10.440955 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:13.602145 11 controller.go:469] backend reload required
I0906 05:49:13.763612 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:16.935457 11 controller.go:469] backend reload required
I0906 05:49:17.107115 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:20.268754 11 controller.go:469] backend reload required
I0906 05:49:20.433657 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:23.602011 11 controller.go:469] backend reload required
I0906 05:49:23.778793 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:26.935830 11 controller.go:469] backend reload required
I0906 05:49:27.093896 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:30.268740 11 controller.go:469] backend reload required
I0906 05:49:30.439507 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:33.601973 11 controller.go:469] backend reload required
I0906 05:49:33.762256 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:36.935667 11 controller.go:469] backend reload required
I0906 05:49:37.093202 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:40.268757 11 controller.go:469] backend reload required
I0906 05:49:40.426784 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:43.602111 11 controller.go:469] backend reload required
I0906 05:49:43.759018 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:46.935454 11 controller.go:469] backend reload required
I0906 05:49:47.093846 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:50.268879 11 controller.go:469] backend reload required
I0906 05:49:50.426165 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:53.602204 11 controller.go:469] backend reload required
I0906 05:49:53.755958 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:49:56.935414 11 controller.go:469] backend reload required
I0906 05:49:57.093557 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:00.273564 11 controller.go:469] backend reload required
I0906 05:50:00.440482 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:03.602156 11 controller.go:469] backend reload required
I0906 05:50:03.768584 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:06.935545 11 controller.go:469] backend reload required
I0906 05:50:07.088680 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:10.268781 11 controller.go:469] backend reload required
I0906 05:50:10.431437 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:13.605643 11 controller.go:469] backend reload required
I0906 05:50:13.776033 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:16.935409 11 controller.go:469] backend reload required
I0906 05:50:17.098991 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:20.268684 11 controller.go:469] backend reload required
I0906 05:50:20.427672 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:23.601962 11 controller.go:469] backend reload required
I0906 05:50:23.763189 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:26.942046 11 controller.go:469] backend reload required
I0906 05:50:27.098816 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:30.268765 11 controller.go:469] backend reload required
I0906 05:50:30.433523 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:33.601981 11 controller.go:469] backend reload required
I0906 05:50:33.765012 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:36.935245 11 controller.go:469] backend reload required
I0906 05:50:37.089712 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:40.274349 11 controller.go:469] backend reload required
I0906 05:50:40.432071 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:43.601970 11 controller.go:469] backend reload required
I0906 05:50:43.761934 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:46.935574 11 controller.go:469] backend reload required
I0906 05:50:47.094975 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:50.268455 11 controller.go:469] backend reload required
I0906 05:50:50.435859 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:53.601832 11 controller.go:469] backend reload required
I0906 05:50:53.760351 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:50:56.935345 11 controller.go:469] backend reload required
I0906 05:50:57.094708 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:00.268619 11 controller.go:469] backend reload required
I0906 05:51:00.430192 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:03.601771 11 controller.go:469] backend reload required
I0906 05:51:03.765121 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:06.935396 11 controller.go:469] backend reload required
I0906 05:51:07.093327 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:10.268610 11 controller.go:469] backend reload required
I0906 05:51:10.434336 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:13.602013 11 controller.go:469] backend reload required
I0906 05:51:13.758158 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:16.935286 11 controller.go:469] backend reload required
I0906 05:51:17.094022 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:20.269858 11 controller.go:469] backend reload required
I0906 05:51:20.439926 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:23.602384 11 controller.go:469] backend reload required
I0906 05:51:23.762992 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:26.935228 11 controller.go:469] backend reload required
I0906 05:51:27.092975 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:30.274346 11 controller.go:469] backend reload required
I0906 05:51:30.439364 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:33.602067 11 controller.go:469] backend reload required
I0906 05:51:33.768272 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:36.935415 11 controller.go:469] backend reload required
I0906 05:51:37.099071 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:40.268460 11 controller.go:469] backend reload required
I0906 05:51:40.434276 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:43.603014 11 controller.go:469] backend reload required
I0906 05:51:43.762608 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:46.935752 11 controller.go:469] backend reload required
I0906 05:51:47.100730 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:50.268811 11 controller.go:469] backend reload required
I0906 05:51:50.430542 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:53.605699 11 controller.go:469] backend reload required
I0906 05:51:53.767363 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:51:56.938559 11 controller.go:469] backend reload required
I0906 05:51:57.096201 11 controller.go:478] ingress backend successfully reloaded...
I0906 05:52:00.271451 11 controller.go:469] backend reload required

[aledbf]

@jerryjxj please add the flag --v=2 to see what's changing in the configuration

[jerryjxj]

@aledbf
Firstly, It seems that the generated configuration is not stable though the content is the same. There are screens of logs that denote the upstreams are changed just because the upstream order is changed.

After I set flag sort-backends to start ingress, no config difference log exists. But I still see the backend reload required every 3 seconds.

[jerryjxj]

Some other findings:
After I first started ingress, I found after 14 minutes, the console stopped to emit logs. (There are only 28 ingresses in the K8s cluster)

Then I tried to change an ingress, the logs contued again and it lasts 2 minutes.

[aledbf]

@jerryjxj please post the logs, I cannot help you to debug this issue without information

[martynd]

@aledbf
This issue appears to have been resolved by this mornings aledbf/nginx-ingress-controller:0.222 build. As I was previously testing with 0.217 I also checked 0.221 just to be sure of the resolution point.

Thank you

Regarding the original questions in case there is any remaining relevance:
Interestingly --v=2 didn't show any differences.
Upping it to 3 showed the json representation of the data which it seemed to think had changed but when diff'd it also matched.