Globalnet: Ensure that iptable rule is actually deleted before trying ipset deletion #1746
Comments
|
This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions. |
|
This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions. |
|
Maybe I can work on this :) |
Excellent, go for it! |
|
Maybe-related: #2447 |
|
This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions. |
|
@cyclinder are you still planning on working on this? |
|
Hi @dfarrell07, I'm sorry for the delay, I thought this issue had been fixed, I can work on this next weekend, I'm busy with work these days... |
|
Hi @sridhargaddam @dfarrell07, Sorry for the delay, I'm working on this. the iptable rules are actually deleting, so ipset can't destroy. I think we just need sleep or retry, refer to weaveworks/weave#3882, WDYT? |
We already have a retry mechanism in the code. Wrt adding sleep, I dont think its a good idea. |
|
@sridhargaddam where is the retry mechanism in the code? I haven't found it, Can you tell me more details? thanks. |
|
We use resource syncer mechanism from Admiral project , one of the fields is Transform, Transform is pointer to Function that returns Boolean if event needs to be requeued For GlobalNet egress we set resource syncer Trasnform function to controller.process controller.process Calls to onDelete function if Delete fails we return True to requeue |
|
Understood, thanks for the explanation @yboaron we can eventually delete all these rules, If we don't want the logs could be misleading, a simple way is to change the logs, something like: |
I didn't see it recently |
This handles the case where a set is destroyed right after associated IP table rule(s) are deleted where the latter may not be complete yet. Retry up tp 2 seconds. Fixes submariner-io#1746 Signed-off-by: Tom Pantelis <[email protected]>
This handles the case where a set is destroyed right after associated IP table rule(s) are deleted where the latter may not be complete yet. Retry up to 2 seconds. Fixes submariner-io#1746 Signed-off-by: Tom Pantelis <[email protected]>
|
I think it's reasonable to internally retry in |
This handles the case where a set is destroyed right after associated IP table rule(s) are deleted where the latter may not be complete yet. Retry up to 2 seconds. Fixes submariner-io#1746 Signed-off-by: Tom Pantelis <[email protected]>
This handles the case where a set is destroyed right after associated IP table rule(s) are deleted where the latter may not be complete yet. Retry up to 2 seconds. Fixes #1746 Signed-off-by: Tom Pantelis <[email protected]>
In a Globalnet deployment on OCP, while running e2e tests, the following errors are noticed.
Some of the e2e tests create GlobalEgressIPs with PodSelectors. After the test is completed, when the e2e test tries to delete the GlobalEgressIPs, Globalnet controller initially deletes the associated iptable rules and then tries to delete the corresponding ipset. However, its seen that deleting the iptable rule is taking few milliseconds for the actual deletion to happen in the kernel and in the meantime when Globalnet controller tries to delete the ipset, it gets an error message as shown below.
Globalnet controller has a retry mechanism and on the next retry it succeeds.
It would be good if we can figure out a way to ensure that iptable rule is actually deleted before trying to delete the ipset in the first iteration itself as otherwise the error logs could be misleading while debugging any real issue.
The text was updated successfully, but these errors were encountered: