Adding pod store for flow-exporter and flow-aggregator to query pod information #5185
Conversation
yuntanghsu
force-pushed
the
pod_store
branch
6 times, most recently
from
ed5a7fe
to
b4bb920
Compare
yuntanghsu
changed the title
pkg/util/podstore/podstore.go
Outdated
| type PodStorage struct { | ||
| curPod cache.Indexer | ||
| prevPod cache.Indexer | ||
| // Could be other lighter implementations |
There was a problem hiding this comment.
this is not a super useful comment as it is. Maybe we could add a benchmark in podtsore_test.go and see if the current implementation scales well to a very large number of Pods.
There was a problem hiding this comment.
I've removed it. I also add a benchmark for func BenchmarkGetPodByIPAndTime. Do you think it is sufficient if I only test this function?
There was a problem hiding this comment.
GetPodByIPAndTime does not use the workqueue, so that benchmark doesn't seem relevant to the choice of workqueue.DelayingInterface.
If we wanted a valid benchmark, it would have to generate a lot of Pod Add & Delete events.
There was a problem hiding this comment.
Do you mean we should generate a lot of Pod Add & Delete events in BenchmarkGetPodByIPAndTime ? I'm not quite sure how to test workqueue.DelayingInterface as currently it's only called when we have Delete events. And do you know how many events we should generate based on the number of Pods?
There was a problem hiding this comment.
you could create 1,000 Pods in a fake apiserver, then delete all of them at once.
There was a problem hiding this comment.
I've updated the benchmark test. But I'm still thinking that even I generate lots of Pod Add/Delete events, I still can't tell whether workqueue.DelayingInterface plays a role here? When we have Delete events, the workqueue is only used by s.podsToDelete.AddAfter(pod, DelayTime) in the DeleteFunc. Is that because the workqueue can occupy some resource when we have Delete events?
Or should I have another benchmark for workqueue.DelayingInterface?
yuntanghsu
force-pushed
the
pod_store
branch
3 times, most recently
from
3541f81
to
5a37928
Compare
yuntanghsu
force-pushed
the
pod_store
branch
9 times, most recently
from
a80aa59
to
2ad8557
Compare
yuntanghsu
force-pushed
the
pod_store
branch
4 times, most recently
from
e675cbd
to
6126121
Compare
|
/test-all |
yuntanghsu
force-pushed
the
pod_store
branch
3 times, most recently
from
40dfbdb
to
c22aae4
Compare
|
/test-all |
1. Add Pod store which can be used to store current and previous Pods information. 2. Modify unit test Signed-off-by: Yun-Tang Hsu <[email protected]>
|
/test-all |
tnqn
added
the
action/release-note
In this PR, we are adding a Pod store, which can be used to fetch Pod information by using ip and the connection start time.
The Pod store has the information for existing pods and the information for Pods deleted within 5 minutes.
Pod store is used to increase the accuracy for the records sent from flow-exporter to flow-aggregator.
Motivation:
Since connections are polled from conntrack table in flow-exporter every 5s and are sent to flow-aggregator every 5s, there will be at least 5s delay between the time the connection is established and the time we fetch Pod information in flow-exporter or flow-aggregator (we fill pod information using srcIP or dstIP in flow-exporter and flow-aggregator)
In theory, it is possible for the mapping (ip to pod) to be incorrect due to this 5s delay if there is very high Pod churn and an IP address is reused very quickly: