Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Performance about NetworkPolicy #4349

Closed
zsxsoft opened this issue Jul 30, 2024 · 9 comments · Fixed by #4359
Closed

[BUG] Performance about NetworkPolicy #4349

zsxsoft opened this issue Jul 30, 2024 · 9 comments · Fixed by #4359
Labels
bug Something isn't working

Comments

@zsxsoft
Copy link

zsxsoft commented Jul 30, 2024

Kube-OVN Version

v1.12.19

Kubernetes Version

v1.27.4

Operation-system/Kernel Version

TencentOS Server 4.0
6.6.6-2401.0.1.tl4.4.x86_64

Description

I have a cluster with 4 nodes, ~100 subnets, ~200 pods. I've found that my pods often take more than 10 minutes to obtain a DHCP lease. Therefore, I've been debugging this issue.

My kube-ovn-controller.log looks like:

I0729 00:42:15.706804       7 network_policy.go:430] UpdateNp Egress, allows is [same 2001 ips here with different order], excepts is [], log false
I0729 00:42:24.415878       7 network_policy.go:430] UpdateNp Egress, allows is [same 2001 ips here with different order], excepts is [], log false
I0729 00:42:37.562676       7 network_policy.go:430] UpdateNp Egress, allows is [same 2001 ips here with different order], excepts is [], log false
I0729 00:42:47.649768       7 network_policy.go:430] UpdateNp Egress, allows is [same 2001 ips here with different order], excepts is [], log false
I0729 00:43:05.396868       7 network_policy.go:430] UpdateNp Egress, allows is [same 2001 ips here with different order], excepts is [], log false
I0729 00:43:14.578186       7 network_policy.go:430] UpdateNp Egress, allows is [same 2001 ips here with different order], excepts is [], log false
I0729 00:43:25.556697       7 network_policy.go:430] UpdateNp Egress, allows is [same 2001 ips here with different order], excepts is [], log false
... ~100 lines later
E0729 00:51:30.973962       7 ovn-nb-acl.go:604] more than one acl with same 'parent np.net.xxxx.vm direction from-lport priority 2000 match inport == @np.net.xxxxx.vm && ip'
E0729 00:51:30.973999       7 network_policy.go:501] failed to set egress acl log for np vm/np-net-xxxxx, more than one acl with same 'parent np.net.xxxxx.vm direction from-lport priority 2000 match inport == @np.net.xxxxx.vm && ip'
I0729 00:51:33.956182       7 network_policy.go:142] handle add/update network policy vm/np-net-xxxxx

My dashboard "Work Queue Depth" is look like this
image

So I think my problem is about networkpolicy.

I have ~100 network policies, one is special:

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: egress-allow-internet-and-some-pod
  spec:
    egress:
    - to:
      - ipBlock:
          cidr: 0.0.0.0/0
          except:
          - 10.0.0.0/8
    - to:
      - podSelector:
          matchLabels:
            allow-ingress: "true"
    policyTypes:
    - Egress

Others(>=100) look like this, to make pods in the same subnet can connect to each other:

- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: np-net-SUBNET-A
    namespace: vm
  spec:
    egress:
    - to:
      - podSelector:
          matchLabels:
            net-SUBNET-A: "1"
    podSelector:
      matchLabels:
        net-SUBNET-A: "1"
    policyTypes:
    - Egress

This seems to indicate that there are some performance issues with the NetworkPolicy under this configuration. I haven't confirmed this yet, nor have I constructed a minimal reproduction environment.

I'm attempting to use SecurityGroup as a replacement, hoping this could help.

Steps To Reproduce

/

Current Behavior

/

Expected Behavior

/
@zsxsoft zsxsoft added the bug Something isn't working label Jul 30, 2024
@zsxsoft
Copy link
Author

zsxsoft commented Jul 30, 2024 

kubectl ko nbctl acl-list egress.allow.internet.and.some.pod.vm
from-lport  2001 (inport == @egress.allow.internet.and.some.pod.vm && ip && ip4.dst == $egress.allow.internet.and.some.pod.vm.egress.allow.IPv4.0 && ip4.dst != $egress.allow.internet.and.some.pod.vm.egress.except.IPv4.0) allow-related [after-lb]
from-lport  2001 (inport == @egress.allow.internet.and.some.pod.vm && ip && ip4.dst == $egress.allow.internet.and.some.pod.vm.egress.allow.IPv4.0 && ip4.dst != $egress.allow.internet.and.some.pod.vm.egress.except.IPv4.0) allow-related [after-lb]
from-lport  2001 (inport == @egress.allow.internet.and.some.pod.vm && ip && ip4.dst == $egress.allow.internet.and.some.pod.vm.egress.allow.IPv4.0 && ip4.dst != $egress.allow.internet.and.some.pod.vm.egress.except.IPv4.0) allow-related [after-lb]
from-lport  2001 (inport == @egress.allow.internet.and.some.pod.vm && ip && ip4.dst == $egress.allow.internet.and.some.pod.vm.egress.allow.IPv4.0 && ip4.dst != $egress.allow.internet.and.some.pod.vm.egress.except.IPv4.0) allow-related [after-lb]
from-lport  2001 (inport == @egress.allow.internet.and.some.pod.vm && ip && ip4.dst == $egress.allow.internet.and.some.pod.vm.egress.allow.IPv4.0 && ip4.dst != $egress.allow.internet.and.some.pod.vm.egress.except.IPv4.0) allow-related [after-lb]
from-lport  2001 (inport == @egress.allow.internet.and.some.pod.vm && ip && ip4.dst == $egress.allow.internet.and.some.pod.vm.egress.allow.IPv4.0 && ip4.dst != $egress.allow.internet.and.some.pod.vm.egress.except.IPv4.0) allow-related [after-lb]
from-lport  2001 (inport == @egress.allow.internet.and.some.pod.vm && ip && ip4.dst == $egress.allow.internet.and.some.pod.vm.egress.allow.IPv4.0 && ip4.dst != $egress.allow.internet.and.some.pod.vm.egress.except.IPv4.0) allow-related [after-lb]
from-lport  2001 (inport == @egress.allow.internet.and.some.pod.vm && ip && ip4.dst == $egress.allow.internet.and.some.pod.vm.egress.allow.IPv4.0 && ip4.dst != $egress.allow.internet.and.some.pod.vm.egress.except.IPv4.0) allow-related [after-lb]
from-lport  2001 (ip4.dst == 10.1.2.1) allow-stateless [after-lb]
from-lport  2001 (ip4.dst == 10.1.3.1) allow-stateless [after-lb]
from-lport  2001 (ip4.dst == 10.1.4.1) allow-stateless [after-lb]
from-lport  2001 (ip4.dst == 10.1.5.1) allow-stateless [after-lb]
from-lport  2001 (ip4.dst == 10.1.6.1) allow-stateless [after-lb]
from-lport  2001 (ip4.dst == 10.1.7.1) allow-stateless [after-lb]
from-lport  2001 (ip4.dst == 10.1.8.1) allow-stateless [after-lb]
from-lport  2000 (inport == @egress.allow.internet.and.some.pod.vm && ip) drop [after-lb]
from-lport  2000 (inport == @egress.allow.internet.and.some.pod.vm && ip) drop [after-lb]
from-lport  2000 (inport == @egress.allow.internet.and.some.pod.vm && ip) drop [after-lb]
from-lport  2000 (inport == @egress.allow.internet.and.some.pod.vm && ip) drop [after-lb]
from-lport  2000 (inport == @egress.allow.internet.and.some.pod.vm && ip) drop [after-lb]
from-lport  2000 (inport == @egress.allow.internet.and.some.pod.vm && ip) drop [after-lb]
from-lport  2000 (inport == @egress.allow.internet.and.some.pod.vm && ip) drop [after-lb]
from-lport  2000 (inport == @egress.allow.internet.and.some.pod.vm && ip) drop [after-lb]
  to-lport  2001 (ip4.src == 10.1.2.1) allow-stateless
  to-lport  2001 (ip4.src == 10.1.3.1) allow-stateless
  to-lport  2001 (ip4.src == 10.1.4.1) allow-stateless
  to-lport  2001 (ip4.src == 10.1.5.1) allow-stateless
  to-lport  2001 (ip4.src == 10.1.6.1) allow-stateless
  to-lport  2001 (ip4.src == 10.1.7.1) allow-stateless
  to-lport  2001 (ip4.src == 10.1.8.1) allow-stateless

@hongzhen-ma
Copy link
Collaborator

1、The first log shows that there's sth wrong with the netpol vm/np-net-xxxxx. Is it possible to provide the content about netpol vm/np-net-xxxxx?
2、Is it convenient to provide the complete log file of kube-ovn-controller? The log file exists in directory /var/log/kube-ovn on the node where the kube-ovn-controller pod is located.

@zsxsoft
Copy link
Author

zsxsoft commented Jul 31, 2024

np-net-xxx is np-net-subnet-a attached in the main thread.

- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: np-net-SUBNET-A
    namespace: vm
  spec:
    egress:
    - to:
      - podSelector:
          matchLabels:
            net-SUBNET-A: "1"
    podSelector:
      matchLabels:
        net-SUBNET-A: "1"
    policyTypes:
    - Egress

@zsxsoft
Copy link
Author

zsxsoft commented Jul 31, 2024

head 400 lines here

I0729 00:42:15.706804       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:42:24.415878       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:42:37.562676       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:42:47.649768       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:43:05.396868       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:43:14.578186       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:43:25.556697       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:43:44.707751       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:43:53.308151       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:44:05.892742       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:44:14.972281       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:44:24.423128       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:44:33.677599       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:44:44.002198       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:44:55.686077       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:45:04.365864       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:45:12.962251       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:45:23.068879       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:45:33.873440       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:45:46.579278       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:45:55.907598       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:46:05.681336       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:46:16.650111       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:46:25.813229       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:46:34.512755       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:46:43.522341       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:46:55.470602       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:47:05.376358       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:47:16.680576       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:47:25.809364       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:47:38.083310       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:47:47.634717       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:47:56.481908       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:48:05.739772       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:48:19.297683       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:48:29.334444       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:48:38.700149       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:48:47.898933       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:48:58.102503       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:49:10.220402       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:49:19.441502       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:49:28.590998       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:49:39.889886       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:49:50.919579       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:50:00.852887       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:50:09.621900       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:50:18.228923       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:50:32.089372       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:50:41.778501       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:50:50.563090       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:50:59.841063       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:51:10.046820       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:51:20.677774       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
I0729 00:51:30.857806       7 network_policy.go:430] UpdateNp Egress, allows is [2001 ips here], excepts is [], log false
E0729 00:51:30.973962       7 ovn-nb-acl.go:604] more than one acl with same 'parent np.net.NET-1.vm direction from-lport priority 2000 match inport == @np.net.NET-1.vm && ip'
E0729 00:51:30.973999       7 network_policy.go:501] failed to set egress acl log for np vm/np-net-NET-1, more than one acl with same 'parent np.net.NET-1.vm direction from-lport priority 2000 match inport == @np.net.NET-1.vm && ip'
I0729 00:51:33.956182       7 network_policy.go:142] handle add/update network policy vm/np-net-NET-2
I0729 00:51:33.956655       7 pod.go:138] namespace vm's namedPort portname is metrics with info &{9177 [251 pods here]} 
I0729 00:51:34.366509       7 network_policy.go:430] UpdateNp Egress, allows is [29 ips], excepts is [], log false
I0729 00:51:34.494553       7 network_policy.go:430] UpdateNp Egress, allows is [29 ips], excepts is [], log false
I0729 00:51:34.606982       7 network_policy.go:430] UpdateNp Egress, allows is [29 ips], excepts is [], log false
E0729 00:51:34.634261       7 ovn-nb-acl.go:604] more than one acl with same 'parent np.net.NET-2.vm direction from-lport priority 2000 match inport == @np.net.NET-2.vm && ip'
E0729 00:51:34.634297       7 network_policy.go:501] failed to set egress acl log for np vm/np-net-NET-2, more than one acl with same 'parent np.net.NET-2.vm direction from-lport priority 2000 match inport == @np.net.NET-2.vm && ip'
I0729 00:51:34.666415       7 network_policy.go:142] handle add/update network policy vm/np-net-NET-3
I0729 00:51:34.666734       7 pod.go:138] namespace vm's namedPort portname is metrics with info &{9177 [251 pods here]} 
I0729 00:51:34.710760       7 network_policy.go:430] UpdateNp Egress, allows is [4 ips], excepts is [], log false
I0729 00:51:34.725710       7 network_policy.go:430] UpdateNp Egress, allows is [4 ips], excepts is [], log false
E0729 00:51:34.735655       7 ovn-nb-acl.go:604] more than one acl with same 'parent np.net.NET-3.vm direction from-lport priority 2000 match inport == @np.net.NET-3.vm && ip'
E0729 00:51:34.735688       7 network_policy.go:501] failed to set egress acl log for np vm/np-net-NET-3, more than one acl with same 'parent np.net.NET-3.vm direction from-lport priority 2000 match inport == @np.net.NET-3.vm && ip'
E0729 00:54:39.185347       7 subnet.go:2238] subnet net-NET-4 has 1 v4 ip in use, while the v4 using ip range is empty
E0729 00:54:39.185394       7 subnet.go:921] inconsistency detected in status of subnet net-NET-4 : subnet net-NET-4 has 1 v4 ip in use, while the v4 using ip range is empty
E0729 00:54:39.185447       7 subnet.go:246] error syncing 'net-NET-4': subnet net-NET-4 has 1 v4 ip in use, while the v4 using ip range is empty, requeuing
I0729 00:56:05.862055       7 subnet.go:983] delete u2o interconnection policy route for subnet net-NET-5
I0729 00:56:05.862142       7 vpc.go:147] handle status update for vpc 
I0729 00:56:05.862180       7 subnet.go:2760] logical router  already deleted
E0729 00:56:05.862259       7 ovn-nb-logical_router_route.go:340] not found logical router ""
E0729 00:56:05.862283       7 ovn-nb-logical_router_route.go:145] not found logical router ""
E0729 00:56:05.862296       7 vpc.go:711] del vpc  static route failed, not found logical router ""
E0729 00:56:05.862312       7 subnet.go:2882] failed to add static route, not found logical router ""
E0729 00:56:05.862328       7 subnet.go:990] failed to delete static route for underlay to overlay subnet interconnection net-NET-5, not found logical router ""
E0729 00:56:05.862393       7 subnet.go:274] error syncing 'net-NET-5': not found logical router "", requeuing
I0729 01:02:06.203633       7 service.go:58] enqueue delete service vm/vm-VM-1
I0729 01:02:09.241572       7 pod.go:248] enqueue delete pod vm/vm-VM-1-0
I0729 01:02:09.241660       7 network_policy.go:142] handle add/update network policy vm/egress-allow-internet-and-some-pod
I0729 01:02:09.241675       7 pod.go:907] handle delete pod vm/vm-VM-1-0
I0729 01:02:09.241689       7 network_policy.go:142] handle add/update network policy vm/np-net-NET-6
I0729 01:02:09.242317       7 pod.go:138] namespace vm's namedPort portname is metrics with info &{9177 [250 pods here]} 
I0729 01:02:09.249427       7 pod.go:138] namespace vm's namedPort portname is metrics with info &{9177 [250 pods here]} 
I0729 01:02:09.273480       7 pod.go:1002] gc logical switch port vm-VM-1-0.vm
I0729 01:02:09.273808       7 ovn-nb-logical_switch_port.go:685] delete logical switch port vm-VM-1-0.vm with id VM-2 from logical switch net-NET-6
I0729 01:02:09.350392       7 pod.go:1002] gc logical switch port vm-VM-1-0.vm.libvirt-exporter.vm.ovn
I0729 01:02:09.351875       7 ovn-nb-logical_switch_port.go:685] delete logical switch port vm-VM-1-0.vm.libvirt-exporter.vm.ovn with id 61c25e82-fe43-4e7a-b2fd-8415485ec66a from logical switch libvirt-exporter
I0729 01:02:09.374357       7 pod.go:1008] release all ip address for deleting pod vm/vm-VM-1-0
I0729 01:02:09.374380       7 pod.go:1030] delete cr ip 'vm-VM-1-0.vm.libvirt-exporter.vm.ovn' for pod vm/vm-VM-1-0
I0729 01:02:09.379216       7 pod.go:1030] delete cr ip 'vm-VM-1-0.vm' for pod vm/vm-VM-1-0
I0729 01:02:09.384413       7 subnet.go:496] release v4 IP1 mac XXXXXXXXXX from subnet net-NET-6 for vm/vm-VM-1-0, add ip to released list
I0729 01:02:09.384479       7 subnet.go:496] release v4 IP2 mac XXXXXXXXXX from subnet libvirt-exporter for vm/vm-VM-1-0, add ip to released list
I0729 01:02:09.384613       7 pod.go:438] take 142 ms to handle delete pod vm/vm-VM-1-0
I0729 01:02:09.560681       7 network_policy.go:430] UpdateNp Egress, allows is [18 ips here], excepts is [], log false
I0729 01:02:09.757261       7 network_policy.go:430] UpdateNp Egress, allows is [18 ips here], excepts is [], log false
I0729 01:02:09.898990       7 network_policy.go:430] UpdateNp Egress, allows is [18 ips here], excepts is [], log false
E0729 01:02:09.942402       7 ovn-nb-acl.go:604] more than one acl with same 'parent np.net.NET-6.vm direction from-lport priority 2000 match inport == @np.net.NET-6.vm && ip'
E0729 01:02:09.942443       7 network_policy.go:501] failed to set egress acl log for np vm/np-net-NET-6, more than one acl with same 'parent np.net.NET-6.vm direction from-lport priority 2000 match inport == @np.net.NET-6.vm && ip'
I0729 01:02:17.985190       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:17.999859       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.001440       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.042052       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.043407       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.060085       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.061508       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.075688       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.077293       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.092171       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.093611       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.108277       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.109725       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.142675       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.144039       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.160676       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.162255       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.177388       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.178722       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.193860       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.195199       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.209938       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.211512       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.224520       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.241232       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.257584       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.259066       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.274160       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.275620       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.360849       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.362682       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.441318       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.442897       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.461497       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.462966       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.477294       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.478797       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.553431       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.558719       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.665239       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.666719       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.678788       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.680159       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.694457       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.696056       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.710086       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.711612       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.754415       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.755842       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.770420       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.772096       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.786810       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.788424       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.802898       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.804396       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.818993       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.820465       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.851761       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.853413       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.867975       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.869694       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.884331       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.885715       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.900145       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.901676       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.916473       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.917955       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.945442       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.946930       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.963931       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.965879       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.979371       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.981077       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:18.995406       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:18.996834       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.010224       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.011600       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.024529       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.039230       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.057578       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.059272       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.072655       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.074285       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.088891       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.090340       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.103602       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.105254       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.119721       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.121203       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.144106       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.152042       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.165156       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.166522       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.178529       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.180046       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.195133       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.196462       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.211222       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.212585       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.239608       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.296672       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.312072       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.313432       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.328834       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.339572       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.353225       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.354629       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.369732       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.371144       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.451763       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.455083       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.552993       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.554596       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.569110       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.570444       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.584258       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.585820       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.598688       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.600238       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.614167       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.615425       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.650827       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.652296       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.667239       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.668781       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.683403       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.685356       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.699786       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.701221       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.715823       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.717387       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.744704       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.746421       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.758593       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.760093       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.774152       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.775412       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.789784       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.791181       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.805369       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.806765       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.819355       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.820834       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.850165       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.851641       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.865606       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.866914       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.881057       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.882744       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.896576       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.898058       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.912415       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.913975       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.950454       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.951862       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.965097       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.966552       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.978984       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.980728       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:19.994790       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:19.996038       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.010176       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.067321       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.081741       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.083282       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.097477       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.098914       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.113048       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.114514       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.128786       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.130685       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.144078       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.145542       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.162252       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.163632       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.177615       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.178874       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.192848       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.194407       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.208865       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.210300       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.226648       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.228278       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.258418       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.259879       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.357500       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.359797       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.440076       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.441969       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.465140       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.466471       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.480853       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.482301       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.496483       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.497955       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.512252       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.513683       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.525208       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.526614       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.556679       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.558255       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.572582       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.574007       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.588374       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.589986       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.601692       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.603285       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.617467       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.618885       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.652638       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.654125       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.666682       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.668112       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.682610       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.684091       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.698545       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.699860       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.713936       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.715565       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.739234       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.740799       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.761258       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.762669       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.777011       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.778495       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.791615       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.793002       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.807365       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.808756       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.822985       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.824446       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.857358       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.858906       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.873207       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.874710       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.888922       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.890399       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.904632       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.905939       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.920010       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.921340       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.953171       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.954739       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.968991       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.970437       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:20.984574       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:20.986045       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:21.000249       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:21.001853       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:21.016093       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
I0729 01:02:21.017456       7 network_policy.go:430] UpdateNp Egress, allows is [0.0.0.0/0], excepts is [10.0.0.0/8], log false
I0729 01:02:21.031775       7 network_policy.go:430] UpdateNp Egress, allows is [], excepts is [], log false
E0729 01:02:21.346939       7 ovn-nb-acl.go:604] more than one acl with same 'parent egress.allow.internet.and.some.pod.vm direction from-lport priority 2000 match inport == @egress.allow.internet.and.some.pod.vm && ip'
E0729 01:02:21.346976       7 network_policy.go:501] failed to set egress acl log for np vm/egress-allow-internet-and-some-pod, more than one acl with same 'parent egress.allow.internet.and.some.pod.vm direction from-lport priority 2000 match inport == @egress.allow.internet.and.some.pod.vm && ip'
I0729 01:02:36.002425       7 pod.go:907] handle delete pod vm/vm-VM-1-0
I0729 01:02:36.017834       7 pod.go:1008] release all ip address for deleting pod vm/vm-VM-1-0
I0729 01:02:36.017862       7 pod.go:1030] delete cr ip 'vm-VM-1-0.vm.libvirt-exporter.vm.ovn' for pod vm/vm-VM-1-0
I0729 01:02:36.020084       7 pod.go:1030] delete cr ip 'vm-VM-1-0.vm' for pod vm/vm-VM-1-0
I0729 01:02:36.022233       7 pod.go:438] take 19 ms to handle delete pod vm/vm-VM-1-0
I0729 01:06:32.848173       7 service.go:58] enqueue delete service vm/vm-VM-3
I0729 01:06:32.975130       7 network_policy.go:563] handle delete network policy vm/np-net-NET-4
I0729 01:06:34.517811       7 endpoint.go:106] update add/update endpoint vm/vm-VM-4
I0729 01:06:34.519128       7 service.go:58] enqueue delete service vm/vm-VM-4
I0729 01:06:34.928038       7 endpoint.go:106] update add/update endpoint vm/vm-VM-5
I0729 01:06:34.960683       7 service.go:58] enqueue delete service vm/vm-VM-5
I0729 01:06:35.038920       7 network_policy.go:563] handle delete network policy vm/np-net-NET-5
I0729 01:06:35.262995       7 pod.go:248] enqueue delete pod vm/vm-VM-3-0
I0729 01:06:35.263085       7 pod.go:907] handle delete pod vm/vm-VM-3-0
I0729 01:06:35.263117       7 network_policy.go:142] handle add/update network policy vm/egress-allow-internet-and-some-pod
I0729 01:06:35.272123       7 pod.go:138] namespace vm's namedPort portname is metrics with info &{9177 [249 pods here]} 
I0729 01:06:35.294679       7 pod.go:1002] gc logical switch port vm-VM-3-0.vm
I0729 01:06:35.295160       7 ovn-nb-logical_switch_port.go:685] delete logical switch port vm-VM-3-0.vm with id e9231f17-ba82-4e83-b068-6d70af5e1bf5 from logical switch net-NET-4
I0729 01:06:35.316078       7 pod.go:1002] gc logical switch port vm-VM-3-0.vm.libvirt-exporter.vm.ovn
I0729 01:06:35.316487       7 ovn-nb-logical_switch_port.go:685] delete logical switch port vm-VM-3-0.vm.libvirt-exporter.vm.ovn with id fa6690e9-1958-47f5-b68e-c32e85564e26 from logical switch libvirt-exporter
I0729 01:06:35.336651       7 pod.go:1008] release all ip address for deleting pod vm/vm-VM-3-0
I0729 01:06:35.336682       7 pod.go:1030] delete cr ip 'vm-VM-3-0.vm.libvirt-exporter.vm.ovn' for pod vm/vm-VM-3-0
I0729 01:06:35.341723       7 pod.go:1030] delete cr ip 'vm-VM-3-0.vm' for pod vm/vm-VM-3-0
I0729 01:06:35.346143       7 subnet.go:496] release v4 XXXXXXXX mac XXXXXXXXX from subnet libvirt-exporter for vm/vm-VM-3-0, add ip to released list
I0729 01:06:35.346244       7 subnet.go:496] release v4 XXXXXXXX mac XXXXXXXXX from subnet net-NET-4 for vm/vm-VM-3-0, add ip to released list
I0729 01:06:35.346297       7 pod.go:438] take 83 ms to handle delete pod vm/vm-VM-3-0
I0729 01:06:35.372247       7 subnet.go:338] format subnet net-NET-4, changed false
I0729 01:06:35.382931       7 subnet.go:983] delete u2o interconnection policy route for subnet net-NET-4
I0729 01:06:35.383042       7 vpc.go:147] handle status update for vpc ovn-cluster
I0729 01:06:35.452250       7 subnet.go:1010] delete policy route for centralized subnet net-NET-4
I0729 01:06:35.452292       7 subnet.go:2574] delete policy route for router: ovn-cluster, priority: 31000, match ip4.dst == XXXX/24
I0729 01:06:35.613480       7 subnet.go:2605] delete policy route for centralized subnet net-NET-4
I0729 01:06:35.613523       7 subnet.go:2485] delete policy route for router: ovn-cluster, priority: 29000, match ip4.src == XXXX/24
I0729 01:06:35.714596       7 ipam.go:319] delete subnet net-NET-4
I0729 01:06:37.357804       7 pod.go:248] enqueue delete pod vm/vm-VM-4-0
I0729 01:06:37.357857       7 pod.go:907] handle delete pod vm/vm-VM-4-0
I0729 01:06:37.387868       7 pod.go:1002] gc logical switch port vm-VM-4-0.vm.libvirt-exporter.vm.ovn
I0729 01:06:37.388330       7 ovn-nb-logical_switch_port.go:685] delete logical switch port vm-VM-4-0.vm.libvirt-exporter.vm.ovn with id fda5ba61-b78f-43b6-a0d5-a1dd24e1a358 from logical switch libvirt-exporter
I0729 01:06:37.410063       7 pod.go:1002] gc logical switch port vm-VM-4-0.vm
I0729 01:06:37.410514       7 ovn-nb-logical_switch_port.go:685] delete logical switch port vm-VM-4-0.vm with id 0dfce526-d8ed-417d-8836-ac0813fd1dd9 from logical switch net-NET-5
I0729 01:06:37.430721       7 pod.go:1008] release all ip address for deleting pod vm/vm-VM-4-0
I0729 01:06:37.430741       7 pod.go:1030] delete cr ip 'vm-VM-4-0.vm.libvirt-exporter.vm.ovn' for pod vm/vm-VM-4-0
I0729 01:06:37.435334       7 pod.go:1030] delete cr ip 'vm-VM-4-0.vm' for pod vm/vm-VM-4-0

@zsxsoft
Copy link
Author

zsxsoft commented Jul 31, 2024

From the dashboard, I can't find any association
image

@hongzhen-ma
Copy link
Collaborator

Thanks for the information. I will have a test in my environment.

@zsxsoft
Copy link
Author

zsxsoft commented Jul 31, 2024

I tried to use SecurityGroup to replace NetworkPolicy np-net-xxx, but it seems hard to make SG and other NP work together.

That's another problem, just a side note.

@hongzhen-ma
Copy link
Collaborator

The logs still looks confusing. Are you in the WeChat group? We can communicate the issue directly.

To join the WeChat group, you can visit https://kubeovn.github.io/docs/stable

@hongzhen-ma
Copy link
Collaborator

image

多网卡的情况下，多个子网，都被列出来过滤了(最开始过滤子网，是要区分 IPv4 和 IPv6，方便添加地址到不同的 address-set 中去)，在两个子网都是 IPv4 的情况下，循环重复创建了 ACL，出现了 issue logs 最开始的报错。

This was referenced Aug 1, 2024
Closed
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@zsxsoft @hongzhen-ma