nat-gateway-pod 已经删除，但是subnet ovn-vpc-external-network仍然有两个ip没有被清理，从而导致subnet无法删除

[zhangcongjiang]

我在测试非默认vpc出外网功能，使用了multus-cni 0.3.1版本，配合macvlan做双网卡出外网，测试很顺利，但是在删除配置的时候，遇到了一些问题，具体表现如下：subnet ovn-vpc-external-network有两个ip残留，且查询eip也发现eips01和eipd01依然存在，subnet ovn-vpc-external-network也无法删除，通过kubectl get ips 无法查询到使用中的那两个ip地址，预期中eip,subnet都应该可以正常删除

实际上出外网的subnet无法删除，使用的两个eip也无法删除，且get ips查不到对应的数据

Steps to Reproduce the Problem

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: test-vpc-1
spec:
  namespaces:
  - ns1
  staticRoutes:
    - cidr: 0.0.0.0/0
      nextHopIP: 10.0.1.254
      policy: policyDst

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovn-vpc-external-network
  namespace: kube-system
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "macvlan",
      "master": "ens224",
      "mode": "bridge",
      "ipam": {
        "type": "kube-ovn",
        "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
        "provider": "ovn-vpc-external-network.kube-system"
      }
    }'

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: ovn-vpc-external-network
spec:
  protocol: IPv4
  provider: ovn-vpc-external-network.kube-system
  cidrBlock: 10.0.0.0/24
  gateway: 10.0.0.254  #实际上这个gateway ping不通，不可达
  excludeIps:
  - 10.0.0.201..10.0.0.254

kind: ConfigMap
apiVersion: v1
metadata:
  name: ovn-vpc-nat-gw-config
  namespace: kube-system
data:
  image: 'kubeovn/vpc-nat-gateway:v1.10.4'
  enable-vpc-nat-gw: 'true'

kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
  name: gw1
spec:
  vpc: test-vpc-1
  subnet: net1
  lanIp: 10.0.1.254
  selector:
    - "kubernetes.io/hostname: k8s-node02"
    - "kubernetes.io/os: linux"

---
kind: IptablesEIP
apiVersion: kubeovn.io/v1
metadata:
  name: eips01
spec:
  natGwDp: gw1
  v4ip: 10.0.0.2
---
kind: IptablesSnatRule
apiVersion: kubeovn.io/v1
metadata:
  name: snat01
spec:
  eip: eips01
  internalCIDR: 10.0.1.0/24

kind: IptablesEIP
apiVersion: kubeovn.io/v1
metadata:
  name: eipd01
spec:
  natGwDp: gw1
  v4ip: 10.0.0.3
---
kind: IptablesDnatRule
apiVersion: kubeovn.io/v1
metadata:
  name: dnat01
spec:
  eip: eipd01               # eip name
  externalPort: '8888'
  internalIp: 10.0.1.2
  internalPort: '80'
  protocol: tcp

2. 上面是我的配置文件，一次通过apply -f 命令执行上述yaml文件发现snat和dnat已经生效，且登陆nat-gateway-pod能看到iptables规则的存在
3. 之后依次通过kubectl delete -f 删除上述配置，通过kubectl get eip 发现eip没有被移除，并且subnet删除不掉，通过kubectl get subnet 发现有两个IP地址正在使用中
   4.```bash
   k get subnet ovn-vpc-external-network
   NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS
   ovn-vpc-external-network ovn-vpc-external-network.kube-system ovn-cluster IPv4 10.0.0.0/24 false false false distributed 2 198 0 0 ["10.0.0.201..10.0.0.254"]

```bash
k get eip
NAME     IP    MAC   NAT   NATGWDP   READY
eipd01                     gw1       false
eips01                     gw1       false

Additional Info

• Kubernetes version:

  Output of kubectl version:

  (Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.9", GitCommit:"c1de2d70269039fe55efb98e737d9a29f9155246", GitTreeState:"clean", BuildDate:"2022-07-13T14:26:51Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}
  

Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.9", GitCommit:"c1de2d70269039fe55efb98e737d9a29f9155246", GitTreeState:"clean", BuildDate:"2022-07-13T14:19:57Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"})

- kube-ovn version:

(1.10.4)

- operation-system/kernel version:

**Output of `awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release`:**
**Output of `uname -r`:**

("CentOS Linux 7 (Core)"
4.19.12-1.el7.elrepo.x86_64)

<在整个配置中，我的物理网关10.0.0.254实际上是不存在的，因为我只需要pod和k8s集群中的节点能通信即可>

[zhangzujian]

Thanks for the information! We'll fix it ASAP.

[bobz965]

您好，eip 是基于 eip crd来维护的，不在ip中维护。 可以尝试基于kubectl delete eip eips01 eipd01 尝试下先清理下eip

[zhangcongjiang]

您好，eip 是基于 eip crd来维护的，不在ip中维护。 可以尝试基于kubectl delete eip eips01 eipd01 尝试下先清理下eip

无法删除

[bobz965]

您好，eip 是基于 eip crd来维护的，不在ip中维护。 可以尝试基于kubectl delete eip eips01 eipd01 尝试下先清理下eip

无法删除

您好，目前认为应该是这种情况

1. 如果nat gw pod 已经删除，eip确实是无法删除的，因为没有pod，无法执行操作了，需要先清理eip，再清理pod