Flatcar with container openvpn-client not wokring properly

[lmq1999]

Description

Flatcar with openvpn client container not wokring

Impact

Unable to use flatcar with openvpn

Environment and steps to reproduce

1. Set-up:
   Flatcar image: flatcar_production_openstack_image.img

pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3975.2.0
VERSION_ID=3975.2.0
BUILD_ID=2024-08-05-2103
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3975.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3975.2.0:*:*:*:*:*:*:*"

2. Task: Running Openvpn-client container (I have tried a lot docker openvpn-client on github and build one myself)

2.1: Install docker-compose

curl -SL https://github.com/docker/compose/releases/download/v2.29.1/docker-compose-linux-x86_64 -o /opt/bin/docker-compose
chmod +x /opt/bin/docker-compose

2.2: Write docker-compose VPN file

version: "3.3"
services:
  vpn:
    image: cr-hn-1.bizflycloud.vn/31ff9581861a4d0ea4df5e7dda0f665d/openvpn-client@sha256:2defe3062e65ad0ecf43bc747d60d1274ebbf7a45100c5cf5448fd7114caac80
    volumes:
      - /etc/openvpn/kengine.conf:/vpn/kengine.conf
      - /dev/net/tun:/dev/net/tun
    cap_add:
    - NET_ADMIN
    restart: always
    network_mode: "host"

2.3: Write openvpn-client file

client
dev kengine
dev-type tap
reneg-sec 0
proto tcp-client
remote xxx.xx.xx.xx xxxxx
resolv-retry infinite
nobind
<ca>
-----BEGIN CERTIFICATE-----
.........
-----END CERTIFICATE-----

</ca>
<key>
-----BEGIN PRIVATE KEY-----
..........
-----END PRIVATE KEY-----
</key>
<cert>
-----BEGIN CERTIFICATE-----
......
-----END CERTIFICATE-----

</cert>
<tls-auth>
#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
...........
-----END OpenVPN Static key V1-----

</tls-auth>
remote-cert-tls server
key-direction 1
script-security 3
keepalive 10 60
persist-key
persist-tun
comp-lzo
verb 3

3. Action(s):
   a. Run the docker-compose

pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # docker-compose -f docker-compose-2.yaml up
WARN[0000] /root/docker-compose-2.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
WARN[0000] Found orphan containers ([openvpn-client root-openvpn-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
[+] Running 1/0
 ✔ Container root-vpn-1  Created                                                                                                                                                                                                                                                                                                                                                       0.0s 
Attaching to vpn-1
vpn-1  | + exec sg vpn -c 'openvpn --cd /vpn --config /vpn/kengine.conf --script-security 2 --redirect-gateway def1                 '
vpn-1  | Mon Aug 12 08:30:58 2024 OpenVPN 2.4.9 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 20 2020
vpn-1  | Mon Aug 12 08:30:58 2024 library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
vpn-1  | Mon Aug 12 08:30:58 2024 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
vpn-1  | Mon Aug 12 08:30:58 2024 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
vpn-1  | Mon Aug 12 08:30:58 2024 TCP/UDP: Preserving recently used remote address: [AF_INET]123.31.11.151:10001
vpn-1  | Mon Aug 12 08:30:58 2024 Socket Buffers: R=[131072->131072] S=[16384->16384]
vpn-1  | Mon Aug 12 08:30:58 2024 Attempting to establish TCP connection with [AF_INET]123.31.11.151:10001 [nonblock]
vpn-1  | Mon Aug 12 08:30:59 2024 TCP connection established with [AF_INET]123.31.11.151:10001
vpn-1  | Mon Aug 12 08:30:59 2024 TCP_CLIENT link local: (not bound)
vpn-1  | Mon Aug 12 08:30:59 2024 TCP_CLIENT link remote: [AF_INET]123.31.11.151:10001
vpn-1  | Mon Aug 12 08:30:59 2024 TLS: Initial packet from [AF_INET]123.31.11.151:10001, sid=b30c806f 6b1db9ce
vpn-1  | Mon Aug 12 08:30:59 2024 VERIFY OK: depth=1, CN=bke-vpn
vpn-1  | Mon Aug 12 08:30:59 2024 VERIFY KU OK
vpn-1  | Mon Aug 12 08:30:59 2024 Validating certificate extended key usage
vpn-1  | Mon Aug 12 08:30:59 2024 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
vpn-1  | Mon Aug 12 08:30:59 2024 VERIFY EKU OK
vpn-1  | Mon Aug 12 08:30:59 2024 VERIFY OK: depth=0, CN=bke-vpn
vpn-1  | Mon Aug 12 08:30:59 2024 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
vpn-1  | Mon Aug 12 08:30:59 2024 [bke-vpn] Peer Connection Initiated with [AF_INET]123.31.11.151:10001
vpn-1  | Mon Aug 12 08:31:00 2024 SENT CONTROL [bke-vpn]: 'PUSH_REQUEST' (status=1)
vpn-1  | Mon Aug 12 08:31:00 2024 PUSH: Received control message: 'PUSH_REPLY,route-gateway 10.99.0.1,ping 20,ping-restart 60,ifconfig 10.99.0.4 255.255.240.0,peer-id 0,cipher AES-256-GCM'
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: timers and/or timeouts modified
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: --ifconfig/up options modified
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: route-related options modified
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: peer-id set
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: adjusting link_mtu to 1659
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: data channel crypto options modified
vpn-1  | Mon Aug 12 08:31:00 2024 Data Channel: using negotiated cipher 'AES-256-GCM'
vpn-1  | Mon Aug 12 08:31:00 2024 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
vpn-1  | Mon Aug 12 08:31:00 2024 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
vpn-1  | Mon Aug 12 08:31:00 2024 ROUTE_GATEWAY 103.107.182.1/255.255.255.0 IFACE=eth0 HWADDR=fa:16:3e:c1:ee:2b
vpn-1  | Mon Aug 12 08:31:00 2024 TUN/TAP device kengine opened
vpn-1  | Mon Aug 12 08:31:00 2024 TUN/TAP TX queue length set to 100
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip link set dev kengine up mtu 1500
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip addr add dev kengine 10.99.0.4/20 broadcast 10.99.15.255
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip route add 123.31.11.151/32 via 103.107.182.1
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip route add 0.0.0.0/1 via 10.99.0.1
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip route add 128.0.0.0/1 via 10.99.0.1
vpn-1  | Mon Aug 12 08:31:00 2024 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
vpn-1  | Mon Aug 12 08:31:00 2024 Initialization Sequence Completed

This seem working but actually not, the ip addr add dev kengine 10.99.0.4/20 broadcast 10.99.15.255 is not working so there is no ip on the kengine

112: kengine: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/ether 52:a3:e8:10:4c:96 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ac5b:61ff:fe54:98f6/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

b. I have to manualy add myself and the VPN finally work

112: kengine: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
  link/ether 52:a3:e8:10:4c:96 brd ff:ff:ff:ff:ff:ff
  inet 10.99.0.4/20 brd 10.99.15.255 scope global kengine
     valid_lft forever preferred_lft forever
  inet6 fe80::ac5b:61ff:fe54:98f6/64 scope link proto kernel_ll 
     valid_lft forever preferred_lft forever

Testing

pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # ping 10.99.0.1
PING 10.99.0.1 (10.99.0.1) 56(84) bytes of data.
64 bytes from 10.99.0.1: icmp_seq=1 ttl=64 time=2.50 ms
64 bytes from 10.99.0.1: icmp_seq=2 ttl=64 time=1.42 ms
64 bytes from 10.99.0.1: icmp_seq=3 ttl=64 time=1.45 ms
^C
--- 10.99.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.416/1.786/2.496/0.502 ms
pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # ping 10.99.0.2
PING 10.99.0.2 (10.99.0.2) 56(84) bytes of data.
64 bytes from 10.99.0.2: icmp_seq=1 ttl=64 time=2.11 ms
64 bytes from 10.99.0.2: icmp_seq=2 ttl=64 time=2.75 ms
64 bytes from 10.99.0.2: icmp_seq=3 ttl=64 time=2.14 ms
^C
--- 10.99.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.109/2.334/2.752/0.295 ms

but since i want automaticly run VPN and I cant add it manual all the time so this problem need look up to
4. Error:

Not add IP properly on VPN interface

Expected behavior

The IP on interface automaticlly added

Additional information

None

[jepio]

Can you share the output of networkctl status from the host? Does the device show up as unmanaged?

Are you able to create a minimal reproducer for this?

[lmq1999]

Can you share the output of networkctl status from the host? Does the device show up as unmanaged?

Are you able to create a minimal reproducer for this?

pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # networkctl status
● Interfaces: 1, 2, 3, 7, 8, 12, 14, 16, 24, 20, 22, 18, 30, 28, 26, 33, 34, 4, 113
         State: routable                                    
  Online state: online                                      
       Address: 103.107.182.222 on eth0
                10.20.4.127 on eth1
                172.17.0.1 on docker0
                10.200.0.159 on cilium_host
                172.18.0.1 on br-098937d94946
                fe80::f816:3eff:fec1:ee2b on eth0
                fe80::f816:3eff:fefd:b77d on eth1
                fe80::42:96ff:fe5b:9fc4 on docker0
                fe80::40d7:33ff:fe7a:910b on cilium_net
                fe80::48a5:55ff:fe9d:3bb2 on cilium_host
                fe80::e037:abff:fe5c:6eb3 on lxc99eb9c4d3eb4
                fe80::6c06:47ff:fe44:2265 on lxcd81121c9fde0
                fe80::90dc:a0ff:fe41:bea0 on lxce0fba8614383
                fe80::b842:75ff:fee7:cec3 on lxc6c022ee53b5c
                fe80::40f3:58ff:fe97:219a on lxc21e65491adf6
                fe80::a448:14ff:fe58:3189 on lxc7df0526efc64
                fe80::1070:18ff:fe05:e8c0 on lxc59b2200808e0
                fe80::6c7e:31ff:fe1f:316d on lxcc625a36d34a6
                fe80::5c8e:d2ff:fe18:106 on lxc6316b6423586
                fe80::8463:c4ff:fe74:5fb1 on lxcb04e3a7afb05
                fe80::2cc2:e2ff:fe77:6d04 on lxc_health
                fe80::42:97ff:fe94:1333 on br-098937d94946
                fe80::d489:f4ff:fe2a:18e4 on kengine
       Gateway: 103.107.182.1 on eth0
           DNS: 8.8.8.8
                45.124.93.62
Search Domains: openstacklocal

Aug 12 08:31:00 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained carrier
Aug 12 08:31:00 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.
Aug 12 08:31:02 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained IPv6LL
Aug 12 09:46:24 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Link DOWN
Aug 12 09:46:24 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Lost carrier
Aug 12 09:46:24 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: DHCPv6 lease lost
Aug 12 09:46:30 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Link UP
Aug 12 09:46:30 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained carrier
Aug 12 09:46:30 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.
Aug 12 09:46:31 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained IPv6LL

here
I can give you the client full config if you want
I have tried other openvpn-client and build 1 myself but seem they have common error not adding ip on itnerface

[jepio]

sorry, i meant networkctl list

[jepio]

but this already confirms my suspicion that networkd is trying to manage your device:

systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.

[lmq1999]

but this already confirms my suspicion that networkd is trying to manage your device:

systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.

IDX LINK            TYPE     OPERATIONAL SETUP      
  1 lo              loopback carrier     unmanaged
  2 eth0            ether    routable    configured 
  3 eth1            ether    routable    configured 
  4 docker0         bridge   no-carrier  unmanaged
  7 cilium_net      ether    degraded    unmanaged
  8 cilium_host     ether    routable    unmanaged
 12 lxc99eb9c4d3eb4 ether    degraded    unmanaged
 14 lxcd81121c9fde0 ether    degraded    unmanaged
 16 lxce0fba8614383 ether    degraded    unmanaged
 18 lxc6c022ee53b5c ether    degraded    unmanaged
 20 lxc21e65491adf6 ether    degraded    unmanaged
 22 lxc7df0526efc64 ether    degraded    unmanaged
 24 lxc59b2200808e0 ether    degraded    unmanaged
 26 lxcc625a36d34a6 ether    degraded    unmanaged
 28 lxc6316b6423586 ether    degraded    unmanaged
 30 lxcb04e3a7afb05 ether    degraded    unmanaged
 33 lxc_health      ether    degraded    unmanaged
 34 br-098937d94946 bridge   no-carrier  unmanaged
113 kengine         ether    degraded    configuring

19 links listed.

look like is not working properly, so how to fix this problem

[jepio]

for a quick fix create /etc/systemd/network/kengine.network with these contents:

[Match]
Name=kengine

[Link]
Unmanaged=yes

and then systemctl restart systemd-networkd.

[lmq1999]

for a quick fix create /etc/systemd/network/kengine.network with these contents:

[Match]
Name=kengine

[Link]
Unmanaged=yes

and then systemctl restart systemd-networkd.

wow ok it worked for now, tks for your support

[jepio]

I'll leave this issue open as we should exclude tun/tap interfaces from networkd management by default.

[krishjainx]

@jepio Hey Jeremi! How are you doing? I'm thinking of adding a new configuration file with the following content:

[Match]
Kind=tun tap

[Link]
Unmanaged=yes

This configuration will match all tun and tap interfaces and set them as unmanaged. I can make a PR if this looks good to you :)

[jepio]

Sure, try it out. The PR needs to be against flatcar/init and flatcar/bootengine (for consistency).

[krishjainx]

Sure, try it out. The PR needs to be against flatcar/init and flatcar/bootengine (for consistency).

Cool, I started a discussion here: link