automated rules removed and not properly added back on docker service restart

[robby-d]

Running into an issue on an Ubuntu 22.04 x64 host (using the latest ufw-docker-automated, built from source on master) where the ufw container rules established is deleted and not properly recreated on a service docker restart / systemctl restart docker command.

E.g. - prior to docker service restart:

root@services:/# ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
[...] more static rules
8443/tcp (v6)              ALLOW       Anywhere (v6)

172.17.4.4 5044/udp        ALLOW FWD   10.0.0.0/16                # graylog-graylog-1:66528c03bad1 Beats
172.17.4.4 5140/udp        ALLOW FWD   10.0.0.0/16                # graylog-graylog-1:66528c03bad1 Syslog
172.17.4.4 5555/udp        ALLOW FWD   10.0.0.0/16                # graylog-graylog-1:66528c03bad1 Custom
172.17.4.4 12201/udp       ALLOW FWD   10.0.0.0/16                # graylog-graylog-1:66528c03bad1 GELF
172.17.4.4 5044/tcp        ALLOW FWD   10.0.0.0/16                # graylog-graylog-1:66528c03bad1 Beats
172.17.4.4 5140/tcp        ALLOW FWD   10.0.0.0/16                # graylog-graylog-1:66528c03bad1 Syslog
172.17.4.4 5555/tcp        ALLOW FWD   10.0.0.0/16                # graylog-graylog-1:66528c03bad1 Custom
172.17.4.4 12201/tcp       ALLOW FWD   10.0.0.0/16                # graylog-graylog-1:66528c03bad1 GELF
172.17.5.2 25/tcp          ALLOW FWD   10.0.0.0/16                # postfix-postfix-1:d5bf6f38bad2 SMTP
172.17.5.2 587/tcp         ALLOW FWD   10.0.0.0/16                # postfix-postfix-1:d5bf6f38bad2 SMTPS
172.17.5.2 25/tcp          ALLOW FWD   192.168.0.0/24             # postfix-postfix-1:d5bf6f38bad2 SMTP
172.17.5.2 587/tcp         ALLOW FWD   192.168.0.0/24             # postfix-postfix-1:d5bf6f38bad2 SMTPS
172.17.7.2 8443/tcp        ALLOW FWD   10.0.0.0/16                # step-ca-step-ca-1:10a514908c50
172.17.7.2 8443/tcp        ALLOW FWD   192.168.0.0/24             # step-ca-step-ca-1:10a514908c50

After sudo service docker restart:

root@services:/# ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
[...] more static rules
8443/tcp (v6)              ALLOW       Anywhere (v6)

Running service ufw-docker-automated restart will successfully re-create the custom rules.

It seems that ufw-docker-automated has code to detect a disconnect from tcp. Here's what I see in /var/log/syslog after the docker service restarts:

Jan 11 20:24:02 services ufw-docker-automated[995]: 2023/01/11 20:24:02 ufw-docker-automated: Event error: unexpected EOF
Jan 11 20:24:07 services ufw-docker-automated[995]: 2023/01/11 20:24:07 ufw-docker-automated: Trying to reconnect..
Jan 11 20:24:07 services ufw-docker-automated[995]: 2023/01/11 20:24:07 ufw-docker-automated: Reconnected to the Docker Engine.

However, inbound rules are not correctly readded post-service restart. Once I manually restart the ufw-docker-automated service /var/log/syslog shows:

Jan 11 20:24:11 services systemd[1]: Stopping Ufw docker automated...
Jan 11 20:24:11 services systemd[1]: ufw-docker-automated.service: Deactivated successfully.
Jan 11 20:24:11 services systemd[1]: Stopped Ufw docker automated.
Jan 11 20:24:11 services systemd[1]: ufw-docker-automated.service: Consumed 55.967s CPU time.
Jan 11 20:24:11 services systemd[1]: Started Ufw docker automated.
Jan 11 20:24:11 services ufw-docker-automated[675485]: 2023/01/11 20:24:11 ufw-docker-automated: Connected to the Docker Engine.
Jan 11 20:24:11 services ufw-docker-automated[675485]: 2023/01/11 20:24:11 ufw-docker-automated: Adding inbound rule: /usr/bin/sudo ufw route allow proto tcp from 10.0.0.0/16 to 172.17.7.2 port 8443 comment step-ca-step-ca-1:10a514908c50
Jan 11 20:24:11 services ufw-docker-automated[675485]: 2023/01/11 20:24:11 ufw-docker-automated: Adding inbound rule: /usr/bin/sudo ufw route allow proto tcp from 192.168.0.0/24 to 172.17.7.2 port 8443 comment step-ca-step-ca-1:10a514908c50
[...] and so on [...]

Not familiar with Go code, but it seems like the service tries to reconnect to the Docker UNIX domain socket (e.g. /var/run/docker.sock) but perhaps binds to an old or inactive version of it?

Anyhow, I "fixed" this issue by adding PartOf=docker.service to my systemd ufw-docker-automated.service file, as so:

[Unit]
Description=Ufw docker automated
Documentation=https://github.com/shinebayar-g/ufw-docker-automated
After=network-online.target ufw.service containerd.service
Wants=network-online.target
Requires=ufw.service
PartOf=docker.service

[Service]
# To manage ufw rules, binary has to run as a root or sudo privileged user.
User=root
# Provide /path/to/ufw-docker-automated
ExecStart=/usr/local/bin/ufw-docker-automated
Restart=always

[Install]
WantedBy=multi-user.target

A better fix might be more robust reconnection detection in the code, perhaps.

[shinebayar-g]

Hi, thanks for the report. I see what's going on here. https://github.com/shinebayar-g/ufw-docker-automated/blob/main/main.go#L64 this code is syncing existing containers, but only executed on the startup. I should call the function on reconnect as well.

PartOf is a nice workaround. Probably worth mentioning on the docs as well.

[robby-d]

Ah, I see that now, great. Seems like an easy fix.

[shinebayar-g]

Yep, will upgrade dependencies and cleanup some things and cut a new release this weekend.

[robby-d]

Cool deal. One minor nit: would be good to have instructions for building from source in the README. Wasn't hard, but not being familiar with building Go code, it took a bit of Googling.

[shinebayar-g]

That's fair. Do you prefer to build yourself instead of downloading from releases page?

[robby-d]

Yeah, given the level of integration into the system (running as root with interaction with docker containers) I prefer to compile directly from the source code on Github, which I give a quick review to. One less thing to worry about. :)

[florie1706]

Sorry if I'm joining in here. I also have problems with Ubuntu 22.04 since short. Unfortunately, the workaround does nothing for me. No more rules are added.

./ufw-docker-automated

only shows

2023/01/20 20:32:08 ufw-docker-automated: Connected to the Docker Engine.

and

sudo journalctl -u ufw-docker-automated -f

shows only this, even on a container restart

Jan 20 05:52:45 nas sudo[635511]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Jan 20 05:52:46 nas sudo[635511]: pam_unix(sudo:session): session closed for user root
Jan 20 20:27:40 nas systemd[1]: Stopping Ufw docker automated...
Jan 20 20:27:40 nas systemd[1]: ufw-docker-automated.service: Deactivated successfully.
Jan 20 20:27:40 nas systemd[1]: Stopped Ufw docker automated.
Jan 20 20:27:40 nas systemd[1]: Started Ufw docker automated.
Jan 20 20:28:45 nas ufw-docker-automated[2957078]: 2023/01/20 20:28:45 ufw-docker-automated: Connected to the Docker Engine.
Jan 20 20:28:45 nas sudo[2966573]:     root : PWD=/ ; USER=root ; COMMAND=/usr/sbin/ufw show added
Jan 20 20:28:45 nas sudo[2966573]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Jan 20 20:28:45 nas sudo[2966573]: pam_unix(sudo:session): session closed for user root

[shinebayar-g]

Hey guys, sorry that I didn't fix this sooner. I just tagged a new release v0.11.0 Tested the behavior with sudo service docker restart, this time ufw-docker-automated will properly sync with docker containers again on reconnect. Also implemented a new logging library for easier debugging.

[robby-d]

No problem, thank you for the update! I’ll give it a try shortly and let you know if any problems.

Unrelated, I was sometimes seeing an issue with duplicate rules — i.e. rules would sometimes not be deleted when the container was stopped and restarted and you’d end up with two (or more) of the same rule after awhile. Might have just been an issue with named rules (where some title was set after the rule port number)…I’ll see if I can reproduce it and post an issue if so.

[shinebayar-g]

Sounds good. In general ufw itself doesn't accept duplicate rules, so I'm currently relying on that behavior. If I receive duplicate rule event from Docker socket, I'm simply forwarding them to ufw itself. If you're seeing duplicates, perhaps they're different containers? Other than that it shouldn't be possible to see duplicates.