Best Practices for Docker and UFW

[kaysond]

There is a well known security concern when running Docker on a Ubuntu host that uses UFW as its main firewall: Docker's manipulation of iptables bypasses the rules created by UFW, enabling access by default to containers with ports mapped despite UFW being enabled. Though there are existing methods of securing the network (e.g. binding to 127.0.0.1), the extra security of UFW can be desirable.

See:

• docker and ufw serious problems moby/moby#4737
• https://stackoverflow.com/questions/30383845/what-is-the-best-practice-of-docker-ufw-under-ubuntu
• https://www.mkubaczyk.com/2017/09/05/force-docker-not-bypass-ufw-rules-ubuntu-16-04/
• https://www.techrepublic.com/article/how-to-fix-the-docker-and-ufw-security-flaw/
• https://www.linux.com/news/how-fix-docker-and-ufw-security-flaw/
• https://svenv.nl/unixandlinux/dockerufw/
• https://blog.viktorpetersson.com/2014/11/03/the-dangers-of-ufw-docker.html

Most of these recommend disabling iptables manipulation with --iptables=false and manually configuring the rules as necessary.

More recently, two other workarounds have surfaced which do not use this flag and seem to be more robust:

• https://github.com/chaifeng/ufw-docker
• https://gist.github.com/rubot/418ecbcef49425339528233b24654a7d

These are at least a year old now, and despite the many results it's still not common knowledge as searches continue: https://trends.google.com/trends/explore?q=docker%20ufw&geo=US

It seems that the Docker team isn't interested in addressing this on their end, so the purpose of this issue is to request community feedback, determine best practices, and create a PR to hopefully add something to the documentation.

[binaryfire]

Hi @kaysond, there's an existing thread about this here: #690. If you could add this info there it would be great

[AkihiroSuda]

Rootless mode seems also useful as a workaround for this

[binaryfire]

Hey @AkihiroSuda, thanks a lot for your suggestion. The only posts I've found referencing rootless mode are using nightly builds so it seems this is an experimental feature at the moment and not ready for production? Are there any official docs and stable builds available?

Also would you be able to expand on the pros and cons of rootless vs regular modes or direct me to some articles which explain this clearly?

Thanks for your time. Appreciate any insights you might have to offer :)

[AkihiroSuda]

The only posts I've found referencing rootless mode are using nightly builds

Not only for nightly builds
https://download.docker.com/linux/static/stable/x86_64/docker-rootless-extras-19.03.2.tgz

it seems this is an experimental feature at the moment and not ready for production

Yes, it is likely to remain experimental until we can complete migration to cgroup2 (around the end of the year)

Also would you be able to expand on the pros and cons of rootless vs regular modes or direct me to some articles which explain this clearly?

https://www.slideshare.net/AkihiroSuda/dockercon-2019-hardening-docker-daemon-with-rootless-mode

[shinebayar-g]

Hi guys, I just made a little script that solves the ufw firewall and docker iptables issue. You can take a look at repo here. It's heavily based on chaifeng/ufw-docker's ufw rules, but automates the manual process. All you have to do is run the container with special label UFW_MANAGED=TRUE. Please let me know your feedbacks. Of course code is crap, but hopefully it works.

[pySilver]

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

[kaysond]

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

I actually like this a lot more than https://github.com/chaifeng/ufw-docker
Seems like it properly interfaces ufw to docker iptables rules without unncessary assumptions.

[nathan-march]

The only thing that helped me: https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

Something to be aware of this solution (and every other one I've found so far) is that they're filtering the dest-port after the nat translation has been applied. This means if you have two containers listening on 443 internally but exposed on different external ports, they'll both be open.

In my case I have a pub container on 443->443 and an admin container on 443->8443, so none of these workarounds manage to close off that 8443 port.

[shinebayar-g]

@nathan-sav I'm using ufw-docker. You can give container a static ip address and write specific rule for that IP. In that way your problem can be solved. For example, I have multiple mysql containers running on default 3306, exposed on 3306, 3307, 3308 externally, and they all have different ufw rules. Unfortunately containers' ip address must be static, otherwise IP specific rules doesn't make sense. To solve that problem, I started ufw-docker-automated.

[nathan-march]

@nathan-sav I'm using ufw-docker. You can give container a static ip address and write specific rule for that IP. In that way your problem can be solved. For example, I have multiple mysql containers running on default 3306, exposed on 3306, 3307, 3308 externally, and they all have different ufw rules. Unfortunately containers' ip address must be static, otherwise IP specific rules doesn't make sense. To solve that problem, I started ufw-docker-automated.

Yeah unfortunately that doesn't work for me, I don't want to manage container ips like that and I don't like the idea of needing a script to watch for IP changes and update rules appropriately.

I think I have a better solution in the works, I'll follow up if/once I'm successful :)

[nathan-march]

Ok, here's what I came up with and this works much better for me:

• You can restart ufw/docker at will and they don't clobber each other
• Allowed ports only line up to the external ports in docker, allowing 443 in your firewall won't allow access to any port that's mapped to 443 inside a container
• You do not get to use the ufw tool to manage the allowed ports (I use ansible to build my after.rules with all my rules in it, don't care about the CLI)

It works by tagging the allowed packets in the NAT table and then allowing them later on in the filter table, rather than doing the matching at filter time (when you're dealing with already natted packets).

It all fits into /etc/ufw/after.rules:

*nat
:PREROUTING ACCEPT [0:0]

# Flush rules before we start so things don't get duplicated on ufw reload
-F PREROUTING

###### Here you add the allowed ports for your docker containers ######
-A PREROUTING -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x5/0xffffffff
-A PREROUTING -p tcp -m tcp --dport 443 -j MARK --set-xmark 0x5/0xffffffff

# Debug
#-A PREROUTING -j LOG --log-prefix "NAT-PRE:" --log-level 6

# Replace the docker rule that was here before
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER

# Commit nat table
COMMIT

*filter
:ufw-after-input - [0:0]
:ufw-after-output - [0:0]
:ufw-after-forward - [0:0]
# Need to create the DOCKER-USER chain now, in case docker isn't running yet so the below commands don't fail
:DOCKER-USER - [0:0]

# don't log noisy services by default (UFW DEFAULT)
-A ufw-after-input -p udp --dport 137 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp --dport 138 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp --dport 139 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp --dport 445 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp --dport 67 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp --dport 68 -j ufw-skip-to-policy-input

# don't log noisy broadcast (UFW DEFAULT)
-A ufw-after-input -m addrtype --dst-type BROADCAST -j ufw-skip-to-policy-input

###### Here you add the allowed ports for regular traffic (not docker) ######
-A ufw-after-input -s 216.XXX.XXX.XXX -j ACCEPT
-A ufw-after-input -p TCP --dport 22 -s 10.0.0.0/8 -j ACCEPT

# Flush rules before we start so things don't get duplicated on ufw reload
-F DOCKER-USER

# Just allow outgoing
-A DOCKER-USER -s 172.18.0.0/16 -j ACCEPT

# Debug
#-A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j LOG --log-prefix "ALLOW CONNTRACK:" --log-level 6

# Allow packets after connection gets brought up, since they don't hit the docker nat rules and won't get tagged / allowed
-A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Debug
#-A DOCKER-USER -m mark --mark 0x5 -j LOG --log-prefix "ALLOW DOCKER-USER:" --log-level 6
#-A DOCKER-USER -m mark ! --mark 0x5 -j LOG --log-prefix "DROP DOCKER-USER:" --log-level 6

# Drop anything that didn't get marked or previously accepted
-A DOCKER-USER -m mark ! --mark 0x5 -j DROP

# Replace the default docker rule
-A DOCKER-USER -j RETURN

# Commit filter table
COMMIT

There might be bugs but so far it seems to work reliably, I've restarted docker/ufw a bunch of times with various combinations of them being enabled at boot. Feedback welcome!

[kaysond]

Something to be aware of this solution (and every other one I've found so far) is that they're filtering the dest-port after the nat translation has been applied. This means if you have two containers listening on 443 internally but exposed on different external ports, they'll both be open.

This is more a fundamental part of iptables than anything to do with docker. It's just because the filter tables come after nat.

There might be bugs but so far it seems to work reliably, I've restarted docker/ufw a bunch of times with various combinations of them being enabled at boot. Feedback welcome!

Marking is definitely a good workaround (annoying that you can't just drop in nat but such is life). Be careful about -A DOCKER-USER -s 172.18.0.0/16 -j ACCEPT though, because docker can use a variety of subnets, including 192.168.0.0/16

[nathan-march]

This is more a fundamental part of iptables than anything to do with docker. It's just because the filter tables come after nat.

Agreed, I just mentioned it so other people wouldn't be caught off guard after reading this thead.

Marking is definitely a good workaround (annoying that you can't just drop in nat but such is life). Be careful about -A DOCKER-USER -s 172.18.0.0/16 -j ACCEPT though, because docker can use a variety of subnets, including 192.168.0.0/16

Excellent point, I'll see if I can come up with a cleaner way to handle that :)

[kaysond]

Excellent point, I'll see if I can come up with a cleaner way to handle that :)

Might just be able to do --src-type LOCAL. Not sure if it handles the veth tunnels though

[pySilver]

In my case I have a pub container on 443->443 and an admin container on 443->8443, so none of these workarounds manage to close off that 8443 port.

I've heard about this problem. So in other words when you do
docker run -p 443:443 some_image & docker run -p 443:8443 maybe_different_image both 443 and 8443 of host machine become exposed despite the fact that UFW allows only 443? Is that related to fact that docker image exposes 443 originally or because first image is mapped to 443 ?

[nathan-march]

I've heard about this problem. So in other words when you do
docker run -p 443:443 some_image & docker run -p 443:8443 maybe_different_image both 443 and 8443 of host machine become exposed despite the fact that UFW allows only 443? Is that related to fact that docker image exposes 443 originally or because first image is mapped to 443 ?

Correct.

Iptables filtering rules are applied AFTER the nat rules are applied, so iptables first of all maps 8443 -> 443, then the destination port 443 allow rule lets it through. This is why ufw-docker suggests setting manual ips to your containers and putting those ips into the rules.

[kaysond]

@nathan-sav It looks like the conntrack extension might be able to do this in a cleaner way: http://ipset.netfilter.org/iptables-extensions.man.html#lbAO

Check out --ctstate DNAT and --ctorigdstport <port>

[algo7]

There's another work around which uses reverse proxy from Nginx or Apache (installed locally) to map ports to containers bind to localhost. However, it adds extra complexities and reduces the flexibilities of using Docker.

[Pimmelton]

There's another work around which uses reverse proxy from Nginx or Apache (installed locally) to map ports to containers bind to localhost. However, it adds extra complexities and reduces the flexibilities of using Docker.

That's essentially what I ended up doing after quickly tiring of fiddling around with UFW after rules and such, with one simple difference.

I set up a Nginx reverse proxy directly on the host system with a minimal configuration that does nothing but forward all queries (including request headers) directly to a Traefik container bound to a localhost port. Traefik then works as expected, dynamically forwarding to containers as I bring them up and down as if it were being accessed directly. The difference is that my firewall behaves as expected with straightforward ufw commands as in days of yore.

It's working well so far, though I still have to do some fine tuning, like whether to make Nginx the TLS termination proxy or pass that on through to Traefik. I'm inclined to do the former because it suits my particular needs and allows me the flexibility to serve stuff via Nginx directly from the host on another port/virtualhost or whatever.

[gnat]

As rootless becomes more common, this may solve itself to an extent, but still, gaping security hole for newcomers- this really needs better, sane defaults!

[algo7]

As rootless becomes more common, this may solve itself to an extent, but still, gaping security hole for newcomers- this really needs better, sane defaults!

Exactly. It should have mentioned it somewhere on the documentation. I guess ufw is just one of those firewall solutions out there (though quite popular), and Docker thinks that it is not worth mentioning.

[gnat]

Reminder to people following: ufw is just an iptables wrapper. The issue is Docker installs its own chain at the very start of iptables, which effectively ignores all of your firewall rules.

Putting Dockers chain at the end of iptables, or at least, after ufw's chain could solve this (since ufw has no ability to modify chains other than its own).

[arkroan]

Docker already provides integration with Firewalld, respecting the way Firewalld works. I do not think it would be such a stretch to also support UFW and how UFW works, respecting the hierarchy in iptables.

[Na-Cly]

This really hasn't been addressed?

[algo7]

This really hasn't been addressed?

No. At least I see nothing related to this in the last release logs.

[zhen-huan-hu]

Marking is definitely a good workaround (annoying that you can't just drop in nat but such is life). Be careful about -A DOCKER-USER -s 172.18.0.0/16 -j ACCEPT though, because docker can use a variety of subnets, including 192.168.0.0/16

Excellent point, I'll see if I can come up with a cleaner way to handle that :)

I am new to all this but would -A DOCKER-USER -i docker0 -j ACCEPT and -A DOCKER-USER -i br-+ -j ACCEPT be a better workaround?

[algo7]

In 2023 there is still no update AFAIK.

[MathewJohn1414]

What about this solution? #690 (comment)

This worked for me, simple and easy to implement.