Drop privileges after start

[killcity]

Will Fabio eventually get support to use setuid (to change to a non-root user once it's up and running)? I'd rather not have to run another proxy in front of it to alleviate security concerns if at all possible. Is the current model putting that burden on something other than Fabio?

Cheers

[magiconair]

Good point. I have to think about that.

[magiconair]

If I look at golang/go#1435 then there might be a way as long as I don't need to chroot. Open socket, re-exec and pass FD to child process might work. This will become interesting for dynamic listeners :)

[magiconair]

@killcity consider this a first attempt of tackling this. The code is based on the idea from @bgilmore in https://play.golang.org/p/dXBizm4xl3. The only thing I've added is the forwarding of stdout and stderr to make logging work.

I find the code still quite ugly and the whole listener setup could use a makeover but it should work. Would be great if you could test it.

[magiconair]

I've tested this only on OSX so far with sudo ./fabio -user nobody which spawns the root process which then sets up the listeners and passes them on to the child process. The child process will then use the FDs to create the actual proxies. The root parent has a root: prefix in the logs and the child logs the user it runs as. As said in the comment the command line arg will most likely change.

[killcity]

You are too quick :) I will give this a go in the morning (here in the US northeast). Thank so much for working this in. Super awesome!

[magiconair]

Some tests are failing. I'll fix that tomorrow.

[killcity]

Wasn't sure if this was one of the failing tests? TBH, I haven't compiled much in Go.

--> linux/amd64 error: exit status 2
Stderr: # _/root/gopath/fabio-drop-privs
./main.go:50: cfg.Username undefined (type *config.Config has no field or method Username)
./main.go:100: cfg.Username undefined (type *config.Config has no field or method Username)
./main.go:102: cfg.Username undefined (type *config.Config has no field or method Username)
./main.go:145: cfg.Username undefined (type *config.Config has no field or method Username)
./main.go:147: cfg.Username undefined (type *config.Config has no field or method Username)

...

[magiconair]

Yes. I've rebased the code and fixed the failing test. One effect is that the cmd line parameter is now -proxy.user instead of just -user since that picked up the $USER environment variable. Code is still ugly and needs work. Name of cmd like arg will most likely still change. Handle with care :)

[killcity]

Thanks. Going to beat her up a bit! :)

fabio 5850 0.8 0.0 583664 20184 ? Ssl 11:44 0:00 ./fabio-1.3.5-branch_195 -cfg /etc/fabio/fabio.properties

[killcity]

Update: No issues. Been running with this since Dec 1.

[magiconair]

@killcity awesome. I'll spend some time on incorporating this into the master branch then.

[mterron]

An alternative approach:
You can always use linux capabilities to allow the user that runs fabio to bind to low ports (I assume that's what you are after).
Just run
root@fabio$ setcap 'cap_net_bind_service=+ep' $(which fabio)
After that you can run fabio under an unprivileged account and it will work as expected.
That's the way I've been running it for months.

Cheers

[killcity]

This only works on Linux, correct? I would think it would still be nice to have the functionality regardless of operating system, although it does add some slight overhead.

[mterron]

It works on linux. Solaris/Illumos have privileges that are similar and can be applied to the user running fabio

$ /usr/sbin/usermod -K defaultpriv=basic,net_privaddr fabio_user

and check with:

$ grep fabio_user /etc/user_attr
fabio_user::::type=normal;defaultpriv=basic,net_privaddr

or you can apply a privilege set to the process you are going to run (can't remember the exact syntax for this usage):

$ /usr/sbin/ppriv -s LI+NET_PRIVADDR -e fabio

FreeBSD have a sysctl to define the reserved port range :

$ sysctl net.inet.ip.portrange.reservedhigh=79

You can add that to /etc/sysctl.conf so it's applied on boot.

I don't know what the equivalent is for the other BSDs and OS X. I don't think there's anything similar on Windows though, but since you mention setuid I assume you're talking about some sort of unix system.

[killcity]

Cool. I figured most modern unix variants have something that can accommodate. Thoughts on whether this should or shouldn't be built into the app?

[mterron]

With my security hat on, I'd say that fabio should refuse to start as root. Maybe add an explicit option to allow it, so the user have to make a conscious decision to run as root.
Something like this user experience:

$ fabio -registry.consul.addr 192.168.0.100:8500
2017/01/30 11:41:27 [ERR] root user detected. Aborting.
WARNING! You are trying to run fabio as root, this is insecure and not recommended. 
If you understand the security implications of running as root, run fabio with the 
-insecure flag.

$ fabio -insecure -registry.consul.addr 192.168.0.100:8500
2017/01/30 11:41:27 [WARN] Running as root
2017/01/30 11:41:27 [INFO] Runtime config
{
     ...
}
2017/01/30 11:41:27 [INFO] Version 1.3.7 starting
2017/01/30 11:41:27 [INFO] Go runtime is go1.7.3
2017/01/30 11:41:27 [INFO] Using routing strategy "rnd"
2017/01/30 11:41:27 [INFO] Using routing matching "prefix"
2017/01/30 11:41:27 [INFO] Setting GOGC=800
2017/01/30 11:41:27 [INFO] Setting GOMAXPROCS=4
2017/01/30 11:41:27 [INFO] Metrics disabled

[magiconair]

What about windows and macOS? I'm not a fan of building per-os code in. OTOH, the proposed patch might also not work on other platforms.

This could be optional though. You can choose which approach you want to take with forking as the default. This would also leave the door open if Go makes this easier at a later stage.

I like the idea of the root switch and making it the default. However, that will break existing setups and since fabio sits in the hot-path this might result in some angry users who have to scramble to get their sites back up. Only way to introduce this is very carefully and with enough lead time. Maybe issue the warning first and announce when it will become the default. Then make the switch.

[mterron]

@magiconair macOS have no root and I don't think Windows even has a concept of root.

Allowing a user to bind to a low port (I'm guessing all this discussion is related to port 80 and 443, please correct me if I'm wrong) is an administrator's activity and different OSs will have different mechanisms to achieve that, as I've shown in my messages. A system administrator that is deploying internet facing services should be familiar with those mechanisms.

I don't think that logic belongs with the fabio code at all, so no os specific tricks for you to code and maintain. What could be added to fabio is a warning message in case it couldn't bind to a port (maybe it already exists, I haven't run into it).

I agree with giving ample warning about the implementation of the insecure switch as you don't want to break deployments though I'd argue that most people running fabio as root have not weighed in the pros and cons and run it like that "cause it works".
It is better to break a deployment with a clear error message and instructions on what to do next than having your system taken over due to some bug in fabio or golang.
You could even go as far as, depending on the OS, explaining how to fix the issue as I've shown in the comments above.

[magiconair]

Just keeping the patch in sync with master but I'm leaning towards @mterron. This sounds like the better approach.

[stephane-martin]

Well, in OSX you define a service with launchd, and launchd takes care of binding to the low port.

On Linux you can use capabilities to allow the low port binding for a non privileged user (and you can even do that in the systemd service definition nowadays).

On OpenBSD you just don't bind to a low port (instead use PF to port forward from a low port to a "high" port).

So on *NIX at least i would not expect a web proxy to take care of security stuff that are quite clearly OS responsibility...

[magiconair]

I agree with @mterron and @stephane-martin that this is not the responsibility of fabio. Lets do the following instead:

• Compile a set of best-practices per operating system as outlined by @mterron and @stephane-martin and add them to the documentation
• Add a stern warning message to the logs when fabio is run as root (if detectable) and point to the documentation for best-practices on how to solve it. Also indicate that the default behavior will change in the future, i.e. the -insecure switch that @mterron suggests.

@mterron and @stephane-martin would you be willing to provide some documentation snippets for the various OSes you seem to have experience with? Not sure if what is already provided in this ticket is sufficient for the documentation.

[stephane-martin]

For instance on Linux with systemd.

/etc/systemd/system/fabio.service:

[Unit]
Description=Fabio proxy
After=syslog.target
After=network.target

[Service]
LimitMEMLOCK=infinity
LimitNOFILE=65535
Type=simple
# unprivileged uid and gid
User=fabio
Group=fabio
WorkingDirectory=/
ExecStart=/path/to/fabio -cfg /path/to/fabio.conf
Restart=always
# no need that fabio messes with /dev
PrivateDevices=yes
# dedicated /tmp
PrivateTmp=yes
# make /usr, /boot, /etc read only
ProtectSystem=full
# /home is not accessible at all
ProtectHome=yes
# to be able to bind port < 1024
AmbientCapabilities=CAP_NET_BIND_SERVICE
# only ipv4, ipv6, unix socket and netlink networking is possible
# netlink is necessary so that fabio can list available IPs on startup
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK

[stephane-martin]

On *BSD with Packet Filter, if fabio is directly facing internet, the best practice is only listen on a non-privileged port (4343), and use PF to forward trafic:

/etc/pf.conf:

EXT_IF = "eth0"
HTTPS_PORT = 443
HTTPS_PORT_BACKEND = 4343
LOCAL_IP = "127.0.0.1"

...

pass in quick on $EXT_IF inet proto tcp from any to $LOCAL_IP port $HTTPS_PORT rdr-to $LOCAL_IP port $HTTPS_PORT_BACKEND

Running fabio in a jail (FreeBSD) or chroot (OpenBSD) doesn't harm too. You just need to copy the relevant C libraries to the jail (eg. libc.so, ld.so and libpthread.so).

[mterron]

I think that between my original messages and @stephane-martin ones the basics are covered.
You have:

• Generic linux (my setcap comment)
• Linux with systemd
• Illumos/Solaris (2 options, privileges for the user or the process)
• FreeBSD (modify "low ports" limit)
• macOS (using launchd)
• OpenBSD (using PF redirection)

Is there other OS that's missing?
We could add references to the man pages for capabilities and privileges for people that want to understand better what they are doing.

[siepkes]

Even if Fabio could do it I personally would never rely / trust any daemon to drop it's privileges after it's done with them. I remember from the "old" days (say 10 years ago) that a good number of root exploits in Linux used processes which relied on privilege dropping (Apache HTTP server for example). Because even though they might poses root rights for only a couple of milliseconds; At some point in time they will have them; It's an opening. If there is a vulnerability in Fabio the exploiter will only have to wait until the daemon is restarted at some point to obtain root rights.

[kostyrev]

@siepkes so how do you manage Apache or Nginx services now?

[siepkes]

@kostyrev On SmartOS deployments I use SMF to grant the NGINX, Apache, etc. process the net_privaddr privilege. This allows the process to bind to < 1024 ports without root privileges. The process it self is always started as a unprivileged user by SMF. On Linux with systemd you can use the Capabilities= option to achieve the same effect.

So the process never has root rights. It only has the rights to bind to the ports below 1024.

[kostyrev]

You grant read perms on private certificates to apache/nginx user?

[siepkes]

@kostyrev Yes only the NGINX user has read rights on the private key. The certificate is world readable since it is obviously public anyway. Since the worker processes have the private key loaded in their memory when they are spawned there is not much point in shielding them from the actual private key file.

[magiconair]

I've compiled the best practices here in a wiki page: https://github.com/fabiolb/fabio/wiki/Binding-to-Low-Ports

Is anybody up for reviewing this? If yes, then please comment here.

I'm going to create another ticket for not allowing to run fabio as root.

[magiconair]

Thanks everybody for the suggestions!

[ianic]

It seams that after upgrade to the MacOS Mojave it is no more required to be root for binding privileged ports. Reference.

I'm running:

fabio -proxy.addr :80

as non-root user, and it works.

...
2019/04/15 16:52:49 [INFO] HTTP proxy listening on :80