BalancerBattle

BalancerBattle was a load test against load balancers / proxies that support WebSockets. These pieces of technology are vital if you want to scale to multiple servers or processes.

The following technologies were tested:

• http-proxy, version: 0.10.0
• nginx, version: 1.3.15 (development release)
• HAProxy, version: 1.5-dev18 (development release)
• Nothing, just the plain echo server that was used as a control test.

There have been some questions about including hipache. The reason that I have not included them in my tests is because it's build upon the http-proxy. They are currently using a fork of the project but it doesn't contain any performance related patches.

3 different, separate servers were used for testing. All these servers are hosted at joyent.

1. Proxy, a 512mb Ubuntu server. This is the server were all the proxy servers are installed. image: sdc:jpc:ubuntu-12.04:2.4.0
2. WebSocketServer, a 512mb Node.js smart machine that ran our WebSocket echo server. The server is written in Node.js and spread across multiple cores using the cluster module. image: sdc:sdc:nodejs:1.4.0
3. Thor, another 512mb Node.js smart machine with the same specs as above. This was the server were we generated the load from. Thor is a WebSocket load generation tool which we've developed. It's released as open source and available at http://github.com/observing/thor

Configuring the Proxy server

The Proxy server was just a clean, bare bones Ubuntu 12.04 server. These are the steps that were taken to configure and install all the dependencies. To ensure that everything is up to date we have to run.

apt-get upgrade

The following dependencies were installed on the system:

• git for access to the github repositories.
• build-essential for compiling the proxies for source, most of the proxies recently got support for WebSockets or HTTPS.
• libssl-dev Needed for HTTPS support.
• libev-dev Libev required for stud, stud is awesomesss.

apt-get install git build-essential libssl-dev libev-dev

Node.js

Node.js is required for the http-proxy. While it runs on the latest Node.js version for these tests were executed under 0.8.19 to ensure compatibility of all dependencies. It was installed through github.

git clone git://github.com/joyent/node.git
cd node
git checkout v0.8.19
./configure
make
make install

This also installed the npm binary on the system so we can install the dependencies of this project. Run npm install . in the root of this repository and the http-proxy and all it's dependencies are installed automatically.

Nginx

Nginx is already a widely deployed server. It supports proxing of to different back end servers but it did not support WebSockets. This got recently added in to the development branch of Nginx. Therefore we installed the latest development version and compiled from source:

Please note that since writing and testing this, nginx 1.4.0 was shipped and has support for WebSockets. So if you are reading this and want to deploy in production I would advice you to use 1.4.0 instead of the development builds

wget http://nginx.org/download/nginx-1.3.15.tar.gz
tar xzvf nginx-1.3.15.tar.gz
cd nginx-1.3.15
./configure --with-http_spdy_module --with-http_ssl_module \
--pid-path=/var/run/nginx.pid --conf-path=/etc/nginx/nginx.conf \
--sbin-path=/usr/local/sbin --http-log-path=/var/log/nginx/access.log \
--error-log-path=/var/log/nginx/error.log --without-http_rewrite_module

As you can from the options above we've included SSL, SPDY and configured some other settings. This yielded the following configuration summary:

Configuration summary
  + PCRE library is not used
  + using system OpenSSL library
  + md5: using OpenSSL library
  + sha1: using OpenSSL library
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/sbin"
  nginx configuration prefix: "/etc/nginx"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/var/run/nginx.pid"
  nginx error log file: "/var/log/nginx/error.log"
  nginx http access log file: "/var/log/nginx/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

After this it's just a simple make away:

make
make install

HAProxy

HAProxy was already able to proxy WebSockets in tcp mode but it's now also possible to do so in http mode. HAProxy also got support for HTTPS termination. So again, we need to install the development branch.

wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev18.tar.gz
tar xzvf haproxy-1.5-dev18.tar.gz
cd haproxy-1.5-dev18
make TARGET=linux26 USE_OPENSSL=1
make install

Stud

While HAProxy is capable of terminating SSL it's common practise to have stud in front of HAProxy for SSL offloading. So this is something we want to test as well.

git clone git://github.com/bumptech/stud.git
cd stud
make
make install

Now that everything is installed we need to install the configuration files. For Nginx you can copy & paste the nginx.conf from the root of this repository to /etct/nginx/nginx.conf. All the other proxies can be configured on the fly.

Kernel tuning

After all the proxies are installed we need to do some socket tuning. This information was generously stolen from the internet:

vim /etc/sysctl.conf

And set the following values.

# General gigabit tuning:
net.core.somaxconn = 16384
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_syncookies = 1
# this gives the kernel more memory for tcp
# which you need with many (100k+) open socket connections
net.ipv4.tcp_mem = 50576   64768   98152
net.core.netdev_max_backlog = 2500

Benchmarking

There are 2 different tests executed:

1. Load testing the proxies without SSL. This will purely test the performance of WebSocket proxing.
2. Load testing the proxies with SSL. Nobody should be running unsecured WebSockets as they have really bad connectivity in browsers. But this adds overhead of SSL termination to the proxy.

In addition to different tests we're also testing the different amount of connections:

• 2k
• 5k
• 10k

And for the equal results:

• 20k
• 30k

Before each test all WebSocketServer is reset and the Proxy re-initiated. Thor will hammer all the Proxy server with x amount of connection with a concurrency of 100. For each established connection one single UTF-8 message is send and received. After the message is received the connection is closed.

Running

Stud

stud --config stud.conf

HAProxy

haproxy -f ./haproxy.cfg

Nginx

nginx

http-proxy

FLAVOR=http node http-proxy.js

WebSocketServer

FLAVOR=http node index.js

Results

The http-proxy lives up to it's name, it proxies requests and does it quite fast. But as it's build on top of Node.js it quite heavy on the memory. Just a simple node process starts with a 12MB of memory. For the 10K requests it took around 70mb of memory. The overhead was of the HTTP proxy was about 5 seconds if you compare it to control test. The HTTPS test was the slowest of all, but that was expected as Node.js sucks hairy monkey balls in SSL. Not to mention that will put your event loop to a grinding halt when it's under severe stress.

There is a pull request for the http-proxy that will drastically reduce memory consumption. I've manually applied the patch and saw the memory consumption get cut in half. It's still using a lot more compared to Nginx after this patch that can easily be explained because they are all build in pure C.

I had high hopes for Nginx and it did not let me down. It had a peak memory of 10MB and it was really fast. The first time I tested Nginx, it had a horrible performance. Node was even faster in SSL then Nginx, I felt like failure, I genuinely sucked a configuring Nginx. But after some quick tips from some friends it was actually a one line change in the config. I had the wrong ciphers configured. After some quick tweaking and a confirmation using openssl s_client -connect server:ip it was all good and used RC4 by default which is really fast.

Up next was HAProxy, it has the same performance profile as NGINX, but lower on the memory it only required 7MB of memory. The biggest difference was when we tested with HTTPS. It's was really slow and no where near the performance of Nginx. Hopefully this will be resolved as it's a development branch we are testing. Made the same mistake as I did with Nginx, configured the wrong ciphers which was kindly pointed out on HackerNews. In addition to testing HTTPS we also put stud in front of it to see what kind of performance it would yield.

Conclusions

http-proxy it's a great flexible proxy, really easy to extend and build up on. If you deploy this in production I advice to run stud in front of it to take care of the SSL offloading.

nginx and haproxy were really close, it's almost not significant enough to say that one is faster or better then the other. But if you look at it from an operations stand point. It's easier to deploy and manage a single nginx server instead of stud and haproxy.

Please note that lower means better in these results tables.

HTTP

Proxy	Connections	Handshaken (mean)	Latency (mean)	Total
http-proxy	10k	293 ms	44 ms	30168 ms
nginx	10k	252 ms	16 ms	28433 ms
haproxy	10k	209 ms	18 ms	26974 ms
control	10k	189 ms	16 ms	25310 ms

Winner: Both Nginx and HAProxy are really fast and close to each other.

HTTPS

Proxy	Connections	Handshaken (mean)	Latency (mean)	Total
http-proxy	10k	679 ms	62 ms	68670 ms
nginx	10k	470 ms	30 ms	50180 ms
haproxy	10k	464 ms	25 ms	50058 ms
haproxy + stud	10k	492 ms	42 ms	52403 ms
control	10k	703 ms	65 ms	71500 ms

Winner: Both Nginx and HAProxy are really fast and close to each other.

All test results are available at:

https://github.com/observing/balancerbattle/tree/master/results

Notes

• The performance of the http-proxy can vary based on your platform. On Debian distributions it's know to add a 200ms latency on top of it. See 305 & 314
• NGINX has released 1.4 which contains WebSocket support by default. There are no major changes made between 1.4 and the version that is tested here so these results are valid for the 1.4 release as well.

Contributions

All the configuration are in the repository, I'm more then happy to see if we can squeeze more performance out of the servers.