wrk-utils: wrk cluster and stats

wrk-utils is a set of commands and wrk lua scripts to run wrk in clusters and have stats.

This project consisted of two major parts;

Commands, that you can enable and find the help message by sourcing activate script.

stats.lua, that collects stats. You can use stats.lua as the main script or a library.

Features

• Running wrk in clusters using SSH
• Live stats in JSON format, separated by status codes
• Possibility to use stats.lua as a library to have stats in custom scripts
• Live 5xx status report
• Slack integration
• JSON formatted logs for both requests and responses in files
• Commands to Initialize servers and update files
• Commands to execute and stop wrk
• Execute custom commands on servers
• Read IPs from file or stdin (pipe)
• Debug mode

Final stats sample data:

{
    "event": "done",
    "start_time": 1639990232,
    "node": "10.10.1.1",
    "target_url": "https://example.com/list?limit=100",
    "total_completed_requests": 778511,
    "total_sent_requests": 778707,
    "total_timeouts": 2809,
    "connect_error": 0,
    "socket_status_error": 341,
    "read_error": 0,
    "write_error": 0,
    "duration": "484.686049",
    "rps": 1606.22,
    "recv_bytes": "690.31mb",
    "status": {
        "502": 315,
        "200": 778170,
        "504": 26
    }
}

Live stats sample data:

{"event": "response", "node": "10.0.1.2", "thread_id": 1001, "count": 231, "status": {"200": 231}}
{"event": "response", "node": "10.0.1.2", "thread_id": 1000, "count": 234, "status": {"200": 234}}
{"event": "response", "node": "10.0.1.2", "thread_id": 1004, "count": 234, "status": {"200": 234}}
{"event": "response", "node": "10.0.1.3", "thread_id": 1004, "count": 353, "status": {"200": 353}}
{"event": "response", "node": "10.0.1.3", "thread_id": 1000, "count": 354, "status": {"200": 354}}
{"event": "response", "node": "10.0.1.2", "thread_id": 1003, "count": 360, "status": {"200": 360}}
{"event": "response", "node": "10.0.1.2", "thread_id": 1002, "count": 355, "status": {"200": 355}}
{"event": "response", "node": "10.0.1.3", "thread_id": 1001, "count": 351, "status": {"200": 351, "502": 3}}
{"event": "response", "node": "10.0.1.3", "thread_id": 1003, "count": 360, "status": {"200": 370}}
{"event": "response", "node": "10.0.1.3", "thread_id": 1002, "count": 355, "status": {"200": 373}}

Installation & run

1. wrk-utils requires sshpass. Make sure it's installed.
2. Prepare a set of servers with ssh keys OR same ssh username/passwords and provide list of IPs in servers.txt or pipe addresses to commands.
3. Add SSH_USR="<SSH USERNAME>", SSH_PWD="<SSH PASSWORD>" and SLACK_WEBHOOK="<SLACK WEBHOOK URL>" to config.env. Look at config.env.sample.
4. Run . activate.sh in bash or . activate.fish in fish.
5. Now you see can help message and use commands.
6. Run init-servers command to setup wrk on provided servers by copying wrk binary and lua scripts there. (There is a compiled version of wrk in this repository. You can replace it with another one.)
7. You can run sync.file wordlist.txt *.csv to copy any other file you need to servers.
8. Run available-node-count to make sure that all of your servers are ready.
9. Run exec-wrk 0 -t10 -c300 -d600s -s stats.lua 'https://example.com/'
10. You can kill wrk instances using kill-all command.

# 10 is the delay to execute next wrk instances.
# All other params after 10 will pass to wrk on servers.
exec-wrk 10 -t10 -c300 -d600s -s stats.lua 'https://example.com/path/?id=1'

# running all wrk instances at once by setting delay to zero
exec-wrk 0 -t10 -c300 -d600s -s custom.lua 'https://example.com/'

Custom script development with stats

You can use stats.lua as a library to enable stats for your custom scripts is this way:

require('stats') -- load stats.lua into your custom script

function request()

    -- you can add request_logger to the request function
    request_logger(false)

    -- ...

    return wrk.format(nil, '/')

end

function response(status, headers, body)

    -- you need to add response_logger to the response function
    response_logger(status, headers)
    --  ...
end

Checkout examples

Commands

ssh-all: executes a command on all servers

ssh-all 'ps aux | grep something'

ssh-one: executes a command on a random server

ssh-one 'ps aux | grep something'

ssh-all-sudo: executes a command on all servers as sudo

ssh-all-sudo id | wc

ssh-one-sudo: executes a command on a server as sudo

ssh-one-sudo id | wc

kill-all: kills all wrk instances on all servers (friendly)

kill-all

kill-all-force: kills all wrk instances on all servers (force, will lose logs)

kill-all-force

available-node-count: prints number of available servers

available-node-count

active-node-count: prints number of active wrk instances in a loop

active-node-count

live-stats: live stats for all servers (per thread)

live-stats

init-servers: creates wrk directory on servers copies wrk and lua scripts into that

init-servers

sync-file: copies provided files to wrk directory on all servers

sync-file wordlist.txt *.jpg

exec-wrk: executes wrk step by step or at once (first argument is the delay to execute next instance)

exec-wrk 10 -t10 -c300 -d600s -s stats.lua 'https://example.com/path/?id=1'

exec-wrk 0 -t10 -c300 -d600s -s custom.lua 'https://example.com/'

Known issues

• If you want to pass a URL with multiple parameters to wrk-exec, you need to quote that URL twice. e.g. exec-wrk 0 -t1 -c3 -d6s -s stats.lua "'https://example.com/?a=1&b=2'"
• You need to rerun live-stats when a new node comes up.
• stats.lua consider requests with HTTP pipelines as one request. You need to multiply number of requests with number of requests in each pipeline.

TODO

• A dashboard to collect logs
• Some documentations
• More examples
• Improve live stats