Add log rotation for internal services to prevent out of disk issues

[jbeemster]

The NSQ Daemon logs very aggressively which can consume many gigabytes of storage and prevent heavy use of a Snowplow Mini server.

These logs should be truncated or rotated regularly to prevent these issues.

After just 2 weeks of running a server 5.5G of logs were present in just the following file:

/var/log/nsqd.err

cc/ @alexanderdean

[alexanderdean]

Ouch! Thanks @jbeemster, scheduling into next milestone. Let's go with truncation.

[jbeemster]

This can be achieved with logrotate - the following script would set this up for all of the Snowplow apps to keep only 3 hours worth of logs:

# Insert logrotate configuration for Snowplow Services
sudo cat << EOF > /etc/logrotate.d/snowplow-apps
/var/log/snowplow_stream_collector.* /var/log/snowplow_stream_enrich.* /var/log/snowplow_elasticsearch_loader_* /var/log/nsqlookupd.* /var/log/nsqd.* /var/log/iglu_server_0.2.0.* {
        hourly
        rotate 3
        missingok
        notifempty
        create 644 root root
}
EOF
sudo chmod 644 /etc/logrotate.d/snowplow-apps

# Change logrotate cron to hourly
sudo mv /etc/cron.daily/logrotate /etc/cron.hourly
sudo service cron restart

[jbeemster]

@oguzhanunlu the above is not tested fully so please do make sure it functions as expected before locking it in. When I tried to make it work it had some weird side affects like the logs not working properly after rotation so would be good to test it thoroughly!

[oguzhanunlu]

Sure @jbeemster , thanks for the warning!

[jbeemster]

As an update to this - the approach I took to fix this was much less elegant - it essentially involves tailing the log file to a temporary log file with a certain number of lines and then doing an in-place overwrite.

Neither of these will be necessary as soon as everything is Dockerised though!

[oguzhanunlu]

I'll start 0.6.0 right after 0.5.0 release and sp mini 0.6.0 includes dockerizing sp mini. I think we should drop this ticket into 0.6.0, to be cancelled after handling docker issue. Working on something that'll be deleted soon sounds inefficient. What do you think @jbeemster ?