Server Out of Disk

you are here because you got alerted that some server was out of disk.

what you do next depends on which disk is out of space. df -h will show you which disk is full (if you did not get that from the alert already`)

• Docker Disk
• Root Disk

Docker

This drive is where all the docker images and container data lives. to clean it we need to remove containers and/or old images. The easiest first step is to remove image builder and weave containers

1. Ssh into the server
2. Remove image builder containers docker rm `docker ps -a | awk '/image-builder.*Exited/{print $1}'`
3. Remove dangling images: docker rmi $(docker images -f dangling=true -q)
4. Check to see if this helped (you want to maintain atleast 10gb free) df -h | grep docker
5. In order to delete them manually try docker rmi `docker images -aq`
6. If there are volumes on the dock, try deleting old unreferenced volumes with docker volume rm `docker volume ls -f dangling=true | awk '{print $2}'`
7. If you are here all is lost for this server. mark as unhealthy using docks-cli

Root

The root disk is also where the os and logs live. here you can safely delete logs and files in the ~/var/log/ path.

1. Check where the space is getting used the usual suspects du -h -d 0 /var du -h -d 0 /home

2. If /var/log is using too much space first remove archived logs rm /var/log/*.gz

3. If that does not help flush out applications logs 1. echo “” > /var/log/docker-listener.log 1. echo “” > /var/log/charon.log 1. echo “” > /var/log/filibuster.log 1. echo “” > /var/log/krain.log

4. If you are here all is lost for this sever. mark as unhealthy using docks-cli