Killing rsync not handled gracefully / rsync exit code not evaluated

[kerner1000]

When I abort a running backup by
sudo killall rsync BIT does not flag this snapshot with WITH ERRORS!

[Germar]

BIT doesn't handle return codes from rsync yet. It's on my Todo list. But before I need to switch all processes to be called by subprocess.Popen

[protist]

For the record, I ran into this issue too. I had a network connection go down, and my backup was aborted. Despite setting backintime to delete the backup on errors, this meant I had an empty backup directory for that particular timepoint. The next backup was thus not incremental, and took a massive amount of space. Unfortunately this happened several times several months ago, and I only noticed it now.

To fix it, I moved the bad backup(s) to a temporary directory, created a hard-link copy of the last good backup, replaced the changed files, then deleted the old directory.

For example, where 20200719-160001-721 is a bad backup and 20200430-061501-472 is the last good backup:

$ mv 20200719-160001-721 tmp/
$ sudo rsync -aPHAX --link-dest=/path/to/backintime/hostname/root/1/20200430-061501-472 20200430-061501-472/ 20200719-160001-721
$ sudo rsync -aPHAX tmp/20200719-160001-721/ 20200719-160001-721 --delete
$ sudo rm -r tmp/20200719-160001-721

I imagine there are faster ways to do this. For my ~430G backup, the hard-link copy took 15 minutes, and the replacement rsync took 1h 36m (!).

Moving forward, I have a cron job on my server to check the log file for errors (then delete the last backup if necessary). Something like:

#!/bin/sh

set -e

cd /path/to/backintime/hostname/root/1

errors=$(bzcat last_snapshot/takesnapshot.log.bz2 | grep '^\[I\] Take snapshot (rsync: rsync error' -B3)

if [ -n "$errors" ]; then
  echo 'Errors found! Log shows:'
  printf '%s\n\n' "$errors"

  printf '%s' "Removing $(realpath last_snapshot) . . . "
  rm -r "$(realpath last_snapshot)"
  echo removed!
  rm last_snapshot
  ln -s "$(ls 20* -d | tail -1)" last_snapshot
fi

[aryoda]

For the record, I ran into this issue too. I had a network connection go down, and my backup was aborted. Despite setting backintime to delete the backup on errors, this meant I had an empty backup directory for that particular timepoint. The next backup was thus not incremental, and took a massive amount of space.

@protist This annoying side effect may also happen (not in your case here) when the option Take a new snapshot whether there were changes or not was enabled and there were errors preventing to backup all files (the more are missing the more disk space is wasted in the next snapshot because the missing files are copied instead of linked I think).

Anyhow I have now fixed this issue by also catching rsync return codes and process signals like SIGKILL and sending an error message + writing the error into the snapshot log:

See my PR #1502

Edit: With my PR you can now also use the error user callback to e.g. send an email, the error user callback is now called for all error case when taking a snapshot (except external events like a computer crash or power supply failure).

PS: killall sends a SIGTERM (15 on Ubuntu) by default (see man killall) and this signal is now also handled gracefully in BiT (shown as "error" in the [last] snapshot log).