Error because of nfs v4 mount (Borg not able to remove lock.exclusive)

[m4nug]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG

System information. For client/server mode post info for both machines.

borg 1.1.15

Ubuntu 20.04.4

Hardware / network configuration, and filesystems used.

Backup to nfs v4 mounted folder.

How much data is handled by borg?

~600GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg create
borg list
etc. (basically for all commands that interact with the repo files)

Describe the problem you're observing.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

1. Mount a nfs share via nfs4
2. Create a borg repo in a folder of that mount
3. Run a backup

Borg is not able to complete a backup, it throws an error:

[Errno 39] Directory not empty: '/mnt/backup/borg/lock.exclusive'

Borg list dosent work as well:

sudo borg list "/mnt/backup/borg"
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4621, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4553, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 162, in wrapper
    with repository:
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 190, in __enter__
    self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 421, in open
    self.lock = Lock(os.path.join(path, 'lock'), exclusive, timeout=lock_wait, kill_stale_locks=hostname_is_unique()).acquire()
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 356, in acquire
    self._roster.modify(SHARED, ADD)
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 117, in __exit__
    self.release()
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 150, in release
    os.rmdir(self.path)
OSError: [Errno 39] Directory not empty: '/mnt/backup/borg/lock.exclusive'

Platform: Linux nextcloud02 5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 x86_64
Linux: Unknown Linux
Borg: 1.1.15  Python: CPython 3.8.10 msgpack: 0.5.6
PID: 2045942  CWD: /home/nc
sys.argv: ['/usr/bin/borg', 'list', '/mnt/backup/borg']
SSH_ORIGINAL_COMMAND: None

Deleting the "lock.exclusive" file manually and running borg list again immediately creates a new "lock.exclusive" file and throws the same error again.

The only solution to get it working was to mount the share as nfs v3, which led to my conclusion that borg has some issues with nfs v4 mounts.

(The whole issue was initially posted and discussed here: nextcloud/all-in-one#771 where we figured out that the issue is acually caused by borg an not the Nextcloud AIO container which is using borg)

[ThomasWaldmann]

When unlocking, borg does a rmdir on the usually already empty lock.exclusive dir.

Find out why it is not empty in your case, it should be.

BTW, you can find a more recent borg there:

https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/borgbackup?field.series_filter=focal

[ThomasWaldmann]

Any news? An strace log might be also interesting.

[m4nug]

Any news? An strace log might be also interesting.

I'm out of office until next week.
I'll keep you posted as soon as i got time to check it.

[isshwar]

@ThomasWaldmann recently my borg backup jobs are also running in to the same issue. in my case the lock.exclusive directory is empty but nightly job is failing. A rerun again is working fine.
No changes made in the recent past with the setup. not sure, what is happening.

[ThomasWaldmann]

@isshwar if it is empty, this sounds like a different issue, please open a separate ticket about it and give all the details.