syncoid: getting "Too many levels of symbolic links" in snapshot dir on remote host

[jonalbrecht]

I don't think this is a problem with syncoid but I wanted to check if there is a problem either with what I'm doing or with zfs.

I'm on ubuntu 16.04 with the latest updates and I'm using syncoid from master as of Aug 8. After the first incremental syncoid I am getting "Too many levels of symbolic links" errors when trying to list the files in the remote snapshot dir.

Here are the commands I've run with robocat as the source host and roboduck as the remote backup host:

root@roboduck:~# zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
tank   757K  74.1G    19K  /tank

root@robocat:~# zfs create tank/testfs
root@robocat:~# echo "test" > /tank/testfs/testfile01.txt 
root@robocat:~# date; /usr/local/bin/syncoid tank/testfs root@roboduck:tank/testfs
Sun Aug 21 13:41:42 EDT 2016
INFO: Sending oldest full snapshot tank/testfs@syncoid_robocat_2016-08-21:13:41:43 (~ 40 KB) to new target filesystem:
41.5KiB 0:00:00 [18.1MiB/s] [================================>] 102%            

root@roboduck:~# date; ls /tank/testfs/
Sun Aug 21 13:42:03 EDT 2016
testfile01.txt
root@roboduck:~# ls /tank/testfs/.zfs/snapshot/
syncoid_robocat_2016-08-21:13:41:43
root@roboduck:~# ls /tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:41\:43/
testfile01.txt

root@robocat:~# date; /usr/local/bin/syncoid tank/testfs root@roboduck:tank/testfs
Sun Aug 21 13:43:02 EDT 2016
Sending incremental tank/testfs@syncoid_robocat_2016-08-21:13:41:43 ... syncoid_robocat_2016-08-21:13:43:02 (~ 4 KB):
1.52KiB 0:00:00 [5.33KiB/s] [===========>                      ] 38%            

root@roboduck:~# date; ls /tank/testfs/
Sun Aug 21 13:43:11 EDT 2016
testfile01.txt
root@roboduck:~# ls /tank/testfs/.zfs/snapshot/
syncoid_robocat_2016-08-21:13:43:02
root@roboduck:~# ls /tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:43\:02/
ls: cannot open directory '/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/': Too many levels of symbolic links
root@roboduck:~# ls /tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:43\:02/testfile01.txt 
ls: cannot access '/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/testfile01.txt': Too many levels of symbolic links
root@roboduck:~# cat /tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:43\:02/testfile01.txt 
test

On some other zfs filesystems I've been testing with, I've seen the files on the remote filesystem disappear after the first incremental but I don't have a simple repro for that yet. The snapshot dir errors are the same in both cases. I'm guessing this is a problem with zfs but it would be great if someone could have a quick look in case I'm doing something wrong.

[jimsalterjrs]

"too many levels of symlinks" strongly implies that the problem isn't
specifically ZFS at all - your dataset likely contains, well, symlinks,
which on the target wind up pointing to themselves recursively.

To troubleshoot further, look at an ls -l of the parent directory
containing the file or folder you're getting the error from, possibly piped
through grep for concision.

root@roboduck:~# ls 
/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:43\:02/ ls: 
cannot open directory 
'/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/': Too many 
levels of symbolic links

Should be followed with:

root@roboduck:~# ls -l /tank/testfs/.zfs/snapshot | grep 2016-08-21

Also note that the .zfs/snapshots directory is basically a convenience
feature only; have you tried manually mounting the snapshot directly, or
cloning it?

root@roboduck:~# zfs clone pool/dataset/snapshot pool/snapclone
root@roboduck:~# ls /pool/snapclone
root@roboduck:~# zfs destroy pool/snapclone

---

(Sent from my tablet - please blame any weird errors on autocorrect)

On August 21, 2016 14:21:14 jonalbrecht [email protected] wrote:

I don't think this is a problem with syncoid but I wanted to check if there
is a problem either with what I'm doing or with zfs.

I'm on ubuntu 16.04 with the latest updates and I'm using syncoid from
master as of Aug 8. After the first incremental syncoid I am getting "Too
many levels of symbolic links" errors when trying to list the files in the
remote snapshot dir.

Here are the commands I've run with robocat as the source host and roboduck
as the remote backup host:

root@roboduck:~# zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
tank   757K  74.1G    19K  /tank

root@robocat:~# zfs create tank/testfs
root@robocat:~# echo "test" > /tank/testfs/testfile01.txt
root@robocat:~# date; /usr/local/bin/syncoid tank/testfs 
root@roboduck:tank/testfs
Sun Aug 21 13:41:42 EDT 2016
INFO: Sending oldest full snapshot 
tank/testfs@syncoid_robocat_2016-08-21:13:41:43 (~ 40 KB) to new target 
filesystem:
41.5KiB 0:00:00 [18.1MiB/s] [================================>] 102%

root@roboduck:~# date; ls /tank/testfs/
Sun Aug 21 13:42:03 EDT 2016
testfile01.txt
root@roboduck:~# ls /tank/testfs/.zfs/snapshot/
syncoid_robocat_2016-08-21:13:41:43
root@roboduck:~# ls 
/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:41\:43/
testfile01.txt

root@robocat:~# date; /usr/local/bin/syncoid tank/testfs 
root@roboduck:tank/testfs
Sun Aug 21 13:43:02 EDT 2016
Sending incremental tank/testfs@syncoid_robocat_2016-08-21:13:41:43 ... 
syncoid_robocat_2016-08-21:13:43:02 (~ 4 KB):
1.52KiB 0:00:00 [5.33KiB/s] [===========>                      ] 38%

root@roboduck:~# date; ls /tank/testfs/
Sun Aug 21 13:43:11 EDT 2016
testfile01.txt
root@roboduck:~# ls /tank/testfs/.zfs/snapshot/
syncoid_robocat_2016-08-21:13:43:02
root@roboduck:~# ls 
/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:43\:02/
ls: cannot open directory 
'/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/': Too many 
levels of symbolic links
root@roboduck:~# ls 
/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:43\:02/testfile01.txt
ls: cannot access 
'/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/testfile01.txt': 
Too many levels of symbolic links
root@roboduck:~# cat 
/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:43\:02/testfile01.txt
test

On some other zfs filesystems I've been testing with, I've seen the files
on the remote filesystem disappear after the first incremental but I don't
have a simple repro for that yet. The snapshot dir errors are the same in
both cases. I'm guessing this is a problem with zfs but it would be great
if someone could have a quick look in case I'm doing something wrong.

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#48

[jonalbrecht]

Thanks for the quick reply.
This is from the source host:

jalbrecht@robocat:~$ ls -la /tank/testfs/.zfs/snapshot/
total 0
dr-xr-xr-x 2 root root 2 Aug 21 13:43 .
dr-xr-xr-x 1 root root 0 Aug 21 13:41 ..
dr-xr-xr-x 1 root root 0 Aug 21 17:55 syncoid_robocat_2016-08-21:13:43:02

This is from the remote host:

deploy@roboduck:~$ ls -la /tank/testfs/.zfs/snapshot/
total 0
dr-xr-xr-x 2 root root 2 Aug 21 13:43 .
dr-xr-xr-x 1 root root 0 Aug 21 13:41 ..
dr-xr-xr-x 1 root root 0 Aug 21 13:43 syncoid_robocat_2016-08-21:13:43:02
deploy@roboduck:~$ ls -la /tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:43\:02/
ls: '/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/': Too many levels of symbolic links
ls: cannot access '/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/.': Too many levels of symbolic links
ls: cannot access '/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/..': Too many levels of symbolic links
ls: cannot access '/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/testfile01.txt': Too many levels of symbolic links
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
-????????? ? ? ? ?            ? testfile01.txt

I saw this recent zfsonlinux issue. Maybe the cause of what I'm seeing?
openzfs/zfs#4514

[jimsalterjrs]

Possibly. Have you tried accessing the snapshot with mount or clone as I
suggested?

---

(Sent from my tablet - please blame any weird errors on autocorrect)

On August 21, 2016 20:06:31 jonalbrecht [email protected] wrote:

Thanks for the quick reply.
This is from the source host:

jalbrecht@robocat:~$ ls -la /tank/testfs/.zfs/snapshot/
total 0
dr-xr-xr-x 2 root root 2 Aug 21 13:43 .
dr-xr-xr-x 1 root root 0 Aug 21 13:41 ..
dr-xr-xr-x 1 root root 0 Aug 21 17:55 syncoid_robocat_2016-08-21:13:43:02

This is from the remote host:

deploy@roboduck:~$ ls -la /tank/testfs/.zfs/snapshot/
total 0
dr-xr-xr-x 2 root root 2 Aug 21 13:43 .
dr-xr-xr-x 1 root root 0 Aug 21 13:41 ..
dr-xr-xr-x 1 root root 0 Aug 21 13:43 syncoid_robocat_2016-08-21:13:43:02
deploy@roboduck:~$ ls -la 
/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21\:13\:43\:02/
ls: '/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/': Too 
many levels of symbolic links
ls: cannot access 
'/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/.': Too 
many levels of symbolic links
ls: cannot access 
'/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/..': Too 
many levels of symbolic links
ls: cannot access 
'/tank/testfs/.zfs/snapshot/syncoid_robocat_2016-08-21:13:43:02/testfile01.txt': 
Too many levels of symbolic links
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
-????????? ? ? ? ?            ? testfile01.txt

I saw this recent zfsonlinux issue. Maybe the cause of what I'm seeing?
openzfs/zfs#4514

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#48 (comment)

[jonalbrecht]

Sorry for taking so long to reply.

root@roboduck:~# zfs clone tank/testfs@syncoid_robocat_2016-08-21\:13\:43\:02 tank/snapclone
root@roboduck:~# ls /tank/snapclone/
testfile01.txt
root@roboduck:~# cat /tank/snapclone/testfile01.txt 
test

It all looks good

[jimsalterjrs]

Going to close this here; it's not a bug in syncoid. Thanks for the report though.