disappearing files on zfs dataset (with nextcloud desktop client)

[lopezio]

Hi Jürgen, Hi All here.

I'm sorry that I cannot provide deeper logs and insight. But after days of debugging and different trial-and-errors strategies, I have to report that openzfsonosx (running 0.9.4 on Mojave macOS 10.14.6 (18G6042)) has caused data loss in the context of a nextcloud desktop client.

What I can tell so far:

Environment: An encrypted dataset, inside of which I have 4 synced Nextcloud folders.
Expected behaviour: When I add / delete or modify files, the client should synchronize with the server and keep the files equal. Other clients with the same connected folders should reflect the changes.
Actual behaviour: With some folders (but not all!!), the client seems to believe that they have been deleted and deletes them on the server and on the other clients. In some cases, when adding files to the directory they would be found by the desktop client, uploaded, and then immediately deleted. On the local side, the folders are still visible in the Finder, but loose the "sync" icon from Nextcloud client.

I tried everymany things until finding out that openzfsonosx was the "culprit": Removing files and re-syncing from server from scrap; copying to a new dataset with casesensitivity=insensitive and/or mimic_hfs=on, but no combination worked.
When I finally copied all files to an APFS Volume, the problems all vanished.

I have also at least one confirmed case where rsyncing from that dataset to another resulted in missing files and folders.
I haven't found problems yet with files and folders outside of the Nextcloud desktop client control, but I believe I'll probably find them. Some of the missing folders were found by chance, because I happened to need a file from them and had to resort to a backup. A comparison of the backup and the active copy brought more, isolated, missing folders to light.

Worth of note is that the hardware in use is a Retina MacBook Pro 15" (2015) with a third party ADATA NVME M2 module installed, which has been part of a few panic backtraces in the last weeks - so I cannot rule out hardware problems, however, the problems only appear on the zfs volumes.

If there's anything I can produce to help debugging this, please let me know. It was quite a shocking thing to find out, so that's why I'm sharing it here. Just in case there's anyone experiencing similar issues and/or there are bugs / differences with APFS justifying this behaviour.

Best Regards,

Lorenzo

[lopezio]

Afrer reading a bit along the latest other issues: instinctively, this issue might have some relationship with the behaviours reported in #760 and possibly with #769 . I can't tell whether the st_flags play a role for an app such as the nextcloud desktop client for mac. On the other hand, in the panics I had when transferring big chunks of data from that dataset, one of the kernel modules in backtrace, was com.apple.driver.AppleMobileFileIntegrity, so something along the SIP fuzz might be going on here.