SMB Recycle Bin on mergerfs pool share not working

    • OMV 2.x
    • SMB Recycle Bin on mergerfs pool share not working

      I recently switched my disk pool to mergerfs. Now I realized that the recycle bin stopped working on my smb shares.

      openmediavault/stoneburner uptodate 2.1.20
      openmediavault-unionfilesystems/stoneburner uptodate 2.3

      What I did to reproduce:
      1. Create shared folder on mergerfs pool
      2. Give permissions to a user through a group
      3. Create samba share on this shared folder.
      4. Enable recycle bin
      5. Enable inherited permissions
      6. Log in using smb from Windows 7 client
      7. Create a file on the share
      8. Delete the file
      Expected behavior: File moved to share/.recycle/username/
      Actual behavior: share/.recycle/username is created, but stays empty.

      Why I think this has sth. to do with mergerfs? When I create an analog share directly on a disk and not on the pool, the file is correctly moved to the recycle bin as expected. To give all the details: The disk on which I created the direct share is still part of the pool. However, I am not very experienced with OMV so I do not rule out a general misconfiguration.

      Second non-deterministic deletion issue:
      While analyzing the issue above I sometimes I had the issue with files not being deleted at all. I observed this the last time even on the not-pool share. The deleted file disappeared on the client side while it was actually never deleted on the server. If you refresh the client's view the file is still present. I then restarted the smb daemon and out of a sudden the file was moved to the .recycle bin. I was not able to reproduce this scenario deterministically.
    • Have you checked the logs? I suspect Samba is expecting that a share is a single drive which mergerfs does not currently represent. That is... rename returns EXDEV. I'm betting that Samba is creating the directory path and then tries to `rename` the file into that path. Mergerfs returns EXDEV and then it just gives up. It should be performing a move but probably isn't.

      I'm going to be putting out a new version of mergerfs soon which will handle this better but EXDEV is a totally legit return value and I will limit when it's returned but not entirely.
    • Thanks for investigating @trapexit. Indeed you are absolutely right. I set the "log level" and "debug level" to 10 after your hint. In parallel I tried to find out what Samba is doing. I found some documentation that it hooks the unlink function. I checked out the source code of samba and searched for recycling functionality. Inside the vfs_recycle.c I found the corresponding actions.

      The log file revealed exactly what you assumed:

      Source Code

      1. [2015/12/19 19:41:19.986667, 10] modules/vfs_recycle.c:609(recycle_unlink)
      2. recycle: recycled file name: .recycle/ruben///unke.odg
      3. [2015/12/19 19:41:19.987103, 10] modules/vfs_recycle.c:640(recycle_unlink)
      4. recycle: Moving unke.odg to .recycle/ruben///unke.odg
      5. [2015/12/19 19:41:19.987471, 3] modules/vfs_recycle.c:646(recycle_unlink)
      6. recycle: Move error 18 (Ungültiger Link über Gerätegrenzen hinweg), purging file unke.odg (.recycle/ruben///unke.odg)

      According to /usr/include/asm-generic/errno-base.h 18 is exactly EXDEV. I looked into the source and it reads:

      Source Code

      1. DEBUG(10, ("recycle: Moving %s to %s\n", smb_fname_str_dbg(smb_fname),
      2. smb_fname_str_dbg(smb_fname_final)));
      3. rc = SMB_VFS_NEXT_RENAME(handle, smb_fname, smb_fname_final);

      So obviously you are right in that it calls rename only. I looked into the source code of gnu mv and found a comment that they copy in case a rename does not succeed exactly for the reason you listed (EXDEV). I'll try to contact folks from samba to ask them why this only uses rename. Seems to be a bit naive.

      PS: Just to have it linked in both directions: See also Issue 182 on mergerfs github.

      The post was edited 1 time, last by kranich ().

    • Thanks for investigating. I hadn't had the chance yet but when it comes to moving files around lots of software fail to hand EXDEV correctly. As I said I'm trying to handle it better in a future version of mergerfs.

      I'm trying to provide a hybrid solution which doesn't just straight up replicate the unlink target, copy files, unlink source that mv does on EXDEV.

      If not using a path preserving policy then I'll just clone the target path on the source drive and rename. If a path preserving policy is being used try the rename; if it fails but a create would end up on that drive, clone the path on the source drive from target and then rename. If all that fails then return EXDEV.