mergerfs directory becomes unusable every time I try to list its contents

  • Config

    Hardware

    • Machine: RockPro64 4GB
    • Drives: I have two WB Red 4T (cmr) and a 6T (smr) ext4 formatted.

    Software

    • SnapRAID: The 6T is a SnapRAID parity drive to the 4T data/content drives.
    • UnionFS: The 4Ts are combined into UnionFS using Most Free Space and these options: defaults,allow_other,direct_io,noforget,use_ino

    The problem

    Whenever I try to read from my unionfs filesystem, it does what I am calling a dirty unmount. Whenever I try to do an ls on an NFS mount (if I even get it to NFS mount, that is), or an rsync listing, or even just ls on the /srv entry, I get this:

    rsync listing error

    Code
    # rsync http://localhost/share
    rsync: readdir("." (in share)): Software caused connection abort (103)
    drwxrwsrwx 4,096 2020/04/21 22:10:41 .
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.2]

    an ls

    Code
    # ls -lh /srv/a0ea2e22-75eb-48b4-91de-581e48f9185b/
    ls: reading directory '/srv/a0ea2e22-75eb-48b4-91de-581e48f9185b/': Software caused connection abort
    total 0

    rebooting is the only way I have found to effectively correct this state. doing an fusermount -u / mount does not seem to work well.

    Possibly related symptom

    Whenever I try to list the contents of any shared folders on the unionis via the CLI, it comes back with nothing. No errors, just a blank listing.

    Things I have tried to fix it

    • Rebuild the whole thing several times
    • Only bring in small amounts of data
    • Stop trying to do an NFS mount
    • Searched /r/openmediavault, this forum, youtube videos, the mergefs doc and FAQ
    • Prayed to an eldritch god

    Most recently, I tried doing a simple cp on the command line from the /srv/DIR with 70mb of files straight to a shared folder on the ufs. (so cp /srv/[uuid]/* /sharedfolder/[unionfs_folder]) and I am able to do an ls on the shared folder with no problem and everything seems to work. But then when I do an rsync listing (so rsync rsync://localhost/[unionfs_folder]) it comes back with an empty listing.

    What does work

    Thing is, I can do a non-listing rsync to and from a unionfs folder. I can do an rsync job from OMV between the ufs and a shared directory on a USB drive.

    I have been able to do remote and local rsync to put files on the drives via the union. The filesystem listing from OMV shows the correct amount of space being used both by the individual drives and the unionfs. Plus, doing an ls /srv/dev-disk-by-label-DRIVE shows all of the files I expect, and they are accessible. My data is still there! I just can't access it via the union.


    The SnapRAID content files seem good, too, but I haven't paid a ton of attention to that. A snapraid sync completes without error.


    Also, importantly, everything works fine if I just set up shared folders directly on the drives instead of the union. listing, nfs, rsync. it all works. the issue only surfaces when I bring mergerfs into the equation.

    My question

    HAAALLLP!!!!1


    ok, but seriously, seems like my only solution is to stop using unionfs, but that seems like quitting and I can't accept that just yet.

    • Has anybody experienced this behavior?
    • Am am I doing something dumb and apparently can't read documentation properly?
    • What else can I do to further diagnose this issue?

    Edited 2 times, last by HokieGeek: Added that it all works without mergerfs ().

  • HokieGeek

    Changed the title of the thread from “unionfs becomes unusable every time I try to list its contents” to “mergerfs directory becomes unusable every time I try to list its contents”.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!