Union Filesystems crashes - "Transport endpoint is not connected"

    • Resolved
    • Union Filesystems crashes - "Transport endpoint is not connected"

      Hello again!

      I am trying to set up a datapool for my two SD cards with Union Filesystems, but the plugin crashes every time I am trying to transfer data to the pool. The SD cards are connected via two different USB-ports on my Rock64. In the future, the two SD cards are going to be replaced with two HDDs, but in the meanwhile I am just playing around.

      This is what I do:
      - Mount the to SD cards --> named "ALPHA" & "BRAVO"
      - Create a pool with the UnionFS plugin (version 4.0.2) (Create policy: Most free space.) --> named "DATAPOOL"
      - Create a shared folder connected to the pool --> named "ALLDISKS"
      - Give user1 read/write-permissions to "ALLDISKS"
      - Share the folder "ALLDISKS" with Samba
      - Trying to transfer some data to the shared folder via my Windows computer
      - This error shows up in the WebGUI:

      Source Code

      1. Error #0:
      2. OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; df -PT '/srv/fb97e7c0-e130-4e4d-b1df-33cac946b547' 2>&1' with exit code '1': df: /srv/fb97e7c0-e130-4e4d-b1df-33cac946b547: Transport endpoint is not connected in /usr/share/php/openmediavault/system/process.inc:182
      3. Stack trace:
      4. #0 /usr/share/php/openmediavault/system/filesystem/union.inc(258): OMV\System\Process->execute(Array)
      5. #1 /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc(157): OMV\System\Filesystem\Union->getStatistics()
      6. #2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(612): OMVRpcServiceFileSystemMgmt->{closure}()
      7. #3 /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc(172): OMV\Rpc\ServiceAbstract->asyncProc(Object(Closure))
      8. #4 [internal function]: OMVRpcServiceFileSystemMgmt->enumerateFilesystems(NULL, Array)
      9. #5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
      10. #6 /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc(320): OMV\Rpc\ServiceAbstract->callMethod('enumerateFilesy...', NULL, Array)
      11. #7 [internal function]: OMVRpcServiceFileSystemMgmt->getList(Array, Array)
      12. #8 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
      13. #9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod('getList', Array, Array)
      14. #10 /usr/share/php/openmediavault/rpc/serviceabstract.inc(565): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}('/tmp/bgstatusUX...', '/tmp/bgoutput2h...')
      15. #11 /usr/share/php/openmediavault/rpc/serviceabstract.inc(159): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
      16. #12 /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc(470): OMV\Rpc\ServiceAbstract->callMethodBg('getList', Array, Array)
      17. #13 [internal function]: OMVRpcServiceFileSystemMgmt->getListBg(Array, Array)
      18. #14 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
      19. #15 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('getListBg', Array, Array)
      20. #16 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('FileSystemMgmt', 'getListBg', Array, Array, 1)
      21. #17 {main}
      Display All

      Does anyone know if there are known bugs with the plugin? Should I try an older version of the plugin?



      Additional information
      - The OMV image is downloaded from github.com/ayufan-rock64/linux-build/releases/ (version 0.9.14: gitlab-ci-linux-build-159)
      - I have not tried with any other images yet


      I hope someone can help me with the problem :)

      Best regards
      Marius
    • ryecoaaron wrote:

      What is the output of: dpkg -l | grep mergerfs
      grep 33cac946b547 /proc/mounts
      Thank you for helping! :)

      This is the output:

      Source Code

      1. root@rock64:~# dpkg -l | grep mergerfs
      2. ii mergerfs 2.28.2~debia n-stretch armhf another FUSE union filesystem

      Source Code

      1. root@rock64:~# grep 33cac946b547 /proc/mounts
      2. ALPHA:BRAVO /srv/fb97e7c0-e130-4e4d-b1df-33cac946b547 fuse.mergerfs rw,relatime, user_id=0,group_id=0,default_permissions,allow_other 0 0
    • ryecoaaron wrote:

      Not seeing anything wrong yet. What about grep mergerfs /etc/fstab
      Hm... Do you know any bugs with the plugin?

      Output of the command is:

      Source Code

      1. root@rock64:~# grep mergerfs /etc/fstab
      2. /srv/dev-disk-by-label-ALPHA:/srv/dev-disk-by-label-BRAVO /srv/6ae2eee8-f0ff-4f8b-848e-a35cfb5870c6 fuse.mergerfs defaults,allow_other,direct_io,use_ino,category.create=mfs,minfreespace=4G 0 0
    • Mumbedeus wrote:

      Do you know any bugs with the plugin?
      Not right off hard. This plugin on OMV 4.x has been around a long time. If this was OMV 5.x, it might make sense. Your fstab entry looks ok too. This seems like a potential mergerfs issue. Maybe @trapexit (author of mergerfs) would have an idea?
      omv 5.1.2 usul | 64 bit | 5.3 proxmox kernel | omvextrasorg 5.1.9
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      Mumbedeus wrote:

      Do you know any bugs with the plugin?
      This seems like a potential mergerfs issue. Maybe @trapexit (author of mergerfs) would have an idea?
      It's worth a try. Thank you! :)


      I came over this, which led me to this and this. The armhf versions here are libfuse2_2.9.9-2 and fuse_2.9.9-2. The output of mergerfs -v on my system is:

      Source Code

      1. root@rock64:~# mergerfs -v
      2. mergerfs version: 2.28.2
      3. FUSE library version: 2.9.7-mergerfs_2.28.0
      4. fusermount version: 2.9.7
      5. using FUSE kernel interface version 7.29
      The versions are newer than 2.9.4 (which was recommended here), but it's not the latest release(?)
    • Mumbedeus wrote:

      The versions are newer than 2.9.4 (which was recommended here), but it's not the latest release(?)
      mergerfs has its own fuse libraries compiled in. So, the system fuse libraries shouldn't matter.
      omv 5.1.2 usul | 64 bit | 5.3 proxmox kernel | omvextrasorg 5.1.9
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      Mumbedeus wrote:

      The versions are newer than 2.9.4 (which was recommended here), but it's not the latest release(?)
      mergerfs has its own fuse libraries compiled in. So, the system fuse libraries shouldn't matter.
      Ah, I understand...
      Well, I guess I have to wait for the author, @trapexit, as you recommended :) That said, I have seen several post on the forum describing the exact same problem, but it looks like it never was solved. I hope I will be the lucky man! :thumbsup: Anyway, thank you so far!
    • trapexit wrote:

      I can't comment without details. What mergerfs version? Is it up to date? Can you reproduce it outside OMV?
      Look at this:

      Mumbedeus wrote:

      Source Code

      1. root@rock64:~# mergerfs -v
      2. mergerfs version: 2.28.2
      3. FUSE library version: 2.9.7-mergerfs_2.28.0
      4. fusermount version: 2.9.7
      5. using FUSE kernel interface version 7.29
      The versions are newer than 2.9.4 (which was recommended here), but it's not the latest release(?)

      What do you mean by "Can you reproduce it outside OMV?"?
    • trapexit wrote:

      I really can't comment without the ability to reproduce it. There are lots of reasons why it could have died. Have you tried running mergerfs on the commandline in the foreground to see what the output might be as it dies? Or ... journald logs or something?
      No, I have not tried that as I am not that kind of an expert with Linux. Would you mind guiding me, or maybe link to a webpage showing how to do it?
      Currently everyting is set up from the WebGUI in OMV. When trying to add a folder from my PC with Samba, the pool crashes.
    • trapexit wrote:

      I don't know what the plugin does so I really can't comment on what it may or maynot contribute to the issue.
      Off topic - but I just wanted to pass a note of thanks for your work. Once one spends a bit of time understanding the policies, it's pretty easy to aggregate drives. mergerfs has worked flawlessly for me.

      Thanks again.