MergerFS crashing after OMV 5 to 6 upgrade

  • Since upgrading from OMV 5 to 6, MergerFS is crashing on me every day or so. All the other violumes that make up the MergerFS pool are still online. I am seeing the following messages in the syslog. I can only get the Volume back by rebooting. Any ideas on how to troubleshoot this?


    I am running OMV as a Proxmox VM, with a Dell SAS card running in IT mode passed through directly to OMV. Having been running OMV 5 without any issues for some time.

    Code
    Jun 26 19:40:52 fileserver kernel: [75313.331529] mergerfs[1512]: segfault at 10 ip 000055d4214e07ea sp 00007f51a9dfc3c0 error 4 in mergerfs[55d421493000+5a000]
    Jun 26 19:40:52 fileserver kernel: [75313.331543] Code: 80 7b 02 00 0f 85 24 fe ff ff 49 8d 6f 60 31 db 48 89 ef e8 e8 32 fb ff 4c 89 ee 4c 89 ff e8 6d c6 ff ff 48 89 ef 48 8b 40 20 <4c> 8b 68 10 e8 8d 2d fb ff e9 f6 fd ff ff 0f 1f 84 00 00 00 00 00
    Jun 26 19:40:56 fileserver collectd[1932]: statvfs(/srv/97dd3425-2e5f-4684-9f16-3814a590f3a2) failed: Transport endpoint is not connected
    Jun 26 19:40:56 fileserver smbd[5397]: [2022/06/26 19:40:56.406439,  0] ../../source3/smbd/service.c:166(chdir_current_service)
    Jun 26 19:40:56 fileserver smbd[5397]:   chdir_current_service: vfs_ChDir(/srv/97dd3425-2e5f-4684-9f16-3814a590f3a2) failed: Input/output error. Current token: uid=1000, gid=100, 8 groups: 100 27 999 998 997 1000 995 994
  • I get the following error if I try to view File Systems after the MergerFS Volume goes offline.


  • and this if I try to restart the pool


    Code
    Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; systemctl restart srv-97dd3425\x2d2e5f\x2d4684\x2d9f16\x2d3814a590f3a2.mount' with exit code '5': 
    
    OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; systemctl restart srv-97dd3425\x2d2e5f\x2d4684\x2d9f16\x2d3814a590f3a2.mount' with exit code '5':  in /usr/share/php/openmediavault/system/process.inc:197
    Stack trace:
    #0 /usr/share/openmediavault/engined/rpc/mergerfs.inc(181): OMV\System\Process->execute(Array, 5)
    #1 [internal function]: OMVRpcServiceMergerfs->restartPool(Array, Array)
    #2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
    #3 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('restartPool', Array, Array)
    #4 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Mergerfs', 'restartPool', Array, Array, 1)
    #5 {main}
  • If you go to the plugin and edit the pool, deselect fstab and reboot it may fix it - doing this starts the pool from systemd instead of fstab.

    Inwin MS04 case with 315 W PSU

    ASUS Prime H310i-Plus R2.0 board

    Two port PCI-E SATA card

    8GB Kingston DDR4

    Intel Pentium Coffee Lake G5400 CPU

    Samsung Evo M.2 256GB OS drive

    4x4TB WD Red NAS drives + 5TB Seagate drive - MergerFS pool

    Seagate 4TB USB drive - SnapRAID parity x 2

    Seagate 5TB USB drive

  • If I read this:

    Code
    Jun 26 19:40:52 fileserver kernel: [75313.331529] mergerfs[1512]: segfault at 10 ip 000055d4214e07ea sp 00007f51a9dfc3c0 error 4 in mergerfs[55d421493000+5a000]
    Jun 26 19:40:52 fileserver kernel: [75313.331543] Code: 80 7b 02 00 0f 85 24 fe ff ff 49 8d 6f 60 31 db 48 89 ef e8 e8 32 fb ff 4c 89 ee 4c 89 ff e8 6d c6 ff ff 48 89 ef 48 8b 40 20 <4c> 8b 68 10 e8 8d 2d fb ff e9 f6 fd ff ff 0f 1f 84 00 00 00 00 00
    Jun 26 19:40:56 fileserver collectd[1932]: statvfs(/srv/97dd3425-2e5f-4684-9f16-3814a590f3a2) failed: Transport endpoint is not connected


    I think it is a software problem either in the driver or the kernel itself. Can you choose a different kernel?

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

  • Thanks, I will try disabling the fstab option. I have also recreated the mergerfs volume in case some other weirdness had carried over from my OMV5 config.

  • If I read this:

    Code
    Jun 26 19:40:52 fileserver kernel: [75313.331529] mergerfs[1512]: segfault at 10 ip 000055d4214e07ea sp 00007f51a9dfc3c0 error 4 in mergerfs[55d421493000+5a000]
    Jun 26 19:40:52 fileserver kernel: [75313.331543] Code: 80 7b 02 00 0f 85 24 fe ff ff 49 8d 6f 60 31 db 48 89 ef e8 e8 32 fb ff 4c 89 ee 4c 89 ff e8 6d c6 ff ff 48 89 ef 48 8b 40 20 <4c> 8b 68 10 e8 8d 2d fb ff e9 f6 fd ff ff 0f 1f 84 00 00 00 00 00
    Jun 26 19:40:56 fileserver collectd[1932]: statvfs(/srv/97dd3425-2e5f-4684-9f16-3814a590f3a2) failed: Transport endpoint is not connected


    I think it is a software problem either in the driver or the kernel itself. Can you choose a different kernel?

    Thanks, I think a new kernel version has just been installed with an update. I will see if that helps.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!