mhddfs segfault "transport endpoint is not connected"

watari · 26. Juni 2015

Hello

I'm having a problem with mhddfs crashing every few days or when under heavy load (ie. samba file transfers). I'll get a monit alert saying my pools filesystem does not exist. If I try to access the mount I get "transport endpoint is not connected". remounting corrects the condition, but I want to address the root of the problem if possible. I really like this setup with OMV but, this one problem is really annoying.

Google searches suggest this problem is not limited to OMV but, I haven't found a post or article that offers a good solution.

serverfault.com/questions/677352/mhddfs-randomly-breaks

The above link says there is a bug in mhddfs 0.1.39 and a workaround is to downgrade the package.

https://github.com/vdudouyt/mhddfs-nosegfault

This second link offers a version without the segfault bug. But I found no posts from anyone that uses this, so I'm not sure if it's safe.

I want advice before trying either of those possible solution to be on the safe side.

Any help would be appreciated.

--Info--
OVM 1.19
Running backports kernel 3.16

Code

root@server:~# dmesg |grep mhddfs
[115110.323967] mhddfs[4867]: segfault at 0 ip 000000000040443c sp 00007f3982bb49d0 error 4 in mhddfs[400000+a000]
[258882.505660] mhddfs[25369]: segfault at 2 ip 000000000040443c sp 00007f8940bda9d0 error 4 in mhddfs[400000+a000]

Code

root@server:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=8148690e-95ca-4114-89af-9b8b672a2e71 /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=b2152da0-9542-4d36-8128-4d761a25ca58 none            swap    sw              0       0
/dev/sdb1       /media/usb0     auto    rw,user,noauto  0       0
tmpfs           /tmp            tmpfs   defaults        0       0
# >>> [openmediavault]
UUID=23aadaef-7082-4a2d-8268-fe0cf59c21fe /media/23aadaef-7082-4a2d-8268-fe0cf59c21fe ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=5be3cf2e-ad64-4206-959a-cb1a90d3c9dd /media/5be3cf2e-ad64-4206-959a-cb1a90d3c9dd ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=1fc9981d-8bea-4290-bc69-c7bd40cc6699 /media/1fc9981d-8bea-4290-bc69-c7bd40cc6699 ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=5732d9c2-5f4f-4d49-a687-1d26ed1db953 /media/5732d9c2-5f4f-4d49-a687-1d26ed1db953 ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=2db92dde-7886-4fda-b56b-b6ebc0bd8bdb /media/2db92dde-7886-4fda-b56b-b6ebc0bd8bdb ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=bc20fc92-3af6-4fac-a9bd-bd78e57de063 /media/bc20fc92-3af6-4fac-a9bd-bd78e57de063 ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=4389b04a-8922-44eb-936c-dff1c5f7bc8c /media/4389b04a-8922-44eb-936c-dff1c5f7bc8c ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
mhddfs#/media/23aadaef-7082-4a2d-8268-fe0cf59c21fe,/media/5be3cf2e-ad64-4206-959a-cb1a90d3c9dd,/media/1fc9981d-8bea-4290-bc69-c7bd40cc6699,/media/5732d9c2-5f4f-4d49-a687-1d26ed1db953,/media/2db92dde-7886-4fda-b56b-b6ebc0bd8bdb /media/6cdff609-56d1-44f0-a434-0052bc283885 fuse defaults,allow_other,mlimit=4g 0 0
# <<< [openmediavault]

Alles anzeigen

If more information is needed, just ask.

synci · 26. Juni 2015

Exactly the same problem here, switched back to AUFS but im not really happy with it because of the .wh. files ...
I have also permission problems sometimes with AUFS.

watari · 27. Juni 2015

Yeah.. I don't think I want to fool with AUFS because of the opaque/whiteout files; that sounds like it would make a mess of things. But, from what I understand it's not suppose to make the whiteout files unless the permissions are read-only? That doesn't seem like it would be problem.

cha · 22. Juli 2015

Same problem here: Transport endpoint is not connected (107) - every day.

mhddfs is not usable this time.

HK-47 · 22. Juli 2015

I built and installed the code from mhddfs-nosegfault and have been running the last few weeks. So far I haven't had a single segfault.

ryecoaaron · 23. Juli 2015

Did you have them before?

cha · 23. Juli 2015

Zitat von HK-47

I built and installed the code from mhddfs-nosegfault and have been running the last few weeks. So far I haven't had a single segfault.

I tried this in OMV, but it didn't work. How did you do the build and installation?

ryecoaaron · 23. Juli 2015

Here is the package. If it fixes the problem, I can put this in the repo.

cha · 23. Juli 2015

Thank you! The installation of your package worked, i will see and tell how mhddfs does from now on.

vdudouyt · 30. Juli 2015

Zitat

Same problem here: Transport endpoint is not connected (107) - every day.mhddfs is not usable this time.

We have recently found that sometimes, mhddfs can halt with the following error which is believed to be an internal bug in FUSE:

Zitat

fuse internal error: node 2238037 not found

So please consider to check that you're building mhddfs-nosegfault as least against FUSE-2.9.4. If this problem persists, then please consider to run mhddfs in foreground mode (you might want to use GNU Screen), and collect an output when it halts again.

HotBlack · 9. September 2015

Zitat von ryecoaaron

mhddfs segfault "transport endpoint is not connected"[/url]']Here is the package. If it fixes the problem, I can put this in the repo.

Is updating the repo possible? I have been running the 'no-segfault' version for quite a while now and have not seen the "transport endpoint is not connected" once. It does seem that the pooling plugin is much more stable in MHDDFS mode when making use of this package.

synci · 9. September 2015

Zitat von HotBlack

Is updating the repo possible? I have been running the 'no-segfault' version for quite a while now and have not seen the "transport endpoint is not connected" once. It does seem that the pooling plugin is much more stable in MHDDFS mode when making use of this package.“

Same here, works fine, without "no-segfault" version i got endpoint errors every few hours and i had to reboot my OMV machine.
Im on 2.1.14 now.

ryecoaaron · 9. September 2015

Added amd64 version to stoneburner-testing repo.

synci · 9. November 2015

I have tried mergerfs few days ago, also got transport endpoint errors ... so decided to switch back to mhddfs.

Installed OMV completely new(100% clean), installed latest Union Filessystems plugin (testing repo), created a mhddfs pool with 4x4TB WD Reds.
Saw in the log that mhddfs nosegfault version got correctly installed.
Still transport endpoint errors, i dont know what to do and why this happen.
Problem occurs when downloading some GB and extracting them.
Also when rsync some GB to my OMV backup machine.

What are the correct steps to fullcheck my hardrives for bad sectors/errors on debian, maybe thats the problem ?
Sorry im a linux beginner, but ssh and some more are no problem.

All i can see in syslog is this:

Thanks !

raulfg3 · 9. November 2015

I use mergerFS for several weeks and have no problems, I use it in a 8x2TB=16TB pool, this pool only have one folder=Video, and inside this video Folder I have a lot of folders (series/TV/Films/Manga/etc..)

And as I say, no problem, only notice a small banwith like say here: Unionfilesystem Plugin

synci · 11. November 2015

Litte Update, i got no more transport endpoint errors since monday.
Disabled APM, ALPM and write cache for all my drives.
Unplugged my 2,5" backup USB disk, maybe defective, have to check.

synci · 12. November 2015

In the morning the problem returned ... wihle downloading and postprocessing these.
All my harddrives and my mhddfs pool got dismounted, had to restart. (Is there any quicker solution instead of restart ?)
In the syslog i can see that mkvmerge caused the segfault, maybe thats my problem.
Updated mkvtoolnix now, it was more then 2 years old. (Debian package added to my /etc/apt/sources.list now)

Code

Nov 12 07:30:01 omv /USR/SBIN/CRON[64684]: (root) CMD (/usr/sbin/omv-mkgraph >/dev/null 2>&1)
Nov 12 07:30:01 omv /USR/SBIN/CRON[64685]: (root) CMD (test -x /etc/init.d/anacron && /usr/sbin/invoke-rc.d anacron start >/dev/null)
Nov 12 07:30:01 omv rrdcached[2667]: Received FLUSHALL
Nov 12 07:30:01 omv anacron[64720]: Anacron 2.3 started on 2015-11-12
Nov 12 07:30:01 omv anacron[64720]: Will run job `cron.daily' in 5 min.
Nov 12 07:30:01 omv anacron[64720]: Jobs will be executed sequentially
Nov 12 07:31:54 omv rpc.mountd[2646]: authenticated mount request from 192.168.1.9:803 for /export/DATA (/export/DATA)
Nov 12 07:35:01 omv anacron[64720]: Job `cron.daily' started
Nov 12 07:35:01 omv anacron[64905]: Updated timestamp for job `cron.daily' to 2015-11-12
Nov 12 07:35:26 omv kernel: [70981.373645] nfsd: non-standard errno: -103
Nov 12 07:35:26 omv kernel: [70981.376121] nfsd: non-standard errno: -107
Nov 12 07:35:26 omv kernel: [70981.378481] mkvmerge[64898]: segfault at f72e40 ip 0000000000f72e40 sp 00007ffe2a1520c8 error 15
Nov 12 07:35:26 omv kernel: [70981.378705] nfsd: non-standard errno: -107
Nov 12 07:35:48 omv collectd[3521]: statvfs(/media/0260a948-f882-49a5-9599-dc3eb6aeb1a9) failed: Transport endpoint is not connected
Nov 12 07:35:50 omv kernel: [71005.354822] nfsd: non-standard errno: -107
Nov 12 07:35:50 omv kernel: [71005.432138] nfsd: non-standard errno: -107
Nov 12 07:35:54 omv kernel: [71009.351273] nfsd: non-standard errno: -107
Nov 12 07:35:54 omv kernel: [71009.428376] nfsd: non-standard errno: -107
Nov 12 07:35:58 omv kernel: [71013.347583] nfsd: non-standard errno: -107
Nov 12 07:35:58 omv monit[3662]: 'fs_media_0260a948-f882-49a5-9599-dc3eb6aeb1a9' filesystem doesn't exist
Nov 12 07:35:58 omv monit[3662]: 'fs_media_0260a948-f882-49a5-9599-dc3eb6aeb1a9' trying to restart

Alles anzeigen

Jetzt mitmachen!