OMV server stop responding on network while copying

  • Hello, I've changed all the disk from a running server because one disk of the RAID was dead. So I buy 3 new disks and create a new RAID. Note I keep the same OS, I just remove everything using the OMV interface and create a new RAID with new ones.
    When I copy files from the other NAS to the fresh one, the server stop responding to SSH, HTTPS, SMB after a few minuts of file transfer...


    My command is :
    rsync --verbose --recursive --checksum --itemize-changes --delete --progress --times --exclude '**/.recycle/**' root@othernas:/root/Save10_11_2019/Files /media/91be7395-eeaf-4239-a139-dd30cd9b62da/Files


    On syslog, I see this :


    Nov 13 12:48:59 onyx kernel: [ 7757.296469] r8169 0000:01:00.0: eth0: link up
    Nov 13 12:49:00 onyx kernel: [ 7757.328413] r8169 0000:01:00.0: eth0: link up
    Nov 13 12:49:00 onyx kernel: [ 7757.364473] r8169 0000:01:00.0: eth0: link up
    Nov 13 12:49:00 onyx kernel: [ 7757.396412] r8169 0000:01:00.0: eth0: link up
    Nov 13 12:49:00 onyx kernel: [ 7757.432475] r8169 0000:01:00.0: eth0: link up
    Nov 13 12:49:00 onyx kernel: [ 7757.464472] r8169 0000:01:00.0: eth0: link up
    Nov 13 12:49:00 onyx kernel: [ 7757.496474] r8169 0000:01:00.0: eth0: link up
    Nov 13 12:49:00 onyx kernel: [ 7757.532464] r8169 0000:01:00.0: eth0: link up
    Nov 13 12:49:00 onyx kernel: [ 7757.564478] r8169 0000:01:00.0: eth0: link up


    Nov 13 12:49:00 onyx kernel: [ 7757.600471] r8169 0000:01:00.0: eth0: link up


    But I have no more log. I try to remake the RAID this afternoon.
    Do you know any other file I should watch ? Should I try to reinstall the OS ?
    I will post SMART tests but they looks fine (and the RAID was marked as clean)

  • Try to decompose your Problem by separating Net and Raid and testing them separately.
    Use iperf for net trafic tests and use dd localy for big files on raid.
    check at firt your raid:
    #dd if=/dev/zero of=/srv/path-to-raid/sample1G.txt bs=1G count=1
    and
    #dd if=/dev/zero of=/srv/path-to-raid/sample10G.txt bs=1G count=10 status=progress
    and
    #dd if=/dev/zero of=/srv/path-to-raid/sample100G.txt bs=1G count=100 status=progress
    this creates 3 files in 3 sizes, 1G ,10G and 100G.
    so you can check if the raid is working.
    then... read about testing iperf...https://iperf.fr/ or https://www.linode.com/docs/ne…e-network-speed-in-linux/ as example.

    Equipment: a few Computers, lot's of waste heat, little time and a Pile of work.


    When solving problems, dig at the root instead of hacking at the leaves.

  • Let me sum up the issue, I've just recreated the RAID and got the same :
    - The server is up
    - I recreate the RAID 5 using OMV - 230 mins (is it too long ?) 3x2to
    - I start a new copy at 19:30
    - At 20:43, the server is still running but disconnected from the network.
    - Reboot
    - The copy stopped because network drop


    So akward... And no log from RAID or kernel issue. Just the network "link up" several times before the server stops responding



    Top:




    Syslog
    [code=bash]Nov 13 19:43:41 onyx kernel: [25132.600469] r8169 0000:01:00.0: eth0: link up
    Nov 13 19:43:41 onyx kernel: [25132.632412] r8169 0000:01:00.0: eth0: link up
    Nov 13 19:43:41 onyx kernel: [25132.668538] r8169 0000:01:00.0: eth0: link up
    Nov 13 19:43:41 onyx kernel: [25132.700462] r8169 0000:01:00.0: eth0: link up
    Nov 13 19:43:41 onyx kernel: [25132.732477] r8169 0000:01:00.0: eth0: link up
    Nov 13 19:43:41 onyx kernel: [25132.768472] r8169 0000:01:00.0: eth0: link up
    Nov 13 19:43:41 onyx kernel: [25132.800474] r8169 0000:01:00.0: eth0: link up
    Nov 13 19:43:42 onyx kernel: [25132.836462] r8169 0000:01:00.0: eth0: link up
    Nov 13 19:43:42 onyx kernel: [25132.868477] r8169 0000:01:00.0: eth0: link up[/code]


    Disk : df

    EDIT :
    Oh I didn't see that you answered Rd65. I will dig into the issue with unraid tests
    And I don't think that's a network issue as 165Go were transfered in 2 hours


  • I get no trouble with dd, I created 410Go on each disk without any issue.
    I just get this log in syslog but I think that's still ok


    Nov 13 22:14:55 onyx monit[2595]: 'localhost' loadavg(5min) of 4.1 matches resource limit [loadavg(5min)>2.0]



    So what can I do ? I will start a dd with 1To to confirm 75% of the disk are ok but I don't know what to do..

  • ok, so in my opinion, the raid is not part of the problem.
    But you can test more things.
    install #apt install iperf3
    on your server and on a client depend on os.
    https://iperf.fr/iperf-download.php#windows
    but you can use android, mac or windows too.
    its a command line executable without gui.
    start in a shell #iperf3 -s
    this will be your server.
    now connect with
    a shell, cmd, cli or what ever you use as terminal on your client:
    #iperf3 -c server-ip
    and you will see statistics.
    try #iperf3 -? or -h, you can change blocksizes and more things.
    You can connect local (to your own ip) or you connect via ethernet, wlan or whatever.
    connecting to your own ip (in a second shell) may test the local networkstack, connecting to a server behind a switch tests the switch and the network hardware on both sides.
    you may change the role of client and server... (and open the Win firewall or disable for iperf3 at first)
    now you are able to test your network, including network adapters without trouble from other devices.
    if you identify slow connetions, or broken hardware, you may change networkhardware or doublecheck the configs.
    If it is all fine, the the problem is not the network.
    iperf3 is your friend to testing throuput and reliability of networks!
    use it


    the upper part shows statistics from client side, the lower part are server statistics.
    Lots of Retrys show problems... slow or variant speeds show problems.. and so on.
    This is my wlan link.


    $ iperf3 -t 60 -P 100 -c nas


    this wil test 100 parallel connections for 60 Sec on a 1 GB Line.
    and it say:


    [SUM] 0.00-60.00 sec 6.54 GBytes 937 Mbits/sec 0 sender
    [SUM] 0.00-60.00 sec 6.52 GBytes 934 Mbits/sec receiver
    + 1 mp3 music stream from plex :)


    Take and a look on your cpu stats..
    top - 22:16:34 up 2 days, 21:09, 2 users, load average: 0.67, 0.40, 0.28


    the load should not go through the ceiling


    an older but common way to do these testings is using the echo service on inetd:
    take a look at https://en.wikipedia.org/wiki/Echo_Protocol
    all *nixes support that, but need to install some software too. but iperf3 is the easy way to do that.

    Equipment: a few Computers, lot's of waste heat, little time and a Pile of work.


    When solving problems, dig at the root instead of hacking at the leaves.

    3 Mal editiert, zuletzt von Rd65 () aus folgendem Grund: typo and insert code

  • hmm I don't know if I can install iperf3 now :


    I've just added inside the apt repo list (/etc/apt/sources.list)
    deb http://ftp.de.debian.org/debian sid main
    And so many (unchecked) updates want to be installed ! I'm running a last badblocks test against each disk to be sure and I don't want to interupt that test so I will give a try to network test after.


    But I did the transfer on the other way few days ago (the server I'm facing issue => the other NAS) and had no issue. If I had a network issue, I wouldn't be able to make the first transfer..
    If that's not disks, I will try to install new OMV... But hell!!

  • Hu?
    why you insert things in /etc/apt/sources.list?
    i never told that.
    strange...
    and why sid? buster is stable and strech oldstable....
    verry strange...
    do you use sid as your nas-os?
    one more time... STRANGE!
    thats the way not to do...

    Equipment: a few Computers, lot's of waste heat, little time and a Pile of work.


    When solving problems, dig at the root instead of hacking at the leaves.

  • Oh no I'm used the OMV 4 iso image, I don't know why it just told me the iperf3 package was missing on repo. Imo basic OMV 4.x can't install iperf3 using apt, the OS is too much outdated..


    When I added some additionnal debian repo, updates was ready to be installed but the upgrade system could not install anything of the repo I added. I think it tried to install the wrong version of these softs, the repo I used (sid) was maybe too recent for OMV 4?


    And I've installed iperf3 on the other NAS which is running on the latest OMV version (without adding any repo), I think the 4 was too old to get basic updates.


    I will try again rsync and iperf3 on the (main) NAS, the one I'm facing disconnection issue, after OS reinstall.

  • Is there any difference between the ISO I can find on sourceforge and a basic debian netinstall then installing OMV ?
    Sorry I've already done the OS reinstall, rsync is good (iperf3 -t 60 -P 100 -c) between both servers.
    And my first rsync copy without RAID succeed, so I think the OS reinstall fix my pb

  • original debian images lie on original debian servers, not on sourceforge....
    https://www.debian.org/CD/netinst/
    Dont know what the diference to sourceforge images is ... im trust only in debian images, not even in ubuntu.
    download a iso or img what ever you need fitting your hardware requirement, boot it an install as described in link i've posted.

    Equipment: a few Computers, lot's of waste heat, little time and a Pile of work.


    When solving problems, dig at the root instead of hacking at the leaves.

    2 Mal editiert, zuletzt von Rd65 ()

    • Offizieller Beitrag

    Dont know what the diference to sourceforge images is ... im trust only in debian images, not even in ubuntu.

    I assume he is referring to the OMV images: https://sourceforge.net/projects/openmediavault/files/


    OMV4 is based on Debian Stretch. OMV5 is based on Debian Buster. If you install OMV on top of Debian you will end up with the same system as if you were using the image. Maybe some different packages will be installed.

  • Yeah I refer to the OMV image. This is much more easier to install from the sourceforge site and everything works.


    The copy worked on the RAID; something was definitely wrong with the OS. Thanks for the support Rd65!


  • OMV4 is based on Stretch. OMV5 is based on Debian. If you install OMV on top of Debian you will end up with the same system as if you were using the image. Maybe some different packages will be installed.

    OMV5 on Buster... its all debian.
    but you can even install OMV4 on Buster(backports) with netinstalls... if you dont want to be a betatester for OMV5.

    Equipment: a few Computers, lot's of waste heat, little time and a Pile of work.


    When solving problems, dig at the root instead of hacking at the leaves.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!