Random disk/raid copy slowness

  • Hello.


    I just installed OMV and did a SMB transfer that resulted in speeds from 7MB/s to 9MB/s, which is much slower than acceptable. I unfortunately started out by troubleshooting network connections, replacing switches, etc. However, when that didn't change anything, I started examining the raid setup itself.


    Here are my basic specs:
    I have RAID5 with EXT4 with 4x2TB Red NAS drives and an SSD for the OS. I came over from FreeNAS, where I was getting 40MB/s (at least), but switched because I was using ZFS without ECC RAM and my processor was a little underpowered (AMD c-60 APU) and was maxing out when transferring via SMB. I have 16GB of RAM in the machine.


    After searching the forums I ran a few basic speed tests on the RAID itself, with odd results:



    These tests were all ran directly after eachother, nothing changed in the configuration. It wasn't a fresh reboot of the server either. Going from ~290MB/S to ~5.5MB/S is a LARGE drop...


    EDIT:
    Just ran the hdparm command on the RAID and found this which is almost exactly the transfer speed I was getting over the network:

    Code
    hdparm -t /dev/md0
    
    
    /dev/md0:
     Timing buffered disk reads:  24 MB in  3.18 seconds =   7.54 MB/sec


    I delved a little further, finding a post by datadigger on tests to run:


    Now, it seemed odd that two of my drives were limited to UDMA/33, but datadigger had the same output (I think he might have the same board) and he said the speed didn't differ on those disks from the others, so "came to the conclusion that the board maker has used a port multiplier with different capabilities as the other four SATA ports and they seem to run on PATA-mode (A 40-wire cable is an old style flat cable)."


    Also "AA" isn't enabled on those two drives, which may be a problem?


    Next, he ran speed tests on individual disks, where I once again had another interesting result.



    Notice that ONE TIME sdd had a copy speed of 2.0MB/S. However, all the times after that it worked fine! Does anyone know what might be going on here??

    • Offizieller Beitrag

    Something is going on with the array, is it clean?


    What's the status of cat /proc/mdstat


    Who long since you format the array? ext4 can take some time to initialize having the journal process in constant writing for a day depending on the size.
    You can checj with iostat if any process is writing to the array

  • Code
    root@OMV-Server:/media/4f85211f-0972-4eb4-8ce3-ac7664a3cd4f# cat /proc/mdstat
    Personalities : [raid6] [raid5] [raid4]
    md0 : active raid5 sde[3] sdd[2] sdc[1] sdb[0]
          5860148736 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
    
    
    unused devices: <none>


    I took a quick look at the IOSTAT manpage, as I am not familiar with the tool, but not sure if that output is high or normal?



    The ext4 fs was created about 24 hours ago if that helps....

  • Code
    Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
      TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
      260 be/3 root          0.00 B    204.00 K  0.00 %  0.04 % [jbd2/sda1-8]
    21626 be/4 root          0.00 B      8.00 K  0.00 %  0.00 % rsyslogd -c5
     2217 be/4 root          0.00 B     16.00 K  0.00 %  0.00 % monit -c /etc/monit/monitrc
     1832 be/4 root          0.00 B     84.00 K  0.00 %  0.00 % rrdcached -l unix:/var/run/rrdcached.sock -j /var/lib/rrdcached/journal~-F -b /var/lib/rrdcached/db/ -B -w 900 -f 3600 -p /var/run/rrdcached.pid
        1 be/4 root          0.00 B      0.00 B  0.00 %  0.00 % init [2]
        2 be/4 root          0.00 B      0.00 B  0.00 %  0.00 % [kthreadd]


    Not much showing in

    Code
    iotop -a

    . I see a max of maybe 30kb/s at times (but that is on jbd2/sda), which should be the OS disk, not the RAID.


    Yes, this is what I did.
    1) Backed up my FreeNAS ZFS archive to external USB drives
    2) Powered off PC, unclipped all drives, installed SSD drive
    3) Installed OMV to the SSD drive, shut down PC, plugged in all WD Red drives.
    4) Configured OMV with date/time, etc
    5) Used OMV web-ui to wipe all 4 physical disks
    6) Created RAID5 with my 4 2TB WD Red drives using OMV web-ui
    7) Created EXT4 FS on the RAID
    8) Copied ~250GB to the RAID (from USB) for testing
    9) Started testing RAID and ran into issues.

  • Well I replaced almost all of the sata connections and changed their ports around on the mobo and found that AHCI was not set in the BIOS, so changed that. Then checked all the smart settings and found some CRC errors on one of my drives... seems rather excessive, is this my issue? I'm re-running a full test on the drive now. Before I ran the test i did retry the test and the results were MUCH better, but network tranfers were only aroun 11-12MB/S... which is odd, if the on server transfers appear to be better, but not the network transfers.



    And here is the smartctl report for /dev/sda:

  • CRC issues in one of my 2TB WD RAID drives, not my SSD drive


    Code
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
    
    
    
    
    199 UDMA_CRC_Error_Count    0x0032   200   199   000    Old_age   Always       -       7888


  • Hey..... so thanks so much for your help! I did look at my client and was shocked to realize I wasn't running gigabit. I double checked the switch the client was on and it was a gigabit switch, so I replaced the LAN cable and it went to gigabit immediately, must have been a bad cable. Now I'm getting 60-70MB/s down (haven't tried up yet) and everything appears to be working fine.


    I'll still check on that drive, since I already replaced 4 of the 5 cables in the case, there is a slim chance that the one drive I didn't replace it on is the one that is having issues. I'll have to confirm the serial number to verify the exact drive and check the cable... if a cable replacement and port move on the MB doesn't resolve it I assume it is time to order another drive?

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!