RAID6 Speeds between 12 MB/s and 150 MB/s

1. offizieller Beitrag

    Hi,


    i have a 7x4 TB RAID6.


    I'm just transferring transfer my data from my old HDDs to this RAID via the 8th internal port, so this should not be the issue.


    The problem is, that there are some files which are writing with over 150 MB/s and then there are files which are writing only with 12 MB/s (the files have nearly the same big (>5GB) size). Maybe there is a problem with only some of the disks?


    Example:
    5 to 6 files are at high speed
    then 5 to 6 files are at low speed
    then 5 to 6 files are at high speed
    etc...


    I saw that the speed sometimes slows down during the file transfer (maybe they write to the "faulty" drives), so its not really an file issue.


    Here is my mdstat:

    Code
    Personalities : [raid6] [raid5] [raid4]
md127 : active raid6 sde[0] sdg[6] sdf[5] sdc[4] sdd[3] sdb[2] sda[1]
      19534435840 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]


unused devices: <none>


    Here are my fdisk information (i know that fdisk can't really show the information for 4TB Drives with advanced format):


    For me it looks like the sde and the sdf are maybe the problem? Can i pull them out of the RAID and delete the partitions via webgui?


    sda:

    Code
    Disk /dev/sda: 4000.8 GB, 4000787030016 bytes
255 heads, 63 sectors/track, 486401 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/sda doesn't contain a valid partition table


    sdb:

    Code
    Disk /dev/sdb: 4000.8 GB, 4000787030016 bytes
255 heads, 63 sectors/track, 486401 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/sdb doesn't contain a valid partition table


    sdc:

    Code
    Disk /dev/sdc: 4000.8 GB, 4000787030016 bytes
255 heads, 63 sectors/track, 486401 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/sdc doesn't contain a valid partition table


    sdd:

    Code
    Disk /dev/sdd: 4000.8 GB, 4000787030016 bytes
255 heads, 63 sectors/track, 486401 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/sdd doesn't contain a valid partition table


    sde:


    sdf:


    sdg:

    Code
    Disk /dev/sdg: 4000.8 GB, 4000787030016 bytes
255 heads, 63 sectors/track, 486401 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/sdg doesn't contain a valid partition table


    thx for helping

    • Offizieller Beitrag

    Nope. OMV doesn't use partitions with raid because you don't need to. It works fine using the entire drive. I would wipe every drive with the following:


    dd if=/dev/zero of=/dev/sdX bs=512 count=1000

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Offizieller Beitrag

    Yep.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Before wiping everything, check some parameters (replace md127 by your own RAID identifier) :

    Code
    cat /sys/devices/virtual/block/md127/queue/read_ahead_kb
cat /sys/devices/virtual/block/md127/md/stripe_cache_size


    These parameters are set in the file : etc/udev/rules.d/99-openmediavault-md-raid.rules


    I use 5120 and 16384 personally (I have 8 GB RAM).


    You should also consider switching to Kralizec / Wheezy and use backport kernel in order to have a recent kernel and RAID implementation.


    I use a 7x3To RAID 6 with a GPT partition table and two partitions, it works fine.


    Sometimes, however, SMART selftest cand degrade performances.


    Use dstat to check what happens

    Code
    apt-get install dstat # just once !
dstat -tcndylpg --top-cpu --top-io --disk-util

    omv 2.x omv 3.x (testing) - AMD CPU - RAM ECC 8x4TB RAID6 + 4x3TB RAID5 - SSD 4GB for system

    Hi,


    thanks for reply my settings are (i use 4GB RAM):


    Code
    root@omv:~# cat /sys/devices/virtual/block/md127/queue/read_ahead_kb
5120
root@omv:~# cat /sys/devices/virtual/block/md127/md/stripe_cache_size
8192


    and I even use Krazilec/Wheezy


    Is there any difference between using partitions against the whole disk?


    any pro/cons?


    And why do you use two partitions?


    Thank you both for your help!

    • Offizieller Beitrag

    There are no pros or cons to using partitions. They just aren't needed and the OMV web interface doesn't create them when creating an array.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Offizieller Beitrag

    Correct.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Thanks...


    i have to copy some files back to my backup and its a torture....


    it's nearly the problem i had before when i wrote to the RAID... now i am reading back from the RAID... and it's very very bad.


    It goes down to 3 MB/s what is a pain in the ass. It doesn't matter what file it is, cause the speed goes up and down in the middle of the transfer. It works for some minutes to go down abruptly for a couple of minutes and back.


    What's going on here? Faulty backup drive (where i'm copying to)? A faulty drive in the RAID?


    high speed (look at the average speed - there were no small files... all over 5 GB each) :



    slow speed (transferrate went down in the middle of the high speed transfer --> average speed is higher then the real live speed):





    smart-data for the backup-drive:


    And the SMART data from a RAID device which maybe looks not so good?! (even it says PASSED):



    In detail:


    OMG... :shock:


    maybe i got a hint to the error:




    Hopefully that are errors caused by the filesystem or are they real hardware issues?


    this are the SMART data for this drive:


    Okay... i set the sdh drive to faulty and now it runs like a bastard :D


    This is a new WD RED Drive... Should i try to wipe it and hope that the drive will work?


    How does WD say "yes this drive is faulty" if i try to replace it via warranty? It is sufficient to say that my RAID is sh** with this drive or what do they need from me?


    I'll do a SMART long check right now and see what it will say.


    thank you :)

    You could try to wipe it with zeros (gparted or systemrescuecd) and try again.
    I´d open an RMA at WD and send it to them. They will replace it without any complaint (as far as I am concerned)
    Just write them some technical stuff like bad sectors and send / print them the smartctl page and they will replace it almost sure. I doubt there are employees who understand that at the RMA section ;)

    Hi there,


    i tried to wipe my sdh with " dd if=/dev/zero of=/dev/sdh bs=512 count=1000" and got the following message:


    Code
    dd: Schreiben von „/dev/sdh“: Eingabe-/Ausgabefehler
577+0 Datensätze ein
576+0 Datensätze aus
294912 Bytes (295 kB) kopiert, 4,75693 s, 62,0 kB/s
root@omv:~# dd if=/dev/zero of=/dev/sdh bs=512 count=1000
dd: Schreiben von „/dev/sdh“: Eingabe-/Ausgabefehler
585+0 Datensätze ein
584+0 Datensätze aus
299008 Bytes (299 kB) kopiert, 5,27799 s, 56,7 kB/s


    As you can see the speed is very slow the wipe of the other drives were between 25 MB/s and 55 MB/s.


    And here is the SMART-Result:


    Code
    === START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%        74         576


    I think this should be enough for WD :)


    thx

    Yup. Just send it in as an RMA... they will replace without problems. But, I think you rather need to contact your vendor if the drive is that new...


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

    Well,


    today i got my recertified drive...


    I created a new RAID6 and now it is syncing...


    BUT:



    I checked all S.M.A.R.T. attributes of my drives but there were no such values?!?


    What the f*** is this? It don't think that 50% of my new(!) drives are faulty?!


    How can i check which drives are affected? I can't connect these hexadecimal values to my /dev/sdX drives (where can i see this?)


    Or are these errors "normal" read errors which are corrected by ECC-RAM or by re-read of the block? :?:

    Here is a result of one of the drives... the other look similar (ATA-Error count is different)


    Are these real drive errors? Because the S.M.A.R.T. values are looking good?


    Unfortunately, yes it could happen that those disks are bad from arrival ... you should download the WD test tool for that purpose and run it on the disk before continuing.


    Another hint for performance optimization for your raid:
    There are multiple settings to consider for your raid. That is including file system settings as well as raid settings.


    Find some tutorials here:
    http://forums.openmediavault.org/viewtopic.php?f=10&t=1333
    http://forums.openmediavault.o…pic.php?f=11&t=1417#p6219


    Also for the partition stuff:
    Yes there is an issue with partition, esp. if they are not aligned as your partitions are.


    MD starts at 1MB after a partition and therefore automatically is aligned if you use it on the raw device (/dev/sdf instead of /dev/sdf1).
    If you create a partition at sector 1 and the logical sector size is 512 byte, you will start with the md data at exactly 1MB+512b and all your blocks are misaligned. In reads, that is not painful, but every write to an already written sector on the discs requires two sectors to be read and written again. That will definitely impact your write performance.

    Everything is possible, sometimes it requires Google to find out how.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden