Write speed to OMV drops then resumes to max again, and repeats with very large files

  • Hi,


    I recently installed OMV (4.1.9) on an older gaming PC to try it out and experiment.


    I have configured two 8TB WD RED disks (WD80EFZX-68U) as a raid 1. But I've noticed that the write speed drops to a very low speed and resumes again with large files. I am currently testing with very large files, 40GB
    https://streamable.com/0man6


    What is causing the drop? Read speed is unaffected from what I can see.


    Hardware
    CPU: Intel i7-5930K
    RAM: 16GB

  • I didn't notice that it was a recording. I see what you're referring to now. Do you have another computer to try transferring from to possibly eliminate your sending machine as the cause?

    Yeah, same problem occurs on my other machines, a desktop and laptop (wired). All running windows 10.

  • I have configured two 8TB WD RED disks (WD80EFZX-68U) as a raid 1

    IMO a really bad idea (reasons). Anyway: you need to test your storage locally first to see whether storage performance is ok or not.


    Drops when writing through the network can be due to the kernel's filesystem buffers filling up (now network transfer speeds only been limited by Gigabit Ethernet) and once they need to be flushed to disk the performance drops down to storage performance.


    Personally I use iozone to test for storage performance (in the upper right corner is a search function). Others use dd for the same purpose.

    • Offizieller Beitrag

    It seems that your disk writes are slower than the network throughput. And pdflush throttle/locks other I/O while it flushes the full write cache. That is what you see as a dip/halt in throughput.


    Raid 1 also typically slows down writes compared to single disks. If write performance is very important you might consider using one of the disks for automated backups instead of raid? Possibly using rsync snaphots.


    Less drastic is to make sure you have write back activated on your disks. That might help throughput, possibly at the expense of corrupt disks in case of power failures.


    You might also be able to tweak the write caches by changing the settings for:


    vm.dirty_background_ratio=10


    vm.dirty_ratio=20


    See pages 109-110 in the book "Linux Performance and Tuning Guidelines" or google for other write-ups. Not for newbies...


    pdflush starts writing dirty data in the background when dirty pages/RAM exceed dirty_background_ratio. Typically at 10%.
    pdflush throttle/lock other I/O when dirty pages/RAM exceeds dirty_ratio. Typically at 20%.


    (Dirty pages are also flushed when they get old...)


    Perhaps something like...


    vm.dirty_background_ratio=5


    vm.dirty_ratio=40


    ...might help increase/smooth throughput.


    However tweaking the cache behavior will most likely not help a lot if you are writing very large files, significantly larger than free RAM for the write cache pages. But it could help a lot with smaller random writes over a fast network to slow disks. Especially if you have plenty of unused RAM and an UPS...


    At least it might be interesting to experiment with...

  • IMO a really bad idea (reasons). Anyway: you need to test your storage locally first to see whether storage performance is ok or not.
    Drops when writing through the network can be due to the kernel's filesystem buffers filling up (now network transfer speeds only been limited by Gigabit Ethernet) and once they need to be flushed to disk the performance drops down to storage performance.


    Personally I use iozone to test for storage performance (in the upper right corner is a search function). Others use dd for the same purpose.

    What do you recommend instead of RAID1? I don't have the disks for RAID10 (If I understand it correctly, minimum is 4?) and is ZFS snapshots possible through OVM (I will have to read more about it too)?
    Maybe something like @Adoby mentioned? Daily rsync to the other disk?
    I tried iozone, but running the command sudo apt install iozone3 returns E: Unable to locate package iozone3. What have I missed?


    "write back activated on your disks" - do you mean Write Cache?
    Where can I change these values? Do I need to restart the machine each time or are the changes detected and used immediately?

  • What kind of raid controller are you using? If it is onboard you might try using mdadm.

    No hardware controller, software controller (i think?).


    I have the disks plugged in normally to my motherboard, then I created a RAID1 from within the OMV GUI.

  • I don't have the disks for RAID10

    You have them but since you don't read the replies to your questions it's useless to continue. Again: Home NAS build, FS info (last paragraph!)


    Every time I see someone using RAID-1 in 2018 I feel sad. It's soooooo anachronistic and provides almost no protection.


    Edit: apologies! Just realized that the last link in my old post was broken. Corrected it in the post and here: https://blog.a2o.si/2014/09/07…aid-10-instead-of-raid-1/

    • Offizieller Beitrag

    "write back activated on your disks" - do you mean Write Cache?Where can I change these values? Do I need to restart the machine each time or are the changes detected and used immediately?

    Yes, you are right: Write Cache on the HDDs.


    From the manual:


    Power Options
    Pressing the edit button with a selected disk will give the following options available to set:


    • Advanced power management (APM)
    • Automatic accoustic management (Not all drives support this)
    • Spindown time (ST)
    • Write cache
  • You have them but since you don't read the replies to your questions it's useless to continue. Again: Home NAS build, FS info (last paragraph!)
    Every time I see someone using RAID-1 in 2018 I feel sad. It's soooooo anachronistic and provides almost no protection.


    Edit: apologies! Just realized that the last link in my old post was broken. Corrected it in the post and here: https://blog.a2o.si/2014/09/07…aid-10-instead-of-raid-1/

    My knowledge about raid and linux is limited (still learning and want to learn).
    But isn't RAID 10 with two disks (1 partition each) RAID 0? If one disk fails, all the data is lost? Or i'm I missing something here how RAID 10 works?
    The Web GUI says 4 disk minimum when trying to create a RAID 10 device.


    Yes, you are right: Write Cache on the HDDs.
    From the manual:


    Power Options
    Pressing the edit button with a selected disk will give the following options available to set:


    • Advanced power management (APM)
    • Automatic accoustic management (Not all drives support this)
    • Spindown time (ST)
    • Write cache


    Thank you, just activating Write Cache showed no change. But tweaking the values of vm.dirty_ratio and vm.dirty_background_ratio gave improvements!


    Default values were:

    • vm.dirty_ratio = 20
    • vm.dirty_background_ratio = 1

    I tried different values until I found these worked pretty well:

    • vm.dirty_ratio = 10
    • vm.dirty_background_ratio = 5

    Left is new values and Right is default values


    Will continue to experiment later, and reply with an update if more improvements are made.

    • Offizieller Beitrag

    The default settings were interesting. They are set so that background flush of the cache start almost at once. Might reduce the chance of blocked IO if disk IO is fast enough. But it also reduces the chance of write cache hits that might improve performance.


    If I understand correctly your settings means that pdflush will use more RAM for the write cache before starting to write to disk, and also block other IO earlier/more often but shorter periods of time. That means smaller but more dips, which your results seem to confirm. But there is no difference in average throughput when writing very large files. But shorter frozen IO each time.


    More aggressive use of the write cache would mean that more RAM is used to cache writes. But that would only help random writes since that would increase the chance of more random writes hitting cached writes before they are flushed. And increased up/downs when writing very large files.


    So I suspect you don't really gain any performance at all for your use case with very large sequential write. Just shorter periods of blocked IO. But that might be good for other tasks that are blocked shorter periods?


    I suspect that most users, with many random writes, might benefit from slightly more aggressive use of free RAM for write cache, as I suggested earlier.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!