Write speed to OMV drops then resumes to max again, and repeats with very large files

    • OMV 4.x
    • Resolved
    • Write speed to OMV drops then resumes to max again, and repeats with very large files

      Hi,

      I recently installed OMV (4.1.9) on an older gaming PC to try it out and experiment.

      I have configured two 8TB WD RED disks (WD80EFZX-68U) as a raid 1. But I've noticed that the write speed drops to a very low speed and resumes again with large files. I am currently testing with very large files, 40GB
      streamable.com/0man6

      What is causing the drop? Read speed is unaffected from what I can see.

      Hardware
      CPU: Intel i7-5930K
      RAM: 16GB
    • PangPinne wrote:

      I have configured two 8TB WD RED disks (WD80EFZX-68U) as a raid 1
      IMO a really bad idea (reasons). Anyway: you need to test your storage locally first to see whether storage performance is ok or not.

      Drops when writing through the network can be due to the kernel's filesystem buffers filling up (now network transfer speeds only been limited by Gigabit Ethernet) and once they need to be flushed to disk the performance drops down to storage performance.

      Personally I use iozone to test for storage performance (in the upper right corner is a search function). Others use dd for the same purpose.
    • It seems that your disk writes are slower than the network throughput. And pdflush throttle/locks other I/O while it flushes the full write cache. That is what you see as a dip/halt in throughput.

      Raid 1 also typically slows down writes compared to single disks. If write performance is very important you might consider using one of the disks for automated backups instead of raid? Possibly using rsync snaphots.

      Less drastic is to make sure you have write back activated on your disks. That might help throughput, possibly at the expense of corrupt disks in case of power failures.

      You might also be able to tweak the write caches by changing the settings for:

      vm.dirty_background_ratio=10

      vm.dirty_ratio=20

      See pages 109-110 in the book "Linux Performance and Tuning Guidelines" or google for other write-ups. Not for newbies...

      pdflush starts writing dirty data in the background when dirty pages/RAM exceed dirty_background_ratio. Typically at 10%.
      pdflush throttle/lock other I/O when dirty pages/RAM exceeds dirty_ratio. Typically at 20%.

      (Dirty pages are also flushed when they get old...)

      Perhaps something like...

      vm.dirty_background_ratio=5

      vm.dirty_ratio=40

      ...might help increase/smooth throughput.

      However tweaking the cache behavior will most likely not help a lot if you are writing very large files, significantly larger than free RAM for the write cache pages. But it could help a lot with smaller random writes over a fast network to slow disks. Especially if you have plenty of unused RAM and an UPS...

      At least it might be interesting to experiment with...
      OMV 4, 5 x ODROID HC2, 2 x 12TB, 2 x 8 TB, 1 x 500 GB SSD, GbE, WiFi mesh

      The post was edited 3 times, last by Adoby ().

    • tkaiser wrote:

      PangPinne wrote:

      I have configured two 8TB WD RED disks (WD80EFZX-68U) as a raid 1
      IMO a really bad idea (reasons). Anyway: you need to test your storage locally first to see whether storage performance is ok or not.
      Drops when writing through the network can be due to the kernel's filesystem buffers filling up (now network transfer speeds only been limited by Gigabit Ethernet) and once they need to be flushed to disk the performance drops down to storage performance.

      Personally I use iozone to test for storage performance (in the upper right corner is a search function). Others use dd for the same purpose.
      What do you recommend instead of RAID1? I don't have the disks for RAID10 (If I understand it correctly, minimum is 4?) and is ZFS snapshots possible through OVM (I will have to read more about it too)?
      Maybe something like @Adoby mentioned? Daily rsync to the other disk?
      I tried iozone, but running the command sudo apt install iozone3 returns E: Unable to locate package iozone3. What have I missed?

      Adoby wrote:

      It seems that your disk writes are slower than the network throughput. And pdflush throttle/locks other I/O while it flushes the full write cache. That is what you see as a dip/halt in throughput.

      Raid 1 also typically slows down writes compared to single disks. If write performance is very important you might consider using one of the disks for automated backups instead of raid? Possibly using rsync snaphots.

      Less drastic is to make sure you have write back activated on your disks. That might help throughput, possibly at the expense of corrupt disks in case of power failures.

      You might also be able to tweak the write caches by changing the settings for:

      vm.dirty_background_ratio=10

      vm.dirty_ratio=20

      See pages 109-110 in the book "Linux Performance and Tuning Guidelines" or google for other write-ups. Not for newbies...

      pdflush starts writing dirty data in the background when dirty pages/RAM exceed dirty_background_ratio. Typically at 10%.
      pdflush throttle/lock other I/O when dirty pages/RAM exceeds dirty_ratio. Typically at 20%.

      (Dirty pages are also flushed when they get old...)

      Perhaps something like...

      vm.dirty_background_ratio=5

      vm.dirty_ratio=40

      ...might help increase/smooth throughput.

      However tweaking the cache behavior will most likely not help a lot if you are writing very large files, significantly larger than free RAM for the write cache pages. But it could help a lot with smaller random writes over a fast network to slow disks. Especially if you have plenty of unused RAM and an UPS...

      At least it might be interesting to experiment with...
      "write back activated on your disks" - do you mean Write Cache?
      Where can I change these values? Do I need to restart the machine each time or are the changes detected and used immediately?
    • PangPinne wrote:

      I don't have the disks for RAID10
      You have them but since you don't read the replies to your questions it's useless to continue. Again: Home NAS build, FS info (last paragraph!)

      Every time I see someone using RAID-1 in 2018 I feel sad. It's soooooo anachronistic and provides almost no protection.

      Edit: apologies! Just realized that the last link in my old post was broken. Corrected it in the post and here: blog.a2o.si/2014/09/07/linux-s…aid-10-instead-of-raid-1/

      The post was edited 1 time, last by tkaiser ().

    • PangPinne wrote:

      "write back activated on your disks" - do you mean Write Cache?Where can I change these values? Do I need to restart the machine each time or are the changes detected and used immediately?
      Yes, you are right: Write Cache on the HDDs.

      From the manual:

      Power Options
      Pressing the edit button with a selected disk will give the following options available to set:

      • Advanced power management (APM)
      • Automatic accoustic management (Not all drives support this)
      • Spindown time (ST)
      • Write cache
      OMV 4, 5 x ODROID HC2, 2 x 12TB, 2 x 8 TB, 1 x 500 GB SSD, GbE, WiFi mesh
    • tkaiser wrote:

      PangPinne wrote:

      I don't have the disks for RAID10
      You have them but since you don't read the replies to your questions it's useless to continue. Again: Home NAS build, FS info (last paragraph!)
      Every time I see someone using RAID-1 in 2018 I feel sad. It's soooooo anachronistic and provides almost no protection.

      Edit: apologies! Just realized that the last link in my old post was broken. Corrected it in the post and here: blog.a2o.si/2014/09/07/linux-s…aid-10-instead-of-raid-1/
      My knowledge about raid and linux is limited (still learning and want to learn).
      But isn't RAID 10 with two disks (1 partition each) RAID 0? If one disk fails, all the data is lost? Or i'm I missing something here how RAID 10 works?
      The Web GUI says 4 disk minimum when trying to create a RAID 10 device.

      Adoby wrote:

      PangPinne wrote:

      "write back activated on your disks" - do you mean Write Cache?Where can I change these values? Do I need to restart the machine each time or are the changes detected and used immediately?
      Yes, you are right: Write Cache on the HDDs.
      From the manual:

      Power Options
      Pressing the edit button with a selected disk will give the following options available to set:

      • Advanced power management (APM)
      • Automatic accoustic management (Not all drives support this)
      • Spindown time (ST)
      • Write cache


      Thank you, just activating Write Cache showed no change. But tweaking the values of vm.dirty_ratio and vm.dirty_background_ratio gave improvements!

      Default values were:
      • vm.dirty_ratio = 20
      • vm.dirty_background_ratio = 1
      I tried different values until I found these worked pretty well:
      • vm.dirty_ratio = 10
      • vm.dirty_background_ratio = 5
      Left is new values and Right is default values
      [IMG:https://i.imgur.com/qvROd6q.jpg]

      Will continue to experiment later, and reply with an update if more improvements are made.
    • The default settings were interesting. They are set so that background flush of the cache start almost at once. Might reduce the chance of blocked IO if disk IO is fast enough. But it also reduces the chance of write cache hits that might improve performance.

      If I understand correctly your settings means that pdflush will use more RAM for the write cache before starting to write to disk, and also block other IO earlier/more often but shorter periods of time. That means smaller but more dips, which your results seem to confirm. But there is no difference in average throughput when writing very large files. But shorter frozen IO each time.

      More aggressive use of the write cache would mean that more RAM is used to cache writes. But that would only help random writes since that would increase the chance of more random writes hitting cached writes before they are flushed. And increased up/downs when writing very large files.

      So I suspect you don't really gain any performance at all for your use case with very large sequential write. Just shorter periods of blocked IO. But that might be good for other tasks that are blocked shorter periods?

      I suspect that most users, with many random writes, might benefit from slightly more aggressive use of free RAM for write cache, as I suggested earlier.
      OMV 4, 5 x ODROID HC2, 2 x 12TB, 2 x 8 TB, 1 x 500 GB SSD, GbE, WiFi mesh
    • Users Online 1

      1 Guest