Pinned [Tutorial][Experimental][Third-party Plugin available]Reducing OMV's disk writes, also to install it on USB flash

    • [Tutorial][Experimental][Third-party Plugin available]Reducing OMV's disk writes, also to install it on USB flash

      Please note, ryecoaaron made a plugin to automate this procedure, see this post for details [Tutorial][Experimental]Reducing OMV's disk writes, also to install it on USB flash

      ----------------------back to original first post--------------------

      I'm a bit puzzled that it's none's priority to tame a bit OMV to allow the use of USB flash drives for OS drive like they do for FreeNas. Or even to let hard drives rest a bit.

      Shedding some light on the matter myself then.

      WARNING: This system is experimental, that is, it has not been throughly tested on multiple stations to ensure everything works fine in all situations.
      I CANNOT GUARANTEE DATA SAFETY. DO BACKUPS OR DO NOT TRY IT.
      Volunteers that test this system are welcome.


      Let's track down what files got written often, with a script I took from here info4admins.com/tips-to-spindo…disk-in-debian-or-ubuntu/

      Shell-Script

      1. #!/bin/sh
      2. echo Search whole filesystem for changed files in the last $1 minutes
      3. find /bin /dev /etc /home /lib /media /mnt /opt /root /sbin /srv /tmp /usr /var -cmin -$1

      Place this in a /bin folder you create in your /home folder, then chmod a+x the script to make it executable.
      Then you can call it by writing its name and the number of minutes you want.
      ScriptName 5 will give you an output listing files written in the last 5 minutes
      this is a real-life example:

      Brainfuck Source Code

      1. Search whole filesystem for changed files in the last 5 minutes
      2. /dev/xconsole
      3. /etc/samba
      4. /etc/samba/dhcp.conf
      5. /tmp
      6. /var/spool/postfix/maildrop
      7. /var/spool/postfix/maildrop/8C728FC9A
      8. /var/lib/monit/state
      9. /var/lib/openmediavault/rrd
      10. /var/lib/openmediavault/rrd/cpu-0-week.png
      11. /var/lib/openmediavault/rrd/df-root-week.png
      12. /var/lib/openmediavault/rrd/interface-eth0-hour.png
      13. /var/lib/openmediavault/rrd/cpu-0-day.png
      14. /var/lib/openmediavault/rrd/cpu-0-hour.png
      15. /var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-day.png
      16. /var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-month.png
      17. /var/lib/openmediavault/rrd/cpu-0-year.png
      18. /var/lib/openmediavault/rrd/interface-eth0-week.png
      19. /var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-week.png
      20. /var/lib/openmediavault/rrd/load-day.png
      21. /var/lib/openmediavault/rrd/df-root-month.png
      22. /var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-year.png
      23. /var/lib/openmediavault/rrd/interface-eth0-year.png
      24. /var/lib/openmediavault/rrd/load-hour.png
      25. /var/lib/openmediavault/rrd/df-root-year.png
      26. /var/lib/openmediavault/rrd/cpu-0-month.png
      27. /var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-hour.png
      28. /var/lib/openmediavault/rrd/interface-eth0-month.png
      29. /var/lib/openmediavault/rrd/interface-eth0-day.png
      30. /var/lib/openmediavault/rrd/df-root-hour.png
      31. /var/lib/openmediavault/rrd/df-root-day.png
      32. /var/lib/rrdcached/db/localhost/interface-eth0/if_octets.rrd
      33. /var/lib/rrdcached/db/localhost/interface-eth0/if_packets.rrd
      34. ------------a bunch of entries removed for space reasons----------
      35. /var/lib/rrdcached/db/localhost/cpu-0/cpu-softirq.rrd
      36. /var/lib/rrdcached/db/localhost/cpu-0/cpu-idle.rrd
      37. /var/lib/rrdcached/db/localhost/cpu-0/cpu-steal.rrd
      38. /var/lib/rrdcached/db/localhost/cpu-0/cpu-user.rrd
      39. /var/lib/rrdcached/db/localhost/cpu-0/cpu-nice.rrd
      40. /var/lib/rrdcached/db/localhost/cpu-0/cpu-system.rrd
      41. /var/lib/rrdcached/db/localhost/cpu-0/cpu-interrupt.rrd
      42. /var/lib/rrdcached/db/localhost/cpu-0/cpu-wait.rrd
      43. /var/lib/rrdcached/journal/rrd.journal.1415583612.312057
      44. /var/log/debug
      45. /var/log/auth.log
      46. /var/log/daemon.log
      47. /var/log/syslog
      Display All


      The bulk of that output you see is mostly performance logging to allow OMV to make pretty graphs, and pre-made images of graphs for a page in the webGUI. And it is done pretty often, waay too often.

      So, now we have a general idea of what is going on in the disk and we can start our countermeasures.

      The standard practice is to disable the logging from the application's settings/config file, or to move the whole offending folder to RAM. It means any write in that folder will go in RAM instead than on disk in a completely magical way, but this also means that if the NAS reboots or crashes you will lose all modifications you made to those folders.
      Since on average it's logging or temporary files, it's no big loss.
      If you want to save logs or some folder contents from the tmpfs you made above have a look at this script from Solo0815 [Tutorial][Experimental]Reducing OMV's disk writes, also to install it on USB flash

      Open /etc/fstab with a text editor

      Source Code

      1. sudo nano /etc/fstab

      if yu are already root don't write sudo.
      this is an example fstab

      Source Code

      1. # /etc/fstab: static file system information.
      2. # <file system> <mount point> <type> <options> <dump> <pass>
      3. /dev/sda1 / ext2 noatime,errors=remount-ro 0 1
      4. tmpfs /tmp tmpfs defaults,noatime 0 0

      Add the noatime option to the root partition, delete any "realtime" option. This turns off the logging of the last access time of a file (that causes writes every time a file is read)
      As you see in the example fstab I added only /tmp folder as a tmpfs (ram disk).
      Adding other folders tends to cause problems, sadly. For example, adding /var/cache and /var/log would result in annoyances from apt and nginx and probably something else that will fail horribly or crash if they don't find their things and log files.

      We need a script that mounts the appropriate folders as tmpfs and fills them with data and directory structure from a backup, then at shutdown takes the data from the tmpfs and makes a backup on disk (or not, it really matters only when booting).
      Like explained here debian-administration.org/article/661/A_transient_/var/log

      Thankfully, there is a premade package of scripts called fs2ram that does that automagically for a list of folders in its config. Not available on Wheezy, but available in Sid, here you find the .deb, install manually. packages.debian.org/sid/admin/fs2ram
      from terminal it's this (if the package is updated in the repos and the link breaks, use the link above to find the correct download link to use):

      Source Code

      1. wget http://ftp.us.debian.org/debian/pool/main/f/fs2ram/fs2ram_0.3.12_all.deb
      2. sudo dpkg -i fs2ram*

      (if you are already root don't write sudo)

      Once installed it will ask you if you prefer to save logs at shutdown or just to rebuild folder structure on boot in the tmpfs. Choose the one you prefer.

      Time to work on folder /var/lib/openmediavault/rrd and the /var/spool
      All this stuff goes directly in fs2ram's own config file, that looks suspiciously like fstab.

      Source Code

      1. sudo nano /etc/fs2ram/fs2ram.conf
      to open it and the follwoing is an example:

      Source Code

      1. #<file system> <mount point> <script> <script option> <type> <options>
      2. tmpfs /var/log keep_file_content - tmpfs
      3. tmpfs /var/tmp keep_file_content - tmpfs
      4. tmpfs /var/lib/openmediavault/rrd keep_file_content - tmpfs
      5. tmpfs /var/spool keep_file_content - tmpfs
      6. tmpfs /var/lib/rrdcached/ keep_file_content - tmpfs
      7. tmpfs /var/lib/monit keep_file_content - tmpfs
      8. tmpfs /var/lib/php5 keep_folder_structure - tmpfs
      Spot the differences between this list and the standard list in your config file.

      You could theoretically add also tmpfs /var/cache keep_file_content - tmpfs, but I find it unnecessary.
      I commented it out because I have only 512 MB of ram in my NAS, and /var/cache contains the apt-get package cache (and little else).
      If you want to keep it in a tmpfs make sure you are running apt-get autoclean in a cronjob, or you are keeping the package cache size in check somehow.

      Old tutorials usually talk about symlinking the mtab with /proc/mounts, but this was already done upstream in Debian, so we don't need to do anything about it.

      Done. Reboot the system for changes to take effect, and if it does boot up again and is still operational, pat yourself on the shoulder, Good Job!

      Look in logs to see if f2ram is reporting errors about some folder, and if there is any program complaining that it cannot start because it needs something.
      Some programs/services will complain saying they did not find file X, check if the file has been created or not, those error reports are usually just a warning.

      If I missed something or you have a file that I don't have (quite possible, as I'm running OMV on a Zyxel nsa325v2, ARM debian) please post below and I'll try to keep the OP updated.

      The post was edited 14 times, last by bobafetthotmail ().

    • Interesting work :) I think not many have pursued this because SSDs are so cheap and you can use it on a usb port. As you changed file box shows, it isn't OMV causing the problem really. It is Debian logging and collectd.
      omv 4.1.13 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • As you changed file box shows, it isn't OMV causing the problem really. It is Debian logging and collectd.
      Wat? Bulk of the writing is done in folder "openmediavault" and in folder called "rrcached" which is a caching daemon that stores performance data, used to make the pretty graphs as I said. You cannot realistically claim that it's not OMV's fault, man.

      That said, I'm not accusing OMV of anything, I'm just asking some hints about disabling this logging for my setup, and documenting my steps for posterity. Allowing OMV to be run on a USB stick can only be beneficial for its popularity.

      It just amazes me. This never ending quest to be a cheap bastard. I don't see why people would want to use a crappy medium when dealing with data that is likely important.
      Probably because FreeNas can do it. You know, dozens of disks, ECC ram, Supermicro mobos, ZFS file system... yet it is designed to be run from a USB flash drive. And if you ask in their forums, they usually use USB flash drives as OS disk.

      What makes OMV so special? Nothing. The fact it needs a HDD or a SSD as system disk to run what is basically an appliance is embarassing imho.

      With just a couple tweaks all unnecessary disk writes drop dramatically and you can use a USB flash drive safely, just like with plain Debian.
      There are tutorials to optimize Debian to run on CF cards without killing them, like this bernaerts.dyndns.org/linux/75-…bian-server-compact-flash
      It's not even hard

      FYI, USB flash drives have wear leveling, and had since 2008 at least. Wear leveling does not save them from death if there is something that constantly writes stuff on them for months on end, of course. (that's around 1-5TB of total writes vs nearly 100 TB for SSDs, probably because of differences in cheaper kinds of NAND technology used and in sheer capacity available to level wear in a SSD).

      Sources that say usb drives have wear leveling, dating back to 2008.
      zdnet.com/blog/storage/usb-drive-life-fact-or-fiction/849
      usenix.org/event/fast10/tech/full_papers/boboila.pdf
      bress.net/blog/archives/114-Ho…s-a-Flash-Drive-Last.html

      The post was edited 2 times, last by bobafetthotmail ().

    • You are free to use FreeNAS.

      You are funny. What good does wear leveling do if the stick has no where near the max read/writes as a SSD? Typically a user would be inclined to use a smaller usb stick too, thus less cells. Less cells equals faster death. On a SSD the whole disk is being used by the wear leveling. Most people do not realize this. I know what works and what fails. My SSD in my main system will be 3 years old in January (running 24/7). Most people using sticks have been wearing out within a few months, or less. I can only think of one user that made it just over 1 year. It is not worth the headache on a normal system. You will end up spending way more on the cost of usb sticks. Also, a good percentage of the users here are not concerned with the cost/benefits of usb stick vs. ssd. They are more concerned with stability. If you think using a SSD in OMV vs. a USB stick is an embarrassment, given all the benefits of OMV and the fact it's free, you have issues. OMV is special and so is the lead developer. If it were not I would not be here. :)

      PS- The average user cannot do the things you list above. And we do not plan on supporting it. Reliability and Stability are at the fore.
    • tekkb wrote:

      You are free to use FreeNAS.
      I'm also free to use OMV and post tutorials to make it better. Being linux what it is, if it does not my bidding i bend it to my bidding. Since I like to think I do stuff in Open source spirit I document what I do if someone could be interested.

      What good does wear leveling do if the stick has no where near the max read/writes as a SSD?
      Wear leveling ain't magic. a SSD has hundreds of GB of NAND to level the wear, a USB drive, not that much.
      Compared to a NAND on a embedded board that nlacks wear leveling it's still a MAJOR difference.

      Most people using sticks have been wearing out within a few months, or less.
      Considering that without any of the optimizations listed in the OP (and the stuff I'll do in the future) there is around 1MB of writes per few minutes due to OMV doing its own loggings and stuff, it ain't surprising.

      Your system is not optimized to reduce writes, as simple as that. I'm changing this.

      Also, a good percentage of the users here are not concerned with the cost/benefits of usb stick vs. ssd. They are more concerned with stability.
      USB drives aren't inherently less stable, just not suited for a high-write environment.
      Why you put them in a high-write environment? That's user error, don't blame the hardware.

      PS- The average user cannot do the things you list above.
      I was suspecting it.
      The plan was trying to attract more advanced users to get some feedback and maybe speed up the work.
      Feel free to move the thread in a place where more advanced users dwell.

      If I find the time and a smart way to deal with this issue, I might write a plugin for the masses that can't follow the tutorial.
      And we do not plan on supporting it.
      Your compliance is not necessary nor required (I just asked for hints, not a feature request), although would be certainly welcome.

      The post was edited 2 times, last by bobafetthotmail ().

    • It's really interesting to see that in this forum it's possible to find post like this OMV to the nines (hinting at the use of OMV for production/enterprise use) and posts trying to use tweak the system so that USB keys can be used for the OS...
      I wonder how many people using FreeNas in enterprise environment work with a USB stick for the OS. I don't think that many.

      instead of compromising the stability of the system would it not be better, faster, safer etc... to resize the OS partition so that you can use the OS drive for data as well as OMV?
      I'm aware that this is not a good solution since a system drive failure=possible data loss and vice versa (and people should be advised against using this solution!) but for some people with little room for expansion it may be better than relying on a cheap USB device...?
    • You give people something cool, you find it running on anything from teapots to spaceships. A good sign I guess.

      I wonder how many people using FreeNas in enterprise environment work with a USB stick for the OS. I don't think that many.
      If you look at Supermicro motheroards and HP prolian microservers, you see that both categories of products have an internal USB port soldered near the Sata connectors. That's expensive professional hardware, I doubt they place an internal USB port to attach a bluetooth dongle.

      Still, in the industrial sector they use DOMs (basically a industrial-rated SSD), don't know what Freenas users do in the commercial sector.

      instead of compromising the stability of the system would it not be better, faster, safer etc... to resize the OS partition so that you can use the OS drive for data as well as OMV?
      I'ts certainly another option, especially if you make a raid1 with identical partitions you place on all your drives. Grub boots a raid1 system afaik. In the embedded world... not that much.
      I'm still not intrigued by the excessive logging. The disk should go in spindown when not needed, not stay up forever just because it is logging its own activity.

      Due to the nature of mdadm, you can make a raid 1 of USB drives, or simply keep another ready as a backup. But frankly, as long as you aren't using a chinese nobrand pos drive it should last forever as long as the writes are kept in check.

      The post was edited 1 time, last by bobafetthotmail ().

    • If you use raid 1 with usb sticks, it won't help. Each drive gets the same amount of writes. As soon as one drive starts getting bad sectors, the corrupted data will be mirrored to the other drive.
      omv 4.1.13 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • Is your system working from a USB stick with the reduced writes? Are you going to report on the stability of your setup?
      It is working from usb stick, yes. If something bad happens I will report it.

      Before changing the title of the thread to “tutorial” you should at least run some trials and show that the described process produces a working/reliable system.
      A good point. Will add a disclaimer stating that for now it's experimental.

      ryecoaaron wrote:

      If you use raid 1 with usb sticks, it won't help. Each drive gets the same amount of writes.
      Yeah, but failure point is random.

      As soon as one drive starts getting bad sectors, the corrupted data will be mirrored to the other drive.
      flash drives tend to lock and become read-only when failed.
    • I have had many sd cards fail when using them with my RPi (building OMV images). Rarely do they do the same thing. Sometimes you get read only. Most of the time, you get sporadic errors when trying to do something in the web interface. So, I still think raid1 will not help.
      omv 4.1.13 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • Maybe I can interject with a comment :) Let's distinguish between the capability and principle of (not) using USB flash drives as boot disks. As stated, OMV is aligned with the principle that the normal functioning of the platform requires a hard drive for the operating system, and USB flash drives can not sustain the amount of data being written. It's a decent choice and it has been clearly stated - just as some other choices regarding which other functionalities make sense for a file server and should be made available through plug-ins, and which not.

      Yet here we have a system designed for the flexibility to meet any niche requirement or even personal quirk :D As long as someone wants to invest time in adding the new capability and making it work, they're free to do so, and this creates no obligation for OMV. Personally I salute anyone mad curious enough to get their hands dirty, even if the projected end result is a Frankenstein. I mean, I have a friend who purchased an used router - a specialized device custom built for enterprise customers of a telecom company, focusing on reliable 3G and enhanced VPN capabilities. He wants to hack it to add a parallel printer interface to it and make it a print server. He's definitely bonkers for even considering such a radical change for the existing device, but hey - kudos to him for trying. :)

      This being said, the option to boot from a flash drive and having a way to minimize writes (to it) does sound appealing. However, I'd raise you this proposal: why not use the USB flash drive for the OS and user settings, and reserve some space on the storage disks for work files, logs, cache and so on?

      Normally a file server will have storage attached to it if you intend to use it, so you could block 100MB or whatever seems fit for temporary data (or persistent storage flushed periodically from the ramdisk). And if the server doesn't have any storage attached, then it doesn't fulfill its purpose anyways and you can safely assume that it can use a ramdisk when powered on and disregard most of that data when powered off. (Insert here the discussion about which changes need to be written on the flash disk and retained from one boot to the other, ie patches and updates, vs. which temporary files, uptime statistics etc. can be ignored.)
    • "Glowing days. Don't cry because they are over. Smile because they happened." - Confucius

      Server: 1x 32GB SSD (system) - 5x 2TB Data - 1x 2TB Snapraid-Parity - latest OMV 1.x
      No Support through PM
      Tutorials --- Howto install OMV-Extras --- Upgrade/Update-Problems --- If autoshutdown doesn' -work
    • I have had many sd cards fail when using them with my RPi (building OMV images). Rarely do they do the same thing.
      Dunno. USB drives I've killed on purpose (scripting with rsync) simply went read-only after their "death". Maybe USB devices are better?

      puterfixer wrote:

      why not use the USB flash drive for the OS and user settings, and reserve some space on the storage disks for work files, logs, cache and so on?
      Most of the temporary stuff I detected should never have gone to disk in the first place, period.
      OMV generates ridicolous amounts of stuff through rrcached, and since they are not useful on reboot that's just keeping the drive working for no reason. Even my crappy embedded nas has 512 mb of ram, let's use a fraction of it for a good reason, shall we?

      I can see logs being flushed to internal disk every now and then, it's just a simple rsync cronjob. Might whip together a script or something, later.

      Have a look at this also: phpbb.openmediavault.org/viewt…2461d19b7b91bdbb9bd6e6fb8
      oh boy. hundreds of mb of temp stuff per hour? Good to see I'm not the only one noticing an issue. :D
    • Your rsync script probably keeps writing when small problems start which might push it to read only. These are easily noticed when you are interacting with the system. Before my RPi, I never killed an SD card but had many usb sticks fail.

      OMV uses the default settings for collectd and that is why they go to the disk. Just turn off collectd (rrdcache) if it is that big of an issue. Most of the graphs (cpu, load, memory) are probably of little use on arm system since they are typically high doing anything. With collectd disabled, there really isn't that much logging either. Just because a file was modified in the last five minutes doesn't mean it was a lot of writes.
      omv 4.1.13 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • Shedding some light on the matter myself then, because my NAS has only two bays and I don't exactly feel like having both drives constantly spinning just to write logs and random trash, nor having a secondary USB HDD wasted for a 3-4GB partition.


      Also, good hard drives don't need to be powered down. It is bad for them. They are designed to run 24/7. I use Enterprise class Hitachi's which are never spun down. If they make it to me in one piece I never had an issue with them.
    • ryecoaaron wrote:

      Just turn off collectd (rrdcache) if it is that big of an issue. Most of the graphs (cpu, load, memory) are probably of little use on arm system since they are typically high doing anything. With collectd disabled, there really isn't that much logging either.
      I´m really thinking about turning them off, too. I never cared about any graphs on my system...
      I´ve created a feature request: bugtracker.openmediavault.org/view.php?id=1193
      OMV stoneburner | HP Microserver | 256GB Samsung 830 SSD for system | 4x 2TB in a RAID5
      OMV erasmus| Odroid XU4 | 5TB Data drive | 500GB Backup drive