[Tutorial][Experimental][Third-party Plugin available]Reducing OMV's disk writes, also to install it on USB flash

Please note, ryecoaaron made a plugin to automate this procedure, see this post for details [Tutorial][Experimental]Reducing OMV's disk writes, also to install it on USB flash

----------------------back to original first post--------------------

I'm a bit puzzled that it's none's priority to tame a bit OMV to allow the use of USB flash drives for OS drive like they do for FreeNas. Or even to let hard drives rest a bit.

Shedding some light on the matter myself then.

WARNING: This system is experimental, that is, it has not been throughly tested on multiple stations to ensure everything works fine in all situations.
I CANNOT GUARANTEE DATA SAFETY. DO BACKUPS OR DO NOT TRY IT.
Volunteers that test this system are welcome.

Let's track down what files got written often, with a script I took from here http://info4admins.com/tips-to…disk-in-debian-or-ubuntu/

#!/bin/sh
echo Search whole filesystem for changed files in the last $1 minutes
find /bin /dev /etc /home /lib /media /mnt /opt /root /sbin /srv /tmp /usr /var -cmin -$1

Place this in a /bin folder you create in your /home folder, then chmod a+x the script to make it executable.
Then you can call it by writing its name and the number of minutes you want.
ScriptName 5 will give you an output listing files written in the last 5 minutes
this is a real-life example:

Search whole filesystem for changed files in the last 5 minutes
/dev/xconsole
/etc/samba
/etc/samba/dhcp.conf
/tmp
/var/spool/postfix/maildrop
/var/spool/postfix/maildrop/8C728FC9A
/var/lib/monit/state
/var/lib/openmediavault/rrd
/var/lib/openmediavault/rrd/cpu-0-week.png
/var/lib/openmediavault/rrd/df-root-week.png
/var/lib/openmediavault/rrd/interface-eth0-hour.png
/var/lib/openmediavault/rrd/cpu-0-day.png
/var/lib/openmediavault/rrd/cpu-0-hour.png
/var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-day.png
/var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-month.png
/var/lib/openmediavault/rrd/cpu-0-year.png
/var/lib/openmediavault/rrd/interface-eth0-week.png
/var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-week.png
/var/lib/openmediavault/rrd/load-day.png
/var/lib/openmediavault/rrd/df-root-month.png
/var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-year.png
/var/lib/openmediavault/rrd/interface-eth0-year.png
/var/lib/openmediavault/rrd/load-hour.png
/var/lib/openmediavault/rrd/df-root-year.png
/var/lib/openmediavault/rrd/cpu-0-month.png
/var/lib/openmediavault/rrd/df-media-a4db3b26-75a4-4677-b32e-0abbe0e9c62c-hour.png
/var/lib/openmediavault/rrd/interface-eth0-month.png
/var/lib/openmediavault/rrd/interface-eth0-day.png
/var/lib/openmediavault/rrd/df-root-hour.png
/var/lib/openmediavault/rrd/df-root-day.png
/var/lib/rrdcached/db/localhost/interface-eth0/if_octets.rrd
/var/lib/rrdcached/db/localhost/interface-eth0/if_packets.rrd
------------a bunch of entries removed for space reasons----------
/var/lib/rrdcached/db/localhost/cpu-0/cpu-softirq.rrd
/var/lib/rrdcached/db/localhost/cpu-0/cpu-idle.rrd
/var/lib/rrdcached/db/localhost/cpu-0/cpu-steal.rrd
/var/lib/rrdcached/db/localhost/cpu-0/cpu-user.rrd
/var/lib/rrdcached/db/localhost/cpu-0/cpu-nice.rrd
/var/lib/rrdcached/db/localhost/cpu-0/cpu-system.rrd
/var/lib/rrdcached/db/localhost/cpu-0/cpu-interrupt.rrd
/var/lib/rrdcached/db/localhost/cpu-0/cpu-wait.rrd
/var/lib/rrdcached/journal/rrd.journal.1415583612.312057
/var/log/debug
/var/log/auth.log
/var/log/daemon.log
/var/log/syslog

The bulk of that output you see is mostly performance logging to allow OMV to make pretty graphs, and pre-made images of graphs for a page in the webGUI. And it is done pretty often, waay too often.

So, now we have a general idea of what is going on in the disk and we can start our countermeasures.

The standard practice is to disable the logging from the application's settings/config file, or to move the whole offending folder to RAM. It means any write in that folder will go in RAM instead than on disk in a completely magical way, but this also means that if the NAS reboots or crashes you will lose all modifications you made to those folders.
Since on average it's logging or temporary files, it's no big loss.
If you want to save logs or some folder contents from the tmpfs you made above have a look at this script from Solo0815 [Tutorial][Experimental]Reducing OMV's disk writes, also to install it on USB flash

Open /etc/fstab with a text editor

sudo nano /etc/fstab

if yu are already root don't write sudo.
this is an example fstab

# /etc/fstab: static file system information.
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/sda1      /              ext2 noatime,errors=remount-ro                     0     1
tmpfs          /tmp           tmpfs      defaults,noatime            0     0

Add the noatime option to the root partition, delete any "realtime" option. This turns off the logging of the last access time of a file (that causes writes every time a file is read)
As you see in the example fstab I added only /tmp folder as a tmpfs (ram disk).
Adding other folders tends to cause problems, sadly. For example, adding /var/cache and /var/log would result in annoyances from apt and nginx and probably something else that will fail horribly or crash if they don't find their things and log files.

We need a script that mounts the appropriate folders as tmpfs and fills them with data and directory structure from a backup, then at shutdown takes the data from the tmpfs and makes a backup on disk (or not, it really matters only when booting).
Like explained here https://www.debian-administrat…/661/A_transient_/var/log

Thankfully, there is a premade package of scripts called fs2ram that does that automagically for a list of folders in its config. Not available on Wheezy, but available in Sid, here you find the .deb, install manually. https://packages.debian.org/sid/admin/fs2ram
from terminal it's this (if the package is updated in the repos and the link breaks, use the link above to find the correct download link to use):

wget http://ftp.us.debian.org/debian/pool/main/f/fs2ram/fs2ram_0.3.12_all.deb
sudo dpkg -i fs2ram*

(if you are already root don't write sudo)

Once installed it will ask you if you prefer to save logs at shutdown or just to rebuild folder structure on boot in the tmpfs. Choose the one you prefer.

Time to work on folder /var/lib/openmediavault/rrd and the /var/spool
All this stuff goes directly in fs2ram's own config file, that looks suspiciously like fstab.

sudo nano /etc/fs2ram/fs2ram.conf

to open it and the follwoing is an example:

#<file system>  <mount point>   <script>                <script option> <type>  <options>
tmpfs            /var/log        keep_file_content       -               tmpfs
tmpfs            /var/tmp        keep_file_content       -               tmpfs
tmpfs           /var/lib/openmediavault/rrd     keep_file_content       -               tmpfs
tmpfs           /var/spool                      keep_file_content       -               tmpfs
tmpfs           /var/lib/rrdcached/             keep_file_content       -               tmpfs
tmpfs           /var/lib/monit                  keep_file_content       -               tmpfs
tmpfs           /var/lib/php5                   keep_folder_structure   -               tmpfs

Spot the differences between this list and the standard list in your config file.

You could theoretically add also tmpfs /var/cache keep_file_content - tmpfs, but I find it unnecessary.
I commented it out because I have only 512 MB of ram in my NAS, and /var/cache contains the apt-get package cache (and little else).
If you want to keep it in a tmpfs make sure you are running apt-get autoclean in a cronjob, or you are keeping the package cache size in check somehow.

Old tutorials usually talk about symlinking the mtab with /proc/mounts, but this was already done upstream in Debian, so we don't need to do anything about it.

Done. Reboot the system for changes to take effect, and if it does boot up again and is still operational, pat yourself on the shoulder, Good Job!

Look in logs to see if f2ram is reporting errors about some folder, and if there is any program complaining that it cannot start because it needs something.
Some programs/services will complain saying they did not find file X, check if the file has been created or not, those error reports are usually just a warning.

If I missed something or you have a file that I don't have (quite possible, as I'm running OMV on a Zyxel nsa325v2, ARM debian) please post below and I'll try to keep the OP updated.

Quote

As you changed file box shows, it isn't OMV causing the problem really. It is Debian logging and collectd.

Wat? Bulk of the writing is done in folder "openmediavault" and in folder called "rrcached" which is a caching daemon that stores performance data, used to make the pretty graphs as I said. You cannot realistically claim that it's not OMV's fault, man.

That said, I'm not accusing OMV of anything, I'm just asking some hints about disabling this logging for my setup, and documenting my steps for posterity. Allowing OMV to be run on a USB stick can only be beneficial for its popularity.

Quote

It just amazes me. This never ending quest to be a cheap bastard. I don't see why people would want to use a crappy medium when dealing with data that is likely important.

Probably because FreeNas can do it. You know, dozens of disks, ECC ram, Supermicro mobos, ZFS file system... yet it is designed to be run from a USB flash drive. And if you ask in their forums, they usually use USB flash drives as OS disk.

What makes OMV so special? Nothing. The fact it needs a HDD or a SSD as system disk to run what is basically an appliance is embarassing imho.

With just a couple tweaks all unnecessary disk writes drop dramatically and you can use a USB flash drive safely, just like with plain Debian.
There are tutorials to optimize Debian to run on CF cards without killing them, like this http://bernaerts.dyndns.org/li…bian-server-compact-flash
It's not even hard

FYI, USB flash drives have wear leveling, and had since 2008 at least. Wear leveling does not save them from death if there is something that constantly writes stuff on them for months on end, of course. (that's around 1-5TB of total writes vs nearly 100 TB for SSDs, probably because of differences in cheaper kinds of NAND technology used and in sheer capacity available to level wear in a SSD).

Sources that say usb drives have wear leveling, dating back to 2008.
http://www.zdnet.com/blog/stor…-life-fact-or-fiction/849
https://www.usenix.org/event/f…h/full_papers/boboila.pdf
http://www.bress.net/blog/arch…s-a-Flash-Drive-Last.html

Quote from tekkb

You are free to use FreeNAS.

I'm also free to use OMV and post tutorials to make it better. Being linux what it is, if it does not my bidding i bend it to my bidding. Since I like to think I do stuff in Open source spirit I document what I do if someone could be interested.

Quote

What good does wear leveling do if the stick has no where near the max read/writes as a SSD?

Wear leveling ain't magic. a SSD has hundreds of GB of NAND to level the wear, a USB drive, not that much.
Compared to a NAND on a embedded board that nlacks wear leveling it's still a MAJOR difference.

Quote

Most people using sticks have been wearing out within a few months, or less.

Considering that without any of the optimizations listed in the OP (and the stuff I'll do in the future) there is around 1MB of writes per few minutes due to OMV doing its own loggings and stuff, it ain't surprising.

Your system is not optimized to reduce writes, as simple as that. I'm changing this.

Quote

Also, a good percentage of the users here are not concerned with the cost/benefits of usb stick vs. ssd. They are more concerned with stability.

USB drives aren't inherently less stable, just not suited for a high-write environment.
Why you put them in a high-write environment? That's user error, don't blame the hardware.

Quote

PS- The average user cannot do the things you list above.

I was suspecting it.
The plan was trying to attract more advanced users to get some feedback and maybe speed up the work.
Feel free to move the thread in a place where more advanced users dwell.

If I find the time and a smart way to deal with this issue, I might write a plugin for the masses that can't follow the tutorial.

Quote

And we do not plan on supporting it.

Your compliance is not necessary nor required (I just asked for hints, not a feature request), although would be certainly welcome.

You give people something cool, you find it running on anything from teapots to spaceships. A good sign I guess.

Quote

I wonder how many people using FreeNas in enterprise environment work with a USB stick for the OS. I don't think that many.

If you look at Supermicro motheroards and HP prolian microservers, you see that both categories of products have an internal USB port soldered near the Sata connectors. That's expensive professional hardware, I doubt they place an internal USB port to attach a bluetooth dongle.

Still, in the industrial sector they use DOMs (basically a industrial-rated SSD), don't know what Freenas users do in the commercial sector.

Quote

instead of compromising the stability of the system would it not be better, faster, safer etc... to resize the OS partition so that you can use the OS drive for data as well as OMV?

I'ts certainly another option, especially if you make a raid1 with identical partitions you place on all your drives. Grub boots a raid1 system afaik. In the embedded world... not that much.
I'm still not intrigued by the excessive logging. The disk should go in spindown when not needed, not stay up forever just because it is logging its own activity.

Due to the nature of mdadm, you can make a raid 1 of USB drives, or simply keep another ready as a backup. But frankly, as long as you aren't using a chinese nobrand pos drive it should last forever as long as the writes are kept in check.

Quote

Is your system working from a USB stick with the reduced writes? Are you going to report on the stability of your setup?

It is working from usb stick, yes. If something bad happens I will report it.

Quote

Before changing the title of the thread to “tutorial” you should at least run some trials and show that the described process produces a working/reliable system.

A good point. Will add a disclaimer stating that for now it's experimental.

Quote from ryecoaaron

If you use raid 1 with usb sticks, it won't help. Each drive gets the same amount of writes.

Yeah, but failure point is random.

Quote

As soon as one drive starts getting bad sectors, the corrupted data will be mirrored to the other drive.

flash drives tend to lock and become read-only when failed.

Quote

I have had many sd cards fail when using them with my RPi (building OMV images). Rarely do they do the same thing.

Dunno. USB drives I've killed on purpose (scripting with rsync) simply went read-only after their "death". Maybe USB devices are better?

Quote from puterfixer

why not use the USB flash drive for the OS and user settings, and reserve some space on the storage disks for work files, logs, cache and so on?

Most of the temporary stuff I detected should never have gone to disk in the first place, period.
OMV generates ridicolous amounts of stuff through rrcached, and since they are not useful on reboot that's just keeping the drive working for no reason. Even my crappy embedded nas has 512 mb of ram, let's use a fraction of it for a good reason, shall we?

I can see logs being flushed to internal disk every now and then, it's just a simple rsync cronjob. Might whip together a script or something, later.

Quote

Have a look at this also: phpbb.openmediavault.org/viewt…2461d19b7b91bdbb9bd6e6fb8

oh boy. hundreds of mb of temp stuff per hour? Good to see I'm not the only one noticing an issue.