Evaluation of File System Performance on OMV
Introduction
I wanted to completely reinstall all my NAS using OMV. Doing so I decided it would be a good opportunity to think about the used file systems, backup and how I could use my existing HDDs the best possible way.
I did some extensive testing. Most of them in a NAS with 6 HDDs and with most of the file systems available in OMV. I tested Ext4, XFS, JFS and BTRFS. I did not do any tests with ZFS.
I tested different types of data ranging from very small files to multi gigabyte files. I also explicitly tested btrfs send/receive, since I had realized, that this has its own I/O-usage profile and was not covered good enough by just copying some files.
Since I have HDDs of different sizes, I was interested in building a raid-5 with those HDDs. BTRFS is capable of doing so, but (as of this writing in summer 2024) this is not yet considered stable for production use. Therefor I was also looking for some alternatives. I found something called Sliced Hybrid Raid (SHR). I also did some modifications on that concept, mainly trying to improve performance.
To cut a long story short (tl;dr)
- Ext4 and JFS are slower than XFS and BTRFS.
- As long as you only deal with small files, that fit into the Linux cache (in my case up to a size of 2-3 GB), the underlying technology doesn’t really matter. They are all similarly fast (due to the file cache).
- MergerFS has a visible impact on performance.
- For multi gigabyte files you will notice the different technologies of redundancy. Native Raids are all similarly fast, no matter if built with mdadm, lvm or BTRFS. In contrast, SHR and the Sliced solutions have a significant dropdown in performance (up to 10 times slower in my case). BTRFS Raid-0 on top of Slices was the fastest of them (only factor 2 slower).
- Used with a big enough cache on SSD (lvmcache or bcache) SHR and the Sliced solutions are all similar fast again. I’d recommend bcache applied on the slices for this scenario.
I have attached to this entry post, the raw data as cvs file (the .txt file) and an Excel File (the .zip file) with the same data and the visualization, I used for this guide. I also wanted to upload a PDF version of this guide, but the PDF file is slightly bigger than 1 MB and I wasn't allowed to upload it.