Pool configuration and format type suggestion

1st Official Post

    Hi I'm about to build a new nas since a long time, my last omv configuration it's basically going strong since 6 years so it's been some time since I build one from scratch.

    It is composed basically by 3 hard drivers in raid1 formatted in ext4 (1+1 and one hotspare all of them of 1 tb each) for the users data and documents and another hard drive of 2tb for backup through rsnapshot all of them formatted in ext4.

    As of now I think this configuration is not the best and I was thinking to ask for some suggestions for a build that could make sense for my use case.

    What I want to achive is to have enough space for the users to put documents and files (no video nor jellyfin nothing like that it will be used mostly as a nas to serve internally files for work and a couple of services like duckdns, nginx and nextcloud to serve remote collaborators)

    I was wondering what is as of today the best configuration possible to achive with the hardware I have?

    my costraints are:

    3x hard disk wd red 2tb each (that I would like to use for data and\or redundancy, I might get a 4th one if totally necessary, but I would rather not)

    1x hard disk 4tb ironwolf (that I would like to use to backup all the data from the pool)


    • My questions are which technology to use?

      For the data raid? which one? or mergerfs+snapraid? or even more zfs?! and what format? ext4, btrfs, something else?

      For the backup solution of the biggest drive instead is using rsnapshot still the best way to go or something else?


      Thank you in advance for any suggestion :)
    Quote from Wek

    For the backup solution of the biggest drive instead is using rsnapshot still the best way to go or something else?

    I had rsnapshot (on Ext4) as my backup solution in place for years. Recently I decided to switch all my NASes to OMV. Doing so I also decided to go with BTRFS and using the mechanisms of BTRFS for backup.

    I found btrbk a useful backup solution for BTRFS.


    The other solution I had in mind for backup was Borgbackup. Would have been my plan B if BTRFS didn't work out.

    There is some tool for Borgbackup to import old rsnapshot backups.


    When using BTRFS have in mind, that it is completely different than Ext4 (e.g. no need for mdadm for raid, BTRFS manages this on its own). You must familiarize yourself with it before you use it.

    TerraMaster T6-423

    Qnap TS-853A

    Qnap TS-451+

    Qnap TS-259 Pro+

    • Official Post

    I'm running ZFS on a few servers. In the ZFS world, I've replaced drives and done pool upgrades without issues. Over the years, when drives began to fail, ZFS data integrity features kept the primary server's data clean.
    I've setup automated snapshots that I've used for retrieving prior versions of files or for deleted file restorations. (Supporting -> Document.)

    Frankly, I can't speak to BTRFS. While I have BTFRS running in RAID1, I haven't had it long enough to evaluate drive replacement, data integrity features, restorations from snapshots, etc.

    __________________________________________________________________

    If you're going to keep your backup disk within the same server, or connected with a drive dock, you might consider rsync'ing a 4TB backup disk to a 4TB pool. It's easy to set up as outlined -> here and, in the event of failure of 4TB pool, it's easy to transfer shares to the backup disk (minutes not hours). This will restore your server quickly and easily, until the pool can be repaired. When the pool is repaired, if data is lost, it can be rsync'ed back to the pool, from the backup disk, by reversing the rsync command line.

    (BTW: the only difference in the command line for rsync'ing a ZFS pool to a single disk backup, is rsync'ing the pool's mount point (the source) to the disk (the destination).
    Here's an example command line:
    rsync -av --delete /ZFS1/ /srv/dev-disk-by-uuid-dfa2df17-9764-4a47-8e9d-9a36ff74ca37/

    thank you very much Quacksalber I already got btfrs on my linux workstation, so I'm some familiar with it, I just didn't know if it was now mature enough to integrate it to OMV, from what I remembered a while ago btrfs wasn't fully supported yet in the GUI for things like raid construction\recostruction and snapshots, and I had to do it through the shell instead.

    Is this situation changed in omv7?

    and thanks also to crashtest I was valuating about zfs as well especially in conjunction with ecc ram to build something secure for data integrity,
    my main concern with it is from what I can see in omv is only supported through an omv-extras plugin of openzfs is that right?

    How's your experience with it? I'm a bit concerned about future development or stability of it being built on an external plugin, can you tell me your experience?

    Plus as well what is the situation for support of it? is it manageable from the GUI the raid\restoration\snapshots features or is something that could be done only through shell (I saw your documentation related to automatic snapshots thanks)?

    Kind regards really useful food for thoughts :)

    • Official Post
    Quote from Wek

    1. and thanks also to crashtest I was valuating about zfs as well especially in conjunction with ecc ram to build something secure for data integrity,

    2. my main concern with it is from what I can see in omv is only supported through an omv-extras plugin of openzfs is that right?

    1. When it comes to data integrity, I don't think it gets better than ZFS. To say that ZFS is well developed and mature is an understatement. ZFS is the Granddaddy of copy-on-write check summed file systems. I've actually tested ZFS with errors introduced using a drive sector editor. ZFS' data integrity features work and they work well. On the other hand BTRFS, whether it's admitted or not, still has some fairly serious issues with little to no progress being made on correcting them.

    2. Yes. The plugin is necessary because of licensing nonsense. That's the simple truth of it. OpenSolaris and ZFS were published under an open source license as of 2005. The argument boils down to (get this) the type of "free license" ZFS is under. It's that petty.

    In practical terms:
    Ubuntu integrates ZFS - Debian does not. Because OMV is built on Debian, the plugin approach is needed for legal reasons. Personally, I'm of the opinion that ZFS is not going anywhere. The openZFS project is sponsored by Lawrence Livermore Labs (US Government)which means the project has deep pockets support.

    Quote from Wek

    How's your experience with it? I'm a bit concerned about future development or stability of it being built on an external plugin, can you tell me your experience?

    After acclimating to OMV (ver2), I went in search of best practices for data intergrety. That led to a server with ECC RAM and ZFS. I believe I've been running ZFS since ver3 or 4. In other words, several years. I've upgraded pools a few times, replaced a couple of aging drives, pulled back deleted files and previous file versions from snapshots, etc. All of it went smoothly.


    In my experience, ZFS will let you know that a drive is becoming marginal before SMART starts registering issues. After a scrub, you'll see corrected data and errors. It's laid out by drive. It two cases, over a bit of time, it became pretty obvious that a drive needed to be replaced. In the bottom line, nothing was corrupted (as in unrecoverable errors) and I haven't lost anything that would force me to recover from backup.

    At this point, I'm running 3 ZFS pools with 1 vdev each. 2 are Zmirrors and 1 is RAIDZ1 (that's the mdadm RAID equivalent of RAID5). My preference is for Zmirrors (the equivalent of RAID1).

    Quote from Wek

    1. Plus as well what is the situation for support of it?
    2. is it manageable from the GUI the raid\restoration\snapshots features or is something that could be done only through shell (I saw your documentation related to automatic snapshots thanks)?

    1. 90 to 95% of anything you might want to do, as far as setting features, could be done using the GUI. Mostly, once the pool and each filesystem is set up, it's on autopilot. However, it is important to setup certain features before copying data into a pool or filesystem. Otherwise files may have mixed attributes.

    I will say that I prefer to remove a drive and add a replacement using the command line. That insures accuracy. Given the maturity of ZFS there's plenty of resources out there for commands and to support basic operations, like drive replacement.


    2. Snapshots are displayed in the GUI, even those taken by automated packages like the one I documented (zfs-auto-snapshot). As shown below, snapshots can be cloned, rolled back or deleted in the GUI. While it might be useful to go back to a time before a ransomware attack, I have yet to roll back or clone a filesystem snapshot. (But I appreciate that the feature is there, if needed.)

    What I have done is made snapshots directories visible and I've used a network client to selectively pull older or deleted files and folders from a given snapshot. How to retrieve files and folders is in the zfs-auto-snapshot document.


    _____________________________________________________________________________________________________

    With the above noted, there is a learning curve associated with what ZFS is and what it can do, but that understanding is not unlike what would be needed to understand BTRFS.

    If you do decide to go with ZFS, I would proceed as follows:


    - Install the kernel plugin.
    - Install the Promox kernel, select it and boot up on it. (This kernel has ZFS modules built into it.)
    - Remove non-proxmox kernels (Optional)
    - At this point, install the ZFS plugin.
    _____________________________________________

    - Create a pool.

    After creating the pool, run the following on the command line (you could do this in the GUI but it's easier on the CLI)

    Sub in your pool name, in place of ZFS1

    zfs set aclmode=passthrough ZFS1

    zfs set aclinherit=passthrough ZFS1

    zfs set acltype=posixacl ZFS1

    zfs set xattr=sa ZFS1

    zfs set compression=lz4 ZFS1


    - Highlight the pool, in the GUI, and create filesystems.
    (A "filesystem" is the equivalent of a Linux folder, at the root of the pool, but it has editable ZFS properties and separate snapshots.)
    ____________________________________________________________________

    On the other hand, ZFS is not EXT4. While it's fairly easy to get started, a learning curve is involved. Obviously there's more to ZFS, when compared to a regular filesystem, but that's the entire point of it - "more features".

    thank you crashtest for the detailed explanation, the zfs approach seems really good and interesting I will fireup a test machine and study it, before deploy but if it's well integrated in omv as you are saying seems a great solution.

    Last question, the utilization of zfs snapshots from what I got from you, it makes redundant to setup an hard disk dedicated solely to backup with rsnapshot plugin then? (or borgbackup) or am I mistaken your words? so from my OP the setup with a 4tb hard disk with the function of just backup for the files in the pool in this case would be just redundant? if I use zfs snapshots?

    Wek If you're thinking about using BTRFS raid profiles, keep in mind that only profiles raid1 and raid10 are production ready. With just three data drives that means you could only use BTRFS raid1 profile which would give 3TB of available storage on 3 x 2TB HDDs.


    If you used ZFS, a raidz1 pool on 3 x 2TB HDDs gives you 4TB available storage and single disk redundancy the same as mdadm RAID5.


    The zfs set commands listed above and no longer necessary. Changes were made to the zfs plugin some months ago so that pools created via the WebUI already have these settings. e.g:


    Code
    root@omvt:~# zpool history | grep create
2024-09-01.09:39:48 zpool create -o ashift=12 -o failmode=continue -o autoexpand=on -O atime=off -O acltype=posix -O xattr=sa OMVpool mirror scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-1
root@omvt:~# zfs get compression OMVpool
NAME     PROPERTY     VALUE           SOURCE
OMVpool  compression  lz4             local
root@omvt:~#

    The dataset properties of aclmode and aclinherit can be left at their defaults unless you actually make use of posix.1e type ACLs. Using ACLs is a rabbit hole that most OMV user should avoid. The acltype is only set, along with xattr, for compatability with the design of OMV which expects all filesystem to support ACLs without necessarily using them.


    ZFS snapshots, like BTRFS snapshots, are not a substitute for proper backups on an external device. The 4TB HDD you mention could be that external device.

    Great Krisbee thank you very much for the heads up, and answered my other doubt sounds a good plan then, about ACLs is not a problem in this case since I tend to not use them, despite I know my way around them, I prefer to keep it simple, so I just used permissions for the share share folders for users and so far it's sufficient for what the production needs, but great to know :)

    I really appreciate all your inputs, it sounds like I will give it a whirl then.
    Also thanks for the explanation of the proxmox kernel, I was wondering what is that all about.
    It seems from your comments that is a cool addition to manage zfs

    • Official Post
    Quote from Wek

    Last question, the utilization of zfs snapshots from what I got from you, it makes redundant to setup an hard disk dedicated solely to backup with rsnapshot plugin then? (or borgbackup) or am I mistaken your words?

    As already noted, you'll need backup regardless. There are plenty of hardware failure scenarios that could wipe out a ZFS pool.

    On the the hand, whether that backup should be versioned, such as what would be available with rsnapshot or borgbackup could be debated. Since ZFS provides versioned files in it's pool / filesystems a simple, one-to-one, rsync'ed backup would be fine in my opinion.

    I've been meaning to set up ZFS to send snapshots to another pool, to get the full spectrum of snapshots take at my primary server, sent and available at a backup server, but I haven't got around to doing it.

    to be honest my backup concern was more from a "non techy" users perspective,

    exactly the ones that will use the nas.


    Since tehre are some people in the team that have the tendency to delete files that they shouldn't touch, and everytime they call me to make their files reapper, not even the recycle of the samba share is sufficient to mitigate their armageddon.

    ...so my idea to get rid of all this cumbersome of managing every single cry and destuction, is to create a rotational backup on another hard disk and with the file browser plugin make a pratical access for them to access the backup and retrieve whatever they can manage to destroy.

    All of this without giving them any access to the OMV gui or even worst to shell to manage zfs :°D

    I guess this way would be almost idiot proof for them to not cause too many issues, plus I retain the zfs snapshot for myself when they really really really really mess up...other than that...only GOD can save them :D

    • Official Post
    Quote from Wek

    Since tehre are some people in the team that have the tendency to delete files that they shouldn't touch, and everytime they call me to make their files reapper, not even the recycle of the samba share is sufficient to mitigate their armageddon.

    ZFS and zfs-auto-snapshot is the exact prescription you're looking for.

    We'll use a "documents" share as an example. zfs-auto-snapshot can be setup to take daily (31 each), weekly (7 each) and monthly (12) snapshots. (While these are default retention periods, all of this is customizable. Note that if all time intervals are active we're taking about a LOT of snapshots. As a matter of personal preference; to minimize snapshot confusion when I need to find something, I don't capture "frequent", "hourly" or "weekly".) As it is, when capturing 31 daily's and 12 monthly's for the pool and all filesystems, we're looking at lot of snapshots.

    12 monthly snapshots means that the prior states of the file system (with all files and folders) are retained for up to 1 entire year. That means all files and even their prior version states are captured and retained, even if they're deleted by the user. After 1 year, user deleted files are finally purged. I can't imagine retaining more than a years worth of monthly snapshots but the retention period could be extended to two years or more.

    File or folder retrieval is easy, as shown -> here in the zfs-auto-snapshot doc. It's a matter of unhiding the .zfs snapshot directory at the server, going to a network client and finding the snapshot named with the desired time frame, then copy and paste the deleted or older version file back into the current filesystem. There would be no need to go to backup.
    ________________________________________________________________________________________

    Backing up data to another server is another matter but, in a business environment, a good argument could be made for replicating the pool and filesystems to another server, using the ZFS send command. Since that I have multiple backups and I'm not in need of such comprehensive snapshot options on a backup server, I go with simple backup.


    Quote from Wek

    I guess this way would be almost idiot proof for them to not cause too many issues, plus I retain the zfs snapshot for myself when they really really really really mess up...other than that...only GOD can save them :D

    I would have loved to have these options way back in the day. If I had to retrieve something for a bonehead user, I had to pull it from tape.

    crashtest definitely I got you the zfs snapshot feature seems perfect, but again I would use it just for me, as it involves going in the shell making the .zfs folder visible etc etc retrieve the file and hide back everything, this would require my interventon.

    instead the 4th drive backup solution will be basically already set it up with the file browser plugin so they can recover whatever they want without my help ever.

    And of course I will use the snapshots of zfs for myself to recover a more serious disaster if I need :)

    Seems doable

    Quote from Wek

    btrfs wasn't fully supported yet in the GUI for things like raid construction\recostruction and snapshots, and I had to do it through the shell instead.

    With the exception of Raid-5 & 6 you can create all RAIDs using the Web-UI. There is also some functionality for snapshots (on the same disk), but not for backups (on another disk).

    I have not found any support in the Web-UI for manipulation of RAIDs in BTRFS (adding, removing, replacing disks; growing, shrinking).

    The advantage of BTRFS is, that it can combine disks of different sizes in one RAID.

    TerraMaster T6-423

    Qnap TS-853A

    Qnap TS-451+

    Qnap TS-259 Pro+

    Like 1

    thank you Quacksalber I will look into this as well as it seems cool, although the issue that omv doesn't support snapshots and managing raids through the GUI was the problem here so I guess I will go the zfs way
    My last concern of all of it.

    Is there a way to "migrate" the configurations of fstab to the new pool created in zfs or to the new btrfs system without recreate every link or mount inside the GUI for applications\services\plugins that where using some specific UUID?

    What I mean is after all the tests I will do how can I switch smoothly to the new filesystem without make omv lost its mind?

    - As of now I have my main os on a separate nvme, it also contains home folders for users in ad hoc partition and docker in another partition, so that will remain the same etc etc. formatted in ext4 and I will keep it this way, so no problem here.
    The problem arises when I have to migrate all the users\nas data to the new pool and reconnect everything to the new UUID path.

    • The files itself of course will be done through rsync to respect ownerships etc.etc. that will be easy (is there any faster way that I'm not thinking of?)
    • Docker compose\containters wise, should be streight forward to migrate to the new pool as I used the "global.env" file so I guess I will just change the UUID of the variables to new pool created and everything should be back online out of the box.
    • What about share folders of the samba service? and all the other plugins how can I reconnect them to the new pool\UUID smoothly? is there a way or the only way is recreating all the shares from scratch on the new pool, or is there a faster way to do it maybe modify the fstab?
    • Basically what would be the fastest way to migrate sambashares\plugins\and everything that point\map to specific old ext4 UUIDs to the new pool that will be created?

    Wek Time to pause and think hard about what you might gain from adopting BTRFS or ZFS and what is involved in making the transition. AFAIK there is no simple way to migrate an OMV configuration when a major change in the storage configuration takes place. FYI, ZFS does not use the fstab.


    How important are the oft quoted features of COW filesystems to you ( snapshots, bit-rot protection, data integrity via checksums, combine volume manger and fileystem, etc) ? Do the RAID like features of BTRFS or ZFS at least equal or surpass that of linux mdadm software raid? How easy is it to administer BTRFS and ZFS in OMV? Basic questions such as what happens with a failing or failed drive, what warnings are there re: degraded arrays, how do I replace drives etc , all need an answer.


    A few more positives and negatives re: BTRFS and ZFS.


    BTRFS - unlike mdadm and ZFS there is no daemon monitor. There will be no messages re: degraded BTRFS arrays in the WebUI. The combination of disks of different sizes in one RAID1 may not give you the available storage you'd first think. E.g 2TB + 2TB + 8TB gives 4TB not 6TB. (see https://carfax.org.uk/btrfs-us…dg=1&d=8000&d=2000&d=2000 ) OMV creates a subvol for new shared folder created on a BTRFS filesystem. The use of snapshots means any SAMBA shares are automatically configured to show "previous versions" on Windows

    via "shadow copies". Your Windows user have a chance to recover previous verision of data via this mechanism. For the OMV administrator, the same applies when working directly on OMV. A combination of cp --reflinks and server side copy makes this an efficient process when done via SAMBA shares.


    ZFS - ZED and OMV notifications combine to give comprehensive warnings re: zfs pool problems, as do the zpool details in WebUI. ZFS raidz expansion is meant to finally appear in openzfs 2.3 due soon. So it will be possible to go from a 3 disk to a 4 disk raidz1 array. You can combine disk of different sizes but there will be wasted space. Like BTRFS, ZFS based SAMBA shares can be configured for "shadow copies" but it is not done for you by OMV.


    Dealing with a failing or failed drive is going to be a manaul process at the CLI for both BTRFS and ZFS.

    Krisbee this is exacrtly why I'm asking :) so thank you for your input, really useful, also the testing part I hope will help me out if it's worth it, I'm more prone to zfs to be honest since data integrity and a more robust "easier" way to manage zfs pool for restore appeals to me more than btrfs since doing pools of different hard disks doens't really make too much difference for me.

    I'm not worried about recreate arrays if something fails through cli for zfs if at least I have notitification from the omv about it, that if I'm not mistaken you are saying monitoring and notification are supported on the GUI for zfs am I right?

    So it seems the way to go, about the migration is what concern me the most infact I hoped there was a simpler way, this still concern me to be fast since I don't have many days to do the process only a weekend.


    From your msg, zfs still seems the good option though more features for data integrity especially having the ecc ram, notification and monitor of issues from omv if I'm not mistaken, and recreation from the cli of the zfs array seems acceptable.


    I have nonetheless to substitute the ext4 raid, as I discover the previous admin used SMR disks for the raid not that great for raid...so I guess at the end an easy migration wouldn't be possible even if I keep the old system intact since the UUID of the new disks would be different nonetheless and a "forced fail of the raid" to rebuild on a new one with SMR disks is not that great a solution either...

    So I guess PAIN is the only doable way :D


    mmmmmh

    Wek Don't use SMR drives with ZFS. Here's an example of drive replacement in ZFS where zpool commands are interspersed with journalctl output. The point being to show output of ZED ( zfs event daemon) which by using OMV's notification system gets turned into corresponding emails.



    ECC RAM is the memory of choice on servers, irrespective of the filesystem used. The way ZFS uses memory via its ARC is a particular feature. I suppose I should ask how much memory is installed on your OMV host.

    yep maybe I didn't explain myself correctly, the SMR disks were from a previous owner and I'm about to substitute them with all CMR that's why my questions about a more "friendly" migration.


    About the ecc ram, the server already has it, so my point was having that ram already it could be even more worth it to switch to zfs as well, the system has 32 gb of ram, which is plenty for the amount of users with zfs and the few services active on the server, so shouldn't be a problem.

    I guess I'll go into zfs modality then test out a few weeks to understand better zfs tools and then migrate everything.
    Plus I like the fact I can easily manage\control zfs integrity from what I see, instead of using something like mergerfs+snapraid


    I don't know why, but snapraid never convinced me too much I tried it in a previous build.

    So my doubt really right now is just if go mergerfs+snapraid configuration or alltogether to the zfs route.


    I guess for something work related (not home usage) zfs would be better, I was a bit worried about it being part of a plugin system as you said (for maintenance longevity), but I guess is something that will be ported over and over on new omv versions.

    You need two plugins to use mergerfs+snapraid. It's popular with users of for slowly changing, or unchanging, bulk data. Its well suited to storing things like large media libraries where real time RAID is not crucial. The nature and usage pattern of your data ought to guide you on suitability of mergerfs+snapraid. The risk in OMV is that the maintenance of all extra plugins is down to basically one person.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login