Choice of filesystem and RAID with large drives?

    Hi all,


    I have 6 old 4TB drives running RAID 6 (via mdadm, not supermicro motherboard) which are running out of room, and I'm running 3 new 16TB drives through their paces over the next week to make sure that they'll hold out.


    My knowledge is a bit antiquated and incomplete on EXT4 vs XFS vs openZFS, and I need some help on making a good long term decision. Board, processor and memory are not an issue in supporting any of these. I'm willing to learn something new.


    Goal is to build these 3 new drives into a raid group and I'd like to make the best decision possible with regards to data availability and integrity - I'm less concerned with speed. Minimal administration is a plus. I'll also be looking for a backup solution for this new setup, but that's a separate topic that I think I can manage on my own.


    I've been doing a lot of research on RAID selection and it seems that, with these larger drives, RAID 5 and 6 may be obsolete right off that bat. I'm not interested in RAID 10 due to cost.


    I'd appreciate anyone sharing their knowledge and experience, and/or point me to some educational resources. I've only found mixed information and lots of opinions and feelings on FS that I just don't have enough background in.


    Thanks in advance!

    hi,

    for this hdd size, i think raid 6 is the very more secure solution (because of 2 parity disks), but you need to buy another one to have the right number of hdds. And result size will be not awesome. But it's the cost of raid6.

    Raid 6, double parity, 4x16Tb, 32TB available for use.


    with ZFS, Z1, 4x16tb, 40Tb avalaible.

    with ZFS, Z2, 4x16TB, 27Tb avalaible.


    need more ram.

    I would go for zfs if you want integrity and backup is easy by shipping snapshots to somewhere else.

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

    Genna and Zoki Thanks for your inputs. I'm totally comfortable with my RAID 6 on the RAID of 4TB disks that I already have. They are also small (and inexpensive) enough that I have two revolving backup disks. One on-site and one-offsite that get swapped around on schedule.


    I forgot to cite the website that I got this from, but thought it was interesting:

    Quote

    Instead of using vanilla RAID, I would recommend looking at ZFS. Whilst it uses the same data layout principles as RAID ("ZFS mirror" = RAID 10, "raidz1" = RAID 5, "raidz2" = RAID 6, "raidz3" = 3 parity blocks) it has the advantage of being both a filesystem and volume manager which allows it to have some optimizations and features that aren't possible when the filesystem and volume manager are separate systems.

    For example, ZFS can detect bit-rot caused by bad sectors by regularly running a scrub operation which verifies the checksums of data blocks and repairs them from the other replica/parity blocks if necessary. Additionally, rebuilds can be more efficient since the filesystem knows which blocks are in use and need to be rebuilt, whereas traditional RAID must rebuild every block on the drive since it knows nothing about the filesystem on top of it.

    I imagine that in most ZFS would be out of the question (and probably overkill) for most home use cases. This is looking interesting for mine.

    and look at zfs send / receive for backups. This makes off site backups on file system level cheap

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

    hi,

    ZFS seems great.


    but i have this question :

    if ZFS is very helpfull in this case, why it is always a plugin and not implemented in OMV first ?


    and a second,

    How many RAM do you need for a comfortable system ?

    Check license conditions for ZFS. It is not part of debian distribution.

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

    MiddleMan Advocates of zfs would say use it on anything from single disk laptops to multi-disk monster storage servers. But for the home user, it has its “costs” as pointed out here: https://louwrentius.com/the-hi…fs-for-your-home-nas.html


    Personally, I’ve been happy to use it with zfs mirrors, single or striped, rather than using mdadm. If you had chosen to upgrade to 6 x 8TB HDDs, I wouldn’t hesitate to recommend zfs, creating a pool with all 6 disks in a single vdev configured as raidz2. But with just three 16TB disks, the choice is not so clear cut, as the options for a usable zfs pool layout are limited.


    I can see the trend to large HDD size is attractive to home users who want capacity without the need to combine multiple drives, but they have a down side in RAID use where build/rebuild times become very long with the increased risk of failures. This is true of both mdadm RAID5, or say using your three 16TB in a zfs raidz1 configuration.


    I don’t know how much life may be left in your 4TB drives, or how many SATA ports you have available, but had you thought about creating two zfs pools, a raidz2 from your existing 6 x 4TB drives, and a singe mirror pool from two of your new 16TB drives? You’d have the spares on hand to replace failed disks, assuming you had the separate capacity for backup.


    For more information about zfs, I’d recommend this presentation given by Jim Salter:

    External Content www.youtube.com
    Content embedded from external sources will not be displayed without your consent.
    Through the activation of external content, you agree that personal data may be transferred to third party platforms. We have provided more information on this in our privacy policy.


    His articles in arstechnica are also worth reading: https://arstechnica.com/inform…-storage-and-performance/

    Regarding the rebuild / resilver process zfs is superior to RAID, as only the used data has to be copied in zfs whereas the whole disk has to be copied when using raid.


    If it comes to ordinary home users, I would advocate for something simpler than raid / zfs, but if primary focus is on " best decision possible with regards to data availability and integrity" only raid or zfs remain. And If you have to choose between mdadm and zfs I tend to go for zfs. (Made up my mind at the beginning of the year, so it may change).

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

    I definitely will. I have a bit of studying to do on this but openZFS has piqued my interest. If I choose to use it, based on what I'm seeing so far, the effort will take time to plan also - zfs vdevs can't (currently) be made larger. The effort may also include how to tune, depending on use case.

    Hi all.

    I decided to order another disk (for 4 total). I also converted them all to 4Kn from 512e after contacting manufacturer support for the utility and manual.

    I work with some very seasoned UNIX administrators and had separate discussions with each. While they all recommended what is the equivalent of RAID 10, I decided on RAIDZ2 for 2 reasons: cost and I don't need high performance. Neither setup is infallible, in any regard. So, will still maintain rotating off-site backups.

    Both new and old sets of drives are now RAIDZ2 (2 zpools), and still much faster than the 1GB NIC.

    To answer earlier question I have 48GB ECC RAM and 60% in use. Also, 10 year old (bought used) quad core Xeon never went above 40% throughout the entire migration.

    rsync was used to: move files around, ensure data didn't get messed up in the move, and to retain permissions.


    The OMV ZFS plugin is great. I was hesitant to not use shell, but it really makes everything a breeze.


    I passed the following in ssh (as root) after setup:

    zfs set atime=off zpool

    zfs set compression=lz4 zpool

    zfs set recordsize=X zpool/myfs, where X=

    • 64K for VMs (I have one small VM for Jellyfin)
    • 128K general, mixed filetypes and sizes (default)
    • 1M video and large (space consuming) pictures

    Used zfs get all to verify that settings took. Am not planning on any additional tuning.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login