ZF1 vs (RAID1 MDADM) vs (1 master HD and rsync to slave HD) combined with a SSD to speedup random read of small files AND yes I always use a USB HD as backup stored at another location

1. offizieller Beitrag

    Hello all,


    I am building an OMV system intended to do more than only file storage. When I did a backup copy of the existing data from the old server I saw the data speed dropping significantly when copying lots of small files. The data speed dropped when copying from the old server over LAN to a "temp backup pc" and it was the same when writing directly from the old server to an USB drive.


    The old server has got a mint install with a RAID1 configuration I do not want to spend too much time analyzing the old system as I will hopefully stop using it soon. But I think it would be helpful for me when someone points me in the right direction for the new server, I did read a lot, but some treads seem to be relatively old and with limited knowledge I can not judge which is the right way to go. At this time I did a basic test of OMV on the new hardware and it seems to work I did not attach the HDD's because they are still in the old server.


    This is what I did,


    I did install OMV on the M2 ssd (which I also hope to use as a HDD cache)

    did add a test sdd just to see if OMV works and did put a file share on it.

    tested a few big 650 MB files over LAN and they are transferred with about 85 MB/s


    while testing the new system I did experience the slow speed for the small files as described above. Then I did read some among this one :


    SSD Cache (bcache) for OpenMediaVault?


    I think they basically say if you want to speed up copying a lot of small files you need to use ram or a ssd to cache your hd with bcache and only ZF1 can do that. But I could be wrong.


    I have got a USB drive for backups which I will store in a different location. I see RAID as an redundancy measure not as a backup.



    Hardware available for the system:

    mainbord vga card Ryzen cpu

    32 gig ECC

    2 x 4 TB hdd

    1 M2 ssd

    a sata SSD (for testing but if I need it I could use it I ques)


    for hardware description please see the end of this thread:

    Review my NAS configuration (Ryzen, ECC)





    So the question is with the hardware I have, what is the best approach for the OMV install? MDADM or ZF1 or something else ? When the hd's are in the new system I really don't want to change the main setup over again, at this time I can still change things.


    Hope you can point me in the right direction thanks for the help!!

    • Offizieller Beitrag

    I not sure what the concern about transferring lots of small files would be. Once they're loaded onto the new server, that type of content is usually static. The mass copy operation, of numerous small files, is a one time event.


    When it comes to caching hard drives or the parallel I/O of a RAID array, note that your bottle neck will be your network connection. I checked your config, and since you didn't mention it, I'm assuming you have a 1GB network. In that case, forget about RAID for increasing drive throughput. There won't be any noticeable benefit. The I/O of one (1) hard drive can saturate a 1GB network connection.

    __________________________________________

    When it comes to type of filesystem, I personally prefer ZFS. ZFS is disk aggregation, a file system, and LVM all in one. ZFS, by default, will use available ram for file caching. A ZIL drive would speed up writing but, again, you're still going to be limited by the network speed of 1GB.

    One of the notable differences between ZFS and mdadm RIAD is ZFS can detect and correct bit-rot. For example, if a tradtional RAID1 drive is going bad, data can be silently corrupted as it slowly dies. ZFS, on the other hand, can detect and repair this type of corruption.

    Zitat von crashtest

    I not sure what the concern about transferring lots of small files would be. Once they're loaded onto the new server, that type of content is usually static. The mass copy operation, of numerous small files, is a one time event.


    When it comes to caching hard drives or the parallel I/O of a RAID array, note that your bottle neck will be your network connection. I checked your config, and since you didn't mention it, I'm assuming you have a 1GB network. In that case, forget about RAID for increasing drive throughput. There won't be any noticeable benefit. The I/O of one (1) hard drive can saturate a 1GB network connection.

    __________________________________________

    When it comes to type of filesystem, I personally prefer ZFS. ZFS is disk aggregation, a file system, and LVM all in one. ZFS, by default, will use available ram for file caching. A ZIL drive would speed up writing but, again, you're still going to be limited by the network speed of 1GB.

    One of the notable differences between ZFS and mdadm RIAD is ZFS can detect and correct bit-rot. For example, if a tradtional RAID1 drive is going bad, data can be silently corrupted as it slowly dies. ZFS, on the other hand, can detect and repair this type of corruption.


    Hello crashtest, thank you for your reply!


    The thing is with RAID1 on the old Mint server I sometimes get low transfer rates when it copies a lot of small files, then the speed goes down to about 10MB/s or even less! while large files of 650MB go at a speed of 85MB/s. So in this case the bottle neck is not the HD (?) but it might be a stupid mistake I made in some setting or something else. As this is a fresh install I try to see if I can do it better this time. So I think the first thing I have to do is to make a choice between ZFS and mdadm and then try to tweak the results if necessary.


    Still not sure which one to use


    is ZFS hard to setup in OMV compared to mdadm?

    In past days I did install a LVM system for fun but those where the days when had time to nerd around (liked that very much :) ) If it is LVM can I later just add another HD without reworking the whole setup?

    • Offizieller Beitrag

    A collection of small files, scattered to different locations on a hard drive would, understandably, slow down I/O. If that's what you're reading and writing most of the time, saying you're a photographer or some other commercial interest with a LOT of small files, a RAID5 equivalent might help.

    Zitat von The_Others

    is ZFS hard to setup in OMV compared to mdadm?

    Not really. If you're interested, I can guide you through it. While it's my opinion, ZFS is far better than mdadm RAID. Few would disagree.


    Zitat von The_Others

    If it is LVM can I later just add another HD without reworking the whole setup?

    I see LVM as somewhat risky. If a drive in a volume fails, there's no redundancy. The only thing I would trust when dealing with a failure would be a full comparison to backup for a selective recovery or, possibly, a full recovery from backup just to be sure.

    Yes you can add "drives" to a ZFS pool, but, once set up you can't downsize a ZFS array. It can only get larger. The absolute safest array is called a "zmirror" or the ZFS equivalent of RAID1 (but far better than mdadm RAID1). You can continue to add mirrors, consisting of two drives, to the pool. Here's a short piece on how to -> enlarge a ZFS pool. It's pretty easy in the GUI. You'd just need enough sata ports and the drive slots to do it.

    (You could add a single drive to a ZFS pool but it's a real bad idea. It's best to enlarge a pool with a pair of disks or more.)
    _____________________________________________

    With that said, you have another choice that's very flexible. SNAPRAID (provides for full drive recovery and even files and folders) + Merger FS (aggregates drives under a single mount point.) Both are available as plugin's, in OMV.

    The great thing about SNAPRAID is you can turn it on or off at will. Data is unaffected. (Obviously it needs to be on and up-to-date for any kind of recovery and it requires one drive as the parity drive. The parity drive must be as large, or larger, as the largest drive to be protected.) The maintenance is a bit more complicated, when compared to ZFS, but the setup can be done at any time using a simple file system, on individual drives, like EXT4.


    Mergerfs (call the Unionfilesystems plugin) makes a collection of drives appear to be one. It has a couple of considerations but, essentially, it's mostly transparent.


    Other than the SNAPRAID rule for the parity drive (must be the largest), both SNAPRAID and Mergerfs don't care about drive size, drives can be added at any time, and as little as one drive can be added at a time. If this sounds interesting, there's more detail on what SNAPRAID + Mergerfs offers here -> a Media Server Article The articles does a better job, than I can, of describing the features.
    _________________________________________________


    **Edit** Looking at this thread's title; if you're interested in an Rsync disk-to-disk mirror, that's super easy to do and much better than mdadm RAID1. The User Guide outlines the advantages over RAID1, how to set it up, and how to recover from a drive failure quickly.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden