creating a single large filesystem from 16 x 8TB drives in the best way

  • Hi,


    I want to create a single large volume from 16 x 8TB drives using openmediavault_3.0.36 (downloaded the iso) on a system with 16 x 8TB drives connected to an ASRock X99 Extreme 11 motherboard.


    What I have done:
    -created 2 RAID6 arrays (each one consisting of 8 drives) since I have heard it's not good to have too many drives in a single array.
    -now there are 2 devices but to have a single file system you need 1 device. So I have combined the 2 RAID6 arrays into a single array using RAID0 (stripe) on top and it looks like it worked.


    Questions:
    Is there a better way to create a large volume from my 16 drives? (I can still start over at this point)
    Is RAID0 on top of RAID6 ok in general? I want to avoid complications in the future and nested arrays may be bad.
    Is it correct that 2 RAID6 with 8 drives is better than 1 RAID6 with 16?


    Thanks in advance!

  • Do yourself a favor. If you don't need redundancy, use the unionfilesystem plugin to pool these drives. If you do need redundancy, use a hardware raid controller.


    I can't imagine how you are going to back this up...

    omv 5.5.22 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.4.4
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • You should have a look to zfs. As an example you could build two raid-z2-vdevs with 8 disks per vdev. It's comparable to raid60. ;)

    ---------------------------------------------------------------------------------------------------------------
    frontend software - tvos | android tv | libreelec | win10 | kodi krypton
    frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
    -------------------------------------------
    backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
    backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
    ---------------------------------------------------------------------------------------------------------------------------------------

    Edited once, last by hoppel118 ().

  • @ryecoaaron Thanks, yes, redundancy is needed (2 failed drives should be tolerated). Backup is not an issue, it will happen continuously via rsync.


    I'd like to work with what I have (or is it such a disaster that it's not possible without getting problems later on? Things look like they are working so far). Given that, I'd appreciate if you could give me a tip regarding the RAID questions, even if a hardware raid controller would be more suited for the job.


    @hoppel118 I have considered ZFS under Freenas with the exact config you mention, but ran into an issue with not being able to replicate NFS paths as they were before (Freenas requires /mnt/data/...). But ZFS under OMV may be different - I will check out if I can get around the issue under OMV.

  • You should ask the question why too many devices in One Raid are not advisable.
    Then, you should ask yourself whether the same does or does not apply to your Setup.
    ou do have 16 devise in One Raid.
    Also, you should think about why your approach seems uncommon.
    Last but not least, every filesystem corruption would endanger your whole Data and nur just Parts of it.


    You could consider Snapraid.


    Where are you backing up to?
    Rsync does not do versioning. What if Malware corrupts (encrypts) your Data?
    After the next Backup/rsync, your Backup would contain the same corrupt Data.

  • @hoppel118 I have considered ZFS under Freenas with the exact config you mention, but ran into an issue with not being able to replicate NFS paths as they were before (Freenas requires /mnt/data/...). But ZFS under OMV may be different - I will check out if I can get around the issue under OMV.

    Report your bugs or issues with the omv-zfs plugin in the following thread:


    http://forum.openmediavault.or…FS-Plugin-use-ZFS-on-OMV/


    Maybe someone there can help you. ;)


    Greetings Hoppel

    ---------------------------------------------------------------------------------------------------------------
    frontend software - tvos | android tv | libreelec | win10 | kodi krypton
    frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
    -------------------------------------------
    backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
    backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
    ---------------------------------------------------------------------------------------------------------------------------------------

  • I'd like to work with what I have (or is it such a disaster that it's not possible without getting problems later on? Things look like they are working so far). Given that, I'd appreciate if you could give me a tip regarding the RAID questions, even if a hardware raid controller would be more suited for the job.

    I did give you a tip. Don't use software raid and use hardware raid. Definitely don't put software raid on top of software raid. You will regret that. zfs might be ok but not mdadm. If you can afford all those 8tb drives and space to back them up to, a hardware raid card is a small cost.

    omv 5.5.22 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.4.4
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • I did give you a tip. Don't use software raid and use hardware raid. Definitely don't put software raid on top of software raid. You will regret that. zfs might be ok but not mdadm. If you can afford all those 8tb drives and space to back them up to, a hardware raid card is a small cost.

    Thanks, got it now :) . I will consider getting controllers but the system is already over budget and it's not only up to me. As an alternative, what would you think about 2 separate Raid6 software arrays if I'd be willing to get 2 separate file systems out of it?


    Report your bugs or issues with the omv-zfs plugin in the following thread:
    http://forum.openmediavault.or…FS-Plugin-use-ZFS-on-OMV/


    Maybe someone there can help you. ;)


    Greetings Hoppel

    Thanks, will do if I go down the ZFS route!

  • As an alternative, what would you think about 2 separate Raid6 software arrays if I'd be willing to get 2 separate file systems out of it?

    It would probably take a week to sync. Assuming the server is on all the time, it should be fine.

    omv 5.5.22 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.4.4
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Just wanted to provide an update about what I ended up doing: I used ZFS to create a single raid-z2 pool made up of 2 vdevs of 8 drives each ( by first creating a pool of 8 drives in the OMV GUI and then selecting "Exapand Pool" and adding the next 8 drives). The first 9 TB have been copied to new file systems created in the pool.
    Thanks @ryecoaaron and @hoppel118 for advice.


    Fyi, here are my versions:


    dpkg -l |grep zfs
    ii debian-zfs 7~jessie amd64 Native ZFS filesystem metapackage for Debian.
    ii libzfs2 0.6.5.7-8-jessie amd64 Native ZFS filesystem library for Linux
    ii openmediavault-zfs 3.0.3.5 amd64 OpenMediaVault plugin for ZFS
    ii zfs-dkms 0.6.5.7-8-jessie all Native ZFS filesystem kernel modules for Linux
    ii zfsutils 0.6.5.7-8-jessie amd64 command-line tools to manage ZFS filesystems
    dpkg -l |grep openm
    ii openmediavault 3.0.46 all Open network attached storage solution
    ii openmediavault-keyring 1.0 all GnuPG archive keys of the OpenMediaVault archive
    ii openmediavault-omvextrasorg 3.3.3 all OMV-Extras.org Package Repositories for OpenMediaVault
    ii openmediavault-zfs 3.0.3.5 amd64 OpenMediaVault plugin for ZFS

  • I am happy to see your decision for zfs!

    ---------------------------------------------------------------------------------------------------------------
    frontend software - tvos | android tv | libreelec | win10 | kodi krypton
    frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
    -------------------------------------------
    backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
    backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
    ---------------------------------------------------------------------------------------------------------------------------------------

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!