Posts by peterkeyzers

    4 drive enclosures are $100 and up, USD. If you're using USB 3.0 or better (both ends), you're getting reasonable throughput and it's working well, you might consider using what you have.

    I was thinking about that, but RE: Disk not showing under RAID management tab made me reconsider somewhat. This seems like one of those things where folks have very different opinions, and not having much experience in this area, not sure entirely which way to go. Will have a good sleep on it :)

    Latest update time. New adapter card arrived, but seeing the same issues, so looks like it's more than likely a fault with the box. A review from 6 years ago of the same unit model describes the same problem, but none of the recent reviews do. So perhaps just unlucky... So looks like I'm in the market for a new enclosure now :)

    Thanks all for your advice here!

    People will just tick the box and ignore the warning then. If you really want to make the mistake of using raid on usb drives, just create the array from the command line. You can do everything else in the web interface. Just try to remember that you using something designed for high availability (raid) with something that has no clue what that means (usb).

    OK, that's a good way of describing the issue. I was really just thinking about "hey, it connects just fine, so why not?", but the design considerations help to reset my thinking. Thanks Aaron!

    votdev this is likely a dumb question, but what's the main reason not to allow USB drives? USB3 drives a MB port or adapter card seem pretty solid, although USB sticks plugged into the front of a box wouldn't be :) I'm more than likely missing something though.

    How feasible would it be to have a checkbox option somewhere users could tick to enable USB drives, along with a warning that if you tick it, you have to give up support on it, sacrifice a pet, etc?

    If the disks are connected via USB, then they are not listed in the UI with reason. You have to create the RAID manually in this case.


    I had a similar issue, and ended up creating a USB ZFS pool manually via the CLI. THis worked fine, but when I clicked on the ZFS UI section, and tried to apply changes (when OMV picked up the presence of the new ZFS pool), the UI crashed with a bad gateway exception, SSH connections were dropped, and I had to reboot the server by hand.

    I couldn't find any logs in /var/log/openmediavault, and there was nothing obvious in syslog when I rebooted. The only solution was to revert the change in the UI, because each time I clicked Apply, it crashed the box.

    I can reproduce this at will, but don't know how to send any logs for debugging/fixes - can you advise?




    Based upon the above 2 comments, when you attempted to create a pool via the GUI, OMV was unable to locate any usb drives

    Whilst using USB drives for this is probably not recommended it should be doable but not advisable. That would leave you with sourcing a 'branded' eSata card -> StarTech

    Found the reference: RE: Disk not showing under RAID management tab

    For £25 and next day delivery, I think it's worth a punt :)

    BTW, any ideas on where to look for logs for some form of OMV crash that's not well handled by the UI? I poked around /var/log/openmediavault but couldn't find anything in there...

    Thanks for the pointers, will have a read at lunchtime and see what I can narrow it down to! Your patience and advice is much appreciated!

    Neither could I, reminds me of the long thread where the OP had used a cheap Chinese generic sata card, in cases like this the best option is not the cheapest and a 'branded' card such as StarTech that has their own site and support is the better option.

    But if this was me I'd stick with USB3 and I found this from back in 2013

    Main issue there is that the OMV GUI doesn't support USB drives for ZFS. When I tried to add the pool I'd created via CLI, it actually crashed most of the server (SSH connections died, bad gateway to the UI, etc), so I had to physically reset the machine! Couldn't even find any logs to submit for a bug report... :(

    OK, so another update after nearly a week!

    I've managed to run several drives in the main server box itself, running mergerfs + snapraid quite happily, no issues at all. Then switched to a USB3-connected box with ZFS, 2 2-disk mirrored vdevs, so a total of just over 5TB of usable space. Uploaded 2TB of data, have done several reboots, and two full scrubs, and no problems at all.

    So looks like the next thing to try is swapping out the eSATA card. If you have any solid recommendations, would be glad to hear them! But will do some reading online to try and find some good ones to pick from...

    All of your HDD's have the same problem? Statically, that's a very long shot. However, if syncthing under Docker was used for the copies, that's worth following up. Rsycn, which is clean and reliable, can be used to copy host to host.

    I still like the interface possibility. Going USB3 eliminates the eSATA expansion card (which, notably, has some negative reviews for longevity) and there's a change of interface in the ICYBOX. A few possibilities are eliminated.

    Time for an update!

    Seems a bit random which ones were failing, which is definitely a red flag. Syncthing is used to sync between different machines, some of which aren't rsync-friendly (e.g. overseas with non-tech-literate folks), so is a bit of a requirement. Worth mentioning that both the Icybox and card are new (e.g. a couple of months old) so longevity would hopefully not be an issue here - although that doesn't rule out faulty equipment in the first place!

    Current experiment is with 3 drives mounted directly in the server, syncthing syncing between a few hosts. So far, no issues at all, will leave running for a day or so. Have rebooted a couple of times so far, no obvious explosions. If this looks good, will try USB3 from the Icybox to isolate whether it's the card or the box causing the issue...


    So, more testing! Setup a 3 disk merger FS system, with a single parity disk. All 3TB disks. Copied around 1TB of data yesterday, using epmfs as the mergerfs policy, ended up rebalancing using the mergerfs tools. Ran sync and 100% scrub, no problems reported.

    Tried again this morning, wiping files (not formatting the disks though), and remounting the mergerfs filesystem with mfs instead of epmfs. Then tried copying over another 1TB of data. This time, I see the following part-way through:

    Jun 30 14:15:01 core kernel: [ 9087.053899] EXT4-fs error (device sdc1): ext4_find_extent:920: inode #114559341: comm mergerfs: pblk 458260480 bad header/extent: invalid magic - magic d8ff, entries 57599, max 4096(0), depth 17994(0)

    Guessing this is an HDD problem of some sort, or should I be nuking the disks altogether with a secure format and ext4 filesystem creation?

    Didn't think to mention earlier, but I'm using syncthing (running under Docker) to "copy" the files across. Is it possible this is causing an issue?

    Haven't tried the USB3 connection yet... Also, if there's better recommendations for an interface card (ideally not costing hundreds of pounds :)), would love to hear!




    Thanks for the detailed response!

    SMART stats are all good, did a long test for each drive with no problems found. This is an installation of OMV that's around 2 years old, and no problems apart from this one, so if the ISO was bad, would have seen it before now. No encryption is used.

    Funnily enough, the first problem with the RAID array, I checked the disks, and one of them was a SMR drive. I replaced this with a Seagate Ironwolf NMR, which reported no SMART errors. So think I'm good on the types of disks in use.

    I'm running the 4 drives from an ICYBOX 8-bay JBOD box. Connected via eSATA, not USB. Bought a new connector cable for it last Friday (shielded, just in case), but no change there. The server (a Dell T40) has one of these for the eSATA connection. I checked the seating of it today at lunchtime, all seems to be fine.

    So after upping the sleeps as follows in /etc/defaults/zfs:

    I rebooted, and recreated an empty pool (using an ashift of 12). Without even creating a dataset or writing anything into it, I got a selection of read and write errors, and the pool quickly made itself unavailable.

    This is in stark contrast to last week, where for 4 or 5 days, I had perfect behaviour (I nervously checked zpool status many times a day). The scrub ran for some time (12h) and reported no errors.

    My suspicion is something hardware related, but just in case, I'm testing out a mergerfs+snapraid setup, just to rule out ZFS as the cause, in case it's a bit sensitive. I'm really starting to run out of ideas though!

    I captured the zdb ouptut if that's of any use?



    OK, so reporting back after a week. Re-setup the pool, followed advice in the thread here, copied 2TB of data to it, ran a scrub, and 0 errors reported.

    Ran my scheduled weekly reboot, and I'm immediately seeing array errors.

    I'm completely stumped as to why a reboot could cause this. The scheduled job is nothing more than running reboot with no additional params as root. Any thoughts/suggestions appreciated at this point, because it's incredibly frustrating!

    Update: Investigating this and this as they seem promising.


    Yeah, when you put it that way.... :) Thanks for your help, I'll do it the proper way and try again!

    I tried that / in the path entry and that was what seemed to blow up, in that the error seemed to be trying to change permissions at the root of my os drive. So I therefore tried to use the trove device and typed a path of music, even though trove/music is a formal dataset. I don't think it should have messed anything up though?

    Probably though, the error was due to what you originally corrected me on, so I think if I apply the acl settings, it should be ok. Will nuke the pool and start again from scratch, as it's been resilvering for over a day and hasn't made any progress at all, so is stuck somewhere...

    The only think that I can think of, that might result in data corruption, is a hardware fault or a misconfiguration. Also, you might look at your drives' SMART stats. ZFS can correct corruption with a scrub "but" data must be written to disk, without corruption, the first time.

    It's all been very odd. SMART is all fine, the data was all written smoothly (just over 2TB), zpool status returned no errors. (I've spent a week testing different configurations which have all survived reboots). Then, after this Sunday's scheduled reboot, bam.

    I'm using an Icybox 8-bay JBOD enclosure with 4 drives in it, connected via eSata, not USB. Using drive Id, not names, so trying to follow all the good advice. 2 2-disk mirror vdevs, and about 20 different datasets with varying record sizes. So nothing too outlandish.

    I did a mixture of creating the pool in OMV, but the datasets on the CLI. Would hope that wouldn't affect anything though.

    One potentially interesting point. My layout was (roughly) as follows:

    trove (pool)

    -> development (dataset)

    -> music (dataset)

    -> docker (dataset)

    -> config (dataset)

    Because I wanted these shares to be visible at the levels in my diagram, I had to create the share as:

    device: trove

    mountpoint: music

    as there wasn't an option to use trove/music as the device and no mountpoint. Would have hoped this wouldn't cause issues, but am open to trying something else!

    Do you think the failed command above could have messed things up?

    Ah, didn't realise this. Thanks for letting me know. I had already copied 2TB of data to the pool, but in a separate problem, it seems to have become degraded/corrupted after the server's weekly reboot, which I'm somewhat puzzled by. So I might need to recreate the pool anyway, in which case I can apply this :)


    Having had my RAID setup fall over recently, I've been in the process of setting up a ZFS filesystem instead (thanks geaves!). It's going OK so far, but have noticed a bug when trying to set the ACL on a shared folder for the ZFS pool to allow users to write to a shared folder:

    Error #0:
    OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; setfacl --remove-all --recursive -M '/tmp/setfaclpq5Tzs' -- '/srv/test/music/' 2>&1' with exit code '1': setfacl: /srv/test/music/: Operation not supported in /usr/share/openmediavault/engined/rpc/
    Stack trace:
    #0 /usr/share/php/openmediavault/rpc/ Engined\Rpc\ShareMgmt->Engined\Rpc\{closure}('/tmp/bgstatuss1...', '/tmp/bgoutputqH...')
    #1 /usr/share/openmediavault/engined/rpc/ OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure), NULL, Object(Closure))
    #2 [internal function]: Engined\Rpc\ShareMgmt->setFileACL(Array, Array)
    #3 /usr/share/php/openmediavault/rpc/ call_user_func_array(Array, Array)
    #4 /usr/share/php/openmediavault/rpc/ OMV\Rpc\ServiceAbstract->callMethod('setFileACL', Array, Array)
    #5 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('ShareMgmt', 'setFileACL', Array, Array, 1)
    #6 {main}

    The operation itself seems to succeed, in that I can now write via the share, but the error is a bit odd. Am I doing something wrong? Happy to provide more info if need be.



    You will have to delete smb shares, then delete the shared folders, break the array (this might be possible by just deleting it rather than removing one drive, then another before deleting it) delete the file system in relation to the array. In essence that should work, you would then have to wipe each drive, create the array, create the file system etc.

    When I moved from OMV4 to 5 instead of attempting an upgrade I did a clean install, took me less than day to get back up and running and I changed to using ZFS :)

    Hah, I did the same, although given this isn't a major upgrade, there's a bunch of stuff I'd rather not spend the time re-setting up, got little enough time as it is! Will try it the manual way, and if I get stuck, the nuke and restart option is always there...

    Thanks for your assistance, much appreciated!