Strange behaviour when installing OMV

TL;DR

When installing OMV on big USB-Sticks (e.g. 64 GB), the installation fails if the system partition is formatted either with EXT4 or BTRFS.

Long Story

I recently decided to switch to BTRFS for the system partition, to be able to create snapshots before an update. To have enough space left for a few snapshots, I bought myself USB-Sticks with 64 GB.

I installed Debian using the official installation media. The installation was successful.

I then installed OMV using this script:

wget -O - https://github.com/OpenMediaVault-Plugin-Developers/installScript/raw/master/install | sudo bash

The installation of OMV failed and the file system went read only.

I first suspected a faulty USB-Stick, but I tested it with 4 sticks of that size - always the same.

I only could get it to work with an 8 GB USB-Stick. 16 GB also fails.

I then suspected that maybe the NAS could have a problem with its USB port. Therefore I tried it on on two other NASes (I even bought myself another one, because at first I was really convinced, that the NAS was broken).

But all NASes (all from QNAP, but different models and ages) behaved the same. Installation on 8 GB was fine, installation on 16 GB or 64 GB stick failed.

I even tried both architectures (64 bit & 32 bit), same for both.

That was when I started to really test it systematically and also write it all down...

Seems this behavour is depending on the size of the system partition AND on the file system.

XFS, JFS and EXT3 work fine. OMV can be installed even on big sticks and it works.

On EXT4 and BTRFS the installation fails.

Out of curiosity I also tested the official installation media provided for OMV.

Since it uses EXT4, it also fails -- well sort of.

The installation itself works well, but after a reboot of the system, I start to get error messages.

I also tested different sizes for the system partition on the 64 GB Stick. A size of 7,2 GB for the system partition still works, a size of 7,5 GB fails.

To make that clear: I can get a 64 GB stick to work, if either I use one of XFS, JFS or EXT3 OR if I create on the big stick a small system partition.

I also created a separate small partition of 1 GB for /boot, but that doesn't change anything: When the system partition is bigger than 8 GB and contains as FS EXT4 or BTRFS, the installation fails (the FS on the /boot partition can be any).

Spinning disks and SSDs (NVMe) don't seem to be affected by this.

I have another NAS, where I have the system partition on a NVMe SSD on a partition of about 160GB, which works fine. And I also tested an installation on a 55 GB Partition on a SATA HDD which also worked fine.

If the installation fails, it always fails at the same place, no matter if EXT4 or BTRFS is used.

In the attached file I provide the last lines on the shell of one of the failed attempts (using EXT4) and the output of dmesg.

Any help how I can get OMV installed on a 64 GB USB-Stick on BTRFS would be greatly appreciated.

If more logs or anything else is needed, I can reproduce this anytime and provide any needed information.

Thanks in advance.

Quote from wenzhuo

Do a badblocks check of it on a computer first to make sure it's fine.

Of course I did check the USB-Sticks. I checked all of them and I checked them multiple times.

They are all fine.

I have seen those message also, that's why I also suspected faulty USB-Sticks at first. But currently I am completely convinced that I don't have any problem with that. I don't know what triggers those messages, but I rather have the impression they are not the root cause but some follow-on problem. But I might be wrong with that...

Quote from wenzhuo

Then try connecting it to the NAS via a powered USB 3.0 hub.

Can't do that -at least not right now- since I don't have such a hub.

Quote from sfcampbell

I genuinely respect the goal you're trying to accomplish: but I think the limitation you're running into is the USB disk platform in general -- as far as I understand they're really only functional in very limited fat16/fat32 (even NTFS 🤮 formats are finitely available and farcically incomplete) in a USB storage device. I think only alternative platforms i.e. SD, SATA, m-SATA, or M.2 may be able to perform with mainstream formats like you're [justifiably] pursuing.

Hm, that would mean, that an external USB-Drive (HDD or SSD) wouldn't work either. I will try to test that. (but I first need to get my hands on such a drive...)

I don't have a true external USB-disk currently (will get one at the weekend).

But I DO have an external USB-SATA-Bridge (for emergencies), and I attached an old Seagate HDD to it and connected it with my NAS.

Installation of Debian and OMV succeeded.

Seems to me, that the USB subsystem isn't the culprit...

...but I will test again with a true USB-Disk at the weekend.

Quote from macom

Do you have any evidence for that statement? I run OMV from USB stick with btrfs for years.

Would you tell me, how big your system partition is?

Today I did a test with an external SSD (attached via USB).

It worked without any problems.

That also rules out any power topics, since that SSD got its power only via the USB connector.

So my current hypothesis is, that the throughput might be the crucial thing here.

So far one of the important criteria for those USB-Stickst for OMV was that they are small in size (i.e. short).

But looking at the specs, it seems, that sticks those are rather small.

My current stick has a (measured) write rate of 20 MB/s.

The SSD had 290 MB/s.

I have currently ordered two other sticks with significantly more speed.

Will report here..

Quote from Soma

Some food for thoughts

Thanks, will have a look at it.

Yes, I booted my PC from those sticks without problems.

Quote from wenzhuo

Also, why not get a powered USB 3.0 hub

because I'm 98% sure, that's not the root cause behind all this.

As I said, I have ordered faster sticks. One already arrived.

It's from Transcend and spec says: 700 MB/s write and 1050 MB/s read speed.

Measured read / write speed is as follows: 136 MB/s write and 308 MB/s read.

With this stick everything works as expected.

...still waiting for others to arrive.

Quote from Soma

It was working flawless but the system was set in full with SNAPPER, prior to install OMV.

I am using btrbk for my daily backups and could probably use that also for the system partition, but you can achieve this much simpler:

You simply need to create a shared folder, that points to / (the system partition). The you can use the OMV Web-UI for it.

(I have this in place for a NAS that is running from NVME SSDs)

Quote from Soma

The idea of the root subvolume on btrfs is to revert back in case of a borked OS.

True. And I use it only to create the snapshots.

So far I had no need to revert back to a prev. snapshot, but I would probably also use native mechanisms.

And for those it doesn't matter which way the snapshots were created.

As explained earlier in this thread, I experienced some strange behaviour, when installing first Debian and then OMV on a boot stick on my Qnap NAS.

In the meantime I’ve tested more than 10 different USB sticks to see, if I could find any systematic behind that. The results range from “Everything works as expected” to “Boot Stick physically(?) damaged”. Yes, that’s true. I somehow managed to damage that stick. I was no longer able to write data on it nor to format it. That was reproducible. I damaged 4 of them…

This is what I did as my test.

• Format the USB Stick with GPT partition table (using Rufus).
• Check if whole stick is writable. This test also gave me the “Measured Throughput” (using h2testw.exe).
• Install Debian.
  During installation I created the following partitions:
  - 11 MB Grub
  - 2 GB /boot (XFS)
  - 55 GB / (BTRFS)
  - rest swap
• Let the installer do its job
• Reboot
• Install OMV

During my tests I also experimented with mdadm Raid. I know, it’s not at all supported and I advise against it for production purposes. I only wanted to have some other means to put some “stess” on the system and to see if that leads to some outcome. Fact is, that adding a bitmap can be seen as an additional test, since this was only successful on some of the sticks. And all sticks that were able to handle that bitmap, also were able to handle the installation of OMV.

Unfortunately, it must be the system drive (carrying /). Only formatting two sticks with the given partitioning scheme and mounting them with some data always worked. To get different results, you need to prepare the Raid-1 using the Debian Installer and format the FS with BTRFS. Then after the reboot, wait until the initial sync is ready. Then issue “mdadm --grow /dev/md1 --bitmap=internal”. If it comes back immediately with some message indicating that it was done, you are lucky. If not, the system hangs and needs a hard reset.

So, the procedure to test the bitmap is this:

• If not yet done: Format the USB Stick with GPT partition table (using Rufus).
• Just to be completely sure check again if whole stick is writable. (using h2testw.exe).
• Install Debian.
  During installation I created the following partitions:
  - 11 MB Grub
  - 2 GB /boot (Raid)
  - 55 GB / (Raid)
  - rest swap
• Create Raid-1 for /boot and /
• Create FS for both (XFS for /boot; BTRFS for /)
• Let the installer do its job
• Reboot
• Wait until the initial sync is done
  (cat /proc/mdstat)
• mdadm --grow /dev/md1 --bitmap=internal

In the following table you can see my results.

(Installation of Debian always succeeded, therefore it’s not mentioned in the table)

My personal favourite is “Samsung FIT”, since it is (physically) small and passed all tests.

Unfortunately, I couldn’t see any systematic behind it. So, the reason for all this remains blurry…

I did at least once in the past, but I did not use it for this test.

I can't remember, why I decided against it then.

Maybe I will repeat the test with that tool, if I get curious enough...