How best to use BTRFS for both OS and Data in OMV6?

Soma · Feb 15th 2023

Quote from geaves

You should write this stuff up, this is beginning to make some sort of sense, you have a better way of explaining it than the stuff I've read

I read this as a real layman and try to write in the same way I understand it.

Although, sometimes I have a hard time understanding myself,

Maybe Krisbee can also give some input on what profiles can work best in some specific cases.

As per OMV use, I would like to have an idea on what profile would fit better on a disk that is meant to hold only MEDIA to share on the LAN (Videos, Music, etc). These are usually, "static" files on disk just to be "Read".

So, does it make any sense in having bitrot protection on this?

Or, is this the exact situation that requires bitrot protection?

And on the inverted situation:

A disk where a lot of writtings happen all the time? I already know how and why to avoid "write amplification" due to databases constant writtings.

But, what about the other different scenarios:

A webserver, for eg? Or file browser with lots of files upload/download? Nextcloud with it's cloud function?

Still in the grey zone area and swimming in "muddy waters", on some subjects.

Krisbee · Feb 15th 2023

With relatively few hours using and reading about BTRFS, I'm not sure I should be advising others. But to answer Soma, I'd say the profile choice for storing "media" type data is between single or raid1. If the total required storage space is greater than any one individual drive you have, then it must be raid1 if you want redundancy. I don't see raid10 being useful for streaming media data unless the number of simultaneous clients demands the higher performance, and of course you need a minimum of 4 drives.

You could argue with the size of modern drives that maybe only a single HDD is needed by a lot of home users, plus whatever is used for backups. I'd want to at least detect bit-rot if I had taken many hours to rip a personal CD/DVD collection to disk, so a BTRFS single profile is still useful if you only had one HDD.

One stated advantage of BTRFS is you can combine disks of unequal size. But there is a gothca described here: https://wiki.tnonline.net/w/Btrfs/Profiles under "size restrictions with multiple drives" which is why the "btrfs disk usage calculator" referred to by Soma is a reminder that available space is not simply half of the sum of the individual disk capacity in a raid1 profile. For example, throwing together a 500GB, 1TB & 2TB drive in a raid1 profile gives an approx pool of 1.5TB and not 1.75TB. You effectively loose 500GB on the 2TB drive . If you had two 4TB drives in a raid1 and wanted to expand storage by adding a 10TB drive, then only an additional 8TB could be used. (This figures are approx as metadata storage is ignored).

Adding or removing drives from a BTRFS array or changing profiles on the fly as Soma described needs to be treated with care.

Using BTRFS for write intensive storage in the case of databases and virtual machines is questionable. For copy-on-write filesystems fragmentation and "write amplification" via read-modify-write cycles are a fact of life. The later can prematurely wear out typical consumer SSDs. ZFS can partly mitigate "write amplification" via tunable filesystem recordsize and zvol volblocksize. BTRFS does not have this. Performance suffers and the common advice to disable CoW, which in turn disables checksumming, in these cases is IMHO not an answer.

Using docker on btrfs is another area to think about.

Krisbee · Feb 16th 2023

Soma A few additional comments. Just re-read the debian BTRFS WIKI. It’s hardly a glowing endorsement of BTRFS with many warnings and recommendations. One thing I took note of was that the WIKI says this:

“Btrfs' raid10 profile is currently unoptimised and usually performs identically to or worse than the same disks in raid1 profile. Given the raid10 profile's added complexity, it is clear that raid1, should continue to be preferred at this time.”

If true, that really limits the choice of which BTRFS profiles are of practical use.

The WIKI also includes this recommendation:

“Use two (ideally three) equally sized disks, partition them identically, and add each partition to a btrfs raid1 profile volume.

Alternatively, dedicate 1/3 of the disks for holding backups, because not much benefit in throughput or iops is yet gained by using btrfs raid1.
~~Alternatively use raid1c3 or raid1c4~~ (Qu Wenruo, linux-btrfs, 2022-06-07) “

So, in other words, should you bother with a raid1 profile at all?

Returning to the question of storing “write once, ready many times” data, In this case, BTRFS snapshots may be of little importance other than a means to perform BTRFS send/receive type backups. Do you really need “real-time” raid on such data? You get checksums with mergerfs +snapraid which has made it a popular choice among OMV users.

In the case of mdadm raid, using dm-integrity is supposed to provide error checking and error correction. Maybe geaves can comment on the use of mdadam raid + dm-integrity, it’s not something I’ve seen mentioned on the forum.

geaves · Feb 16th 2023

Quote from Krisbee

can comment on the use of mdadam raid + dm-integrity, it’s not something I’ve seen mentioned on the forum.

TBH not something I've used and have little knowledge of other than what I've read and for home use it would be pushing unnecessary complexity into the setup.

Before OMV I had used a Raid setup on Ubuntu and Debian, I even had symlinks from one server to another, but I never had an issue, I ran a regular backup and SMART tests which allowed me to ensure the integrity of my data and hardware.

With regard to file systems some offer more 'bang for your buck', but again in a home environment is the extra 'bang' a necessity for home use when users are coming into a home nas for the first time, from a windows or mac background.

OMV has always worked on the KIS principle, but since I started using it from V3 it has evolved, but it still has it's KIS principle, 'it does what it says on the tin'. But a user is given options as to how they want their system setup, but to maintain that KIS principle users should be encouraged how to 'look after' their data, because I think in some cases there is a 'set and forget' mentality. Then when it goes down the sh!tter they come here seeking help.

Compared to Windows my Linux knowledge doesn't scratch the surface, but it comes from real world usage and getting my own arse out of trouble Soma real world usage of BTRFS and the context he's laid it out in has helped me a little to understand it, would I use it, No, for me it goes beyond the KIS principle for home use.

What I like about Linux is it's ability to run on any hardware, it doesn't require you to run huge amounts of ram, xeon processor, (or the latest generation) or a server that sounds like a jet engine when booting up. Stick to KIS for the newly initiated and let the advanced users play with the stuff that would simply baffle a new user.

I once said in the forum on a thread, take a RPi install Raspberry Pi OS connect a hard drive, create some shares, add users if necessary plug into your network, it's still a NAS

Soma · Feb 16th 2023

Quote from geaves

the KIS principle

Now, it was me who was feeling on a different planet, had to search for this,

Quote from geaves

I once said in the forum on a thread, take a RPi install Raspberry Pi OS connect a hard drive, create some shares, add users if necessary plug into your network, it's still a NAS

I can vouche for this,

My Nextcloud is so tuned (on the Pi) that I see almost no difference when opening the Photos page (with lots of thumbnails) on the Pi or on a regular PC (low specs, of course) running NC on it's defaults..

Of course, if someone is using OMV to serve and transcode the "latest-greatest" 8K movies/tvshows, etc... just because it's nice to say that you have them available, then a bleeding edge hardware is needed.

If you use the NAS as a holder for your(not only but mainly) "really important DATA that your world will end if you lose it", then it makes more sense in having the simplicity of beeing able to recover it in a heartbeat, instead of having to figure out: what do I do now????

This "playing around" on BTRFS is just a way to gain some understanding, especially on RAID (area that I didn't had ANY knowledge until I had access to a Synology at work that went FUBAR and the whole Windfarm INFO with it. We recovered the INFO from Central, no worries).

It is also a way to experiment new things but, in truth, it's more of a pass time for the days I don't have work to do,

Although I want to know more and more about it, in the end, my practical use of BTRFS will end up as the same use as any other FS, be it ext4, ZFS, XFS, NTFS, FAT32, etc:

Drives with DATA, in the x-type format to hold y-amount of data with z-easiness for clone/backup.

After figuring out what is the easiest, less troublesome type to use, I'll just stick to it and forget the rest.

In the end, ext4 from a newbie perspective wins BIG TIME.

Krisbee · Feb 16th 2023

I wouldn't disagree with either of the last two posts. It will be interesting to see how far votdev goes with adding BTRFS functions to OMV and what changes between OMV6 and OMV7. For example, might mdadm raid move to a plugin and BTRFS become the core RAID1 method?

Soma · Feb 16th 2023

Quote from Krisbee

For example, might mdadm raid move to a plugin and BTRFS become the core RAID1 method?

I believe this would be the best in the sense that BTRFS functionalities surpass the one's for MDADM.

Of course, this implies some knowledge from USERS when it starts adding profiles to the mix.

But a simple -mraid1 -draid1 with timely scrubs will clear errors on the SPOT. (and still survive a disk failure)

Convert this to -mDUP -draid1 and you gain space with some resilience (but loose the ability to survive a disk failure)

I mean, the possibilities are immense for those who want/have the knowledge on their functions and really need them.

Can we say the same of MDADM?!?

geaves · Feb 16th 2023

There would be no need to move mdadm to a plugin, BTRFS is already a file system option, the most sensible thing would be to add a BTRFS Raid1 function along with BTRFS Raid profiles for those that want to use it.

We're now moving into the realms of a previous topic/discussion

crashtest · Feb 16th 2023

Quote from Krisbee

if you use BTRFS and that snapshots give you access to previous versions of data via SMB/CIFS shares where shadow copies are configured and check sums can offer self-healing for certain raid profiles.

If you're willing to set aside the licensing issue, I have all of the above with ZFS with fully customizable and automated snapshots, and self healing scrubs. Once it's set up, it pretty close to self maintaining. I've found drive change outs to be easy, moving the pool between OMV versions is straight forward, and selective file/folder restorations (past versions or deletes) are simple.

Quote from Soma

I would like to have an idea on what profile would fit better on a disk that is meant to hold only MEDIA to share on the LAN (Videos, Music, etc). These are usually, "static" files on disk just to be "Read".
So, does it make any sense in having bitrot protection on this?
Or, is this the exact situation that requires bitrot protection?

I believe a good argument would rest on, "how long does the user want to keep error free data"? I have some data items that go back to Windows 98 and even earlier. Over that span of time, the chance of hard drive failures and/or data destruction by silent corruption becomes a certainty. Thankfully, with the use of backup, monitoring of hard drives (replacing them when they became questionable) and a generous portion of "luck", I've managed to maintain older data stores. That's part of the reason why I tested and adopted ZFS for data drives. Most of what I used to do manually is now automated.

For boot drives:
ryecoaaron mentioned one of his laptops that boots from ZFS. I think he said that running apt triggers a boot drive snapshot, before changes are made to the boot drive. In the event that an update went south, a feature like that would be useful for restoring an earlier state.

ryecoaaron · Feb 16th 2023

Quote from crashtest

mentioned one of his laptops that boots from ZFS. I think he said that running apt triggers a boot drive snapshot, before changes are made to the boot drive. In the event that an update went south, a feature like that would be useful for restoring an earlier state.

Seems like zsys (what my laptop was using - reinstalled to 23.04 without zfs) is not going to be maintained anymore. Timeshift (maintained by Linux Mint) can do this with ext4 or btrfs. I thought about a timeshift plugin but you can see how far that got.

Soma · Feb 16th 2023

Quote from crashtest

I believe a good argument would rest on, "how long does the user want to keep error free data"?

Well, this made me re-think my whole perspective on what is my final goal for DATA protection.

The original question was about MEDIA data, but now, I have to consider that it's the least important DATA to protect.

Movies, TVshows, Music can be as volatile as one wants. In practical terms, you can always get it "somewhere" so, even if the DATA is gone, all you lose is time to "fetch" it again.

When you said, "since windows98", the bell rang and I only remembered the huge pile of burned CDRs that have gone to waste for beeing stored for a long time and can't be read anymore due to read errors.

The only DATA that I can't afford to lose is the gazillion photos, documents from the "Golden Years".

My wife is even "worst" than me: the DATA she has is ten times more than mine.

With this in mind, I (now) realize that the BACKUP is only as secured as the ORIGINAL.

So, the ORIGINAL has to be the most secured and protected part of the equation: if it corrupts on the original and you don't see it or realize , it will transpire to the BACKUP.

Adopting the (3)(2)(1) backup strategy from BackBlaze:

In practical terms, for "THE REAL DATA THAT MATTERS" and to keep the most basic and easy way, I believe a solution with a BTRFS RAID1 on 2x SSDs (3) then BTRFS SEND to another HDD/SSD on a different device (2) that will then be configure with MegaNZ cloud hosting (1) is the one for me.

Adding SNAPSHOTS on it, doesn't make me much sense since the files won't have much change once they're written (OK, if the idea is also prevent ransomware, maybe it's better have it but that's what the other copies exist for also).

What is mandatory here is the RAID1 profile that gives immediate repair on SCRUB on the ORIGINAL. Add a 3rd drive and change the profile to RAID1c3 and there's 3x copies of any file.

So, thank you crashtest for putting my noodle-in-motion,

Quote from crashtest

For boot drives:
ryecoaaron mentioned one of his laptops that boots from ZFS. I think he said that running apt triggers a boot drive snapshot, before changes are made to the boot drive. In the event that an update went south, a feature like that would be useful for restoring an earlier state.

I also have it but in BTRFS on the Pi,

It was fun to do only. In real terms (on a SBC point-of-view) it doesn't replace the "powerdown, clone, flash, boot" easiness.

On x86_64 I have no idea,

crashtest · Feb 16th 2023

Quote from ryecoaaron

Seems like zsys (what my laptop was using - reinstalled to 23.04 without zfs) is not going to be maintained anymore. Timeshift (maintained by Linux Mint) can do this with ext4 or btrfs. I thought about a timeshift plugin but you can see how far that got.

That's kind of a shame about dropping ZFS boot drive support. I'd almost be willing to bet that it was because users were mucking it up. Support can have a long tail.

I had planned to look at Timeshift on the CLI, during snow days, but this winter has been very mild so I've been doing other things outside. (I've been setting up bee hives for spring swarms.)

Quote from Soma

With this in mind, I (now) realize that the BACKUP is only as secured as the ORIGINAL.
So, the ORIGINAL has to be the most secured and protected part of the equation: if it corrupts on the original and you don't see it or realize , it will transpire to the BACKUP.

You nailed it here. That concept is CRUCIAL. What's the point in backing up even slightly corrupted data? I'm running a server with ECC RAM partly because of theoretical cosmic ray "bit-flips". (While the actual reason is debatable, RAM bit-flips are a reality.) In the same vein, along with errors from magnetic media degradation, bit's change polarity on hard drives as well.

This is where a ZFS mirror shines. Top level data is maintained in a pristine state. I've tested ZFS with artificial bit-rot inserted into a file, using a sector editor. It works. ZFS recovers it. From my primary server, I backup to a variety of platforms. One backup is running a ZFS mirror, but I also maintain an R-PI4 (with SnapRAID) and an HC4 for user doc's and forum support.

Quote from Soma

The original question was about MEDIA data, but now, I have to consider that it's the least important DATA to protect.

Well, I look at it like this, what's my time worth? If the task of collecting it up and sorting it out again takes more than 3 or 4 hours, I'll back up it up. These days, 4TB drives are relatively cheap and they have lot of capacity. -> 4TB Seagate. (SMR drives are a bit cheaper and well suited to be a backup destination.)

I'm also noticing, as time passes, that media from back in the day is disappearing from the Internet. These days, archives more that 7 years old may be hard to find. (7 years, interestingly, is roughly the max life of a hard drive.)

Quote from Soma

In real terms (on a SBC point-of-view) it doesn't replace the "powerdown, clone, flash, boot" easiness.

On x86_64 I have no idea,

I do the same thing with X86 - AMD64, cloning thumbdrives, and SBC SD-cards. It's dirt simple and it works. Restores are a couple of minutes, just plug it in and boot.

Griffo · Feb 27th 2023

Quote from Krisbee

Griffo AFAIK, there's no reason to re-create a pre-existing BTRFS RAID, but snapshots created via the WebUI my not be compatible with the use snapper or btrbk.

Actually I found that the snapshot button was not available on "inherited" BTRFS volumes, but were on a fresh one made via the GUI.

But now I find that i can create snapshots, but the UI does not display them

e.g - can see them via CLI, these were created by clicking the + button in the GUI.

root@lusus:~# btrfs subvolume list /srv/dev-disk-by-uuid-6598e7b1-cf34-4aa2-bbe6-f90ef25c706a

ID 9203 gen 174000 top level 5 path .snapshots

ID 9204 gen 173999 top level 9203 path .snapshots/vault_2023-02-25T11:48:05

ID 9205 gen 174000 top level 9203 path .snapshots/vault_20230225T115147

but cannot see them in the GUI

votdev · Feb 27th 2023

Quote from Griffo

Actually I found that the snapshot button was not available on "inherited" BTRFS volumes, but were on a fresh one made via the GUI.

But now I find that i can create snapshots, but the UI does not display them

e.g - can see them via CLI, these were created by clicking the + button in the GUI.
root@lusus:~# btrfs subvolume list /srv/dev-disk-by-uuid-6598e7b1-cf34-4aa2-bbe6-f90ef25c706a
ID 9203 gen 174000 top level 5 path .snapshots
ID 9204 gen 173999 top level 9203 path .snapshots/vault_2023-02-25T11:48:05
ID 9205 gen 174000 top level 9203 path .snapshots/vault_20230225T115147

but cannot see them in the GUI

Display More

Can not reproduce that behaviour. Have you tried to clear the browser cache?

Griffo · Feb 28th 2023

Quote from votdev

Can not reproduce that behaviour. Have you tried to clear the browser cache?

Yes and i've tried different browsers

I just tried again, same behaviour

I just tried on a different share and it works.. I'll not try to troubleshoot the non-working share in case you want any further details.

chente · Mar 16th 2023

This thread may not be the right one, but it's the closest I've found.

My question is: Is it possible to recover data from a single BTRFS disk without Raid only with the metadata (DUP or whatever it's called)? My brain tells me that it is not possible, but I find conflicting information on google. Maybe someone here can tell you quickly.

My motivation is to set BTRFS on my backups or not.

I currently have redundancy on my main system with a Raidz1. Unfortunately it is not possible to do a guaranteed raid 5 in BTRFS, and a raid 1 costs a lot of money. This Raidz1 fixes file degradations by bitrot. None of my backups (ext4 without redundancy) fix the bitrot. I fix this problem with, so to speak, a monthly "scrub" using rsync with --checksum (sometimes this has modified already saved files). This monthly rsync operation lasts much longer than the rest, but it corrects possible transfer errors, bitrot, etc. This avoids having additional protection in the backups, I take advantage of the protection of the main system and "transfer" it to the backups without additional hard drives.

I'm fine with this solution, but if BTRFS can do anything else without disk redundancy I'd consider implementing it on backups.

Krisbee · Mar 16th 2023

chente

https://wiki.tnonline.net/w/Btrfs/Profiles - DUP mode protects against data or metadata corruption, but not disk failures.

So better than ext4 on a single device as BTRFS scrub detects errors but there's no disk redundancy. But you could argue that zfs on a single disk as backup enables the use the more efficient incremental zfs send/receive instead of rsync from your main pool. Of course, you then loose flexibility as the backup can only be read on another zfs system.

If your raidz1 is a way of using multiple disk and real time redundancy is not crucial, you can stand the problem on its head and use a non-redundant pool on you main system and backup to a redundant pool.

chente · Mar 16th 2023

Quote from Krisbee

DUP mode protects against data or metadata corruption, but not disk failures.

Thank you. That was what he needed to confirm. I will study it better.

Quote from Krisbee

use a non-redundant pool on you main system and backup to a redundant pool.

This makes no sense. If one of the two systems is redundant, it better be the main one.

Krisbee · Mar 16th 2023

Quote from chente

Thank you. That was what he needed to confirm. I will study it better.
This makes no sense. If one of the two systems is redundant, it better be the main one.

Why? Which can you absolutely not afford to lose? Backup or main? In raidz1 scenario, your data is gone if you lose more than one drive. Raidz1 is not recommend because of chances of a second disk failure during pool resilver after replacing a failed drive. The worst happens, you lose another disk on your main pool and turn to your backup. Now, your backup without redundancy better work.

chente · Mar 16th 2023

Quote from Krisbee

Raidz1 is not recommend because of chances of a second disk failure during pool resilver after replacing a failed drive.

From that approach this conversation no longer makes sense, because it would not make sense to use raidz1.

If I lose the second drive during recovery I have the backup, that's why I do it.

Let's assume that raidz1 will work as expected. Then it will be easier and more logical to recover the system by replacing a disk than by copying everything from the backup.

I'm not trying to convince you of anything, I'm just responding to your comment.

Participate now!