OpenMediaVault 0.6 Where are we going !

OMV is targeted for SOHO environment. Tell me how ZFS fits in that target scenario.

We can talk about encryption, but even that isn't that much of a high need feature. Look at the Plugin section what the users ask for ...

Zitat von "k567890"

[...]
PROS
-It is the best files system.

How do you define "best"? For me so far its XFS over EXT4, can't tell what ZFS would make better for me.

Zitat von "k567890"

-It has top-notch raid.

So does my hardwarecontroller.

Zitat von "k567890"

-It has encryption.

Linux allways has LUKS.

Zitat von "k567890"

-It has data integrity features that no-other raid does.

But it only has them or needs them because it uses deduplication. If you don't have that, you don't need more data integrety features besides a parity (or two, speaking of RAID6).

Zitat von "k567890"

-It is dirt easy...and I mean !EXTREMELY! easy to use with FreeNAS.

If its that what you need, you have to stick with FreeNAS for the moment (ZFS Plugin IS in development). However, what other things does FreeNAS offer that are better than what OMV offers?

Zitat von "k567890"

CONS
-Some left-wing die hard communists don't like it because it is not a member of the party.

A NAS without zfs is an amateur NAS at best. No offense, just my opinion.

No offense taken. But this is a NAS Software developed for a SOHO environment. Some people just forget about that or simply ignore that. And tell me who the fuck needs ZFS in that kind of enviroment?

Edit: Not to mention another CON for ZFS: The massive amount of RAM that is needed... A fileserver doesn't have to have much RAM (except Databaseservers). But if it comes to ZFS it has to have big amounts, which makes it much more expensive to use than any other technology around!

Greetings
David

Zitat

Edit: Not to mention another CON for ZFS: The massive amount of RAM that is needed... A fileserver doesn't have to have much RAM (except Databaseservers). But if it comes to ZFS it has to have big amounts, which makes it much more expensive to use than any other technology around!

That is not quite true. ZFS uses normal amounts of ram. It is only when deduplification is enabled that it needs 1GB of ram per TB of data. Deduplication is an eccentric feature that at most is useful in data centers. Alas, it is there if needed. Since it is not needed by 99.99% of people / businesses, there is actually no memory con. Simply disable deduplipication (disabled by default).

Zitat

But it only has them or needs them because it uses deduplication. If you don't have that, you don't need more data integrety features besides a parity (or two, speaking of RAID6).

No, not quite. Deduplipication is a separate issue than the data integrity feature of zfs. zfs has a built in mechanism that allows it to know when data has been corrupted on the hdd. So for example if you have a raid 1, 5, or what ever... when you try to access some data, zfs auto detects if on a particular disk that is being read from the data tha is being read is corrupt (ex. bitrot = happens a lot with these modern flaky drives). In such a case it auto fixes the issue by taking from good copy and either overwriting bad bits or remapping data. This is a must feature these days for home, soho, enterprise (all).

Zitat

Linux allways has LUKS.

The big picture is that everything has to work together. Raid, encryption, data integrity features. The technologies need to be compatible. On top of that, not only have all of these technologies in zfs been made compatible they have been designed/written from scratch to work together providing efficiency when it comes to performance and elegance when it comes to simplified usage and administration. Nothing can match it. XFS and EXT4 are only file systems. ZFS is a file system + disk managmanet software layer with data integrity and raid (among other features).

Zitat

So does my hardwarecontroller.
most commercial NAS boxes don't have ZFS. I would also say a server with hardware raid controller is not amateur either.

A good hardware raid controller costs $500+. Before you were talking about focusing on home and soho. It should be obvious to you why zfs if superior in soho.
Cost + easier to config. Also, zfs scales to much more drives than your hardware raid controller. You canjust keep adding drives to zfs. It is superior to your raid controller in every way. Add a UPS for $200 and you have real protection. Your baterry backed hardware raid controller still needs A ups... but a UPS + zfs offer full protection.

Zitat

OMV is targeted for SOHO environment. Tell me how ZFS fits in that target scenario.

Every point I made in this post explains that.

Aside from that, it has been said that most users don't need encryption. Are you kidding me? 100% of users need encryption.

I do agree that most users don't need raid... but they want it... and the key to making a product successful is to give your users what they want.

Why do you need encryption for your movies and pictures?

First off, only you know what "you" need. I don't think there is point in me telling you what you need because you know that the best, so let me focus on what I need instead.

I store documents, projects, code, etc. I shouldn't have to explain why important files need to be encrypted... the case is obvious.

Aside from that, if I wanted to store my family pictures and home movies in omv, personally I would definitely want it encrypted.

Zitat

PROS
- uses any filesystem (pretty sure ext and xfs are WELL tested).
- allow for different size drives to be used (zfs is not good at this).
- not all files are lost if more than the number of parity drives fail.
- power consumption is the best since only the drive you are reading from has to spin
- memory is very low since snapraid isn't running all the time
- Works with an already filled disk

CONS
-No encryption but that may be fixed with a plugin as well.

Alles anzeigen

Being able to use any file system is nice if true (prob is). Diff size hdds is of questionable benefit. As you said, most don't need or use raid, of the ones that do they buy matched hdds usually. So this caters to a very small group. I think it is silly to place an 80GB, 160GB and 2TB hdd in raid 5. If if this is even possible, I don't think people should be encourage to try. An eccentric feature.

-The not all files are lost point is interesting but I question just how much would not be lost. Just supply a realistic number of parity drives or don't run raid. That is my opinion. If you can't afford it, you shouldn't run it.

The major point in my opinion is what I've stated in the reply post above. The key is that you can't easily have encryption + raid + data integrity features (automatic data healing when blocks get corrupt) with glued together pieces without a lot of hassle and complexity on the part of the developers or the users. And if one piece in the chain changes it breaks the chain. For this reason zfs was designed from scratch to meet all these basic common goals. It is already made for you. Why do you want to reinvent the wheel?

I'm not against you doing what you are doing... just to be clear. My major gripe with omv is that it focuses of eccentric plug-ins and ignores basic functionality that a NAS should have (like good encryption = TrueCrypt or ZFS). Also, don't underestimate the value of automatic data block corruption healing of zfs (THIS IS AN EPIC FEATURE). It is by far the most important data feature invented since Microsoft made popular the journaled file system. I believe the new Microsoft file system also has some sort of data integrity feature like this... if true that would mean that there are only two usable file systems... ZFS and the new Microsoft filesystem. I recommend you stop wasting time on XFS and EXT4, those are garbage legacy file systems. Linux will have to dump them, it is just a matter of when. They will have to come up with a file system with data interity just like they had to add journaling (ext3 in 2001) after Microsoft came up with NTFS (1993).

Zitat von "k567890"

That is not quite true. ZFS uses normal amounts of ram. It is only when deduplification is enabled that it needs 1GB of ram per TB of data. Deduplication is an eccentric feature that at most is useful in data centers. Alas, it is there if needed. Since it is not needed by 99.99% of people / businesses, there is actually no memory con. Simply disable deduplipication (disabled by default).

Then what are the benefits of using ZFS over mdam?

Zitat von "k567890"

No, not quite. Deduplipication is a separate issue than the data integrity feature of zfs. zfs has a built in mechanism that allows it to know when data has been corrupted on the hdd. So for example if you have a raid 1, 5, or what ever... when you try to access some data, zfs auto detects if on a particular disk that is being read from the data tha is being read is corrupt (ex. bitrot = happens a lot with these modern flaky drives). In such a case it auto fixes the issue by taking from good copy and either overwriting bad bits or remapping data. This is a must feature these days for home, soho, enterprise (all).

Hmm... I don't know how well my controller handles that (and yes, this is just my humble opinion to use a hardware raid controller, others do software raid just fine) but every week I do a verify of my data integrity.

Zitat von "k567890"

The big picture is that everything has to work together. Raid, encryption, data integrity features. The technologies need to be compatible. On top of that, not only have all of these technologies in zfs been made compatible they have been designed/written from scratch to work together providing efficiency when it comes to performance and elegance when it comes to simplified usage and administration. Nothing can match it. XFS and EXT4 are only file systems. ZFS is a file system + disk managmanet software layer with data integrity and raid (among other features).

Well, I don't see how ZFS is superior to mdadm (or a hardwareraidcontroller) + XFS/EXT4 then? Except the realtime check for data integrity!

Zitat von "k567890"

I didn't say that my NAS is an amateur NAS, but I prefer it this way. I only said that OMV is targeted for amateuers (and also intermediates of course!), but is far from a professional Solution!

A good hardware raid controller costs $500+. Before you were talking about focusing on home and soho. It should be obvious to you why zfs if superior in soho.
Cost + easier to config. Also, zfs scales to much more drives than your hardware raid controller. You canjust keep adding drives to zfs. It is superior to your raid controller in every way. Add a UPS for $200 and you have real protection. Your baterry backed hardware raid controller still needs A ups... but a UPS + zfs offer full protection.

My Controller cost me 200 €. But it was a used one (both the old and the new), so I don't want to argue on that point.
A BBU only is needed if you are scared of data loss due to enabled write cache. I'm not scared of this so neither do I have a BBU nor a UPS. My NAS also doesn't run 24/7 unlike other users' NAS systems.

Zitat von "k567890"

Every point I made in this post explains that.

Zitat von "k567890"

Aside from that, it has been said that most users don't need encryption. Are you kidding me? 100% of users need encryption.

It may be that every user should use encryption these days, but clearly a large part of them don't need encryption. It just adds another layer of complexity which scares most people away and at least in my scenario it would aggravate my NAS use. I used encryption myself for many years, both on my NAS and my PC, but I stepped away from it 3 years ago.

Zitat von "k567890"

I do agree that most users don't need raid... but they want it... and the key to making a product successful is to give your users what they want.

Yeah, they want it because they think that it improves theire data security while they totally forget that only a backup can secure theire data! Don't misjudge the capabilities of a RAID with the security of a real BACKUP!

Greetings
David

Why do I need zfs when I backup to another server?

Zitat von "k567890"

[...]
I store documents, projects, code, etc. I shouldn't have to explain why important files need to be encrypted... the case is obvious.

Aside from that, if I wanted to store my family pictures and home movies in omv, personally I would definitely want it encrypted. [...]

Don't get me wrong, did you thought about a Truecrypt container? Which lies on a OMV share and is mounted locally on your system.

Zitat von "k567890"

[...]
Being able to use any file system is nice if true (prob is). Diff size hdds is of questionable benefit. As you said, most don't need or use raid, of the ones that do they buy matched hdds usually. So this caters to a very small group. I think it is silly to place an 80GB, 160GB and 2TB hdd in raid 5. If if this is even possible, I don't think people should be encourage to try. An eccentric feature.

A common question on the forums: "How can I do a raid with my 160GB, 320GB and 500GB drive???". Nuff said? Also, have a look at greyhole which is designed for this use environment. Many people just want to use theire old drives rather than buy a couple of new drives because they either don't have the money or don't have that much need of space.

Zitat von "k567890"

-The not all files are lost point is interesting but I question just how much would not be lost. Just supply a realistic number of parity drives or don't run raid. That is my opinion. If you can't afford it, you shouldn't run it.

Thats the benefit of snapraid, that even if two drives fail (where when using raid5 ALL of your data would be lost) you still have the data on the working drives.

Zitat von "k567890"

The major point in my opinion is what I've stated in the reply post above. The key is that you can't easily have encryption + raid + data integrity features (automatic data healing when blocks get corrupt) with glued together pieces without a lot of hassle and complexity on the part of the developers or the users. And if one piece in the chain changes it breaks the chain. For this reason zfs was designed from scratch to meet all these basic common goals. It is already made for you. Why do you want to reinvent the wheel?

Nobody wants to reinvent the wheel. Volker Theile is the only active developer on OMV itself. He set himself some goals and made some decisions how he wants to design OMV and were he thinks his target should be. His target is clearly a SOHO environment and therefore he denied to implement ZFS himself, alltough he improved the storage backend to enable the plugin developers to be able to support ZFS via a plugin!

Zitat von "k567890"

I'm not against you doing what you are doing... just to be clear. My major gripe with omv is that it focuses of eccentric plug-ins and ignores basic functionality that a NAS should have (like good encryption = TrueCrypt or ZFS). Also, don't underestimate the value of automatic data block corruption healing of zfs (THIS IS AN EPIC FEATURE).

Lets just say (even if the chance of this is rather low) that one file (which is split between two drives + parity drive) gets corrupt, but not just the block on one hard drive but simultaniously on two drives. I don't see how even ZFS can recover this while using a RAID5 like environment.

Zitat von "k567890"

It is by far the most important data feature invented since Microsoft made popular the journaled file system. I believe the new Microsoft file system also has some sort of data integrity feature like this... if true that would mean that there are only two usable file systems... ZFS and the new Microsoft filesystem. I recommend you stop wasting time on XFS and EXT4, those are garbage legacy file systems. Linux will have to dump them, it is just a matter of when. They will have to come up with a file system with data interity just like they had to add journaling (ext3 in 2001) after Microsoft came up with NTFS (1993).

I highly doubt your statement above. Thats like saying all manufactures have to jump onto BTX because ATX is at the end of its cooling capacity. Five years later even Intel buried it...

Greetings
David

ZFS has built in RAID and volume management capabilities (It covers what can be done with software RAID and LVM). You don't use mdam with it.

Your hardware controller doesn't handle it at all. Only ZFS can handle data and meta data corruption detection and automatic fixing. If your blocks start to go bad on one of your HDDs on your hardware raid you are shit out of luck. You can't fix it. There is no way you can fix it. The only thing that can save your ass is a backup that is outside of your raid (as you said).

Yep, trust me... I know what RAID is and is not. I agree with you about most peoples misplaced trust in it.

Zitat

Why do I need zfs when I backup to another server?

You don't. Again, the point is guys, you have a fully crafted elite top of the line solution with pretty much no cons. It does everything for you in an elegant way. Use the features you want only. The point is with just a few developers you are wasting man hours and after all that work you don't have features that many people want. I can't make it more plain. Make zfs work (this is only one thing) and every other major NAS feature is done for you free because it has it all.

ZFS should not be a plugin. This should be a core feature...or at least a core plugin that comes with omv.

Last time I checked ZFS was not a core feature in most commercial nas devices. There is work going on to implement zfs in OMV. So why the big discussion? It will not speed up development.

There was no intention to turn it into a big discussion. It just turned out that way. We're simply discussing its value on the totem pole based on its pros and cons.

Zitat von "k567890"

ZFS has built in RAID and volume management capabilities (It covers what can be done with software RAID and LVM). You don't use mdam with it.

Thats why i asked what are the benifits of ZFS via mdadm+XFS/EXT4 are.

Zitat von "k567890"

Your hardware controller doesn't handle it at all. Only ZFS can handle data and meta data corruption detection and automatic fixing. If your blocks start to go bad on one of your HDDs on your hardware raid you are shit out of luck. You can't fix it. There is no way you can fix it. The only thing that can save your ass is a backup that is outside of your raid (as you said).

Then I'm glad to tell you that I CAN fix this errors. And my Hardwarecontroller takes care of that because I use WD REDs (look it up, they improve this behaviour a lot, because they don't try to correct as long as normal drives) which leave the sector correction to the hardwarecontroller!

Zitat von "k567890"

Yep, trust me... I know what RAID is and is not. I agree with you about most peoples misplaced trust in it.

Good.

Zitat von "k567890"

You don't. Again, the point is guys, you have a fully crafted elite top of the line solution with pretty much no cons. It does everything for you in an elegant way. Use the features you want only. The point is with just a few developers you are wasting man hours and after all that work you don't have features that many people want. I can't make it more plain. Make zfs work (this is only one thing) and every other major NAS feature is done for you free because it has it all.

ZFS should not be a plugin. This should be a core feature...or at least a core plugin that comes with zfs.

See my statement for this. At least thats Volkers opinion on that.

Greetings
David

^ you are right, I didn't read the linked info but I will.

I'll tell you this though now... I just don't see how it is possible (and I'm pretty sure it is not since it would defy all logic)to have a 120GB, 320GB, and a 2TB hard drive in raid and not have data loss if the 2TB dies. Obviosussly this is only possible if you are not using the full capacity of these drives... you prob have to limit yourself to 120GB per drive (the smallest drive). As such, it makes the whole thing a joke.

Zitat

you lose the data only on the failed disks. All the data in the other disks is safe.

That sounds like data isn't striped... so yo lose the performance benefits of raid.

So basically it isn't even raid and shouldn't even be called that. That's like calling a donkey a Ferari.
Really, something like this needs a new name. It isn't proper to call it raid. Never the less, it is an interesting product.

OMV is a great project, but I think in order to answer the question of where the project is going, you have to look at the current users of OMV. I believe (I may be wrong) that most OMV users are either
1. people who want a powerful top of the line home NAS and are able/willing to pay for it, or
2. people who want a simple home NAS but do not want to spend $ on a new NAS appliance. The GUI and ability to repurpose old hardware is a big attraction for these users.

OMV caters to both groups currently because it combines a low resource using OS+software combo with high end features. The questions of whether ZFS should be included in OMV and whether it should be 64bit only illustrates this split of user groups. Many of Group 1 (the high end users) want ZFS and 64bit only since it would maximize the potential of their high end hardware, while for many of Group 2 (low end hardware users) switching to ZFS and 64bit only might prevent them from using OMV on their low end hardware.

I myself was using OMV since it came out (and FreeNAS before that), but have since switched to running a NAS using Raspbian on a Raspberry Pi. The hardware is totally low end - as low as it can go but suits my needs. To me (again I could be wrong), if the project wants more users and to attract a lot of attention, it needs to choose one of the groups and run with it. Either cater fully to the high end hardware group and be known as the best high performance NAS software, or go for the huge number of users who want a low end low cost NAS to run on their old hardware. A version for the Raspberry Pi wouldn't hurt (hint).

Either way I am still a big fan of the project.

I think one of the most important things I've learned from the WHS 2011 community is how important drive pooling is. The ability to have redundancy with varied size drives is a point I've tried to drive home to all the mods and Volker. If we want to increase our user base we need to make Greyhole a priority. I also think we should take more time in evaluating a project before time is wasted on it as a plugin. We should evaluate it for bugs and fit within OMV. I was just spending some time tonight on ideas for plugins. It would be nice to have some of them I looked at but they are not ready. I went through this process with Serviio a while back. Serviio needs to get their web interface integrated into the core of the project, not a 3rd Party side project. Tonight I was going back over Mobile PC Monitor and some others. Developers of projects should be aware of the possibility of their projects being included in other projects like OMV. They should have repos setup for their projects, especially when considering upgrades. The projects that just have a tar file are ridiculous. I have a very high opinion of Boudreau and his Greyhole project. In economics there is a term opportunity cost. I feel time spent on OwnCloud plugin would have been better spent on Greyhole. Volker invested a lot time on the Owncloud plugin and it is still buggy. Again, we need to spend a little more time on the frontend assessing programs before devoting a good deal more time on plugin development. I have to mention that almost all plugin work currently is being done by Ryeco, Sub and Volker.

Being that OMV is based on Debian I feel we can successfully cater to a varied group of users. The armhf and armel users it is a bit more difficult. I don't think it is the goal of the project to serve the very low end devices. If it was there would not be work on the backend to enable zfs.