Concept Design Questions (SnapRAID + Greyhole)

davidh2k · 26. Februar 2014

I'm thinking about a concept for a future NAS upgrade and/or suggesting this concept to others and have some questions about that too.

Concept:

Using greyhole and snapraid (Raid5 or even RAID6 liek) in combination (+maybe use AUFS)

I just read about an User testing that kind of scenario where he places a copy of all his fotos on each drive in the "array" and other files like media.

I like the idea of adding improving the level of data protection by using snapraid instead of a normal raid configuration. Until some months this software was totally unknown to me, so i need to ask some questions about how they work.

1.) Does greyhole allways balance files?
2.) Can it fill one hard disk and when full switch to another?
3.) Does it support some kind of intelligent balancing based on individual folders?
3.1.) You all know how much media I have. So I don't want to end up - even if the risk for that is so small - with only "some" episodes of a Season of a TV Series, instead of the whole season. So can greyhole ensure that files that belong togther, are stored together? The more I think over that, it may not be neccessary, but anyways good to know!
But I just think restoring a file here and there is more nasty than just to recover a whole season!
4.) How much CPU Power does snapraid need?
4.1.) How much CPU Power is needed for RAID6/ q-parity?
5.) Would AUFS be neccessary in any way? To pool the media into one big folder for sharing it to media devices?

Maybe later I will add more questions if I thought about this a bit more.

Thank you

Greetings
David

tekkb · 26. Februar 2014

1) No, but you can with the balance button.
2) If you add new disk you just use balance and the full disk will not be full anymore
3) Don't think so

4&4.1) snapraid should take less power then raid 6. not all disks have to spin in snapraid.
5) i've thought about aufs a lot. I just don't need if using Greyhole. You only need 1 media share with Greyhole and the real files will be moved to the pooled disks and symmlinks left in the media share.

PS- I talked to Ryeco about SnapRaid and Greyhole quite a bit ago. Other people using the combination like it.

davidh2k · 26. Februar 2014

Zitat von "tekkbebe"

1) No, but you can with the balance button.

Ah okay. So its not enabled by default.

Zitat von "tekkbebe"

2) If you add new disk you just use balance and the full disk will not be full anymore

From my writing you should know that especially I don't want it to balance, but rather group files together that belong together, so i won't hit that button then I guess.

Zitat von "tekkbebe"

3) Don't think so
4&4.1) snapraid should take less power then raid 6. not all disks have to spin in snapraid.

Especially want to know how CPU intense it is. Don't want to suggest to someone to go with a celeron CPU and then need 100% of the CPU Power for Snapraid Q-Parity, if used.

Zitat von "tekkbebe"

5) i've thought about aufs a lot. I just don't need if using Greyhole. You only need 1 media share with Greyhole and the real files will be moved to the pooled disks and symmlinks left in the media share.

If Greyhole supports it then I'm fine with that. Just read some people using AUFS to group theire media disks together, so I thought it may be neccessary for something.

Zitat von "tekkbebe"

PS- I talked to Ryeco about SnapRaid and Greyhole quite a bit ago. Other people using the combination like it.

Thank you. The thought about using it just popped to my mind today. It sounds like a real alternative that could suit my future system(s)!

Greetings
David

tekkb · 26. Februar 2014

The files are copied into the same folder structure with Greyhole. So it is not hard to find anything.

I would ask Solo on CPU usage. I don't think it is that intensive.

Greyhole is pretty neat. I've thought up new ways to use it. Once I do some more testing I'll write about it.

davidh2k · 27. Februar 2014

Zitat von "tekkbebe"

The files are copied into the same folder structure with Greyhole. So it is not hard to find anything.

Well, that was just in case of Disk Failure. I have no doubt that it's not hard to find anything while its running. :lol:

Zitat von "tekkbebe"

I would ask Solo on CPU usage. I don't think it is that intensive.

Well, no doubt for Snapraid ála RAID5, but I'm interested in the CPU Ressources needed for Q-Parity (RAID6 like). Ralf?

Zitat von "tekkbebe"

Greyhole is pretty neat. I've thought up new ways to use it. Once I do some more testing I'll write about it.

Allways happy to hear new stuff.

Greetings
David

tekkb · 27. Februar 2014

No Daivd, if you have disk failure and choose to make a backup (2 copies) you could attach the disk to another system and not have problems finding your files. Once you use it you will understand. And even if you had to use SnapRaid to restore a disk your files would be easy to find again in the .greyhole folder in the root of the restored drive.

ryecoaaron · 27. Februar 2014

Zitat von "davidh2k"

Well, no doubt for Snapraid ála RAID5, but I'm interested in the CPU Ressources needed for Q-Parity (RAID6 like). Ralf?

Snapraid isn't realtime raid. It only uses cpu resources when you tell it to do something.

davidh2k · 27. Februar 2014

Zitat von "tekkbebe"

No Daivd, if you have disk failure and choose to make a backup (2 copies) you could attach the disk to another system and not have problems finding your files. Once you use it you will understand. And even if you had to use SnapRaid to restore a disk your files would be easy to find again in the .greyhole folder in the root of the restored drive.

I was especially poiting to the situation where more than 1/2 drives fail. Also, can I acces the files via Commandline directly, and see them how they are presented by greyhole, in case I want to copy files locally from my storage to an external driver per-say.

Greetings
David

davidh2k · 27. Februar 2014

Zitat von "ryecoaaron"

Snapraid isn't realtime raid. It only uses cpu resources when you tell it to do something.

Would you enlighten me, how is the parity generation triggered then? How often is it done?

Greetings
David

tekkb · 27. Februar 2014

Yes, you can get to the files like they are in a normal folder. Try in a vm and you will not ask so many questions....hehe :lol: :lol:

davidh2k · 27. Februar 2014

Well, I just summarized my thoughts and questions about that while I was at work, so I thought I just throw them at you to get onto your nerves...

Greetings
David

davidh2k · 27. Februar 2014

Aaron: Would you recommend a mixed setup with AUFS and Greyhole according to this post?:

http://forums.openmediavault.o…wtopic.php?p=32469#p32469

Greetings
David

ryecoaaron · 27. Februar 2014

You either manual sync or use a script in cron/anacron like solo's script.

snapraid and greyhole make sense.
snapraid and aufs make sense.
greyhole and aufs doesn't make sense since they both create pools.

davidh2k · 27. Februar 2014

Well, I just thought it could be usefull to use greyhole for multiple copies of a file and AUFS for media files...

Greetings
David

ryecoaaron · 27. Februar 2014

I guess that would work

gboudreau · 27. Februar 2014

Zitat von "davidh2k"

1.) Does greyhole always balance files?
2.) Can it fill one hard disk and when full switch to another?
3.) Does it support some kind of intelligent balancing based on individual folders?
3.1.) You all know how much media I have. So I don't want to end up - even if the risk for that is so small - with only "some" episodes of a Season of a TV Series, instead of the whole season. So can greyhole ensure that files that belong togther, are stored together? The more I think over that, it may not be neccessary, but anyways good to know!
But I just think restoring a file here and there is more nasty than just to recover a whole season!

Not to make tekkbebe look like he doesn't know what he's saying, but...

Zitat

1.) Does greyhole always balance files?

By default, Greyhole balances free space. ie. when a new file appears, it will choose the drive with the most free space, and put the file there. But there are options to make it do whatever you want. More about that below.

Zitat

2.) Can it fill one hard disk and when full switch to another?

Yes. You need what Greyhole calls drive_selection_algorithm and drive_selection_groups.
With those, you can choose, on a per-share basis if you want, what drives will be picked as the recipient of new files.

For example, say you want to have only one file copy of each file (my guess; since you want to use SnapRaid), you could create drive_selection_groups that contain one drive each, and use those groups in succession in drive_selection_algorithm.
Example config:

Code

drive_selection_groups = HDD1: /mnt/hdd1/gh
	                      HDD2: /mnt/hdd2/gh
	                      HDD3: /mnt/hdd3/gh


drive_selection_algorithm = forced (1xHDD1, 1xHDD2, 1xHDD3) most_available_space

What this translates into is: to choose a drive, pick a drive in the HDD1 group first, if there's no more room in any of the HDDs in that group, then try to pick a HDD from the HDD2 group, etc.

And if you want to control that per share:

Code

drive_selection_groups[ShareName] = ...
drive_selection_algorithm[ShareName] = ...

Zitat

3.) Does it support some kind of intelligent balancing based on individual folders?

Yes. It's what we call sticky_files. Like they stick files together.
And you can define those per share, per folder. And you can also decide to send sticky files onto a specific drive if you'd like (for example, to always have on your USB HDD a copy of the stuff you like to bring with you at parties or whatnot!)

Example conf:

Code

sticky_files = Videos/TV Shows/*/*

This means: all folders that are two folders deep in the Videos share in the TV Shows folder will always 'stick' together, when possible.
For example, all files in "Videos/TV Shows/Heroes/Season 1" would be together, all files in "Videos/TV Shows/Heroes/Season 2" would be together, etc. But "Season 1" could be on one drive, while "Season 2" would be on another.

But since you want to fill each drive sequentially, you won't really need sticky files, since all your files will be together, except when you reach the end of a drive. Either way, you will end up with part of a folder on one drive, and the rest on another drive in that case, because Greyhole will not 'go back' and move the files from the 1st drive to the 2nd, just to keep sticky_files together.
You'll just have to live with this limitation, or you'll need to manually move the file copies each time you fill a drive!

Good luck.

ryecoaaron · 27. Februar 2014

Zitat von "gboudreau"

Not to make tekkbebe look like he doesn't know what he's saying, but...

I think you meant me And you are right, I really don't know much about greyhole. Just ported the plugin.

tekkb · 27. Februar 2014

I thnk I am beyond RAID 5 and SnapRAID. I show you guys something in a while...

davidh2k · 27. Februar 2014

Mister Boudreau,

You are awesome. I will add some more questions to your suggestions later when I'm home from work.

For now, you already improved the useability of Greyhole for me! Just by saying whats possible!

Greetings
David

0rtega · 27. Februar 2014

Zitat von "tekkbebe"

I thnk I am beyon RAID 5 and SnapRAID. I show you guys something in a while...

I'm very curious... better than raid 5 and snapraid ?? I can't wait to know on what are you working!

Jetzt mitmachen!