New in town, looking for advice

  • Hey guys,


    TL;DR? First-timer looking for advice on RAID capabilities and partitioning best practices for setting up a media (and general file) server. Have also bolded direct questions to make this wall of text easier to process.


    I'm one of those people who ends up with a load of random parts from repairing other people's computers, and so after having a clean out of a bunch of stuff and reorgansing the state of my desktop machine, I've decided to start looking in to setting up a NAS drive for the home network. I've never dabbled in any server-y type stuff before, have almost zero exposure to Linux systems (but am happy to learn, I pick stuff up quickly, generally) and so some of this stuff might be stupid, no-brainer stuff, but I've googled around a lot a things keep leading me back here and the community looks to be a good collaborative group :)


    So. Initially I wanted to look at setting up a simple file server to just store/organise my various files in a central location. After a little bit of thought, I figured that it'd be good to actually set the box up to be a media server so then we could stream media from it too (I have an extensive digital music collection and some movies/series lying around). So that's where I started my google journey...


    Hardware


    I know the hardware will generally dictate how I go about assessing things. I have an old Windows 10 machine coming my way, but with the old owner being not-so-technical I only know a little bit about it:


    • Processor: Intel i5 4460 (3.2 Ghz).
    • 8 GB RAM (I assume it's DDR3, possibly 1833 Mhz since it's from a 6-7 gaming-focused build).


    Once I've got the machine in hand then I'll be able to give more detail about the motherboard, etc... but for now, this is the info I'm working with. From what I've read, this should be good enough to achieve what I'm after.


    In terms of storage, I've got 5 (possibly 6) HDDs at my disposal: 2x 1TB, 500GB, 320GB, 160GB (and possibly another 1TB depending on how things go with this). which is where my first hurdle comes in (I know that part of this is dependent on how many SATA connections I have on the motherboard, but I'm assuming I've got 4 to play with and that I could potentially expand the number of SATA ports).


    Storage


    Ideally I'd be pooling my storage using RAID 5 to have some redundancy for possible drive failures, but most things I've read point to a hard requirement for all drives in the array to be the same/similar size. A few things I read point to using SnapRAID in conjunction with mergefs, but also read that SnapRAID doesn't deal nicely with smaller sized files. Does anyone have any info on what is considered small? I don't want to go down the garden path on this and have to walk some of this stuff back. I figure that I could always keep smaller files in a drive not in the array, but want to know what my options are here.


    Streaming


    A lot of the use cases I've seen have people using Plex and while I'm not opposed to using it, I was wondering if there was something else I use/do to serve the media (transcoded on the server end) to client devices? Since I'm dealing with audio and not just video, is there a better solution? A client here could be another computer/laptop, mobile device and possibly a TV (Samsung, a couple of years old, if that matters?).


    File Server


    Assuming I've got my storage solution worked out, are there any tips you guys have for dealing with serving non-media files in this setup? I was thinking that I might have to separate the storage for media vs files so that I don't get any weirdness, but admittedly I'm out of my depth here - I have no experience with networking, but that's kind of the point of this project.


    Webserver


    This is more of a nice-to-have and isn't necessary at all, but I thought it might be cool to try and set up something simple to serve simple web apps on the local network - preferably nodejs based, don't necessarily need all of the overhead of PHP and everything it requires. with the outlined set up, do you guys think this is possible to do on the same box? I could live without this, honestly.


    I apologise in advance if this would be best placed somewhere else, and thanks in advance. I've had a hard time trying to find answers to some of these specific questions.

  • Ideally I'd be pooling my storage using RAID 5 to have some redundancy for possible drive failures, but most things I've read point to a hard requirement for all drives in the array to be the same/similar size.

    If you used raid 5, you would only use the space of the smallest drive on each drive - in your case 160GB on each drive. Redundancy with hand me down, non-server hardware is really a pipe dream. You should just have good backup (raid is not backup).

    A few things I read point to using SnapRAID in conjunction with mergefs, but also read that SnapRAID doesn't deal nicely with smaller sized files. Does anyone have any info on what is considered small?

    I have never heard that. snapraid doesn't run in realtime either. So, this would have very little impact on the system.

    I was wondering if there was something else I use/do to serve the media (transcoded on the server end) to client devices?

    Are you really using a client that can't decode files? Most newer devices can decode on their own.


    Since I'm dealing with audio and not just video, is there a better solution?

    audio is MUCH easier to decode than video and I doubt any device needs that transcoded. I serve all my files via nfs without a media server program.


    Assuming I've got my storage solution worked out, are there any tips you guys have for dealing with serving non-media files in this setup? I was thinking that I might have to separate the storage for media vs files so that I don't get any weirdness, but admittedly I'm out of my depth here - I have no experience with networking, but that's kind of the point of this project.

    I serve all of my files with the same server. The server doesn't care what it is serving.

    do you guys think this is possible to do on the same box?

    Of course it is. Use a docker to keep it separate from your OMV nginx install though. If you need port 80, change OMV's port to use another port or use macvlan on the docker to get its own IP address.

    omv 5.5.5 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.3.5
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Hey, thanks for the detailed blow-by-blow, as I said I'm totally out of my depth with this stuff. Kind of the point of this project I guess, jump in to it as a first pass, learn some stuff with an aim of having ok-ish and then build something better down the line.


    If you used raid 5, you would only use the space of the smallest drive on each drive - in your case 160GB on each drive. Redundancy with hand me down, non-server hardware is really a pipe dream. You should just have good backup (raid is not backup).

    That validates my thinking about the drive sizes. I figured I'd probably use the three 1TB drives for the RAID5 array and have the others doing something else, or maybe not even install them on this machine. Good comment about raid not being the same as a back up, that thought hadn't even entered my mind, honestly. I'd probably have most of the critical things backed up somewhere, so with that in mind, I guess the question is RAID5 even that important for my needs? I guess that's really a question for me to go away and research some more.


    I have never heard that. snapraid doesn't run in realtime either. So, this would have very little impact on the system.

    I ran down a few random avenues exploring options for getting a storage pool solution working with drives of varying size whilst also having some (what I considered) backup. A few different searches came up with various things saying SnapRAID is great for storing large/infrequently changing files, but doesn't deal with the other end of the spectrum so well. If that's not the case then cool. Though given the smallest drive size thing with RAID, may not need SnapRAID specifically and can use the RAID options provided with OMV. That's another thing I need to go away and look at some more.


    Are you really using a client that can't decode files? Most newer devices can decode on their own.

    Realistically, probably not. I may be trying to solve a problem that doesn't exist but I've not looked in to the full support my TVs have, for example.


    audio is MUCH easier to decode than video and I doubt any device needs that transcoded. I serve all my files via nfs without a media server program.

    I serve all of my files with the same server. The server doesn't care what it is serving.

    That's good info to have. I have a tendency to overthink things before I even get started (as you may have guessed).


    Of course it is. Use a docker to keep it separate from your OMV nginx install though. If you need port 80, change OMV's port to use another port or use macvlan on the docker to get its own IP address.

    Cool! I've not used containers at all so that'll be a nice little project for when I get the main server stuff completed.


    This has been super useful. Thanks!

  • A few different searches came up with various things saying SnapRAID is great for storing large/infrequently changing files, but doesn't deal with the other end of the spectrum so well. If that's not the case then cool. Though given the smallest drive size thing with RAID, may not need SnapRAID specifically and can use the RAID options provided with OMV. That's another thing I need to go away and look at some more.

    I see what you were trying to explain. People will say it isn't good for small files that you are changing all the time because it does not run in realtime to provide the redundancy that it is designed to do. This has nothing to do with snapraid really. You could run snapraid syncs all the time to avoid this but I think that would be overkill. You would be better to rsync one drive to another on a frequent basis.


    If you are running this at home, I doubt you need raid. raid is meant to prevent/reduce down time. Most home users could handle having the system down while they restore from backup or configure the system to use a second drive that files were being rsync'd to.

    omv 5.5.5 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.3.5
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Yea, I figured it's more of a production-level requirement and so might not be something worth worrying about, but it's good to learn about these things. It's also one of the big questions that lead me to the conclusion to sign up here after 2 weeks of researching various things around this project, and I'm glad I did. Probably wouldn't have gotten to the raid vs backup thing on my own.

  • These guys are a helpful bunch and there's a lot of good info here.


    What you are building sound like what I built, but better. My core2duo with 4G ram and a few random drives got me going. I added a few new 3T drives to get something with low hours into the machine. One of those is my snapraid parity drive. I worry less about the most recent changes not being 'covered' by a snapraid sync. And as ryecoaaron said, do rsync 'between' snapraid syncs.


    I think you should be fine, but I am having load issues when channels dvr, plex, urbackup, and snapraid are trying to work at the same time. I'm considering a bit more ram. I've had no issues serving other types of files in addition to the media I have stored.


    I am not doing any real transcoding on the machine. But, you could get a reasonable video card for $40 and off-load those duties.


    Have fun!

  • So, having got my hands on the old machine today,, I've been playing around trying to get stuff up and running. Got lost in permission-hell, not 100% sure what the problem was or how I solved it but it seems solved for now. Decided to forego RAIDing entirely, I can just distribute my shared folders among the different drives and maintain that manually, also works to keep this as basic as possible while I learn the ropes, so to speak. Once I get the file/media server bits sorted how I want them then I look at sorting out a proper backup using rsync.


    I've been able to get SMB set up and user with read/write access, can drop files on from my laptop, and read them back fine - I'm using a small video file as a test. So OMV <-> Windows is working fine. Now need to figure out how to get my Samsung TV and phone connecting. A bit of googling suggests that - for the TV at least - I need to play around with some DLNA stuff, so a bit more research to do there. This has a been a fun little project so far, despite the banging of my head against the aforementioned permission issue.


    I am not doing any real transcoding on the machine. But, you could get a reasonable video card for $40 and off-load those duties.

    As @ryecoaaron mentioned, I'm not sure I even need to transcode at the server end. If I did, luckily the machine came with a GTX 760 so could always pop that back it and should be more than capable :)


    If this process has highlighted anything so far, it's tat I should get a bit more comfortable with the Linux console. Had an issue with a HDD with bad sectors, didn't clock it for a while and had to fall back to diagnosing it with my Windows laptop and a HDD dock. I know I could've done the same sort of thing just ssh'ing in to the box and running some stuff.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!