Cannot create RAID, new install

    • OMV 4.x
    • Resolved
    • There are two ways this could be done, that come to mind, with all drives in one box.

      For good fault protection and ease of restoration:

      - UnionFs one 10TB drive together with one 6TB drive, for a total of 16TB.
      - Do the same with the second drive pair.
      - Rsync UnionFs 1 to UnionFs 2
      (Other than the extra piece of combining 2 drives into a UnionFS drive, the Rsync command line is outlined in the guide. It would be a matter of setting up the command line with UnionFs1 as the source and UnionFS2 as the destination.)

      This give you 100% data backup and restorations are easy. If something in the primary pair fails, simply re-point your shared folders to UnionFs 2, with identical content, and you're back on line. (This is in the guide.) However, there's no bitrot protection.
      _________________________________________________________

      For the maximum disk real-estate, with protection and recovery options:

      - UnionFS all drives together 10+6+6, saving one 10TB drive for SNAPRAID. (For SNAPRAID to work, the parity drive must be equal to, or larger, than the largest drive in the array.)
      - This would give you a UnionFS drive of 22Tb
      - SNAPRAID provides decent options for the recovery of the loss of a drive, scrubbing for bitrot and more. Take a look-> here.

      While getting started is easy, recovery is more involved than an Rsync backup. ((In any case, when recovering, it's time to slow down, take a breath and do some reading before doing anything, in any case.))

      The trade off is more disk space and bitrot protection, versus ease of restoration.

      _________________________________________________________

      I'm not telling you what to do, but it makes sense to investigate the options before committing. And remember, as you consider the your options; it's not a matter of "if" a drive will fail, it's a matter of "when".

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
    • telijah wrote:

      Also, I am gonna start this over with blank 10TB drives, which command is best to wipe those drives clean to follow your guide properly? I will remove the share first
      Since you've already done Wipefs, I think the guide (quick wipe) process would do it. But, if you've already set up mdadm RAID, maybe you should do the wipefs command again just to be sure.

      From there, follow the guide. An extra quick wipe takes no time at all.
      (Using the EXT4 format, as it is in the guide, is probably best - that will take a bit of time.) Once you get a 10TB drive setup and your data copied over from the synology; I'll try to provide some pointers for what you decide on.

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
    • Reading the guide, I think I am OK with just the 16TB of usable space, with one 10TB drive rsync'd to the other 10TB, and one 6TB drive rsync'd to the other 6TB drive...

      The following is for my own sanity and to have my plan laid out for migrating from the Synology to OMV

      1. Put the EXT4 on both 10TB drives (sdb and sdc).
      2. Create shared folders and SMB service to them
      3. Stop Plex
      4. Rsync from 6TB Synology share to primary 10TB drive
      5. Setup rsync job between sdb and sdc
      6. Complete Plex migration and restart Plex; point Plex database to new share(s)
      7. Move 6TB drives to OMV; Full wipe; set EXT4; setup rsync between them (presumably sdd and sde)
    • 1. Put the EXT4 on both 10TB drives (sdb and sdc). Yes.

      2. Create shared folders and SMB service to them Yes. (On the primary drive only.)

      3. Stop Plex Maybe? I don't know the in's and outs of Plex, but this makes sense.

      4. Rsync from 6TB Synology share to primary 10TB drive Yes. You'll set up a remote mount, to the Synlogy share (already described).
      When it shows up in Storage, File Systems, then create two shared folders as previously described, one on OMV and one on the Synolgy box.
      Finally, create a LOCAL Rsync job in the GUI, under Services, Rsync, with the source shared folder being the Synology share and the destination shared folder being the OMV share. (Take a look at the Rsync Options I use for a box to box transfer, below.)


      5. Setup rsync job between sdb and sdc Yes. From the guide, you'll set up your own custom Rsync command line, and execute the drive-to-drive command line under System, Scheduled Jobs. Per guidance in the guide, give yourself enough time to intervene if something goes wrong.

      6. Complete Plex migration and restart Plex; point Plex database to new share(s) (Makes sense. You know more about this than I do.)

      Take your time and make sure all is working on OMV the way you want it to, before touching the Synology box. (And I mean like a few weeks to a month or more to be sure of your OMV configuration, all updates are applied, the boot drive is cloned, all is well and it's working as a media and file server the way you want it to.) During that period, the synology box will be a backup/fall back, in the event that something goes wrong. As we all know, "stuff" 8o happens. Be certain before taking the synology box down.

      7. Move 6TB drives to OMV; Full wipe; set EXT4; setup rsync between them (presumably sdd and sde) If you get the first set of drives working, this step will be easy.

      ___________________________________________________________

      Again you could use the synology box as a fully independent backup to OMV, using Remote Mount and (push) Rsync' network shares to synology box. Throw a 10TB drive in it and, with one 6TB drive, you can duplicate 16TB of data on two different machines. I don't know what model you have but a couple synology models can be converted to OMV. (Do a google search with your model and OMV as search terms.)
      ____________________________________________________________

      Rsync Options:
      If the job gets interrupted for some reason, "keep partially transferred files" will allow the next job to pick up where the first job was interrupted. Important for a really large transfer.
      Also, if you add files to the OMV destination, but not the synology box, turn Delete files off. Otherwise if the files don't exist on the synology box (the source) they'll be deleted from the destination (OMV). Once the transfer is complete, this job should be deleted or reversed to push files the other way (for backup).


      That should get you started.

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk

      The post was edited 1 time, last by flmaxey ().

    • flmaxey wrote:

      4. Rsync from 6TB Synology share to primary 10TB drive Yes. You'll set up a remote mount, to the Synlogy share (already described).

      When it shows up in Storage, File Systems, then create two shared folders as previously described, one on OMV and one on the Synolgy box.
      Finally, create a LOCAL Rsync job in the GUI, under Services, Rsync, with the source shared folder being the Synology share and the destination shared folder being the OMV share. (Take a look at the Rsync Options I use for a box to box transfer, below.)

      I for the life of me cannot get my RSYNC from the Synology box to my new drive to work.

      Here are my shares


      And my rsync job


      When the job completes, I get

      Source Code

      1. Please wait, syncing </srv/4fe94ee8-2e58-4596-bca6-4990fbf9a771/SynPlexMedia/> to </srv/dev-disk-by-label-plex01data/PlexMedia01> ...
      2. sending incremental file list
      3. sent 44 bytes received 12 bytes 112.00 bytes/sec
      4. total size is 0 speedup is 0.00
      5. The synchronisation has completed successfully.
      6. Done ...

      And the log shows

      Source Code

      1. Dec 7 00:08:20 telijahnasbox rsync-047f3941-ce76-473d-b9f5-eb9ce2d1ffe1: Please wait, syncing </srv/4fe94ee8-2e58-4596-bca6-4990fbf9a771/SynPlexMedia/> to </srv/dev-disk-by-label-plex01data/PlexMedia01> ...\n
      2. Dec 7 00:08:20 telijahnasbox rsync-047f3941-ce76-473d-b9f5-eb9ce2d1ffe1: \nThe synchronisation has completed successfully.
      So nothing is moving though... I mapped the local 10tb drive in windows and dropped a dummy file, and turned on the option to delete files in destination that are not in source, and it deletes that file... So not sure why it is not working...
    • I hope you didn't erase the synology share - which is what would happen if you got it bass ackwards with the delete switch on. The delete cycle is brutally fast.

      Check the synology share to see if it exists - with files in it.

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
    • flmaxey wrote:

      I hope you didn't erase the synology share - which is what would happen if you got it bass ackwards with the delete switch on. The delete cycle is brutally fast.

      Check the synology share to see if it exists - with files in it.
      I did not get it backwards, everything is still there. Like I said, I placed the test file onto the local 10tb drive, and the rsync job did delete the test file as it does not exist on the Synology drive, so delete action direction was correct.
    • telijah wrote:

      I did not get it backwards, everything is still there. Like I said, I placed the test file onto the local 10tb drive, and the rsync job did delete the test file as it does not exist on the Synology drive, so delete action direction was correct.
      Bro - you don't know how relieved I am, for you... That would have been a disaster. So, let's change things - don't turn the delete switch on in Rsync, until you're sure you have two full copies!

      telijah wrote:

      Please wait, syncing </srv/4fe94ee8-2e58-4596-bca6-4990fbf9a771/SynPlexMedia/> to </srv/dev-disk-by-label-plex01data/PlexMedia01> ..
      Since you have WinSCP installed, use it to navigate to /srv/4fe94ee8-2e58-4596-bca6-4990fbf9a771/SynPlexMedia/. In taking a guess maybe this is the problem. (/srv/4fe94ee8-2e58-4596-bca6-4990fbf9a771/SynPlexMedia/ ) You might be looking into an empty or nonexistent folder on the source side.


      Remember, the path in a shared folder, that's using Remote mount, is "/" This requires editing the default path entry when created.
      In the following, the Device is the Remote Mount. Note the Path.


      telijah wrote:

      Any reason I should not just do a copy from WinSCP on my Windows machine? I have the Synology share and the OMV share mapped in Windows...
      No, you could but WinSCP doesn't do large copy jobs well. You could also simply open the two shares, on a windows machine, and copy and paste in chunks but that will take a long time and is subject to errors. My concern, in either case, would be that files or folders might be missed.

      If going the copy route and you're comfortable on the CLI, I'd use "MC" (midnight commander) with OMV. (It's pretty easy to use, menu driven, and point and click with a mouse works as well.)
      On the command line do:

      apt-get install mc
      then
      mc

      MC copies left to right. Since the remote mount exists, you should be able to navigate to the the following locations and copy files, left to right.

      /srv/4fe94ee8-2e58-4596-bca6-4990fbf9a771/SynPlexMedia/ /srv/dev-disk-by-label-plex01data/PlexMedia01

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
    • Son of a bitch, it was the Path, I didn't think of it the way you spelled it out but yes, in the Shared Folder setup, I had put in SynPlexMedia thinking it needed the name I gave it. Once I changed the Path to just be the forward slash, I can see it creating the file list now (using Preview mode first).

      Thanks! Hopefully rsync is faster as "synching" with WinSCP was going to take it seemed like a few days

      It is running now, with Delete turned off (although I am pretty confident in recognizing the direction of changes :)). I just wish there was a better way of knowing the overall status/progress than the window the OMV gui provides.
    • telijah wrote:

      Son of a bitch, it was the Path, I didn't think of it the way you spelled it out but yes, in the Shared Folder setup, I had put in SynPlexMedia thinking it needed the name I gave it. Once I changed the Path to just be the forward slash, I can see it creating the file list now (using Preview mode first).
      So the problem was,,, it worked EXACTLY the way you configured it.... :D (Just kidding - glad it worked out.)

      Remote Mount takes you "there", to the root of the remote share or "/". No path is required for the shared folder to find the location. If something is added, it goes nowhere. (The default path, that auto-fills based on the name, doesn't help.)

      telijah wrote:

      It is running now, with Delete turned off (although I am pretty confident in recognizing the direction of changes :)). I just wish there was a better way of knowing the overall status/progress than the window the OMV gui provides.
      If you leave the GUI interface up, with the popup Rsync window, it will stay up and scroll away. Rsync is fast, efficient, and far better than WinSCP. Go away,, don't watch it. Depending on what you have it may take hours (1GB) or even a day or two (100mbs). There's a chance that something might happen that interrupts the transfer. (A reboot?) Not a big deal. Just run the job again. The only way to know that all is transferred is to run the job, later, and see nothing transferred.
      (Much like what you saw in the beginning. :) )

      Really, other than the manual pop-up box, there's no need for a progress indicator. It will happen. When automated, after awhile, you'll take it for granted - it's that reliable. And after the first transfer, it's very fast - changes only.
      __________________________________________________________________________________

      Now, I know you want to pull drives out of the Synology box, but (after all is transferred) give some thought to using it as an independent backup device. All you have to do now after the full transfer has taken place (make sure of that), is change the source and destination and (with all due caution) turn delete on.

      After that, on a schedule you specify, Rsync will replicate the additions, deletions, changes, etc., you make on the OMV share, to the Synology share. As of the time of sync, the two shares will be a mirror.

      If you want to use it as cold storage, you could turn the Synology box off, then bring it up and manually run the Rsyn job, from time to time. In such a case, your OMV box could crash and burn, and you'd only lose changes or additions since the last sync.

      BTW - assuming you're running a Windows workgroup LAN:
      If you don't want to see the Synology Box on your network, change its' workgroup name to "backup" (or something different from your current workgroup name.) I will still be on the network, but not visible to users in Network Explorer.

      Cheers.

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
    • You're right, it did work as "setup", so no offense taken :)

      As for the Synology.. I have only two 6tb drives in there, and the two 10tb drives in the OMV box... my plan was to setup a scheduled rsync from the main 10tb to the backup 10tb, and then move those 6tb drives into the OMV box and do the same thing to them. I am keeping the Synology obviously, and that will become my personal cloud storage to offload my personal files from my main PC which has become more of an HTPC server than personal computer.
    • Well, you have a handle on it and know what you want to do.

      My preference is to replicate the server entirely on a second device. If something goes wrong with the first, there's zero down time. Switch to the second. It's already there with identical content.
      ___________________________________________________________

      BTW: you can Rsync Windows network shares in the same manner. It just takes a username and password with at least read access, if pulling, or write access, if pushing.
      I don't use clients for storage because, with UrBackup storing client images, I'd be storing the same thing twice. (Eats up hard disk space on the server side.)

      When configuring a Remote Mount, user names and passwords with "read only" access to a remote network share produce an error, but they still work. "Read only" is my preference because I don't want a backup device to have the ability to change anything at the main server.

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk

      The post was edited 1 time, last by flmaxey ().

    • Users Online 1

      1 Guest