Newbie trying to build a little NAS for family photos

    • Newbie trying to build a little NAS for family photos

      After operating two external drives on my Asus Chromebox for some years, I recently discovered that (at least) one of them is having bad sectors. Since I don't want to jeopardise my digital family memories, I want to switch to a more robust setup with OMV on the Chromebox and two 2 TB Ironwolfs in an external Icy Box case. Since I am new to the OMV family, I appreciate any hints or advice to my planned setup. OMV4 is already installed on the Chromebox and I have made the first steps with it (the HDDs were not yet delivered).

      My plan is to operate the two HDDs separately (i.e. no RAID), writing to disk A and rsync the files to disk B as a backup. I also plan on regularly making an off-site backup to an external volume. I would prefer to let rsync run without --delete flag to avoid any loss due to accidental deletion on drive A.

      In the beginning I was thinking about a RAID 1 setup, but apart from the photos I also automatically copy recordings from TV (from my cable receiver) to drive A , which are then directly uploaded to the cloud and deleted after upload from the drive. So these files do not need to be mirrored and mirroring them only increases writings without any purpose in my opinion. Future usage might change, but currently this is what is planned.

      As file system for the discs I thought about keeping things simple using ext4 on both drives, but I have no experience with the other formats (XFS, ZFS, BTRFS) and if they offer any advantages in my case.

      So if you have any comments/hints on my setup please let me know. If I posted in the wrong section, please move to a more appropriate area.
    • I think your plans sounds excellent. Go for it!

      Using two separate drives, one for shared data and one for rsync backups/snapshots, is a simple and very robust setup.

      If you want to improve the security further you could store the most important images in RAR-archives with the archive protection feature activated. Or use some utility to calculate a checksum for a subfolder and check for changes.

      Another (future?) option is the not only have two separate drives but even two separate NAS. At different locations.
      OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4
    • Thank you both, so be it :)
      I will have a look at rsnapshot. haven't heard of that yet. Sounds well suited for my case as it seems.

      One question about that Icy box encasing (IB-RD3621U3): if I understood correctly, this case also has a RAID controller on board, the settings RAID 0, 1, Single or JBOD/Big can be set with a jumper-linke setup (raidsonic.de/products/external…x_de.php?we_objectID=3589). If I would want to use any of the RAID settings of the drive, how would that needed to be addressed in OMV? I wouldn't set a software RAID on top, would I?
    • If you setup the dual drive cabinet for RAID 1 it will appear in OMV as one single drive. And you won't have to do anything special in OMV.

      If you setup the dual drive cabinet for JBOD then you could either use each disk separately or configure a software RAID. However, since both drives use the same USB cable, performance would be lower with software RAID in OMV.

      But if you setup RAID in the cabinet, if something goes wrong with the cabinet itself, you might loose all data unless you can get a new identical cabinet.

      You could also (if you know how) set it up with btrfs RAID1. That would allow for automatic bitrot protection. But perhaps with low performance since both drives are on the same cable.

      However, OMV doesn't support software RAID over USB, so you would have to set it up yourself from the command line.

      One potential problem with the IB cabinet is that the specs states that it is only compatible with Windows and Mac OS X. That doesn't sound nice. It might still support Linux, but perhaps with very low performance? Or it might work perfectly fine with OMV and Linux, only it hasn't been tested? I don't know. I wouldn't buy it for use with OMV/Linux...

      I use a similar cabinet for one of my RPi4, but from StarTech. It is speced as "OS independent" and I know for a fact that it works fine with OMV on my RPi4. I have it setup for JBOD with two separate independent HDDs. I use it with two old 8TB Archival SMR HDDs for secondary "cold" backups. That is, an extra set of snapshot copies of my most important stuff. I update it only only if needed. Perhaps monthly. It is mostly shut down.

      About rsync and rsnapshot, and my rsync script...

      Snapshots using rsync lets you create backup copies that look just like a full copy of the original. But instead of copying every single file, rsync can identify unchanged files and then "hardlink" to a previous identical copy of the same file. This means that each snapshot consists of a bunch of folders with hardlinks and the occasional updated or new file. It is much faster to create a snapshot like this and it takes up very little storage.

      This means that you can have two similar sized HDDs and use one for shared data and one for backup snapshot. And each snapshot is (appears as) a time-stamped full copy of the original.

      The plugin rsnapshot supports this form of snapshots. it allows you to snapshot shared folders.

      But I prefer to do this outside OMV using cron and my own snapshot script. Typically I use a couple of scripts, on for each major subfolder on the original HDD. For instance one script for music, one for ebooks, one for tv and one for movies.

      github.com/WikiBox/snapshot.sh
      OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

      The post was edited 2 times, last by Adoby ().

    • RAID increases the chance of loosing data (vs separate drive you backup to). Go with using the separate backup drive.

      QUESTION: Has anyone tried the Borg Backup OMV Plug-in? (OPENMEDIAVAULT-BORGBACKUP) I can't find any information or guide or example on how to use it anywhere.
    • Newbie trying to build a little NAS for family photos

      Thanks Adoby for that long answer. The RAID setup was just for information. I will see if the case suits my purpose and if it doesn't work properly, I will send it back. It seems as I don't need the RAID function of the case anyway, but now it is ordered ;)

      How will the speed from one HDD to the other be in such a setup? I mean the files don't need to pass the USB bus twice, don't they? Since I don't copy to both HDDs simultaneously, speed won't be cut in half I hope.
    • Unfortunately I suspect that copying from one HDD to another in a dual USB HDD case will be slower than copying between two separate single USB HDD cases. I think the data has to pass twice through the cable... It might be possible to pass data full speed in two directions simultaneously over USB. Full duplex like a network card. I doubt it...

      But I haven't tested.

      However with regards to rsync snapshots this will only be a big problem for the first snapshot. That might take some extra time. Subsequent snapshots should be pretty fast since most files already are available for hardlinking. It should be fast enough, even at half speed, I think.

      I suspect that the speed of copying from one HDD to another in a dual HDD USB case will still be faster than copying from one NAS to another over GbE. So you should be fine.
      OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4
    • Newbie trying to build a little NAS for family photos

      I happened to order a different external case (IB-RD3662U3S), but from the same brand. Should be here today, I didn't read any reports that these don't work with Linux, so my hopes are still high that I won't encounter any problems.
      In the mean time I already got used to OMV and so far like the capabilities of it.
      I also bought a small UPS, altouhght there are barely any power outages in my area ;)
    • Coxeroni wrote:

      I happened to order a different external case (IB-RD3662U3S), but from the same brand. Should be here today, I didn't read any reports that these don't work with Linux, so my hopes are still high that I won't encounter any problems.
      In the mean time I already got used to OMV and so far like the capabilities of it.
      I also bought a small UPS, altouhght there are barely any power outages in my area ;)
      Wow, that thing is expensive over here ($230 on some sites)...

      If you spent anywhere near that much... I'd ditch it completely, buy an RockPro64, sata card, case, etc. and just install the two drives in there and keep the Chromebook independent of all this.
      Air Conditioners are a lot like PC's... They work great until you open Windows.

      The post was edited 1 time, last by KM0201 ().

    • Newbie trying to build a little NAS for family photos

      First annoyance probably related to the case:
      The drives have a different ID after every reboot, since their serial number is displayed as "RANDOM_some characters", and that characters change with every reboot. Therefore smart monitoring doesn't work after reboot and has to be activated again every time.

      Can I put the /dev/by-label in the /etc/smartd.conf instead of the ID? Or is there any other possibility for solving this?

      The post was edited 3 times, last by Coxeroni ().

    • I assume you talk about the UUID? The Universal Unique ID that is assigned to a block device and stored in the masterblock of that device. Making each drive unique, even when moved.

      If you see that it change between reboots, then something is very seriously wrong with your system. The whole idea behind UUID is that it will uniquely and persistently identify a block device, even after reboots or if new devices are plugged in or the block devices are plugged in using different interfaces/cables/ports.

      What have you done to break this? Do you reformat drives during the boot process?

      I am very impressed by this and I am truly curious about how you did it.
      OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4
    • Coxeroni wrote:

      First annoyance probably related to the case:
      The drives have a different ID after every reboot, since their serial number is displayed as "RANDOM_some characters", and that characters change with every reboot. Therefore smart monitoring doesn't work after reboot and has to be activated again every time.

      Can I put the /dev/by-label in the /etc/smartd.conf instead of the ID? Or is there any other possibility for solving this?
      yeah, something is definitely not right
      Air Conditioners are a lot like PC's... They work great until you open Windows.

    • Sorry for being so unspecific, so here is a bit more info on the subject:

      What I am taking about is the ID as in /dev/disk/by-id/ :

      Source Code

      1. root@chromebox:~# ls -lh /dev/disk/by-id/
      2. total 0
      3. lrwxrwxrwx 1 root root 9 Oct 25 23:05 ata-TREKSTORSSD128GB_AA000000000000008698 -> ../../sda
      4. lrwxrwxrwx 1 root root 10 Oct 25 23:05 ata-TREKSTORSSD128GB_AA000000000000008698-part1 -> ../../sda1
      5. lrwxrwxrwx 1 root root 10 Oct 25 23:05 ata-TREKSTORSSD128GB_AA000000000000008698-part2 -> ../../sda2
      6. lrwxrwxrwx 1 root root 10 Oct 25 23:05 ata-TREKSTORSSD128GB_AA000000000000008698-part5 -> ../../sda5
      7. lrwxrwxrwx 1 root root 9 Oct 25 23:05 usb-Generic_Power_Saving_USB_000000000260-0:0 -> ../../sdd
      8. lrwxrwxrwx 1 root root 10 Oct 25 23:05 usb-Generic_Power_Saving_USB_000000000260-0:0-part1 -> ../../sdd1
      9. lrwxrwxrwx 1 root root 9 Oct 25 23:05 usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 -> ../../sdb
      10. lrwxrwxrwx 1 root root 10 Oct 25 23:05 usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0-part1 -> ../../sdb1
      11. lrwxrwxrwx 1 root root 9 Oct 25 23:05 usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 -> ../../sdc
      12. lrwxrwxrwx 1 root root 10 Oct 25 23:05 usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1-part1 -> ../../sdc1
      Display All
      The part behind the RANDOM changes with every reboot. This is also, what is displayed in the drives section of OMV (s. screenshot attached).

      The UUIDs are different and probably do not change during reboot (haven't checked that). See the "ironwolf" entries.

      Source Code

      1. root@chromebox:~# blkid
      2. /dev/sda1: UUID="3a4bfe58-2d98-4b26-b97a-988100f310a2" TYPE="ext4" PARTUUID="0443930c-01"
      3. /dev/sda5: UUID="8e937b4d-72d8-4a67-bfc8-6812c4afc9da" TYPE="swap" PARTUUID="0443930c-05"
      4. /dev/sdb1: LABEL="ironwolf1" UUID="e8a0ab25-f031-4f04-8faa-f5a8a77950dd" TYPE="ext4" PARTUUID="0d621189-a97d-44ef-b945-0f1bb83ea4ff"
      5. /dev/sdc1: LABEL="ironwolf2" UUID="680ee841-1f68-4a0d-ab99-f287c6a18ac4" TYPE="ext4" PARTUUID="ea15b7be-7425-44b0-b2a3-92049a0aaca4"
      6. /dev/sdd1: LABEL="sd" UUID="da30eda9-c56d-4a67-b216-1933ebf21d74" TYPE="ext4" PARTUUID="328e759b-9aa5-4f88-ba8e-3fab2052b4f8"
      But as I said, the s.m.a.r.t monitoring is directed to the "by-id" devices and therefore fails after reboot, since the drive now has a different id:

      Source Code

      1. root@chromebox:~# cat /etc/smartd.conf
      2. # /etc/smartd.conf
      3. # Configuration file for smartd. Use "man smartd.conf" for more information.
      4. #
      5. # This configuration file is auto-generated.
      6. # WARNING: Do not edit this file, your changes will be lost.
      7. /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__C090397067CD-0:0 -d sat -a -o on -S on -T permissive -W 0,0,0 -I 190 -I 194 -I 231 -n standby,q
      8. /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__C090397067CD-0:1 -d sat -a -o on -S on -T permissive -W 0,0,0 -I 190 -I 194 -I 231 -n standby,q
      9. /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__2CC1D61B6A08-0:0 -d sat -a -o on -S on -T permissive -W 0,0,0 -I 190 -I 194 -I 231 -n standby,q
      10. /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__2CC1D61B6A08-0:1 -d sat -a -o on -S on -T permissive -W 0,0,0 -I 190 -I 194 -I 231 -n standby,q
      11. /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 -a -o on -S on -T permissive -W 0,0,0 -I 190 -I 194 -I 231 -n standby,q
      12. /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 -d sat -a -o on -S on -T permissive -W 0,0,0 -I 190 -I 194 -I 231 -n standby,q \
      13. -s S/../../7/02
      14. /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 -d sat -a -o on -S on -T permissive -W 0,0,0 -I 190 -I 194 -I 231 -n standby,q
      Display All
      Images
      • Bildschirmfoto vom 2019-10-26 11-33-59.png

        36.32 kB, 695×255, viewed 8 times
    • (Had to seprate the posts due to too many characters)

      This is what the smart-log said yesterday when I rebooted. The upper four entries where from former boots with different drive IDs:

      Source Code

      1. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__C090397067CD-0:0 [SAT], open() failed: No such device
      2. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__C090397067CD-0:1 [SAT], open() failed: No such device
      3. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__2CC1D61B6A08-0:0 [SAT], open() failed: No such device
      4. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__2CC1D61B6A08-0:1 [SAT], open() failed: No such device
      5. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698, type changed from 'scsi' to 'sat'
      6. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 [SAT], opened
      7. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 [SAT], TREKSTORSSD128GB, S/N:AA000000000000008698, FW:R0522A0, 125 GB
      8. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 [SAT], not found in smartd database.
      9. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 [SAT], enabled SMART Attribute Autosave.
      10. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 [SAT], SMART Automatic Offline Testing unsupported...
      11. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 [SAT], enabled SMART Automatic Offline Testing.
      12. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 [SAT], is SMART capable. Adding to "monitor" list.
      13. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 [SAT], state read from /var/lib/smartmontools/smartd.TREKSTORSSD128GB-AA000000000000008698.ata.state
      14. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], opened
      15. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], ST2000VN004-2E4164, S/N:Z5285KVW, FW:13621010, 2.00 TB
      16. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], found in smartd database: Seagate IronWolf
      17. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], enabled SMART Attribute Autosave.
      18. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], not capable of SMART Health Status check
      19. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], enabled SMART Automatic Offline Testing.
      20. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], Read SMART Self Test Log Failed
      21. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], no SMART Self-test Log, ignoring -l selftest
      22. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], Read Summary SMART Error Log failed
      23. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], no SMART Error Log, ignoring -l error
      24. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], no ATA CHECK POWER STATUS support, ignoring -n Directive
      25. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], is SMART capable. Adding to "monitor" list.
      26. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], state read from /var/lib/smartmontools/smartd.ST2000VN004_2E4164-Z5285KVW.ata.state
      27. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], opened
      28. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], ST2000VN004-2E4164, S/N:Z5285KKW, WWN:5-000c50-0b57842be, FW:SC60, 2.00 TB
      29. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], found in smartd database: Seagate IronWolf
      30. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], enabled SMART Attribute Autosave.
      31. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], not capable of SMART Health Status check
      32. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], enabled SMART Automatic Offline Testing.
      33. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], no ATA CHECK POWER STATUS support, ignoring -n Directive
      34. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], is SMART capable. Adding to "monitor" list.
      35. Oct 25 23:06:43 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], state read from /var/lib/smartmontools/smartd.ST2000VN004_2E4164-Z5285KKW.ata.state
      36. Oct 25 23:06:43 chromebox smartd[1580]: Monitoring 3 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
      37. Oct 25 23:06:44 chromebox smartd[1580]: Device: /dev/disk/by-id/ata-TREKSTORSSD128GB_AA000000000000008698 [SAT], state written to /var/lib/smartmontools/smartd.TREKSTORSSD128GB-AA000000000000008698.ata.state
      38. Oct 25 23:06:44 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:0 [SAT], state written to /var/lib/smartmontools/smartd.ST2000VN004_2E4164-Z5285KVW.ata.state
      39. Oct 25 23:06:44 chromebox smartd[1580]: Device: /dev/disk/by-id/usb-ST2000VN_004-2E4164_RANDOM__EEA16EF0C17A-0:1 [SAT], state written to /var/lib/smartmontools/smartd.ST2000VN004_2E4164-Z5285KKW.ata.state
      Display All
      Additionally, I am not sure whether smart status can be checked correctly. "not capable of SMART Health Status check" does not sound like everything is ok. Dunno if it makes sense to investigate further or if I just ditch that case (I can still send it back) and look for a different solution.
    • Newbie trying to build a little NAS for family photos

      Had time to think about this over the weekend. I will probably switch to a proprietary solution (QNAP probably), since this USB case probably takes me nowhere.
      My family media is also precious enough to spend a euro or two more, if things run smooth and without hiccups.

      Thanks to everybody that helped to support me, I always appreciate the communities around these kind of projects.

      Cheers
    • Users Online 1

      1 Guest