LUKS + KeyFile + AutoMount? [SOLVED]

    • OMV 3.x
    • Resolved
    • 1activegeek wrote:

      it's ok as I'm happy to test using only keyfiles,
      If you use a keyfile first and then add passphrases using the keyfile to unlock, it works fine.
      omv 4.0.5 arrakis | 64 bit | 4.12 backports kernel | omvextrasorg 4.0.2
      omv-extras.org plugins source code and issue tracker - github.com/OpenMediaVault-Plugin-Developers

      Please don't PM for support... Too many PMs!
    • It looks to be successful in my VM test! Woohoo!! Thank you so much guys in helping me out. Going to output a few pointers in case anyone else stumbles on this and is interested. Once I understood it all, it worked extremely smoothly.

      Adding in a well deserved credit to both @ryecoaaron and @subzero79 for there help in guiding me to getting this running properly. And to Zack Reed as I used his write up found HERE and referenced earlier as the basis and pointers for getting this setup properly.

      End Result:
      1 USB Drive plugged in physically
      3 Disks mapped/mounted with FS of choice
      1 UFS filesystem sitting on top of 2 disks
      SnapRAID running on top of 3 disks

      Important Notes when setting up
      • I think best way to handle USB device is to first insert into the system, and use OMV to mount/format the device. I believe (someone can validate) that if you create the device and format it inside OMV, it will look to remount it properly if you remove it. This is easier than trying to find the UUID, mapping it in FSTAB, and having solid removal strategy. Still to be fully tested. It also reduces the need to create a mapping in fstab.
      • Best option for creating the keyfiles, is to make them locally and then remotely transfer onto the system. This also allows you to create multiple keys to fill into the key slots. HIGHLY SUGGEST setting up 2-3 keys minimum to put into the key slots in case of emergency.
        • Creation for me was done using: dd if=/dev/urandom of=/keyfile bs=1024 count=4 - adjust the naming/locations as you need
        • Once on the destination (OMV), be sure to run chmod 0400 /root/keyfile to restrict access
      • Spend the time to setup and plan out how you want the system running and settle this first (LUKS encrypted devices, Filessytsems, shared folder systems, etc) - I can't tell you how annoying it is to repeat doing these when you message something basic up and have to start from scratch. VM saved my bacon multiple times as I "tested" this out to get it running and learned the problems.
      • Gotcha on keys - as @ryecoaaron pointed out, create the LUKS devices with a keyfile FIRST - so you don't have an issue with passphrase for the first key. This will cause problems and you'll have to scrap your work and start again.
      • Modify the /etc/default/cryptdisks to add in the device mapped for the USB (from /etc/fstab) to the section labeled CYPTDISKS_MOUNT - this will mount your USB before /etc/crypttab is evaluated
      • When editing /etc/crypttab - the name you want to use, is the name of the device from the /dev/mapper/ device upon LUKS mounting. To find this just type blkid | grep mapper and find the lines starting with /dev/mapper - the names should be familiar (for ex: sdb-crypt, sdc-crypt, etc). Your line then looks like sdb-crypt UUID=xxxx-xxxx-xxxx-xxxx-xxxxxxx /location/to/usb/mount/key.disk1 luks - repeat as necessary
      If nothing is amiss, you should now successfully be able to have an encrypted OMV drive setup. Please bear in mind this is only protecting the DATA drives. I'm not planning to or requiring to encrypt the system drive at this point as most of the data retrieved from here is unusable, and not likely to need to be encrypted. Also it's less likely the system drive is going for disk repair.

      The post was edited 1 time, last by 1activegeek ().

    • Just adding in, as subzero pointed out - if you use the USB mapping method without specialized scripting, systemd is creating dependencies to go along with the mounting. Which I see as a good thing. If you remove the key drive, it will automatically lock the disks, and thus the filesystems. The downside to understand here is that any data in process could be corrupted, but positive side - nobody will gain access to the content. I'm not 100% sure on the level of traceability of the key that was in memory once the drive has been locked. I believe it should be 0, but I can't say for certain. So this is essentially a full proof locked down system, pending the ability to extract from memory. Best solution, shut it down properly and remove the key.
    • systemd will try to lock down the drives, but it will fail as they have to be unmount then unlocked, if a process is using files on the path (unmount won't happen then lock will fail also).
      I noticed this as i have two encrypted drives, one wasn't being actively used yet (is new ) and that one got actively kick out once i close the keydisk, the other one refused to be unmounted as was being used by many services
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • subzero79 wrote:

      systemd will try to lock down the drives, but it will fail as they have to be unmount then unlocked, if a process is using files on the path (unmount won't happen then lock will fail also).
      Aha, interesting. Ok good to know. I tested it using a VM system that wasn't running anything, so thats likely why it was able to lock them easily. So the best bet then will be to still shut down the system before removing the key.
    • Has something changed recently that would cause this to fail auto mounting? Or could someone point me to logs that could help identify why the disks are no longer mounting automatically at startup? I've had to recently replace a disk, and in the process when I had to restart the server, I noticed the Encrypted filesystems are no longer auto mounting.

      Any ideas or help?
    • New

      So I finally got around to digging through the log. It has a lot going on that I'm not sure I can pick anything good/bad out of it. What I did find that looks like it's problematic though is the following lines:
      Sep 10 14:46:21 atlantis systemd[1]: Job dev-disk-by\x2dlabel-disk8.device/start timed out.
      Sep 10 14:46:21 atlantis systemd[1]: Timed out waiting for device dev-disk-by\x2dlabel-disk8.device.
      Sep 10 14:46:21 atlantis systemd[1]: Dependency failed for /srv/dev-disk-by-label-disk8.
      Sep 10 14:46:21 atlantis systemd[1]: Dependency failed for File System Check on /dev/disk/by-label/disk8.

      Which looks like something timed out while trying to load. Which my thought would be ok something is missing or not mounting, but I'm not sure what. I can't seem to find anything relative to these device calls any earlier in the journal readout. I'm hesitant to post the whole output as it seems to have some semi-sensitive detail. It appears shortly before this that sda1 thru sdd1 (disks 1-4 which are not encrypted) mount successfully. And slightly before that sdj1 (the USB drive with the keys) is mounted properly as well.


      Source Code

      1. Sep 10 14:46:21 atlantis kernel: scsi 10:0:0:0: Direct-Access SanDisk U3 Titanium 2.16 PQ: 0 ANSI: 2
      2. Sep 10 14:46:21 atlantis kernel: sd 10:0:0:0: Attached scsi generic sg9 type 0
      3. Sep 10 14:46:21 atlantis kernel: sd 10:0:0:0: [sdj] 2006673 512-byte logical blocks: (1.03 GB/980 MiB)
      4. Sep 10 14:46:21 atlantis kernel: sd 10:0:0:0: [sdj] Write Protect is off
      5. Sep 10 14:46:21 atlantis kernel: sd 10:0:0:0: [sdj] Mode Sense: 03 00 00 00
      6. Sep 10 14:46:21 atlantis kernel: sd 10:0:0:0: [sdj] No Caching mode page found
      7. Sep 10 14:46:21 atlantis kernel: sd 10:0:0:0: [sdj] Assuming drive cache: write through
      8. Sep 10 14:46:21 atlantis kernel: EXT4-fs (sdi1): re-mounted. Opts: errors=remount-ro
      9. Sep 10 14:46:21 atlantis kernel: [drm] Initialized
      10. Sep 10 14:46:21 atlantis kernel: iTCO_vendor_support: vendor-support=0
      11. Sep 10 14:46:21 atlantis kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
      12. Sep 10 14:46:21 atlantis kernel: iTCO_wdt: Found a Cougar Point TCO device (Version=2, TCOBASE=0x0460)
      13. Sep 10 14:46:21 atlantis kernel: sdj: sdj1
      14. Sep 10 14:46:21 atlantis kernel: iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
      15. Sep 10 14:46:21 atlantis kernel: sd 10:0:0:0: [sdj] Attached SCSI removable disk
      16. Sep 10 14:46:21 atlantis kernel: [TTM] Zone kernel: Available graphics memory: 8204282 kiB
      17. Sep 10 14:46:21 atlantis kernel: [TTM] Zone dma32: Available graphics memory: 2097152 kiB
      18. Sep 10 14:46:21 atlantis kernel: [TTM] Initializing pool allocator
      19. Sep 10 14:46:21 atlantis kernel: [TTM] Initializing DMA pool allocator
      20. Sep 10 14:46:21 atlantis kernel: SGI XFS with ACLs, security attributes, realtime, no debug enabled
      21. Sep 10 14:46:21 atlantis kernel: XFS (sdb1): Mounting V4 Filesystem
      22. Sep 10 14:46:21 atlantis kernel: XFS (sdd1): Mounting V4 Filesystem
      23. Sep 10 14:46:21 atlantis kernel: XFS (sda1): Mounting V4 Filesystem
      24. Sep 10 14:46:21 atlantis kernel: XFS (sdc1): Mounting V4 Filesystem
      25. Sep 10 14:46:21 atlantis kernel: fbcon: mgadrmfb (fb0) is primary device
      26. Sep 10 14:46:21 atlantis kernel: XFS (sdd1): Ending clean mount
      27. Sep 10 14:46:21 atlantis kernel: XFS (sdc1): Ending clean mount
      28. Sep 10 14:46:21 atlantis kernel: XFS (sdb1): Ending clean mount
      29. Sep 10 14:46:21 atlantis kernel: XFS (sda1): Ending clean mount
      30. Sep 10 14:46:21 atlantis kernel: ipmi_si IPI0001:00: The BMC does not support clearing the recv irq bit, compensating, but the BMC needs to be fixed.
      31. Sep 10 14:46:21 atlantis systemd-fsck[768]: fsck failed with error code 8.
      32. Sep 10 14:46:21 atlantis systemd-fsck[768]: Ignoring error.
      33. Sep 10 14:46:21 atlantis kernel: ipmi_si IPI0001:00: Found new BMC (man_id: 0x002a7c, prod_id: 0x0624, dev_id: 0x20)
      34. Sep 10 14:46:21 atlantis kernel: ipmi_si IPI0001:00: IPMI kcs interface initialized
      35. Sep 10 14:46:21 atlantis kernel: EXT4-fs (sdj1): mounted filesystem with ordered data mode. Opts: user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl
      36. Sep 10 14:46:21 atlantis kernel: intel_rapl: Found RAPL domain package
      37. Sep 10 14:46:21 atlantis kernel: intel_rapl: Found RAPL domain core
      38. Sep 10 14:46:21 atlantis kernel: intel_rapl: RAPL package 0 domain package locked by BIOS
      39. Sep 10 14:46:21 atlantis kernel: Console: switching to colour frame buffer device 128x48
      40. Sep 10 14:46:21 atlantis kernel: mgag200 0000:03:03.0: fb0: mgadrmfb frame buffer device
      41. Sep 10 14:46:21 atlantis kernel: [drm] Initialized mgag200 1.0.0 20110418 for 0000:03:03.0 on minor 0
      42. Sep 10 14:46:21 atlantis systemd[1]: Job dev-disk-by\x2dlabel-disk5.device/start timed out.
      43. Sep 10 14:46:21 atlantis systemd[1]: Timed out waiting for device dev-disk-by\x2dlabel-disk5.device.
      44. Sep 10 14:46:21 atlantis systemd[1]: Dependency failed for /srv/dev-disk-by-label-disk5.
      45. Sep 10 14:46:21 atlantis systemd[1]: Dependency failed for File System Check on /dev/disk/by-label/disk5.
      46. Sep 10 14:46:21 atlantis systemd[1]: Job dev-disk-by\x2dlabel-disk8.device/start timed out.
      47. Sep 10 14:46:21 atlantis systemd[1]: Timed out waiting for device dev-disk-by\x2dlabel-disk8.device.
      48. Sep 10 14:46:21 atlantis systemd[1]: Dependency failed for /srv/dev-disk-by-label-disk8.
      49. Sep 10 14:46:21 atlantis systemd[1]: Dependency failed for File System Check on /dev/disk/by-label/disk8.
      50. Sep 10 14:46:21 atlantis systemd[1]: Job dev-disk-by\x2dlabel-disk7.device/start timed out.
      51. Sep 10 14:46:21 atlantis systemd[1]: Timed out waiting for device dev-disk-by\x2dlabel-disk7.device.
      Display All
      Any direction that can be headed to look for what might be going wrong or causing my delay/timeout?
    • New

      I've seen that error in one of my development vm. Is something to do with systemd escape path, look at x2d before label. These are units auto generated by systemd. At the end I ended up rolling the vm back to original state. There is a bug report as I understand in systemd about this and should be fixed in backport systemd, but as I recall my vm keep failing mount devices even with the bpo systemd.
      Your can try and change dev path for UUID= manually to test.

      edit:This is the bug i saw

      github.com/systemd/systemd/issues/3300

      looks similar but not the same
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server

      The post was edited 1 time, last by subzero79 ().

    • New

      Hmmm ... interesting. Not good news in this case since this is my prod machine. =O I was thinking those labels looked funky with all the x2d all over. Interestingly though, the other disks that show the same oddities, are mounting and working fine upon boot without any issue.

      Where would I attempt to change the UUID manually? I'll have to make some backups and test it out. Just a pain testing as the machine takes a bit to boot from everything going on, and of course downs my apps. I noticed this after recently adding a new disk, copying data, removing bad disk. Not sure if it was the case before that. So perhaps the new disk introduced was causing something wacky as I had to re-order my BIOS start order too. I'll have to dig in there as well to make sure nothing got reassigned funky.
    • New

      Well, not sure what happened here, but I'm back running again. I don't know how, but I think the UUID got modified for the disks perhaps? I didn't have a full copy of the previous and new UUID I don't believe. I thought I matched them up and all worked. So I then attempted fully destroying the current filesystems/disks and re-doing it all. This time, the UUID seemed to match up correctly and it started with no issue. Much faster I might add too. So we'll see if it stays steady, but for now it seems I'm ok. Bad news, I don't really have true solid root cause analysis.