LUKS + KeyFile + AutoMount? [SOLVED]

    • Official Post
    Quote from 1activegeek

    it's ok as I'm happy to test using only keyfiles,

    If you use a keyfile first and then add passphrases using the keyfile to unlock, it works fine.

    omv 7.0-11 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.1 | compose 7.0.1 | cputemp 7.0 | mergerfs 7.0


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Like 1

    It looks to be successful in my VM test! Woohoo!! Thank you so much guys in helping me out. Going to output a few pointers in case anyone else stumbles on this and is interested. Once I understood it all, it worked extremely smoothly.


    Adding in a well deserved credit to both @ryecoaaron and @subzero79 for there help in guiding me to getting this running properly. And to Zack Reed as I used his write up found HERE and referenced earlier as the basis and pointers for getting this setup properly.


    End Result:
    1 USB Drive plugged in physically
    3 Disks mapped/mounted with FS of choice
    1 UFS filesystem sitting on top of 2 disks
    SnapRAID running on top of 3 disks


    Important Notes when setting up

    • I think best way to handle USB device is to first insert into the system, and use OMV to mount/format the device. I believe (someone can validate) that if you create the device and format it inside OMV, it will look to remount it properly if you remove it. This is easier than trying to find the UUID, mapping it in FSTAB, and having solid removal strategy. Still to be fully tested. It also reduces the need to create a mapping in fstab.
    • Best option for creating the keyfiles, is to make them locally and then remotely transfer onto the system. This also allows you to create multiple keys to fill into the key slots. HIGHLY SUGGEST setting up 2-3 keys minimum to put into the key slots in case of emergency.

      • Creation for me was done using: dd if=/dev/urandom of=/keyfile bs=1024 count=4 - adjust the naming/locations as you need
      • Once on the destination (OMV), be sure to run chmod 0400 /root/keyfile to restrict access
    • Spend the time to setup and plan out how you want the system running and settle this first (LUKS encrypted devices, Filessytsems, shared folder systems, etc) - I can't tell you how annoying it is to repeat doing these when you message something basic up and have to start from scratch. VM saved my bacon multiple times as I "tested" this out to get it running and learned the problems.
    • Gotcha on keys - as @ryecoaaron pointed out, create the LUKS devices with a keyfile FIRST - so you don't have an issue with passphrase for the first key. This will cause problems and you'll have to scrap your work and start again.
    • Modify the /etc/default/cryptdisks to add in the device mapped for the USB (from /etc/fstab) to the section labeled CYPTDISKS_MOUNT - this will mount your USB before /etc/crypttab is evaluated
    • When editing /etc/crypttab - the name you want to use, is the name of the device from the /dev/mapper/ device upon LUKS mounting. To find this just type blkid | grep mapper and find the lines starting with /dev/mapper - the names should be familiar (for ex: sdb-crypt, sdc-crypt, etc). Your line then looks like sdb-crypt UUID=xxxx-xxxx-xxxx-xxxx-xxxxxxx /location/to/usb/mount/key.disk1 luks - repeat as necessary

      • [UPDATE 12/7/17] To find the actual UUID you should now run sudo blkid | grep LUKS. You will be returned the lines of the devices and their associated UUID's you'll need.

    If nothing is amiss, you should now successfully be able to have an encrypted OMV drive setup. Please bear in mind this is only protecting the DATA drives. I'm not planning to or requiring to encrypt the system drive at this point as most of the data retrieved from here is unusable, and not likely to need to be encrypted. Also it's less likely the system drive is going for disk repair.

    Just adding in, as subzero pointed out - if you use the USB mapping method without specialized scripting, systemd is creating dependencies to go along with the mounting. Which I see as a good thing. If you remove the key drive, it will automatically lock the disks, and thus the filesystems. The downside to understand here is that any data in process could be corrupted, but positive side - nobody will gain access to the content. I'm not 100% sure on the level of traceability of the key that was in memory once the drive has been locked. I believe it should be 0, but I can't say for certain. So this is essentially a full proof locked down system, pending the ability to extract from memory. Best solution, shut it down properly and remove the key.

    • Official Post

    systemd will try to lock down the drives, but it will fail as they have to be unmount then unlocked, if a process is using files on the path (unmount won't happen then lock will fail also).
    I noticed this as i have two encrypted drives, one wasn't being actively used yet (is new ) and that one got actively kick out once i close the keydisk, the other one refused to be unmounted as was being used by many services

    Quote from subzero79

    systemd will try to lock down the drives, but it will fail as they have to be unmount then unlocked, if a process is using files on the path (unmount won't happen then lock will fail also).

    Aha, interesting. Ok good to know. I tested it using a VM system that wasn't running anything, so thats likely why it was able to lock them easily. So the best bet then will be to still shut down the system before removing the key.

    Has something changed recently that would cause this to fail auto mounting? Or could someone point me to logs that could help identify why the disks are no longer mounting automatically at startup? I've had to recently replace a disk, and in the process when I had to restart the server, I noticed the Encrypted filesystems are no longer auto mounting.


    Any ideas or help?

    So I finally got around to digging through the log. It has a lot going on that I'm not sure I can pick anything good/bad out of it. What I did find that looks like it's problematic though is the following lines:
    Sep 10 14:46:21 atlantis systemd[1]: Job dev-disk-by\x2dlabel-disk8.device/start timed out.
    Sep 10 14:46:21 atlantis systemd[1]: Timed out waiting for device dev-disk-by\x2dlabel-disk8.device.
    Sep 10 14:46:21 atlantis systemd[1]: Dependency failed for /srv/dev-disk-by-label-disk8.
    Sep 10 14:46:21 atlantis systemd[1]: Dependency failed for File System Check on /dev/disk/by-label/disk8.


    Which looks like something timed out while trying to load. Which my thought would be ok something is missing or not mounting, but I'm not sure what. I can't seem to find anything relative to these device calls any earlier in the journal readout. I'm hesitant to post the whole output as it seems to have some semi-sensitive detail. It appears shortly before this that sda1 thru sdd1 (disks 1-4 which are not encrypted) mount successfully. And slightly before that sdj1 (the USB drive with the keys) is mounted properly as well.



    Any direction that can be headed to look for what might be going wrong or causing my delay/timeout?

    • Official Post

    I've seen that error in one of my development vm. Is something to do with systemd escape path, look at x2d before label. These are units auto generated by systemd. At the end I ended up rolling the vm back to original state. There is a bug report as I understand in systemd about this and should be fixed in backport systemd, but as I recall my vm keep failing mount devices even with the bpo systemd.
    Your can try and change dev path for UUID= manually to test.


    edit:This is the bug i saw


    https://github.com/systemd/systemd/issues/3300


    looks similar but not the same

    Hmmm ... interesting. Not good news in this case since this is my prod machine. =O I was thinking those labels looked funky with all the x2d all over. Interestingly though, the other disks that show the same oddities, are mounting and working fine upon boot without any issue.


    Where would I attempt to change the UUID manually? I'll have to make some backups and test it out. Just a pain testing as the machine takes a bit to boot from everything going on, and of course downs my apps. I noticed this after recently adding a new disk, copying data, removing bad disk. Not sure if it was the case before that. So perhaps the new disk introduced was causing something wacky as I had to re-order my BIOS start order too. I'll have to dig in there as well to make sure nothing got reassigned funky.

    Well, not sure what happened here, but I'm back running again. I don't know how, but I think the UUID got modified for the disks perhaps? I didn't have a full copy of the previous and new UUID I don't believe. I thought I matched them up and all worked. So I then attempted fully destroying the current filesystems/disks and re-doing it all. This time, the UUID seemed to match up correctly and it started with no issue. Much faster I might add too. So we'll see if it stays steady, but for now it seems I'm ok. Bad news, I don't really have true solid root cause analysis.

    Quote from subzero79

    I just recently moved all my data to encrypted drives. So basically i have 2x3TB disks encrypted, to unlock them i use another small virtual disk (could be a usb in case of bare metal i guess) that contains the keyfiles. That small virtual drive is encrypted also, so basically I unlock this drive at boot which has the keys to unlock the other drives, at the end of the boot sequence the key disk gets unmounted and closed.
    This was done in a combination of crypttab and a custom script embedded in the initramfs.


    You can do this by using just crypttab and fstab, and let systemd take care......but since systemd auto creates the dependencies at boot whenever you attempt to close the key disk, systemd will close and unmount all the related devices.

    How did you write your script in initramfs so that you could remove the usb key and keep the other encrypted drives mounted?

    • Official Post

    I recently upgraded to stretch and this no longer worked properly, a strange error that didn’t let me type the passphrase at initramfs loading.


    I got rid of that setup and I created a better solution which is to use before-decrypt.target that holds off all relevant services except ssh and disk mounts. Once system boots you can access the server via ssh. A script that I have will:
    - prompt for password of key disk
    - mount the keydisk
    - continue booting until system reaches multi-user.target
    - finally close unmount and close the keydisk.


    If the keydisk is not encrypted(simple usb disk with keyfiles) this can be automated via systemd hotplug.


    The idea of this was extracted from here


    https://blog.iwakd.de/headless-luks-decryption-via-ssh


    Most of it is covered there. But also for Omv you need a fstab mkconf that pipe all mounts requires to decrypt.target

    One thing about the usb stick...many people mount the stick to get the keyfile. but there are better ways...you could place /dev/random infront of the first partition. This way no mount is needed anymore. But check the start of your usb part:

    Code
    Device Boot Start End Sectors Size Id Type
/dev/sdx1 2048 15633407 15631360 7,5G 83 Linux


    Attention: make sure you have enough entropy:

    Bash
    dd if=/dev/random of=/dev/sdx bs=512 seek=1 count=2046
    Bash
    dd if=/dev/sdx bs=512 skip=1 count=8 > tempKeyFile.bin

    Add this keyfile to your luks header!


    Add this option to crypttab entries: "x x x luks,keyfile-size=4096,keyfile-offset=512"


    I tried to auto decrypt a drive being bing mount into a docker container. But neither crypttab or systemd service (-> Before=docker.service) did the trick in time. The bind mount was always empty in container...even the luks drive was opened automatically.


    Does someone know how to properly coordinate with docker on omv?

    Intel Pentium G3460T @ 3GHz
    Debian GNU/Linux 9.2 (stretch)
    Release: 4.0.10-1 Arrakis

    Edited once, last by godfuture ().

    Hi godfuture


    I just came across your interesting method for storing keyfile on a USB. It's appealing because it seems to reduce one layer of operation in the process chain. However I'm having trouble understanding your commands. Could you (or anyone else reading this), please clarify the following?


    • During the creation of the key, you seem to have created random bytes that's 1MB in size with dd parameters "bs=512 seek=1 count=2046". Is there any reason that you did this since you only need 4096 bytes?
    • Where do you actually specify luks to look for this volume header on the USB? Is it in the crypttab entries? I don't see anything that points to a usb device.
    • For crypttab entries, what is this for--> keyfile-offset=512?

    Thanks a lot.



    Quote from rayycy

    Hi godfuture


    I just came across your interesting method for storing keyfile on a USB. It's appealing because it seems to reduce one layer of operation in the process chain. However I'm having trouble understanding your commands. Could you (or anyone else reading this), please clarify the following?


    • During the creation of the key, you seem to have created random bytes that's 1MB in size with dd parameters "bs=512 seek=1 count=2046". Is there any reason that you did this since you only need 4096 bytes?
    • Where do you actually specify luks to look for this volume header on the USB? Is it in the crypttab entries? I don't see anything that points to a usb device.
    • For crypttab entries, what is this for--> keyfile-offset=512?

    Thanks a lot

    1) The real keyfile is created with "dd if=/dev/sdx bs=512 skip=1 count=8 > key" which is only a part of the sectors that were filled with "dd if=/dev/random of=/dev/sdx bs=512 seek=1 count=2046". Basically I filled the space after the MBR till the start of the first partition with true random data to even better hide my key. Thats all.



    2) Yes. In crypttab you have to add the parameters for reading your usb the same way you created it -> "x x x luks,keyfile-size=4096,keyfile-offset=512". The first three x represent the default crypttab parameter. I think they are the crypt volume name, the location of the hard drive and the keyfile location. Mine looks like:
    sdc-crypt /dev/disk/by-uuid/2c5cc311-f87a-40b7-b95f-e06c7905e906 /dev/disk/by-id/usb-USB2.0_Flash_Disk-0:0


    3) This tells luks to start reading the usb stick after the first 512 bytes. This is related to the way the keyfile was created -> "dd if=/dev/sdx bs=512 skip=1 count=8 > key".

    Intel Pentium G3460T @ 3GHz
    Debian GNU/Linux 9.2 (stretch)
    Release: 4.0.10-1 Arrakis

    Like 1
    Quote from godfuture

    1) The real keyfile is created with "dd if=/dev/sdx bs=512 skip=1 count=8 > key" which is only a part of the sectors that were filled with "dd if=/dev/random of=/dev/sdx bs=512 seek=1 count=2046". Basically I filled the space after the MBR till the start of the first partition with true random data to even better hide my key. Thats all.


    2) Yes. In crypttab you have to add the parameters for reading your usb the same way you created it -> "x x x luks,keyfile-size=4096,keyfile-offset=512". The first three x represent the default crypttab parameter. I think they are the crypt volume name, the location of the hard drive and the keyfile location. Mine looks like:
    sdc-crypt /dev/disk/by-uuid/2c5cc311-f87a-40b7-b95f-e06c7905e906 /dev/disk/by-id/usb-USB2.0_Flash_Disk-0:0


    3) This tells luks to start reading the usb stick after the first 512 bytes. This is related to the way the keyfile was created -> "dd if=/dev/sdx bs=512 skip=1 count=8 > key".

    Godfuture, thanks so much for your explanation. I ended up still putting the key on a USB partition that I had to mount, due to my time constraint and the lack of understanding your method. But this will for sure help with future builds. Appreciate your help. Cheers.

    hello guys,


    first thanks for this thread and your time.


    i want also to encrypted my hdd and unlock with a usb key file.


    I have problems with the manual. i write my steps, can you confirm that is right?


    1. i put in my usb key (sde)
    2. dd if=/dev/random of=/dev/sde bs=512 seek=1 count=2046 to create random bytes ( how long takes that?)
    3. dd if=/dev/sde bs=512 skip=1 count=8 > tempKeyFile.bin (this create the keyfile?)


    4. i go to omv luks an create a new luks hdd and upload the key file ( tempKeyFile.bin?)


    5. i find out my uuid from the usb key (sde)


    6. i edit the cryptab file and a the usb keyfile for the hdd (example sdb-crypt uuid /uuid(usb) ?



    i hope that you understand what i mean and thanks for your time!

    Quote from felix

    1. i put in my usb key (sde)

    I don't know what you want to do with this USB stick, besides using it for storing the encryption key. But I formatted the USB stick
    with mbr and added a fat32 partition. This way you might even use it to store something and additionally make it look even more "normal" to intruders. I know, this sounds so paranoid...but I guess you read newspaper ;)

    Quote from felix

    2. dd if=/dev/random of=/dev/sde bs=512 seek=1 count=2046 to create random bytes ( how long takes that?)

    As said, /dev/random uses true random data which is the safest, but has also dependencies -> enough entropy. Means, you need to make sure your entropy stream does not run out of data. The stream gets blocked, if entropy runs out. This would increase the time used to create random numbers enormously. Entropy is normally filled by random input like keyboard, mouse and so on. But these type of inputs are not directly available on server. Threfore get an application that creates entropy by inputs like network data...can't remember the name of the app now. If that sounds too much for you, you might want to read about /dev/random vs /dev/urandom.

    Quote from felix

    3. dd if=/dev/sde bs=512 skip=1 count=8 > tempKeyFile.bin (this create the keyfile?)

    Yes, but on server.

    Quote from felix

    4. i go to omv luks an create a new luks hdd and upload the key file ( tempKeyFile.bin?)

    Instead of transferring the key to client and then upload it back again, you might want to create the luks device on webconsole (starting with password) and add the key by command line later on. But still, its up to you. Starting with keyfile is fine as well.

    Quote from felix

    5. i find out my uuid from the usb key (sde)

    UUID for the partition that should get unlocked and ID for accessing the USB stick during boot for the keyfile.

    Quote from felix

    6. i edit the cryptab file and a the usb keyfile for the hdd (example sdb-crypt uuid /uuid(usb) ?

    Yes. See above in previous comments.

    Intel Pentium G3460T @ 3GHz
    Debian GNU/Linux 9.2 (stretch)
    Release: 4.0.10-1 Arrakis

    Edited once, last by godfuture ().

    Like 1

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login