This is an old revision of the document!
- vmserver
- Jonathan Haack
- Haack's Networking
- netcmnd@jonathanhaack.com
[Spoiler Alert: I changed the initial setup! Read below to skip the pain!]
I have a dual 8-core Xeon SuperMicro server, with 8 HD bays in use, 96GBRAM, SAS to SATA SCU for the hard drives. I calculated that for a 500W PSU, that the RAM would be around 360W at capacity but rarely hit that or even close, that the HDs would often (especially on boot) hit up to 21.3W per drive, or around 150W excluding the boot SSD volume. The motherboard would be 100W, putting me at 610W. This is over, however, I don't expect all the RAM and all the HDs to reach peak, even on boot. After testing and confirming it worked, I went on to OS and setup. I installed Debian on the 120GB SSD, and as for the other 7 drives, my initial idea was to do LUKS first, then zfs, meaning 6 could be mirrors in zfs and I would keep 1 as a spare LUKS crypt for keys, other crap, etc. To create the LUKS crypts, I did the following 6 times, each time appending the last 4 digits of the block ID to the LUKS crypt name:
cryptsetup luksFormat /dev/sda cryptsetup luksOpen /dev/sda sdafc11
You then make sure to use the LUKS label names when making the zpool, not the short names, which can change at times during reboots. I did this as follows:
sudo apt install zfs-utils zpool create -m /mnt/vms vms -f mirror sdafc11 sdb9322 mirror sdc8a33 sdh6444 mirror sde5b55 sdf8066
ZFS by default mount at boot. This is a problem if you don't use auto-unlocking and key files with LUKS to also unlock on boot (and/or a custom script that unlocks). The problem, in this use cases, is ZFS will try to mount the volumes before they are unlocked. The two other options are none/legacy modes, both of which rely on you mounting the volume using traditional methods. But, the whole point of using zfs finally was to not use traditional methods lol, so for that reason I investigated if there was a fix. The closest to a fix is setting cachefile=none boot, but this a) hosed the pool once b) requires resetting, rebooting again and/or manually re-mounting the pool - either of which defeat the point. Using key files, cache file adjustments, etc., and/or none/legacy were all no-gos for me, so in the end, I decided to tolerate that zfs would fail at boot, and that I would zpool import it afterwards.
sudo -i screen su - user [pam_mount unlocks /home for physical host primary user and the spare 1TB vault] ctrl-a-d [detaches from screen]
After unlocking my home directory and the spare 1TB vault, the next step is to unlock each LUKS volume, which I decided a simple shell script would suffice which looks like this mount-luks.sh:
cryptsetup luksOpen /dev/disk/by-uuid/2702e690-...-0c4267a6fc11 sdafc11 cryptsetup luksOpen /dev/disk/by-uuid/e3b568ad-...-cdc5dedb9322 sdb9322 cryptsetup luksOpen /dev/disk/by-uuid/d353e727-...-e4d66a9b8a33 sdc8a33 cryptsetup luksOpen /dev/disk/by-uuid/352660ca-...-5a8beae15b44 sde5b44 cryptsetup luksOpen /dev/disk/by-uuid/fa1a6109-...-f46ce1cf8055 sdf8055 cryptsetup luksOpen /dev/disk/by-uuid/86da0b9f-...-13bc38656466 sdh6466
This script simply opens each LUKS crypt so long as you enter or copy/paste your HD password 6 times. After that, one has to re-mount the pool / rebuild the quasi RAID1 mirror/logical volumes with the import command as follows once the volumes are opened:
zpool import vms
Rebooting in this manner takes about 3-5 minutes for the host, and 2 minutes to screen into my user name, detach, and run the mount LUKS script (which also ends the script by importing the pool). The above was the original setup. I changed that below.
– Alternate Setup –
I am now tinkering with native zfs encryption. So, I destroyed the LUKS pools above, made a zfs pool with the same command, but used the regular short-names and no LUKS. Then, after that, I created two datasets that are encrypted, each that unlocks/mounts by pulling a dd-generated key from the spare 1TB crypt (only unlocks with key after boot). This will be better if a hard drive fails on the pool and I need to do hot fixes. Here are the commands I ran after using the above-mentioned zpool command on the regular short-names:
dd if=/dev/random of=/mnt/vault/example.key bs=1 count=32 zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///mnt/vault/example.key pool/dataset
When you create this on the current running instance, it will also mount it for you as a courtesy, but upon reboot, you need to do the following:
zfs load-key pool/dataset zfs mount pool/dataset
Optionally, as you create more of these, once can create a shell script to run to automate this. As for backing up the .img files, I do the following. Getting the syntax just right to run in one script was super tough with tar, which is very finiky with sparse files, zfs, and just gnu/linux in general. Take caution adjusting directories and paths with tar. Here's what finally worked for me:
DATE=`date +"%Y%m%d-%H:%M:%S"` cd /backups cp -ar /vms/vol.img /backups/vol.img_QUICK_.bak bsdtar --use-compress-program=pbzip2 -Scf vol.img_QUICK_.tar.bz2 vol.img_QUICK_.bak mv /backups/vol.img_QUICK_.tar.bz2 /backups/tbs/vol.img_QUICK_$DATE.tar.bz2 rm /egcy/backups/haacksnetworking.img_QUICK_.bak
Additionally, I have an alternate 'SANE' version of the script above, which virsh shutdown domain before copying/tarballing, and that virsh start domain at the end of the archival. After these run, pull the changes to offsite backup using rsync.
sudo rsync -av --log-file=/home/logs/backup-of-vm.log --ignore-existing -e 'ssh -i /home/user/.ssh/id_rsa' root@domain.com:/backups/tarballs/ /media/user/Backups/
Since the workstation is on rsnapshot, I get redundant dailies on its backup that extend beyond the quantity on the physical host (because of space on my primary workstation)
— oemb1905 2021/11/04 02:54
