This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
computing:vmserver [2021/11/11 01:36] – oemb1905 | computing:vmserver [2023/01/22 14:12] – oemb1905 | ||
---|---|---|---|
Line 7: | Line 7: | ||
------------------------------------------- | ------------------------------------------- | ||
- | [Spoiler Alert: | + | I was given a dual 8-core Xeon SuperMicro server (32 threads), with 8 HD bays in use, 96GBRAM, 8x 6TB Western Digital in Raid1 zfs mirror (24TB actual), with a 120GBSSD boot volume stuck behind |
- | Context: Why? I am a math/CS teacher for HS/college levels. I use free software to help my educational needs, and I would like to use self-hosted Big Blue Button to assist me when using the Inquirer-Presenter-Scribe thinking routine. I also figured it could replace my current business website and host 3/4 instances of NMBBB for small schools, etc. And thanks to Kilo Sierra for the kind hardware donation! | + | zpool create -m /mnt/pool pool -f mirror sda sdb mirror sdc sdh mirror sde sdf mirror sdg sdh |
- | [Spoiler Alert: I changed the initial setup! Read below to skip the pain!] | + | zpool export pool |
- | -- First Setup -- | + | zpool import |
- | I have a dual 8-core Xeon SuperMicro server (circa 08-12), with 8 HD bays in use, 96GBRAM, SAS to SATA SCU for the hard drives, and 1 HDD being a 120GBSSD boot volume running Debian Bullseye. I calculated that for a 500W PSU, that the RAM would be around 360W at capacity but rarely hit that or even close, that the HDs would often (especially on boot) hit up to 21.3W per drive, or around 150W excluding the boot SSD volume. The motherboard would be 100W, putting me at 610W. This is over, however, I don't expect all the RAM and all the HDs to reach peak, even on boot. After testing and confirming it worked (many times lol), I went on to install the physical host OS (Debian) and setup the basics of the system (hostname, DNS, etc., basic package installs). Again, used the 120GB SSD for Debian (with a home crypt) and kept the other 7 drives for a pool/LVM/spare crypt. My initial idea was to do LUKS first, then zfs, meaning 6 could be mirrors in zfs and I would keep 1 as a spare LUKS crypt for keys, other crap, etc. To create the LUKS crypts, I did the following 6 times, each time appending the last 4 digits of the block ID to the LUKS crypt name: | + | |
- | cryptsetup luksFormat /dev/sda | + | Now that pool was created, I created two encrypted |
- | cryptsetup luksOpen /dev/sda sdafc11 | + | |
- | + | ||
- | You then make sure to use the LUKS label names when making the zpool, not the short names, which can change at times during reboots. I did this as follows: | + | |
- | + | ||
- | sudo apt install zfs-utils | + | |
- | zpool create -m /mnt/vms vms -f mirror sdafc11 sdb9322 mirror sdc8a33 sdh6444 mirror sde5b55 sdf8066 | + | |
- | + | ||
- | ZFS by default executes its mount commands at boot. This is a problem if you don't use auto-unlocking and key files with LUKS to also unlock on boot (and/or a custom script | + | |
- | + | ||
- | sudo -i | + | |
- | screen | + | |
- | su - user [pam_mount unlocks /home for physical host primary user and the spare 1TB vault] | + | |
- | ctrl-a-d [detaches from screen] | + | |
- | + | ||
- | After unlocking my home directory and the spare 1TB vault, the next step is to unlock each LUKS volume, which I decided a simple shell script would suffice which looks like this mount-luks.sh: | + | |
- | + | ||
- | cryptsetup luksOpen / | + | |
- | cryptsetup luksOpen / | + | |
- | cryptsetup luksOpen / | + | |
- | cryptsetup luksOpen / | + | |
- | cryptsetup luksOpen / | + | |
- | cryptsetup luksOpen / | + | |
- | + | ||
- | This script simply opens each LUKS crypt so long as you enter or copy/paste your HD password 6 times. After that, one has to re-mount the pool / rebuild the quasi RAID1 mirror/ | + | |
- | + | ||
- | zpool import vms | + | |
- | + | ||
- | Rebooting in this manner takes about 3-5 minutes for the host, and 2 minutes to screen into my user name, detach, and run the mount LUKS script (which also ends the script by importing the pool). The above was the original setup. I changed that below. | + | |
- | + | ||
- | -- Alternate Setup -- | + | |
- | + | ||
- | Eventually, I ended up agreeing with a friend that it made no sense to do LUKS first because that would preclude me from rebuilding degraded pools using zfs tools. This is because, if a drive failed, then the pool could never be imported, and thus never used without very complicated tinkering. So, I destroyed the LUKS pools above, made a zfs pool with the same command structure but used the regular short-names only. Then, after that, I created two datasets | + | |
dd if=/ | dd if=/ | ||
Line 68: | Line 35: | ||
zfs list -H -o name -t snapshot | xargs -n1 zfs destroy | zfs list -H -o name -t snapshot | xargs -n1 zfs destroy | ||
- | However, if the data center is compromised physically or their upstream goes down, I also need remote/ | + | Of course, off-site backups are essential. To do this, I use a small script that powers down the VM, uses '' |
- | DATE=date +" | + | DATE=`date +" |
- | cd /backups | + | IMG=" |
- | cp -ar /vms/vol.img | + | for i in $IMG; |
- | bsdtar --use-compress-program=pbzip2 -Scf vol.img_QUICK_.tar.bz2 | + | do |
- | mv /backups/vol.img_QUICK_.tar.bz2 /backups/tbs/vol.img_QUICK_$DATE.tar.bz2 | + | virsh shutdown $i |
- | rm /backups/vol.img_QUICK_.bak | + | wait |
- | find /egcy/ | + | cd /mnt/vms/backups |
+ | cp -ar --sparse=always /mnt/vms/students/$i /mnt/vms/backups/SANE_$i.bak | ||
+ | wait | ||
+ | virsh start $i | ||
+ | bsdtar --use-compress-program=pbzip2 -Scf SANE_$i.tar.bz2 | ||
+ | mv /mnt/vms/backups/SANE_$i.tar.bz2 | ||
+ | rm /mnt/vms/backups/SANE_$i.bak | ||
+ | find /mnt/vms/ | ||
- | In addition to daily live images using the above, script, I also run a 1/3 days version called SANE , which runs virsh shutdown domain before copying/tarballing and then runs virsh start domain at the end of the tarballing. The host is set to keep 30 days worth of images, but you can easily adjust the flag in the last line above to your use case. After these run, pull the changes | + | The script |
- | | + | |
+ | 00 03 1,15 * * /usr/local/bin/sane-vm-backup-students.sh >> /root/sane-vm-backup-students.log | ||
+ | 00 03 2,16 * * /usr/local/bin/ | ||
- | Since the workstation is on rsnapshot, I get redundant dailies on its backup that extend beyond | + | On the off-site backup machine, I pull the tarballs down using a one line rsync script. I adjust |
- | | + | |
- | * [[https://services.haacksnetworking.org|VM1 | + | |
- | * [[https://nmbbb.club|VM2 - NM Big Blue Button]] project (the driving reason for this change) | + | |
- | -- Network Bridge Setup / VMs -- | + | The off-site backup workstation uses rsnapshot, which provides me with months of restore points and thus provides version control for if/when errors are not caught immediately. |
- | Once the physical host was setup, I created two vanilla VMs using the virt-manager GUI with X-forwarding over ssh prior to bringing the server on site. Once those were setup, I headed to the DataCenter figuring I might have to tinker with bridging and network configurations a bit onsite before leaving the device there indefinitely and subject to 24 hour notice emergency KVM. Once there, I worked for about 3 hours configuring the interfaces for bridge mode, ultimately with two physical ethernet cables into the device, one on a non-bridged static IP / interface and the other on a static IP / interface dedicated to bridging. After about a week of thinking back on my Slackware phases, my freeBSD phases and the late 90s and early 2000s, ... AND ... a lot of Stack Exchange tutorials, I decided on the manual command line approach, utilizing no desktop tools to manage interfaces, just stripped down Debian with no network-manager etc., and just manual entries for needed functionality. Here's what I came up with: | + | **** |
+ | -- Network Bridge Setup / VMs -- | ||
+ | |||
+ | Up until now, I've covered how to provision the machines with virt-manager, | ||
+ | | ||
+ | sudo apt install bridge-utils | ||
+ | sudo brctl addbr br0 | ||
sudo nano / | sudo nano / | ||
- | That file should look like this (adjust | + | Now that you have added created the virtual switch, you need to reconfigure |
- | #eth0 (alt name ent8s0g) | + | #eth0 |
auto ent8s0g0 | auto ent8s0g0 | ||
iface ent8s0f0 inet static | iface ent8s0f0 inet static | ||
Line 104: | Line 84: | ||
nameserver 8.8.8.8 | nameserver 8.8.8.8 | ||
- | #eth1 (alt name enp8s0g1) interface for bridge | + | #eth1 [2nd physical port] |
auto enp8s0g1 | auto enp8s0g1 | ||
iface enp8s0g1 inet manual | iface enp8s0g1 inet manual | ||
Line 115: | Line 95: | ||
bridge_ports enp8s0g1 | bridge_ports enp8s0g1 | ||
nameserver 8.8.8.8 | nameserver 8.8.8.8 | ||
+ | | ||
+ | After that, either reboot or '' | ||
- | Once that's done, run '' | + | auto eth1 |
+ | iface eth1 inet manual | ||
- | | + | |
+ | iface br0 inet dhcp | ||
+ | bridge_ports eth1 | ||
- | Reboot the host and ping 8.8.8.8 and google.com | + | The above home-version allows, for example, users to have a virtual machine that gets an ip address on your LAN and makes ssh/xrdp access |
- | sudo nano /etc/network/ | + | echo nameserver 8.8.8.8 > /etc/resolv.conf |
+ | systemctl restart networking.service | ||
- | This file should look like this (adjust | + | Now that the virtual switch is setup, I can now provision VMs and connect them to the virtual switch '' |
+ | |||
+ | sudo virt-install --name=new.img \ | ||
+ | --os-type=Linux \ | ||
+ | --os-variant=debian10 \ | ||
+ | --vcpu=1 \ | ||
+ | --ram=2048 \ | ||
+ | --disk path=/ | ||
+ | --graphics spice \ | ||
+ | --location=/ | ||
+ | --network bridge: | ||
+ | |||
+ | The machine will open in virt-viewer, | ||
+ | |||
+ | virt-viewer --connect qemu:/// | ||
+ | |||
+ | Once you finish installation, configure | ||
auto epr1 | auto epr1 | ||
Line 133: | Line 135: | ||
nameservers 8.8.8.8 | nameservers 8.8.8.8 | ||
- | The VM interface is listed inside the guest/VM as epr1 - but remember, that's connected | + | If you are creating VMs attached |
- | sudo service networking restart | + | |
- | | + | iface epr1 inet dhcp |
+ | |||
+ | If your guest OS uses Ubuntu, you will need to do extra steps to ensure that the guestOS can route. This is because Ubuntu-based distros have deprecated '' | ||
+ | |||
+ | | ||
+ | | ||
sudo apt install resolvconf | sudo apt install resolvconf | ||
sudo nano / | sudo nano / | ||
+ | < | ||
+ | systemctl restart networking.service | ||
- | Enter the name server | + | You should once again execute '' |
- | | + | |
- | At this point, I would probably reboot and then from within | + | Once you grab the '' |
- | --- //[[jonathan@haacksnetworking.com|oemb1905]] 2021/11/09 19:55// | + | virt-clone \ |
+ | --original=clean \ | ||
+ | --name=sequoia \ | ||
+ | | ||
+ | |||
+ | The purpose of this project was to create my own virtualized VPS infrastructure (using KVM and VMs), to run my own production environments and for clients, students, and family. Here's a few to check out: | ||
+ | |||
+ | * [[https:// | ||
+ | * [[https://mrhaack.org|GNU/ | ||
+ | * [[http://space.hackingclub.org|My Daughter' | ||
+ | * [[http:// | ||
+ | |||
+ | That's all folks! Well ... except for one more thing. When I first did all of this, I was convinced that zfs should be within LUKS as it was difficult for me to let go of LUKS / full disk encryption. I've now decided that's insane because of one primary reason. Namely, by putting zfs (or any file system) within LUKS, you lose the hot swapability that you have when zfs (or regular RAID) run directly on the hardware. That would mean that replacing a hard drive would require an entire server rebuild, which is insane. However, it is arguably more secure that way, so if budget and time permits, I've retained how I put zfs inside LUKS in the passage that follows. Proceed at your own risk lol. | ||
+ | |||
+ | -- LUKS FIRST, ZFS SECOND - (LEGACY SETUP, NOT CURRENT) -- | ||
+ | |||
+ | My initial idea was to do LUKS first, then zfs, meaning 6 could be mirrors in zfs and I would keep 1 as a spare LUKS crypt for keys, other crap, etc. To create the LUKS crypts, I did the following 6 times, each time appending the last 4 digits of the block ID to the LUKS crypt name: | ||
+ | |||
+ | cryptsetup luksFormat /dev/sda | ||
+ | cryptsetup luksOpen /dev/sda sdafc11 | ||
+ | |||
+ | You then make sure to use the LUKS label names when making the zpool, not the short names, which can change at times during reboots. I did this as follows: | ||
+ | |||
+ | sudo apt install zfs-utils bridge-utils | ||
+ | zpool create -m /mnt/vms vms -f mirror sdafc11 sdb9322 mirror sdc8a33 sdh6444 mirror sde5b55 sdf8066 | ||
+ | |||
+ | ZFS by default executes its mount commands at boot. This is a problem if you don't use auto-unlocking and key files with LUKS to also unlock on boot (and/or a custom script that unlocks). The problem, in this use cases, is ZFS will try to mount the volumes before they are unlocked. The two other options are none/legacy modes, both of which rely on you mounting the volume using traditional methods. But, the whole point of using zfs finally was to not use traditional methods lol, so for that reason I investigated if there was a fix. The closest to a fix is setting cachefile=none boot, but this a) hosed the pool once b) requires resetting, rebooting again and/or manually re-mounting the pool - either of which defeat the point. Using key files, cache file adjustments, | ||
+ | |||
+ | sudo -i | ||
+ | screen | ||
+ | su - user [pam_mount unlocks /home for physical host primary user and the spare 1TB vault] | ||
+ | ctrl-a-d [detaches from screen] | ||
+ | |||
+ | After unlocking my home directory and the spare 1TB vault, the next step is to unlock each LUKS volume, which I decided a simple shell script would suffice which looks like this mount-luks.sh: | ||
+ | |||
+ | cryptsetup luksOpen / | ||
+ | cryptsetup luksOpen / | ||
+ | cryptsetup luksOpen / | ||
+ | cryptsetup luksOpen / | ||
+ | cryptsetup luksOpen / | ||
+ | cryptsetup luksOpen / | ||
+ | |||
+ | This script simply opens each LUKS crypt so long as you enter or copy/paste your HD password 6 times. After that, one has to re-mount the pool / rebuild the quasi RAID1 mirror/ | ||
+ | |||
+ | zpool import pool | ||
+ | |||
+ | Rebooting in this manner takes about 3-5 minutes for the host, and 2 minutes to screen into my user name, detach, and run the mount LUKS script to mount the pools/ | ||
+ | --- // |