Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision |
computing:vmserver [2022/07/26 19:22] – oemb1905 | computing:vmserver [2023/05/21 21:47] – oemb1905 |
---|
------------------------------------------- | ------------------------------------------- |
| |
What? I put a large physical host at a data center in Northern New Mexico. I purchased a few blocks of ip4 addresses, and set up virt-manager to run guest OSs for my business and other projects. Put simply, I was creating virtualized infrastructure on a physical host I owned and configured to run VPSs for my own stuff, for clients, my kids, for gaming projects, other hacking projects, and so on. I used a data center rather than hosting this at home because a) it's noisy as f, and b) the data center has a symmetric gig. | I was given a dual 8-core Xeon SuperMicro server (32 threads), with 8 HD bays in use, 96GBRAM, 8x 6TB Western Digital in Raid1 zfs mirror (24TB actual), with a 120GBSSD boot volume stuck behind the power front panel running non-GUI Debian. (Thanks to Kilo Sierra for the donation.) My first job was to calculate whether my PSU was up to the task I intended for it. I used a 500W PSU. From my calculations, I determined that the RAM would be around 360W at capacity but rarely hit that or even close, that the HDs would often (especially on boot) hit up to 21.3W per drive, or around 150W total, and that excluded the boot SSD volume. The motherboard would be 100W, putting me at 610W. Since I did not expect the RAM, HDs, and other physical components to concurrently hit peak consumption, I considered it safe to proceed, and figured no more than around 75% of that ceiling would be used at any one time. The next step was to install the physical host OS (Debian) and setup the basics of the system (hostname, DNS, etc., basic package installs). On the 120GB SSD boot volume, I used a luks / pam_mount encrypted home directory, where I could store keys for the zfs pool and/or other sensitive data. I used a nifty trick in order to first create the pools simply with short names, and then magically change them to block ids without having to make the pool creation syntax cumbersome. |
| |
Why? I am a math/CS teacher for HS/college levels. I use free software to help my educational needs, and I would like to use self-hosted services such as but not exclusive to Big Blue Button, Nextcloud, etc., to assist my instruction. Additionally, I run a small IT/education free software business, and since my client load is only around 10-20, I figured the server could also replace my current business website. Lastly, I serve on some non-profit boards and I figured I could move those websites over as well. It would save money, and it would also make the systems much more portable since restoring would as simple as adding a .img to a new virt-manager instance. | **Update**: I am now running a newer server with 48 threads, 12 hard drive bays, 384GB RAM, 4 two-way mirrors of Samsung enterprise SSDs for the primary vm zpool, and 2 two-way mirrors of 16TB platters for the backup zpool and for some mailservers. These are also SAS hard drives now, not SATA. The server can handle 1.5TB of RAM. |
| |
I was given a dual 8-core Xeon SuperMicro server (32 threads), with 8 HD bays in use, 96GBRAM, 8x 6TB Western Digital in Raid1 zfs mirror (24TB actual), with a 120GBSSD boot volume stuck behind the power front panel running non-GUI Debian. (Thanks to Kilo Sierra for the donation.) My first job was to calculate whether my PSU was up to the task I intended for it. I used a 500W PSU. From my calculations, I determined that the RAM would be around 360W at capacity but rarely hit that or even close, that the HDs would often (especially on boot) hit up to 21.3W per drive, or around 150W total, and that excluded the boot SSD volume. The motherboard would be 100W, putting me at 610W. Since I did not expect the RAM, HDs, and other physical components to concurrently hit peak consumption, I considered it safe to proceed, and figured no more than around 75% of that ceiling would be used at any one time. The next step was to install the physical host OS (Debian) and setup the basics of the system (hostname, DNS, etc., basic package installs). On the 120GB SSD boot volume, I used a luks / pam_mount encrypted home directory, where I could store keys for the zfs pool and/or other sensitive data. I used a nifty trick in order to first create the pools simply with short names, and then magically change them to block ids without having to make the pool creation syntax cumbersome. | |
| |
zpool create -m /mnt/vms vms -f mirror sda sdb mirror sdc sdh mirror sde sdf mirror sdg sdh | zpool create -m /mnt/pool pool -f mirror sda sdb mirror sdc sdh mirror sde sdf mirror sdg sdh |
zpool export vms | zpool export pool |
zpool import -d /dev/disk/by-id vms | zpool import -d /dev/disk/by-id pool |
| |
Now that pool was created, I created two encrypted datasets, which is zfs name for encrypted file storage inside the pool. The datasets each unlock by pulling a dd-generated key from the encrypted (and separate) home partition on the SSD boot volume. I set up the keys/datasets as follows: | Now that pool was created, I created two encrypted datasets, which is zfs name for encrypted file storage inside the pool. The datasets each unlock by pulling a dd-generated key from the encrypted (and separate) home partition on the SSD boot volume. I set up the keys/datasets as follows: |
zfs list -H -o name -t snapshot | xargs -n1 zfs destroy | zfs list -H -o name -t snapshot | xargs -n1 zfs destroy |
| |
However, if the data center is compromised physically or their upstream goes down, I also need remote/failover options, so my next task was to find a way to easily take advantage of cp's understanding of sparse files and tar so that I could easily use rsync to bring over tarballs of the VM disks that only utilized actual data, instead of the entire 1TB container. To do this, I used the ''c'' and ''S'' flags in bsdtar, together with bzip2 compression for speed. Make sure to use bsdtar (''sudo apt install libarchive-tools'')! I did this by making the the script that follows, and take care when adjusting this script, as most alterations will break the ability of tar to properly treat the .img file as sparse: | Of course, off-site backups are essential. To do this, I use a small script that powers down the VM, uses ''cp'' with the ''--sparse=always'' flag to preserve space, and then uses tar with pbzip2 ''sudo apt install pbzip2'' compression to save even more space. From my research, bsdtar seems to honor sparsity better than gnutar so install that with ''sudo apt install libarchive-tools''. The ''cp'' command is not optional, moreover, for remember tar will not work directly on an ''.img'' file. Here's a small shell script with a loop for multiple VMs within the same directory. I also added a command at the end that will delete any tarballs older than 180 days. |
| |
DATE=date +"%Y%m%d-%H:%M:%S" | DATE=`date +"%Y%m%d-%H:%M:%S"` |
cd /backups | IMG="vm1.img vm2.img" |
cp -ar /vms/vol.img /backups/vol.img_QUICK_.bak | for i in $IMG; |
bsdtar --use-compress-program=pbzip2 -Scf vol.img_QUICK_.tar.bz2 vol.img_QUICK_.bak | do |
mv /backups/vol.img_QUICK_.tar.bz2 /backups/tbs/vol.img_QUICK_$DATE.tar.bz2 | virsh shutdown $i |
rm /backups/vol.img_QUICK_.bak | wait |
find /egcy/backups/tarballs -type f -mtime +30 -delete | cd /mnt/vms/backups |
| cp -ar --sparse=always /mnt/vms/students/$i /mnt/vms/backups/SANE_$i.bak |
| wait |
| virsh start $i |
| bsdtar --use-compress-program=pbzip2 -Scf SANE_$i.tar.bz2 SANE_$i.bak |
| mv /mnt/vms/backups/SANE_$i.tar.bz2 /mnt/vms/backups/tarballs/$i:_SANE_$DATE:_.tar.bz2 |
| rm /mnt/vms/backups/SANE_$i.bak |
| find /mnt/vms/backups/tarballs -type f -mtime +180 -delete |
| |
In addition to daily live images using the above, script, I also run a 1/3 days version called SANE , which runs virsh shutdown domain before copying/tarballing and then runs virsh start domain at the end of the tarballing. The host is set to keep 30 days worth of images, but you can easily adjust the flag in the last line above to your use case. After these run, pull the changes to offsite backup ``/`` computer using rsync on the offsite host as follows: | The script above can be downloaded here [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/scripts/sane-vm-backup.sh|sane-vm-backup.sh]]. I use multiple copies of the loop script for related groups of VMs on the same physical host, and then stagger when they run with cron to limit simultaneous read/write time as follows: |
| |
sudo rsync -av --log-file=/home/logs/backup-of-vm.log --ignore-existing -e 'ssh -i /home/user/.ssh/id_rsa' root@domain.com:/backups/tarballs/ /media/user/Backups/ | #backup student machines, client machines |
| 00 03 1,15 * * /usr/local/bin/sane-vm-backup-students.sh >> /root/sane-vm-backup-students.log |
| 00 03 2,16 * * /usr/local/bin/sane-vm-backup-clients.sh >> /root/sane-vm-backup-clients.log |
| |
Since the workstation is on rsnapshot, I get redundant dailies on its backup that extend beyond the quantity on the physical host (because of space on my primary workstation). | On the off-site backup machine, I pull the tarballs down using a one line rsync script. I adjust the cron timing of the rsync script to work well with when the tarballs are created. |
| |
| sudo rsync -av --log-file=/home/logs/backup-of-vm-tarballs.log --ignore-existing -e 'ssh -i /home/user/.ssh/id_rsa' root@domain.com:/backups/tarballs/ /media/user/Backups/ |
| |
| The off-site backup workstation uses rsnapshot, which provides me with months of restore points and thus provides version control for if/when errors are not caught immediately. |
| |
**** | **** |
-- Network Bridge Setup / VMs -- | -- Network Bridge Setup / VMs -- |
| |
Once the physical host was setup, I created two vanilla VMs using the virt-manager GUI with X-forwarding over ssh prior to bringing the server on site. Once those were setup, I headed to the DataCenter figuring I might have to tinker with bridging and network configurations a bit onsite before leaving the device there indefinitely and subject to 24 hour notice emergency KVM. Once there, I worked for about 3 hours configuring the interfaces for bridge mode, ultimately with two physical ethernet cables into the device, one on a non-bridged static IP / interface and the other on a static IP / interface dedicated to bridging. After about a week of thinking back on my Slackware phases, my freeBSD phases and the late 90s and early 2000s, ... AND ... a lot of Stack Exchange tutorials, I decided on the manual command line approach, utilizing no desktop tools to manage interfaces, just stripped down Debian with no network-manager etc., and just manual entries for needed functionality. **Here's what I came up with for interfaces on the physical host**: | Up until now, I've covered how to provision the machines with virt-manager, how to backup the machines on the physical host, and how to pull those backups to an off-site workstation. Now I will discuss how to assign each VM an external IP. The first step is to provision the physical host with a virtual switch (wrongly called a bridge) to which VMs can connect. To do this, I kept it simple and used ''ifup'' and ''bridge-utils'' package and some manual editing in ''/etc/network/interfaces''. |
| |
| sudo apt install bridge-utils |
| sudo brctl addbr br0 |
sudo nano /etc/network/interfaces | sudo nano /etc/network/interfaces |
| |
That file should look like this (adjust to your use-case, ofc): | Now that you have added created the virtual switch, you need to reconfigure your physical host's ''/etc/network/interfaces'' file to use the switch. In my case, I used 1 IP for the host itself, and another for the switch, meaning that two ethernet cables are plugged into my physical host. I did this so that if I hose my virtual switch settings, I still have a separate connection to the box. Here's the configuration in ''interfaces'': |
| |
#eth0 (alt name ent8s0g) physical host base-connection | #eth0 [1st physical port] |
auto ent8s0g0 | auto ent8s0g0 |
iface ent8s0f0 inet static | iface ent8s0f0 inet static |
nameserver 8.8.8.8 | nameserver 8.8.8.8 |
| |
#eth1 (alt name enp8s0g1) interface for bridge | #eth1 [2nd physical port] |
auto enp8s0g1 | auto enp8s0g1 |
iface enp8s0g1 inet manual | iface enp8s0g1 inet manual |
bridge_ports enp8s0g1 | bridge_ports enp8s0g1 |
nameserver 8.8.8.8 | nameserver 8.8.8.8 |
| |
Once that's done, run ''ip a'' to make sure your primary interface connects upstream to the Data Center, and also make sure that the interface ''br0'' appears at the bottom and that the secondary interface shows it as bound to the bridge in its output. If you are doing the same overall project, but limited to a home network behind a dhcp router, then do the following assuming eth1 is primary interface: | After that, either reboot or ''systemctl restart networking.service'' to make the changes current. Execute ''ip a'' and you should see both external IPs on two separate interfaces, and you should see ''br0 state UP'' in the output of the second interface ''enp8s0g1''. You should also run some ''ping 8.8.8.8'' and ''ping google.com'' tests to confirm you can route. If anyone wants to do this in a home, small business, or other non-public facing environment, you can easily use dhcp and provision the home/small business server's ''interface'' file as follows: |
| |
auto eth1 | auto eth1 |
bridge_ports eth1 | bridge_ports eth1 |
| |
I included the dhcp / home-version in case self-hosting home users want to try something similar. Okay, not as for as nameservers are concerned, I find sometimes they don't properly populate to resolv.conf, so I do the following so that my ''resolv.conf'' configurations stick and I don't lose upstream DNS. (Note: I do this because Debian - rightfully - still supports manual over-writing of /etc/resolv.conf.) | The above home-version allows, for example, users to have a virtual machine that gets an ip address on your LAN and makes ssh/xrdp access far easier. If you have any trouble routing on the physical host, it could be that you do not have nameservers setup. If that's the case, do the following: |
| |
echo nameserver 8.8.8.8 > /etc/resolv.conf | echo nameserver 8.8.8.8 > /etc/resolv.conf |
| systemctl restart networking.service |
| |
Reboot the host and ping 8.8.8.8 and google.com to ensure you have link and upstream DNS. Next up, it is time to configure the guest / VM machine. I saw a lot of good tutorials online, but most of them got sloppy at this stage as far as interfaces and bridging was concerned, so I'll try to be clear where they were not. When you set up the new VM (not covered here), instead of relying on the NAT-based default network, change the option to "Bridge" (this is in the virt-manager GUI) and enter the name of the bridge, in my case ''br0''. (You can also use ''virsh'' for this step, but why lol - I just use X forwarding and open the GUI.) This step connects the hypervisor NIC to the virtual switch of the bridge on the physical host. Once that's done, spin up the VM and open up the Terminal (the one inside the VM). **In the VM's Terminal, configure the NIC interface as follows**: | Now that the virtual switch is setup, I can now provision VMs and connect them to the virtual switch ''br0'' in virt-manager. You can provision the VMs within the GUI using X passthrough, or use the command line. First, create a virtual disk to your desired size by excuting ''sudo qemu-img create -f raw new 1000G'' and then run something like this: |
| |
sudo nano /etc/network/interfaces | sudo virt-install --name=new.img \ |
| --os-type=Linux \ |
| --os-variant=debian10 \ |
| --vcpu=1 \ |
| --ram=2048 \ |
| --disk path=/mnt/vms/students/new.img \ |
| --graphics spice \ |
| --location=/mnt/vms/isos/debian-11.4.0-amd64-netinst.iso \ |
| --network bridge:br0 |
| |
This file should look like this (adjust to your use-case - and again, this is inside the VM Terminal, and not on the Terminal of the physical host): | The machine will open in virt-viewer, but if you lose the connection you can reconnect easily with: |
| |
| virt-viewer --connect qemu:///system --wait new.img |
| |
| Once you finish installation, configure the guestOS interfaces file ''sudo nano /etc/network/interfaces'' with the IP you intend to assign it. You should have something like this: |
| |
auto epr1 | auto epr1 |
nameservers 8.8.8.8 | nameservers 8.8.8.8 |
| |
The VM interface is listed inside the guest/VM as epr1 - but remember, that's connected to the virtual switch and bridge through the previous steps we did on the virt-manager GUI, so don't worry. After this step, restart the networking service and check to see if your IP address is assigned. If you are doing this in a home set-up, and you want the VM to receive an ip address via the host switch you set up above, then adjust its interface file as follows: | If you are creating VMs attached to a virtual switch on the smaller home/business environment, then adjust the guest OS by executing ''sudo nano /etc/network/interfaces'' and then something like this recipe: |
| |
auto epr1 | auto epr1 |
iface epr1 inet dhcp | iface epr1 inet dhcp |
| |
Now, to ensure DNS works on both the physical host and the VM flawlessly, I manually adjust resolv.conf. | If your guest OS uses Ubuntu, you will need to do extra steps to ensure that the guestOS can route. This is because Ubuntu-based distros have deprecated ''ifupdown'' in favor of ''netplan'' and disabled manual editing of ''/etc/resolv.conf''. So, either you want to learn netplan syntax and make interface changes using its YAML derivative, or you can install the optional ''resolvconf'' package to restore ''ifupdown'' functionality. To do this, adjust the VM provision script above (or use the virt-manager GUI with X passthrough) to temporarily use NAT then override Ubuntu defaults and restore ''ifupdown'' functionality as follows: |
On Ubuntu VMs, which do not by default allow manual over-writing of resolv.conf, I add the resolv.conf package and manage my upstream DNS as follows: | |
| |
| sudo apt install ifupdown |
| sudo apt remove --purge netplan.io |
sudo apt install resolvconf | sudo apt install resolvconf |
sudo nano /etc/resolvconf/resolv.conf.d/tail | sudo nano /etc/resolvconf/resolv.conf.d/tail |
<nameserver 8.8.8.8> | <nameserver 8.8.8.8> |
sudo service networking restart | systemctl restart networking.service |
ip a | |
| You should once again execute ''ping 8.8.8.8'' and ''ping google.com'' to confirm you can route within the guest OS. Sometimes, I find a reboot is required. At this stage, you now have a physical host configured with a virtual switch, and one VM provisioned to use the switch with its own external IP. Both the physical host and guest OS in this scenario are public facing so take precautions to properly secure each by checking services ''netstat -tulpn'' and/or utilizing a firewall. The main things to configure at this point are ssh access so you no longer need to rely on the virt-viewer console which is slow. To do that, you will need to add packages (if you use the netinst.iso). To make that easy, I keep the sources.list on my primary business server: |
| |
| wget https://haacksnetworking.org/sources.list |
| |
| Once you grab the ''sources.list'' file, install ''openssh-server'' and exchange keys, you can now use a shell to ssh into the guestOS henceforward. This means that at this point you are now in a position to create VMs and various production environments at will or start working on the one you just created. Another thing to consider is to create base VMs that have ''interfaces'' and ''ssh'' access all ready to go, and then leverage those to make new instances using ''cp''. Alternately, you can power down a base VM and then clone it as follows: |
| |
| virt-clone \ |
| --original=clean \ |
| --name=sequoia \ |
| --file=/mnt/vms/students/sequoia.img |
| |
At this point, I would probably reboot and/or then from within the VM, ping 8.8.8.8, and then ping google.com to ensure you have link and upstream DNS. Everything should be rosy ;>. Some folks might be concerned about ARP and such, but virt-manager handles that with the gateway entry combined with the bridge, so no need to alter proc and pass traffic, etc. Of course, replace Google's DNS if you so choose, but I had reliability problems with Level 3 during testing myself (sad). As I alluded to above, the point of this project was to create my own virtualized VPS infrastructure, to run my own stuff and for clients. At present, I have ported my business site over, created a teaching nextcloud for Talk with students and for resource sharing, a big blue button instance (that proves to be a major problem and source of pain), a minecraft server, some gamer sites, and some testing VPS for my kids. Here's a few to check out: | The purpose of this project was to create my own virtualized VPS infrastructure (using KVM and VMs), to run my own production environments and for clients, students, and family. Here's a few to check out: |
| |
* [[https://haacksnetworking.org|Haack's Networking]] | * [[https://nextcloud.haacksnetworking.org|Haack's Networking - Nextcloud Talk Instance]] |
* [[https://mrhaack.org|Mr. Haack]] | * [[https://mrhaack.org|GNU/Linux Social - Mastodon Instance]] |
* [[https://space.hackingclub.org|Space]] | * [[http://space.hackingclub.org|My Daughter's Space Website]] |
| * [[http://bianca.hackingclub.org|A Student's Pentesting Website]] |
| |
The last one is my 10 year old daughter's project. It's coming along nicely, and serves as a great way to teach her basic html, CSS, and JS. The next part of this write-up includes how to do the same overall virtualization of infrastructure and VPS leveraging as above, but does so with LUKS first. Read on to see what I originally did ... the reason I ultimately rejected this, moreover, was because you can't use zfs tools when hard drives fail if you do it that way. ;< | That's all folks! Well ... except for one more thing. When I first did all of this, I was convinced that zfs should be within LUKS as it was difficult for me to let go of LUKS / full disk encryption. I've now decided that's insane because of one primary reason. Namely, by putting zfs (or any file system) within LUKS, you lose the hot swapability that you have when zfs (or regular RAID) run directly on the hardware. That would mean that replacing a hard drive would require an entire server rebuild, which is insane. However, it is arguably more secure that way, so if budget and time permits, I've retained how I put zfs inside LUKS in the passage that follows. Proceed at your own risk lol. |
| |
-- LUKS FIRST, ZFS SECOND - (LEGACY SETUP, NOT CURRENT) -- | -- LUKS FIRST, ZFS SECOND - (LEGACY SETUP, NOT CURRENT) -- |
This script simply opens each LUKS crypt so long as you enter or copy/paste your HD password 6 times. After that, one has to re-mount the pool / rebuild the quasi RAID1 mirror/logical volumes with the import command as follows once the volumes are opened: | This script simply opens each LUKS crypt so long as you enter or copy/paste your HD password 6 times. After that, one has to re-mount the pool / rebuild the quasi RAID1 mirror/logical volumes with the import command as follows once the volumes are opened: |
| |
zpool import vms | zpool import pool |
| |
Rebooting in this manner takes about 3-5 minutes for the host, and 2 minutes to screen into my user name, detach, and run the mount LUKS script to mount the pools/datasets, etc. Again, I ultimately rejected this because you cannot use zfs tools when hard drives fail with this setup. | Rebooting in this manner takes about 3-5 minutes for the host, and 2 minutes to screen into my user name, detach, and run the mount LUKS script to mount the pools/datasets, etc. Again, I ultimately rejected this because you cannot use zfs tools when hard drives fail with this setup. |
| |
--- //[[jonathan@haacksnetworking.org|oemb1905]] 2022/07/26 13:07// | --- //[[jonathan@haacksnetworking.org|oemb1905]] 2022/11/12 12:39// |