| Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision |
| computing:vmserver [2022/11/13 01:35] – oemb1905 | computing:vmserver [2024/02/17 20:45] – oemb1905 |
|---|
| ------------------------------------------- | ------------------------------------------- |
| |
| I was given a dual 8-core Xeon SuperMicro server (32 threads), with 8 HD bays in use, 96GBRAM, 8x 6TB Western Digital in Raid1 zfs mirror (24TB actual), with a 120GBSSD boot volume stuck behind the power front panel running non-GUI Debian. (Thanks to Kilo Sierra for the donation.) My first job was to calculate whether my PSU was up to the task I intended for it. I used a 500W PSU. From my calculations, I determined that the RAM would be around 360W at capacity but rarely hit that or even close, that the HDs would often (especially on boot) hit up to 21.3W per drive, or around 150W total, and that excluded the boot SSD volume. The motherboard would be 100W, putting me at 610W. Since I did not expect the RAM, HDs, and other physical components to concurrently hit peak consumption, I considered it safe to proceed, and figured no more than around 75% of that ceiling would be used at any one time. The next step was to install the physical host OS (Debian) and setup the basics of the system (hostname, DNS, etc., basic package installs). On the 120GB SSD boot volume, I used a luks / pam_mount encrypted home directory, where I could store keys for the zfs pool and/or other sensitive data. I used a nifty trick in order to first create the pools simply with short names, and then magically change them to block ids without having to make the pool creation syntax cumbersome. | //vmserver// |
| | |
| | ------------------------------------------- |
| | |
| | I am currently running a Supermicro 6028U-TRTP+ w/ Dual 12-core Xeon E5-2650 at 2.2Ghz, 384GB RAM, with four two-way mirrors of Samsung enterprise SSDs for the primary vdev, and two two-way mirrors of 16TB platters for the backup vdev. All drives using SAS. I am using a 500W PSU. I determine the RAM would be about 5-10W a stick, the mobo about 100W, and the drives would consume most of the rest at roughly 18-22W per drive. The next step was to install Debian on the bare metal to control and manage the virtualization environment. The virtualization stack is virsh and kvm/qemu. As for the file system and drive formatting, I used luks and pam_mount to open an encrypted home partition and mapped home directory. I use this encrypted home directory to store keys for the zfs pool and/or other sensitive data, thus protecting them behind FDE. Additionally, I create file-level encrypted zfs data sets within each of the vdevs that are unlocked by the keys on the LUKS home partition. Instead of tracking each UUID down on your initial build, do the following: |
| |
| zpool create -m /mnt/pool pool -f mirror sda sdb mirror sdc sdh mirror sde sdf mirror sdg sdh | zpool create -m /mnt/pool pool -f mirror sda sdb mirror sdc sdh mirror sde sdf mirror sdg sdh |
| zpool import -d /dev/disk/by-id pool | zpool import -d /dev/disk/by-id pool |
| |
| Now that pool was created, I created two encrypted datasets, which is zfs name for encrypted file storage inside the pool. The datasets each unlock by pulling a dd-generated key from the encrypted (and separate) home partition on the SSD boot volume. I set up the keys/datasets as follows: | Once the pool is created, you can create your encrypted datasets. To do so, I made some unlock keys with the dd command and placed the keys in a hidden directory inside that LUKS encrypted home partition I mentioned above: |
| |
| dd if=/dev/random of=/secure/area/example.key bs=1 count=32 | dd if=/dev/random of=/secure/area/example.key bs=1 count=32 |
| zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///mnt/vault/example.key pool/dataset | zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///mnt/vault/example.key pool/dataset |
| |
| When you create this on the current running instance, it will also mount it for you as a courtesy, but upon reboot, you need to load the key, then mount the dataset using zfs commands. In my case, I created three datasets (one for raw isos, one for disk images, and a last one for backup sparse tarballs). Each one was created as follows: | When the system reboots, the vdevs will automatically mount but the data sets won't because the LUKS keys won't be available until you mount the home partition by logging in to the user that holds the keys. For security reasons, this must be done manually or it defeats the entire purpose. So, once the administrator has logged in to the user in a screen session (remember, it is using pam_mount), they simple detach from that session and then load the keys and datasets as follows: |
| |
| zfs load-key pool/dataset | zfs load-key pool/dataset |
| zfs mount pool/dataset | zfs mount pool/dataset |
| | |
| Once I created all the datasets, I made a script that would load the keys and unlock all of them, then rebooted and tested it for functionality. Upon verifying that the datasets worked, I could now feel comfortable creating VMs again, since the hard drive images for those VMs would be stored in encrypted datasets with zfs. My next task was to create both snapshots within zfs, which would handle routine rollbacks and smaller errors/mistakes. I did that by creating a small script that runs via cron 4 times a day, or every 6 hours: | If you have a lot of data sets, you can make a simple script to load them all at once, etc. Since we have zfs, it's a good idea to run some snapshots. To do that, I created a small shell script with the following commands and then set it to run 4 times a day, or every 6 hours: |
| |
| DATE=date +"%Y%m%d-%H:%M:%S" | DATE=date +"%Y%m%d-%H:%M:%S" |
| /usr/sbin/zfs snapshot pool@backup_$DATE | /usr/sbin/zfs snapshot pool@backup_$DATE |
| |
| The snapshots allow me to perform roll backs when end-users make mistakes, e.g., delete an instructional video after a class session, etc., or what have you. To delete all snapshots and start over, run: | Make sure to manage your snapshots and only retain as many as you can etc., as they will impact performance. If you need to zap all of them and start over, you can use this command: |
| |
| zfs list -H -o name -t snapshot | xargs -n1 zfs destroy | zfs list -H -o name -t snapshot | xargs -n1 zfs destroy |
| |
| Of course, off-site backups are essential. To do this, I use a small script that powers down the VM, uses ''cp'' with the ''--sparse=always'' flag to preserve space, and then uses tar with pbzip2 compression to save even more space. From my research, bsdtar seems to honor sparsity better than gnutar so install that with ''sudo apt install libarchive-tools''. The ''cp'' command is not optional, moreover, for remember tar will not work directly on an ''.img'' file. Here's a small shell script with a loop for multiple VMs within the same directory. I also added a command at the end that will delete any tarballs older than 180 days. | Off-site //full// backups are essential but they take a long time to download. For that reason, it's best to have the images as small as possible. When using ''cp'' in your workflow, make sure to specify ''--sparse=always''. Before powering the virtual hard disk back up, you should run ''virt-sparsify'' on the image to free up the unused blocks on the host and that are not actually used in the VM. In order for the VM to designate those blocks as empty, ensure that you are running fstrim within the VM. If you want the ls command to show the size of the virtual disk that remains after the zeroing, you will need to run ''qemu-img create'' on it, which will create a new copy of the image without listing the ballooned size. the new purged virtual hard disk image can then be copied to a backup directory where one can compress and tarball it to further reduce its size. I use BSD tar and the pbzip2 compression which makes ridiculously small images. GNU tar glitches with the script for some reason. BSD tar can be downloaded with ''sudo apt install libarchive-tools''. I made a script to automate all of those steps for a qcow2 image. I also adapted that to work for raw images. |
| |
| DATE=`date +"%Y%m%d-%H:%M:%S"` | [[https://repo.haacksnetworking.org/haacknet/haackingclub/-/blob/main/scripts/virtualmachines/vm-bu-production-QCOW-loop.sh|vm-bu-production-QCOW-loop.sh]] \\ |
| IMG="vm1.img vm2.img" | [[https://repo.haacksnetworking.org/haacknet/haackingclub/-/blob/main/scripts/virtualmachines/vm-bu-production-RAW-loop.sh|vm-bu-production-RAW-loop.sh]] |
| for i in $IMG; | |
| do | |
| virsh shutdown $i | |
| wait | |
| cd /mnt/vms/backups | |
| cp -ar --sparse=always /mnt/vms/students/$i /mnt/vms/backups/SANE_$i.bak | |
| wait | |
| virsh start $i | |
| bsdtar --use-compress-program=pbzip2 -Scf SANE_$i.tar.bz2 SANE_$i.bak | |
| mv /mnt/vms/backups/SANE_$i.tar.bz2 /mnt/vms/backups/tarballs/$i:_SANE_$DATE:_.tar.bz2 | |
| rm /mnt/vms/backups/SANE_$i.bak | |
| find /mnt/vms/backups/tarballs -type f -mtime +180 -delete | |
| |
| The script above can be downloaded here [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/scripts/sane-vm-backup.sh|sane-vm-backup.sh]]. I use multiple copies of the loop script for related groups of VMs on the same physical host, and then stagger when they run with cron to limit simultaneous read/write time as follows: | On the off-site backup machine, I originally would pull the tarballs down using a one line rsync script. I would adjust the cron timing of the rsync script to work well with when the tarballs are created. |
| | |
| #backup student machines, client machines | |
| 00 03 1,15 * * /usr/local/bin/sane-vm-backup-students.sh >> /root/sane-vm-backup-students.log | |
| 00 03 2,16 * * /usr/local/bin/sane-vm-backup-clients.sh >> /root/sane-vm-backup-clients.log | |
| | |
| On the off-site backup machine, I pull the tarballs down using a one line rsync script. I adjust the cron timing of the rsync script to work well with when the tarballs are created. | |
| |
| sudo rsync -av --log-file=/home/logs/backup-of-vm-tarballs.log --ignore-existing -e 'ssh -i /home/user/.ssh/id_rsa' root@domain.com:/backups/tarballs/ /media/user/Backups/ | sudo rsync -av --log-file=/home/logs/backup-of-vm-tarballs.log --ignore-existing -e 'ssh -i /home/user/.ssh/id_rsa' root@domain.com:/backups/tarballs/ /media/user/Backups/ |
| | |
| | Since then, I've switched to using rsnapshot to pull down the tarballs in some cases. The rsnapshot configurations can be found here: |
| | |
| | [[https://repo.haacksnetworking.org/haacknet/haackingclub/-/tree/main/scripts/rsnapshot|Rsnapshot Scripts]] |
| |
| The off-site backup workstation uses rsnapshot, which provides me with months of restore points and thus provides version control for if/when errors are not caught immediately. | |
| |
| **** | **** |
| Now that the virtual switch is setup, I can now provision VMs and connect them to the virtual switch ''br0'' in virt-manager. You can provision the VMs within the GUI using X passthrough, or use the command line. First, create a virtual disk to your desired size by excuting ''sudo qemu-img create -f raw new 1000G'' and then run something like this: | Now that the virtual switch is setup, I can now provision VMs and connect them to the virtual switch ''br0'' in virt-manager. You can provision the VMs within the GUI using X passthrough, or use the command line. First, create a virtual disk to your desired size by excuting ''sudo qemu-img create -f raw new 1000G'' and then run something like this: |
| |
| #make new vm | sudo virt-install --name=new.img \ |
| sudo virt-install --name=new \ | |
| --os-type=Linux \ | --os-type=Linux \ |
| --os-variant=debian10 \ | --os-variant=debian10 \ |
| --network bridge:br0 | --network bridge:br0 |
| |
| Once the VM is created and attached to the virtual switch (either using the command line or GUI), it is now time to log into the console with X passthrough (or ssh), and configure the guest OS ''interface'' file with the IP you intend to assign it. Once inside the guest OS, execute ''nano /etc/network/interfaces'' and then edit the file with something like this: | The machine will open in virt-viewer, but if you lose the connection you can reconnect easily with: |
| | |
| | virt-viewer --connect qemu:///system --wait new.img |
| | |
| | Once you finish installation, configure the guestOS interfaces file ''sudo nano /etc/network/interfaces'' with the IP you intend to assign it. You should have something like this: |
| |
| auto epr1 | auto epr1 |
| systemctl restart networking.service | systemctl restart networking.service |
| |
| You should once again execute ''ping 8.8.8.8'' and ''ping google.com'' to confirm you can route within the guest OS. Sometimes, I find a reboot is required. At this stage, you now have a physical host configured with a virtual switch, and one VM provisioned to use the switch with its own external IP. Both the physical host and guest OS in this scenario are public facing so take precautions to properly secure each by checking services ''netstat -tulpn'' and/or utilizing a firewall. You are now in a position to create VMs and various production environments at will. One thing I do is I create base VMs that have ''interfaces'' and ''ssh'' access all ready to go, and then I leverage those to make new instances. Make sure to power down the base image so that you don't have an IP conflict at your data center. | You should once again execute ''ping 8.8.8.8'' and ''ping google.com'' to confirm you can route within the guest OS. Sometimes, I find a reboot is required. At this stage, you now have a physical host configured with a virtual switch, and one VM provisioned to use the switch with its own external IP. Both the physical host and guest OS in this scenario are public facing so take precautions to properly secure each by checking services ''netstat -tulpn'' and/or utilizing a firewall. The main things to configure at this point are ssh access so you no longer need to rely on the virt-viewer console which is slow. To do that, you will need to add packages (if you use the netinst.iso). To make that easy, I keep the sources.list on my primary business server: |
| | |
| | wget https://haacksnetworking.org/sources.list |
| | |
| | Once you grab the ''sources.list'' file, install ''openssh-server'' and exchange keys, you can now use a shell to ssh into the guestOS henceforward. This means that at this point you are now in a position to create VMs and various production environments at will or start working on the one you just created. Another thing to consider is to create base VMs that have ''interfaces'' and ''ssh'' access all ready to go, and then leverage those to make new instances using ''cp''. Alternately, you can power down a base VM and then clone it as follows: |
| |
| #clone existing | |
| virt-clone \ | virt-clone \ |
| --original=clean \ | --original=clean \ |
