This is an old revision of the document!
- proxmox
- Jonathan Haack
- Haack's Networking
- webmaster@haacksnetworking.org
proxmox
This tutorial has my notes for setting up Proxmox on Debian GNU/Linux. Ultimately, I set up a PVE instance on my home server as a network bridge and reverse proxy that can host multiple websites/services with different domains and only one external IP address. Later, I will add a another section, which is adapting the reverse proxy and network bridge setup in the PVE instance to an external block of IPs. My first goal was to make sure I could get virsh images into PVE and PVE images out of PVE and into virsh. To import virsh images into PVE, create an empty VM in the gui, then use the following command changing 500 to match your vmid.
qm importdisk 500 hub.jonathanhaack.com.qcow2 <dataset> qm set 500 --scsi0 vms:vm-500-disk-0 [sets boot order to make this primary]
Alternately, if you just want to attach a raw image file, or VDD, to an existing proxmox VM, just do the following:
qm importdisk <vmid> obstorage.img <dataset>
After that, you can add storage in hardware within the VM settings to attach this image. In order to export a VM at the command line, execute:
vzdump <vmID> vzdump <vmID> --dumpdir /mnt/backups/machines/pve-images/<vmname>
This will create a VM in .vma format. Then, you can extract the vma to a raw image as follows:
vma extract /var/lib/vz/dump/vzdump-qemu-101-2023_06_18-12_17_40.vma /mnt/backups/vma-extract
If it is compressed:
zstd -d vzdump-qemu-506-2023_12_24-04_40_17.vma.zst
In order to see where the virtual machine's vdevs are, run the following:
zfs list
To delete one (be careful), use the following
zfs destroy pool/dataset
I've migrated my Windows VM here for testing to avoid cumbersome EFI/TPM setup on virt-manager. Here's a page from ProxMux wiki on best practices for Windows VMs. Once thing I noticed when doing the Windows install is that PVE makes two more vdevs for the TPM / EFI, so you get something like the following in zfs list:
vms/vm-102-disk-0 vms/vm-102-disk-1 vms/vm-102-disk-2
Be careful not to delete those thinking wrongly that they are extraneous - those are obviously required for booting, but it might not be clear upon first look. Also, I chose to install those on the zpool and PVE's default was to store those locally - so they will likely not appear if you did not pick the option to store them on the pool.
To verify sockets and cores
grep "physical id" /proc/cpuinfo | sort -u | wc -l
Had to switch to postfix with my relay, so used satellite and:
[relay.haacksnetworking.org]:587
Created bridge on pve instance with this recipe:
source /etc/network/interfaces.d/*
auto lo
iface lo inet loopback
iface enp0s31f6 inet manual
auto vmbr0
iface vmbr0 inet static
address 10.13.13.2/24
gateway 10.13.13.1
bridge-ports enp0s31f6
bridge-stp off
bridge-fd 0
Make sure the PVE instance, which is also our reverse proxy has the FQDN set up in /etc/hosts properly. Additionally, ensure that your router is assigning the same matching local address (with dhcp) to the PVE as you assign within the PVE. In our situation, pve.domain.com is the proxmux host and has 80/443 passed to it by a port forward on the openWRT router. Only the openWRT router is public facing. The openWRT router is 10.13.13.1 and hands out addresses to clients on the LAN in that subnet. The PVE instance is both a reverse proxy with nginx for clients upstream on the LAN, and also a bridge for the VMs that reside on the PVE hypervisor (as per the bridge config above). Therefore, we want to ensure that both the openWRT router and the PVE instance itself have a static address.
sudo nano /etc/hosts <10.13.13.2 pve.domain.com pve> sudo nano /etc/hostname <pve.domain.com>
Set up configurations for each website on nginx as follows:
sudo apt install nginx sudo nano /etc/nginx/sites-available/music.domain.com.conf
Enter this in the file
server {
server_name music.domain.com;
location / {
proxy_pass http://10.13.13.3;
#proxy_set_header Host music.domain.com;
}
}
Repeat this as needed for each VM within PVE. For example, here's another entry for an instance doing nextcloud instead of music:
sudo nano /etc/nginx/sites-available/nextcloud.domain.com.conf
Enter this in the file
server {
server_name nextcloud.domain.com;
location / {
proxy_pass http://10.13.13.4;
#proxy_set_header Host nextcloud.domain.com;
}
}
To activate this server block,do as follows:
cd /etc/nginx/sites-enabled ln -s ../sites-available/pihole.outsidebox.vip.conf .
Once this is done, set up TLS on the PVE/RP instance as follows:
sudo apt install certbot letsencrypt python3-certbot-nginx sudo certbot --authenticator standalone --installer nginx -d example.domain.com --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx"
If you want to make a proper website and/or encrypt the connection to the PVE instance on the LAN, then make an entry like above for the other domains. This time, however, add a few lines to make it accessible only to your LAN subnet and/or that of your VPN. The allow/deny lines stop the PVE from being publicly accessible.
server {
server_name pve.domain.com;
location / {
proxy_pass http://10.13.13.254;
#lan permit url requests
allow 10.13.13.0/24;
#vpn permit url requests
allow 10.33.33.0/24;
#deny all url requests excep above
deny all;
} }
After entering the above block, make sure to run certbot again for that domain. Once that's done, you need to set up NAT loopback on the openWRT router so that the instances behind the reverse proxy / PVE instance can route back out. To do that You do that by going to Firewall / Port Forward / Edit / Advanced Settings / Enable NAT Loopback. For the devices within the LAN to locate these domains, you must enter host entries on the openWRT router and that's done in DHCP and DNS / Hostnames / Add. Within each VM, make sure to set the FQDN and install your web server of choice. Make sure to also run certbot within the VMs themselves so that matching certificates are installed. Install headers within the VMs as follows, repeat for each VM:
sudo nano /etc/hosts <127.0.0.1 music.example.com music> sudo nano /etc/apache2/sites-enabled/music.example.com.conf <ServerName music.example.com> <Header add myheader "music.example.com"> <RequestHeader set myheader "music.example.com"> sudo nano /etc/hosts <127.0.0.1 nextcloud.example.com music> sudo nano /etc/apache2/sites-enabled/nextcloud.example.com.conf <ServerName nextcloud.example.com> <Header add myheader "nextcloud.example.com"> <RequestHeader set myheader "nextcloud.example.com">
Once this id done, restart your web server. In my case, the VMs are using apache and only the PVE / reverse proxy server is using nginx. After restarting the web servers inside each VM, you now can create the matching certs inside each VM. Here's the commands for VM1:
sudo apt install certbot letsencrypt python3-certbot-apache sudo certbot --authenticator standalone --installer apache -d music.example.com --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2" sudo apt install certbot letsencrypt python3-certbot-apache sudo certbot --authenticator standalone --installer apache -d nextcloud.example.com --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2"
Once that's done, make sure that you have cron jobs set for both VMs and for the reverse proxy server / PVE instance. Just enter the following in crontab for all three:
30 2 * * 1 /usr/bin/certbot renew >> /var/log/le-renew.log
Now the reverse proxy / PVE instance has matching certs to the clients its serving. This project achieves VMs that are publicly accessible, internally accessible, TLS-secured, and separate VMs (database/stack reasons), and using only one external IP.
— oemb1905 2023/12/25 04:27
