Differences

This shows you the differences between two versions of the page.

--- computing:proxmox [2023/12/25 04:28] – oemb1905
+++ computing:proxmox [2024/01/07 00:09] (current) – oemb1905
@@ Line 11: / Line 11: @@
 -------------------------------------------
-Most important thing about proxmox is knowing how to get the VMs in and out of it. Let's start with getting them in. First, create an empty VM in the gui, then use the following command changing 500 to match your vmid.
+This tutorial has my notes for setting up Proxmox on Debian GNU/Linux. Ultimately, I set up a PVE instance on my home server as a network bridge and reverse proxy that can host multiple websites/services with different domains and only one external IP address. Later, I will add a another section, which is adapting the reverse proxy and network bridge setup in the PVE instance to an external block of IPs. My first goal was to make sure I could get virsh images into PVE and PVE images out of PVE and into virsh. To import virsh images into PVE, create an empty VM in the gui, then use the following command changing 500 to match your vmid.
   qm importdisk 500 hub.jonathanhaack.com.qcow2 <dataset>
@@ Line 32: / Line 32: @@
   zstd -d vzdump-qemu-506-2023_12_24-04_40_17.vma.zst
-Optionallly, you can use the python tool [[https://github.com/jancc/vma-extractor/blob/master/vma.py|vma.py]]. I don't see any value in using this, however, as it is much slower than the native tool and is also just in raw format. Example syntax is:
-  python3 vma.py vzdump-qemu-101-2023_06_18-12_17_40.vma /mnt/backups/vma
-In order to see where the virtual machine's zdevs are, run the following:
+In order to see where the virtual machine's vdevs are, run the following:
   zfs list
@@ Line 51: / Line 47: @@
   vms/vm-102-disk-2
-Be careful not to delete those thinking wrongly that they are extraneous - those are obviously required for booting, but it might not be clear upon first look. Also, I chose to install those on the ''zpool'' and PVE's default was to store those locally - so they will likely not appear if you did not pick the option to store them on the pool.
+Be careful not to delete those thinking wrongly that they are extraneous - those are obviously required for booting, but it might not be clear upon first look. After getting images inside and out and spinning up the old Windows VM, I decided a year or so later that it would be worth it to spin PVE up with a network bridge and reverse proxy in order to run it to manage my home network. The home Nextcloud, airsonic, pihole, and other similar devices would be virtualized in the PVE and the PVE bridge and reverse proxy would route traffic to them. I intended to support the proxmox project with a small community license, so first I needed to verify cores:
-To verify sockets and cores
   grep "physical id" /proc/cpuinfo | sort -u | wc -l
-Had to switch to postfix with my relay, so used satellite and:
+While installing proxmox on top of stock debian, postfix asked me how to set it up and I chose satellite system and sent outgoing email to my relay as follows:
   [relay.haacksnetworking.org]:587
-Created bridge on pve instance with this recipe:
+Once PVE is setup, log in locally and create the following inside interfaces. Make sure you are using ifupdown, not netplan.
   source /etc/network/interfaces.d/*
@@ Line 75: / Line 69: @@
       bridge-fd 0
-Make sure the PVE instance, which is also our reverse proxy has the FQDN set up in ''/etc/hosts'' properly. Additionally, ensure that your router is assigning the same matching local address (with dhcp) to the PVE as you assign within the PVE. In our situation, pve.domain.com is the proxmux host and has 80/443 passed to it by a port forward on the openWRT router. Only the openWRT router is public facing. The openWRT router is 10.13.13.1 and hands out addresses to clients on the LAN in that subnet. The PVE instance is both a reverse proxy with nginx for clients upstream on the LAN, and also a bridge for the VMs that reside on the PVE hypervisor (as per the bridge config above). Therefore, we want to ensure that both the openWRT router and the PVE instance itself have a static address.
+Make sure the PVE instance, which is also our reverse proxy has the FQDN set up in ''/etc/hosts'' properly. Make sure openWRT is assigning the same matching local address (with dhcp) to the PVE as you assign within the PVE. Traffic on 80/443 goes from the openWRT router at 10.13.13.1, to the network bridge and reverse proxy PVE instance at 10.13.13.2, and from the there to the rest of upstream. Therefore, we want to ensure that both the openWRT router and the PVE instance itself have a static address so we can easily get to the management GUI for the essential services the PVE instance will be serving. Make sure the PVE instance is named properly and reboot.
   sudo nano /etc/hosts
@@ Line 82: / Line 76: @@
   <pve.domain.com>
-Set up configurations for each website on nginx as follows:
+Inside the PVE instance, set up configurations for each website on nginx as follows:
   sudo apt install nginx
   sudo nano /etc/nginx/sites-available/music.domain.com.conf
-Enter this in the file
+Enter this server block adapted as needed
   server {
@@ Line 101: / Line 95: @@
   sudo nano /etc/nginx/sites-available/nextcloud.domain.com.conf
-Enter this in the file
+And enter something like ...
   server {
@@ Line 121: / Line 115: @@
   sudo certbot --authenticator standalone --installer nginx -d example.domain.com --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx"
-Remember, it the reverse proxy web server does not need to match the upstream web server on the LAN at the two different clients ''10.13.13.3'' and ''10.13.13.4'' because it is just a reverse proxy. It is simply fielding external url requests on port 80/443 for music.domain.com and nextcloud.domain.com and sending those requests upstream to the web servers on those clients which take over at that point. This works fine for browsing these sites when you are external, or not on the 10.13.13.0/24 LAN. But if you are within the LAN, you need to enable NAT loopback on your port forwards to 80/443 on the openWRT router. You do that by going to Firewall / Port Forward / Edit / Advanced Settings / Enable NAT Loopback. Additionally, you must add corresponding entries in hosts within openWRT for each FQDN and the corresponding local IP. To do that, you go to DHCP and DNS / Hostnames / Add. That takes care of being able to access these sites within or without the LAN 10.13.13.0/24, but there's an additional issue. From inside the LAN, Firefox has no issue accessing these sites, but Chrome will refuse access to them since the cert on the reverse proxy does not match the host serving the website content. This is related to rfc 6844 and can be handled by either making openWRT a certificate authority that has permission to do this or by installing matching certificates within the clients. I chose the latter option because I don't want to expose openWRT or tinker with it. Inside the shell in each VM, make sure to set the FQDN, add certbot, and add some header entries:
+If you want to make a proper website and/or encrypt the connection to the PVE instance on the LAN, then make an entry like above for the other domains. This time, however, add a few lines to make it accessible only to your LAN subnet and/or that of your VPN. The allow/deny lines stop the PVE from being publicly accessible.
-  sudo nano /etc/hosts [inside VM1]
+  server {
-  <127.0.0.1 music.domain.com music>
+    server_name  pve.domain.com;
-  sudo nano /etc/apache2/sites-enabled/music.domain.com.conf
+    location / {
-  <ServerName music.domain.com>
+        proxy_pass http://10.13.13.254;
-  <Header add myheader "music.domain.com">
+        #lan permit url requests
-  <RequestHeader set myheader "music.domain.com">
+        allow 10.13.13.0/24;
+        #vpn permit url requests
+        allow 10.33.33.0/24;
+        #deny all url requests excep above
+        deny all;
-That takes care of VM1, now let's do VM2:
+    }
+  }
-  sudo nano /etc/hosts [inside VM2]
+After entering the above block, make sure to run certbot again for that domain. Once that's done, you need to set up NAT loopback on the openWRT router so that the instances behind the reverse proxy / PVE instance can route back out. To do that You do that by going to Firewall / Port Forward / Edit / Advanced Settings / Enable NAT Loopback. For the devices within the LAN to locate these domains, you must enter host entries on the openWRT router and that's done in DHCP and DNS / Hostnames / Add. Within each VM, make sure to set the FQDN and install your web server of choice. Make sure to also run certbot within the VMs themselves so that matching certificates are installed. Install headers within the VMs as follows, repeat for each VM:
-  <127.0.0.1 nextcloud.domain.com nextcloud>
-  sudo nano /etc/apache2/sites-enabled/nextcloud.domain.com.conf
+  sudo nano /etc/hosts
-  <ServerName nextcloud.domain.com>
+  <127.0.0.1 music.example.com music>
-  <Header add myheader "nextcloud.domain.com">
+  sudo nano /etc/apache2/sites-enabled/music.example.com.conf
-  <RequestHeader set myheader "nextcloud.domain.com">
+  <ServerName music.example.com>
+  <Header add myheader "music.example.com">
+  <RequestHeader set myheader "music.example.com">
+  sudo nano /etc/hosts
+  <127.0.0.1 nextcloud.example.com music>
+  sudo nano /etc/apache2/sites-enabled/nextcloud.example.com.conf
+  <ServerName nextcloud.example.com>
+  <Header add myheader "nextcloud.example.com">
+  <RequestHeader set myheader "nextcloud.example.com">
 Once this id done, restart your web server. In my case, the VMs are using apache and only the PVE / reverse proxy server is using nginx. After restarting the web servers inside each VM, you now can create the matching certs inside each VM. Here's the commands for VM1:
@@ Line 143: / Line 150: @@
   sudo apt install certbot letsencrypt python3-certbot-apache
   sudo certbot --authenticator standalone --installer apache -d music.example.com --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2"
-Here are the commands inside VM2:
   sudo apt install certbot letsencrypt python3-certbot-apache
   sudo certbot --authenticator standalone --installer apache -d nextcloud.example.com --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2"
@@ Line 152: / Line 156: @@
 2 * * 1 /usr/bin/certbot renew >> /var/log/le-renew.log
+On one of my VMs, which is running apache and behind the nginx reverse proxy, I got a client sync error for the nextcloud service running on it. It was complaining about the file size. Note: the GNU/Linux clients had no such error. Anyway, the following was needed on PVE / reverse proxy configs to resolve this error:
-This should now make Chrome happy since the VM cert will match the cert of the reverse proxy. Why does this work you might ask? Well, when you run certbot inside the VM, it merely cares whether it can be reached externally where you've declared it to be and since it can, it creates the cert without issue. The reverse proxy / PVE instance itself is also able to handle requests for these domains so certbot likewise has no problems issuing the cert there either. This not only makes the Chrome happy, but it addresses what the Chrome developers and rfc 6844 is concerned about, namely, that without this ... connections inside the LAN could potentially be non-TLS or non-header matching. So, this is a best of both worlds. Once this was done and I confirmed that some simple static websites were up and running and accessible externally and internally with no TLS errors on multiple browsers, etc., I then set up a WordPress instance on music.domain.com to ensure that there were no errors using some form of CMS. There were no errors. I am now migrating to this stack because it allows me to segregate services on different VMs in my home network without having to use one massive VM that has vhosts for everything.
+  sudo nano /etc/nginx.proxy_params
+  <post_max_size 10G;>
+  <upload_max_filesize 10G;>
+  sudo nano /etc/nginx/nginx.conf
+  <client_max_body_size 10G;>
+  <client_body_buffer_size 400M;>
-One consideration to bear in mind is the following. My example uses one domain.com and other subdomains on that, like music.domain.com and nextcloud.domain.com. That makes it very easy to do. If you want to do this but want to do two different domains, it is certainly possible, but involves more NAT and setup on the reverse proxy and PVE instance than I a) need or b) care to do. It adds no value. I simply want separate VMs that are publicly accessible, internally accessible, TLS-secured, and separate VMs (database/stack reasons). This is now solved.
+Now the reverse proxy / PVE instance has matching certs to the clients its serving. This project achieves VMs that are publicly accessible, internally accessible, TLS-secured, and separate VMs (database/stack reasons), and using only one external IP. Now, for managing how you much your instance can handle, here's some tips:
+  * Physical Cores * Threads * Physical CPUs = Assignable Virtual Cores
+  * If using zfs for the storage, estimate 50% of RAM used by it
+If ram is an issue, make sure to throttle arc. In my case, I let zfs run free and tightly provision the VMs. Each rig and use case will be different; adjust accordingly. Above, I already shared the command to determine your physical CPUs / sockets. Here's how to determine all three:
+  lscpu | grep "thread"
+  lscpu | grep "per socket"
+  lscpu | grep "Socket(s)"
+  Thread(s) per core: 2
+  Core(s) per socket: 12
+  Socket(s): 2
+To determine vCPUs, you do threads*cores*physical-sockets, or (12*2)*(12)*(2) = 576 for a Supermicro with 2x Xeon e5-2650 v4 chips. As for RAM, I have 64GB total, so subtracting 4Gb for the OS, I have 32GB left for the VMs with roughly 32GB being allocated for zfs. PVE is very strict in interpreting allocated RAM and will OOM kill VMs if it senses it is short. This is in contrast, for example, with virsh which will allow sysadmins to run how they see fit even at risk of virsh crashing. For this reason, either expand your rig, adjust arc, or tightly provision your VMs, or all three. I personally just tightly provisioned my VMs as I want zfs to use as much RAM as it needs. If you prefer to throttle arc, here's how:
+  sudo nano /etc/modprobe.d/zfs.conf
+  options zfs zfs_arc_max=1073741824 [1 gigabyte example]
+  update-initramfs -u -k all
+  reboot
+Adjust arc throttle to fit your system. If you need to stop a system at the CLI because you can't reboot/shutdown at the GUI-level, then do something like:
+  /var/lock/qemu-server/lock-101.conf
+Next entry ...
- --- //[[jonathan@haacksnetworking.org|oemb1905]] 2023/12/25 04:27//
+ --- //[[jonathan@haacksnetworking.org|oemb1905]] 2023/12/28 05:26//

Haack's Wiki

User Tools

Site Tools

Differences

Page Tools