| Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision |
| computing:synapse [2022/11/21 18:19] – oemb1905 | computing:synapse [2022/12/11 21:48] – oemb1905 |
|---|
| * **Jonathan Haack** | * **Jonathan Haack** |
| * **Haack's Networking** | * **Haack's Networking** |
| * **webmaster@haacksnetworking.org** | * **webmaster@haacksnetworking.org** |
| |
| ------------------------------------------- | ------------------------------------------- |
| <"server_name": "gnulinux.club"> | <"server_name": "gnulinux.club"> |
| |
| | Once Element is installed, you need to enable registration. Matrix no longer allows unchallenged / non-tokened registration without an explicit override, so after seeing how involved token auth was, I added a [[https://developers.google.com/recaptcha/|Google Captcha v2]] instead. |
| |
| | sudo nano /etc/matrix-synapse/homserver.yaml |
| You now need to install postgresql and create a database with a dedicated non-root user: | enable_registration: true [needs to be disabled when creating the db users from command line] |
| | enable_registration_captcha: true |
| | recaptcha_public_key: "yourmomspublickey" |
| | recaptcha_private_key: "yourdadsprivatekey" #On Google, Turn verify origin off |
| | registration_shared_secret: "yourcousinssharedsecret" |
| | federation_client_minimum_tls_version: 1.2 |
| | |
| | At this stage, you could theoretically add a user through the web GUI, however the instance is still using sqlite3 and I preferred something more robust. For that reason, I waited and first created a proper database and dedicated database user, both for scaling and hardening purposes: |
| |
| sudo apt install postgresql | sudo apt install postgresql |
| sudo -u postgres bash | sudo -u postgres bash |
| createuser --pwprompt synapse_user | createuser --pwprompt synapseusr |
| createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse | createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapseusr synapsedb |
| exit | exit |
| | |
| After creating the database, inform synapse of how to reach it in the ''pg_hba.conf'' file as follows: | |
| |
| nano /etc/postgresql/13/main/pg_hba.conf | After recording the password you specified, make sure to configure ''homeserver.yaml'' to use the database instead of the default sqlite3 one. |
| <host synapse synapse_user ::1/128 md5> | |
| sudo systemctl reload postgresql | |
| | |
| It's now time to edit the file ''/etc/matrix-synapse/homeserver.yaml''. Remove the default database configuration, and replace it with the credentials you just made: | |
| |
| | sudo nano /etc/matrix-synapse/homeserver.yaml |
| <database:> | <database:> |
| <name: psycopg2> | <name: psycopg2> |
| <txn_limit: 10000> | <txn_limit: 10000> |
| <args:> | <args:> |
| <user: synapse_user> | <user: synapseusr> |
| <password: secretpassword> | <password: mommalovesU> |
| <database: synapse> | <database: synapsedb> |
| <host: localhost> | <host: localhost> |
| <port: 5432> | <port: 5432> |
| <cp_max: 10> | <cp_max: 10> |
| |
| There are now some options that you can configure based on personal preference. Hack Liberty has its own recommendations, and I agreed with some and not with others. Moreover, I also found that Matrix/Synapse is currently requiring a stricter recipe for the yaml config than their template or even the official docs recommend. I was unable to get to the ''homeserver.yaml'' to work without adding a ''base_url'' line and a Google V2 challenge. Make sure to refer to Synapse's [[https://matrix-org.github.io/synapse/develop/usage/configuration/config_documentation.html|official docs]] for your use case and so that you understand what they each do. Here are the "optional" configurations that I have active, some of which I had to configure to make everything work: | After creating the database, inform synapse of how to reach it in the ''pg_hba.conf'' file as follows: |
| |
| <public_baseurl: "https://gnulinux.club"> | nano /etc/postgresql/13/main/pg_hba.conf |
| <require_auth_for_profile_requests: true> | <host synapsedb synapseusr ::1/128 md5> |
| <limit_profile_requests_to_users_who_share_rooms: true> | sudo systemctl reload postgresql |
| <include_profile_data_on_invite: false> | |
| <allow_public_rooms_over_federation: true> | |
| <allow_profile_lookup_over_federation: true> | |
| <allow_device_name_lookup_over_federation: true> | |
| <enable_registration: True> | |
| <enable_registration_captcha: True> | |
| <recaptcha_public_key: "enter pub key here"> | |
| <recaptcha_private_key: "enter priv key here"> | |
| <registration_shared_secret: "yourmomismykey"> | |
| | |
| In my case, matrix was not currently allowing un-challenged, or un-tokened user registration, so adding the [[https://www.google.com/recaptcha/about/|Google Challenge]] was required in order to make it functional (keep verify origin off). I also had to define the ''base_url'' and ''enable_registration'' explicitly. From researching forums and reddits online, I was able to ascertain that this is because Matrix changes their criteria/allowances depending on exploits, current threats, etc., actively, meaning that what might be allowed on day x is no longer supported on day y, etc. Now that synapse is more or less configured, it's time to install your web server of choice. I prefer apache2 since I have a long history of making reverse proxy configs for it, and even got some changes [[https://gitlab.com/oemb1905|committed]](see June 8, 2022) to the recipes project on gitlab (yay!). Note: Since ACME's ''cert-only'' command always fails for me, whenever I do apache reverse proxies, I first set up a slim website with stock virtualhost, A records on my DNS host, pull in certs on it, then disable it, as follows: | |
| |
| sudo apt install apache2 | Check that the configuration you established is functioning by restarting the service with ''systemctl restart matrix-synapse''. As long as everything is functioning, you can now create an admin user. Temporarily comment out user registration on ''/etc/matrix-synapse/homeserver.yaml'' and restart the service again ''systemctl restart matrix-synapse'' in order to create the user. Note: the user creation will fail if you do not do this. |
| sudo certbot --authenticator standalone --installer apache -d gnulinux.club --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2" | |
| a2dissite 000-default.conf | sudo -u postgres bash |
| | register_new_matrix_user -c /etc/matrix-synapse/homeserver.yaml http://localhost:8008 |
| | |
| Refer to [[https://wiki.haacksnetworking.org/doku.php?id=computing:apachesurvival|apache survival]] if you need help on setting up virtual hosts, etc. and/or apache/LAMP stacks. Once you build the certs and disabled the dummy website, then create two reverse proxy virtual hosts like these: | Record the password you chose in a safe location, and then uncomment the user registration on ''/etc/matrix-synapse/homeserver.yaml'' again and once again restart the service ''systemctl restart matrix-synapse''. This is required because you cannot manually create users while the ''enable_registration: true'' parameter is active. Likely, you will now want to also create a non-admin user for yourself, and the webgui is very convenient for this so just visit element.gnulinux.club and then register. Once you log in, make sure that you can access federated instances, spaces, and rooms. The last thing we need to do is add the ability for the server to send emails so users can recover accounts when needed via email. To do that, first make sure you have a working [[https://wiki.haacksnetworking.org/doku.php?id=computing:exim4|https://wiki.haacksnetworking.org/doku.php?id=computing:exim4]]email server. Please note that an incoming (IMAP) server is not needed, only outgoing (smtp/MTA). Once that is setup, add the following lines to your ''/etc/matrix-synapse/homeserver.yaml'' configuration: |
| |
| * [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/apache/virtualhosts/synapse.conf|virtualhost-80.conf]] | sudo nano /etc/matrix-synapse/homeserver.yaml |
| * [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/apache/virtualhosts/synapse-ssl.conf|virtualhost-443.conf]] | <public_baseurl: "https://matrix.gnulinux.club"> |
| | <email:> |
| | <smtp_host: haacksnetworking.org> |
| | <smpt_port: 25> |
| | <notif_from: "admin@gnulinux.club"> |
| | <require_transport_security: true> |
| |
| Once the reverse proxies are setup, restart the service with ''systemctl restart apache2.service''. It's now time to set up a turn server so clients behind NAT can function properly. To do that, do as follows: | Again, note that there is no need to set up an incoming email server nor tolerate spam on the server you are sending to. In my case, my smtp server / MTA is a relay/MTA (only) with domains that are allowed to send through it explicitly white listed. Now that email is up, let's tweak the upload settings so that people can upload images that are larger than a megabyte and have reasonable purge directives: |
| |
| sudo apt install coturn | sudo nano /etc/matrix-synapse/homeserver.yaml |
| mkdir -p /etc/coturn/certs | <max_upload_size: 1000M> |
| sudo chown -R turnserver:turnserver /etc/coturn/ | <dynamic_thumbnails: true> |
| chmod -R 700 /etc/coturn/ | <media_retention:> |
| pwgen -s 64 1 | <local_media_lifetime: 90d> |
| <it dumps a key> | <remote_media_lifetime: 14d> |
| sudo nano /etc/turnserver.conf | |
| <use-auth-secret> | Also, make sure to add the following line of code within each nginx http server block: |
| <static-auth-secret=YOUR-STATIC-AUTH-SECRET-HERE> | |
| <realm=yoursite.com:5349> | |
| <no-tcp-relay> | |
| <denied-peer-ip=10.0.0.0-10.255.255.255> | |
| <denied-peer-ip=192.168.0.0-192.168.255.255> | |
| <denied-peer-ip=172.16.0.0-172.31.255.255> | |
| <allowed-peer-ip=10.0.0.1> | |
| <user-quota=12> | |
| <total-quota=1200> | |
| <cert=/etc/coturn/certs/fullchain.pem> | |
| <pkey=/etc/coturn/certs/privkey.pem> | |
| sudo systemctl restart coturn | |
| | |
| Now that you have configured a turn server, make sure that synapse knows how to leverage it by adding a configuration block to ''/etc/matrix-synapse/homeserver.yaml'' as follows: | |
| | |
| turn_uris: [ "turns:yoursite.com:5349?transport=udp", "turns:yoursite.com:5349?transport=tcp" ]` | |
| turn_shared_secret: "YOUR-STATIC-AUTH-SECRET-HERE" | |
| turn_user_lifetime: 86400000 | |
| turn_allow_guests: false | |
| | |
| Now, let's make sure that the turn server can use TLS encryption by copying the certs from Let's Encrypt to it's working directory. (Note: I think these should be updated to symlinks.) | |
| | |
| cp /etc/letsencrypt/live/gnulinux.club/fullchain.pem /etc/coturn/certs/ | |
| cp /etc/letsencrypt/live/gnulinux.club/privkey.pem /etc/coturn/certs/ | |
| sudo chown turnserver:turnserver -R /etc/coturn/ | |
| sudo service coturn force-reload | |
| sudo service coturn restart | |
| | |
| So since I could not get token user creation to work, I ended up using Google Challenge instead. Nevertheless, I include this here because I did create this infrastructure, and will leverage it if/when token authentication begins to work again. Make sure to record all keys/passwords, etc., in a secure manager or FDE drive that's offline. Begin by creating two users for the instance, one that's the administrator for it, and another that's used as a proxy for registration requests with a token. | <client_max_body_size 1000M;> |
| |
| register_new_matrix_user -c /etc/matrix-synapse/homeserver.yaml http://localhost:8008 | Alright, so now it's time to carry on with setting up jitsi. Again, as with element and matrix, always check the [[https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-quickstart/|official documentation]] for the latest releases/gpg keys, etc.: |
| |
| After the database users are thus created and passwords specified and recorded, you now need to install matrix/synapse's registration tool with python-pip. | curl https://download.jitsi.org/jitsi-key.gpg.key | sudo sh -c 'gpg --dearmor > /usr/share/keyrings/jitsi-keyring.gpg' |
| | echo 'deb [signed-by=/usr/share/keyrings/jitsi-keyring.gpg] https://download.jitsi.org stable/' | sudo tee /etc/apt/sources.list.d/jitsi-stable.list > /dev/null |
| sudo apt install python3-pip | sudo apt update |
| pip3 install matrix-registration==1.0.0.dev7 | sudo apt install jitsi-meet |
| pip3 install psycopg2-binary | <enter domain jitsi.gnulinux.club> |
| | <generate self-signed> |
| Once that's done, you will need to create a database user and database for the accounts. You will need to alternate between the user and root to do this and might have to tweak permissions. | |
| | |
| sudo -u postgres bash | |
| createuser --pwprompt matrix_reg_user | |
| createdb --owner=matrix_reg_user matrix_reg | |
| | |
| Now that the database registration user and password have been created, configure the synapse/matrix instance to leverage those credentials by appending this additional entry to the ''pg_hba.conf'' file: | Once the install finishes, let's create let's encrypt using the official script from Jitsi, which will automatically configure a turn server and create a server block in nginx for us: |
| |
| nano /etc/postgresql/13/main/pg_hba.conf | /usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh |
| host matrix_reg matrix_reg_user ::1/128 md5 | |
| | |
| You now need to specify how the reverse proxy for registration can find the base url and the synapse/matrix services, etc. To do that, you create a ''config.yaml'' file as follows: | We now need to configure Element to use jitsi, which we do as follows: |
| | |
| sudo nano /home/registration/matrix-registration/config.yaml | |
| | |
| The contents of ''config.yaml'' should look like this: | |
| | |
| * [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/configs/config.yaml|config.yaml]] | |
| | |
| It's now best practice to create a systemd unit for the registration service itself since this is accordingly a public instance and might need the registration toolchain restarted etc. To do that, the official docs call for creating a systemd unit as follows: | |
| | |
| sudo useradd -m registration | |
| sudo nano /etc/systemd/system/matrix-registration.service | |
| | |
| The contents of the ''matrix-registration.service'' file should have locations that match the configurations you made for registration above. Something like the following is recommended: | |
| |
| * [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/configs/matrix-registration.service|matrix-registration.service]] | nano /var/www/element.gnulinux.club/element/config.json |
| | <"preferredDomain": "jitsi.gnulinux.club"> |
| | systemctl restart matrix-synapse |
| | |
| Once the systemd unit is created and the locations in the block linked to your production locations, then enable the unit, reload the units, and start the service. Note: As the first block states, you won't need to individually restart this service as it will restart when you restart the default service ''systemctl restart matrix-synapse.service''. | You can verify that jitsi is working by ''ps -aux | jitsi'' and then visit jitsi.gnulinux.club (adjusted for your domain) to verify you can create and join a new meeting. Additionally, you can navigate to ''etc/turnserver.conf'' to see the entries jitsi created and likewise verify the server block it created at ''/etc/nginx/sites-enabled/jitsi.gnulinux.club''. You are now done! However, make sure to run regular snapshots on a cron job, and consider some simple shell scripts to monitor nginx, postgres, and fail2ban and if they fail, to restart them and email you. I also have a separate script that dumps the entire database daily with a time stamp. Here are the scripts currently in use: |
| |
| systemctl daemon-reload | - Keep Fail2Ban Running:[[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/scripts/fail2ban-restart.sh|fail2ban-restart.sh]] |
| systemctl --user enable matrix-registration | - Keep Postgres Running: [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/scripts/postgres-restart.sh|postgres-restart.sh]] |
| systemctl --user start matrix-registration | - Keep Nginx Running: [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/scripts/nginx-restart.sh|nginx-restart.sh]] |
| | - Nightly DB Dumps: [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/scripts/postgres-dump.sh|postgres-dump.sh]] |
| Once the registration and account creation services are installed and functional, it is now time to create the web client, which will be used for clients who want to register or test out the instance, and also for users to directly create new matrix accounts on your instance. To do that, let's add a dedicated user for element, which is the standard/default web client, and install the element-web content management system: | |
| | |
| sudo useradd -m elementweb | |
| cd /home/elementweb/ | |
| sudo --user elementweb wget https://github.com/vector-im/element-web/releases/download/v1.10.12/element-v1.10.12.tar.gz | |
| sudo --user elementweb tar -xvf element-v1.10.12.tar.gz | |
| mv element-v1.10.12 /var/www/element | |
| cd /var/www/element | |
| nano config.json | |
| | |
| The ''config.json'' file should like something like what I have below. Note: this file is notorious for causing the "host cannot be found error" that is all over forums. Remember to change all the entries to your domain and make sure the ''base_url'' matches what you have in ''/etc/matrix-synapse/homserver.yaml''. Here's what mine looks like: | Also, there's no point in setting this up unless you have regular backups! In my case, since this is a VM, I just use the same script as I use for all my other instances. That script powers down the VM, and copies a sparse file, then tarballs it. After restarting the VM, my backup workstation pulls down the tarballs (also sparse) on a set schedule, keeping approximately 90 days of restore points. The backup script I use is found here and, of course, this runs on the host OS (not the Synapse VM instance): |
| |
| * [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/configs/config.json|config.json]] | - VM Backup Script: [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/scripts/sane-vm-backup.sh|sane-vm-backup.sh]] |
| |
| If you have not already done so, make sure to add an A record for element.domain.tld to your DNS host. It's now time to create an apache virtual host for the subdomain as follows: | Lastly, I also have a hot-spare in case my co-located hardware fails. Swing on by: |
| |
| sudo nano /etc/apache2/sites-enabled/element80.conf | - [[https://element.gnulinux.club|GNU/Linux Club]] |
| sudo nano /etc/apache2/sites-enabled/element443.conf | |
| | |
| Contents: | |
| | |
| * [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/apache/virtualhosts/element.matrix.conf|element80.conf]] | |
| * [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/apache/virtualhosts/element.matrix-ssl.conf|element443.conf]] | |
| | |
| Then create the certs, and enable the virtualhosts: | |
| | |
| sudo certbot --authenticator standalone --installer apache -d element.gnulinux.club --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2" | |
| a2ensite element80.conf | |
| a2ensite element443.conf | |
| systemctl restart apache2.service | |
| | |
| Once the apache web server is restarted, it's time to debug your instance. To do that, remember these two essential skills: | |
| | |
| tail -f /var/log/example.log | |
| sudo apache2ctl configtest | |
| | |
| Between these two debugging tools, you should be in good shape to get everything up and running! Okay, that took a lot to document but it's worth it. Also, if you are serving others in a public facing community, make sure you have [[https://repo.haacksnetworking.org/oemb1905/haackingclub/-/blob/master/scripts/sane-vm-backup.sh|appropriate backup scripts]] for this VM. Utilize rsync or similar to ensure you have those backups offsite as well as on site, and also provision a hot-spare in case your self-hosted or co-located hardware fails. | |
| |
| --- //[[jonathan@haacksnetworking.org|oemb1905]] 2022/11/20 01:58// | --- //[[jonathan@haacksnetworking.org|oemb1905]] 2022/12/11 14:48// |
