| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| computing:btrfsreminders [2026/02/07 20:12] – removed oemb1905 | computing:btrfsreminders [2026/02/07 20:32] (current) – oemb1905 |
|---|
| | Introduction |
| | This tutorial is for Debian users that want to create a JBOD pool using BTRFS subvolumes and its RAID10 equivalent. These types of setups are common and helpful for virtualization environments and hosting multiple services, either for serious home hobbyist use and/or small business level production. These approaches are not designed for enterprise or large-scale production. |
| | |
| | Overview of setups |
| | Encrypting the home partition is essential because it ensures that the pool key is never directly exposed; its behind LUKS on the boot volume and the sysadmin keeps this credential stored in KeePassXC offsite. Thus, the physical layer is protected by LUKS with integrity. As for Pam's mounting utilities, I use [[https://jasonschaefer.com/encrypting-home-dir-decrypting-on-login-pam/|this method]] because it allows for easy remote reboot as there is no need to enter an FDE key in the post-BIOS FDE splash and/or require you to log in to IPMI each time. Instead, you encrypt home and then unlock that in a screen session after remote reboot with ''screen'' then ''su - user'' - after that, detach from the session with ctrl-d. In short, this method provides two advantages, namely, a secure LUKS-encrypted location for keys/credentials that's not exposed if a physical compromise takes place, and using built-in pam and simple UNIX login infra to avoid cumbersome BIOS/IPMI-level FDE unlocking after reboot. Let's install btrfs, LUKS, and identify your hard drives: |
| | |
| | sudo apt-get install cryptsetup libpam-mount btrfs* |
| | ls -lah /dev/disk/by-id/ |
| | |
| | This installs the required packages and provides you the blkid location/mountpoint for your hard drives. You can also use ''blkid'' but I find ''ls'' to be easier here tbh. After identifying your JBOD drives, do something like: |
| | |
| | dd if=/dev/random of=/home/user/vm.key bs=1 count=32 |
| | dd if=/dev/random of=/home/user/wh.key bs=1 count=32 |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35002538a98416870 --key-file /home/user/.unlock/vm.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35002538a98356f30 --key-file /home/user/.unlock/vm.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35002538a983571d0 --key-file /home/user/.unlock/vm.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35002538a98356590 --key-file /home/user/.unlock/vm.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35002538a0840a300 --key-file /home/user/.unlock/vm.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35002538a98356500 --key-file /home/user/.unlock/vm.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35002538a084065d0 --key-file /home/user/.unlock/vm.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35002538a98357220 --key-file /home/user/.unlock/vm.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35000c500d775df03 --key-file /home/user/.unlock/wh.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35000c500d7694517 --key-file /home/user/.unlock/wh.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35000c500d7771943 --key-file /home/user/.unlock/wh.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | cryptsetup luksFormat /dev/disk/by-id/scsi-35000c500cb1689e3 --key-file /home/user/.unlock/wh.key --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf argon2id --pbkdf-memory 4194304 --pbkdf-parallel 4 --iter-time 4000 --sector-size 4096 --use-random |
| | |
| | After you create the crypts on the hard drive and specify the key, you need to mount the volume and assign the unique blkid a dedicated/unique shortname for its mountpoint. You do this as follows: |
| | |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98416870 ssd1 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98356f30 ssd2 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a983571d0 ssd3 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98356590 ssd4 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a0840a300 ssd5 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98356500 ssd6 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a084065d0 ssd7 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98357220 ssd8 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35000c500d775df03 hdd1 --key-file /home/user/.unlock/wh.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35000c500d7694517 hdd2 --key-file /home/user/.unlock/wh.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35000c500d7771943 hdd3 --key-file /home/user/.unlock/wh.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35000c500cb1689e3 hdd4 --key-file /home/user/.unlock/wh.key |
| | |
| | Now that we have mounted the crypts at ''/dev/mapper/ssd#'', we can easily create a filesystem with them and or pool them together as we see fit. I've chosen to replicate RAID10 with BTRFS as closely as possible. It should be noted that it's not a perfect replication of RAID10 since it is chunk-based. In the commands below, we create the pools, verify the pools, and then mount them |
| | |
| | mkdir -p /mnt/vm |
| | mkdir -p /mnt/wh |
| | mkfs.btrfs -f -d raid10 -m raid1 --checksum=xxhash --nodesize=32k /dev/mapper/ssd1 /dev/mapper/ssd2 /dev/mapper/ssd3 /dev/mapper/ssd4 /dev/mapper/ssd5 /dev/mapper/ssd6 /dev/mapper/ssd7 /dev/mapper/ssd8 |
| | mkfs.btrfs -f -d raid10 -m raid1 --checksum=xxhash --nodesize=32k /dev/mapper/hdd1 /dev/mapper/hdd2 /dev/mapper/hdd3 /dev/mapper/hdd4 |
| | mount -o compress-force=zstd:3,noatime,autodefrag,space_cache=v2,discard=async,commit=120 /dev/mapper/ssd1 /mnt/vm |
| | mount -o compress=zstd:3,noatime,autodefrag,space_cache=v2,discard=async,commit=120 /dev/mapper/hdd1 /mnt/wh |
| | btrfs filesystem show /mnt/vm |
| | btrfs filesystem show /mnt/wh |
| | df -h #verify all looks right! |
| | |
| | After the first reboot I set persistent compression. I did this because I was getting errors trying to do it on initial pool build. Here's what I do for compression: |
| | |
| | btrfs property set /mnt/vm compression zstd:3 |
| | btrfs property set /mnt/wh compression zstd:3 |
| | |
| | Once that's done and you've rebooted a few times and tested things a few times, you can safely make a mount script for remote rebooting. This way, you reboot and then log in to your user and detach, run a simple script to unlock and mount the BTRFS subvolumes ... and you are done! Create ''nano /usr/local/bin/btrfs-mount-datasets.sh'' and some ''chmod 750 /usr/local/bin/btrfs-mount-datasets.sh'' and enter something like: |
| | |
| | #!/bin/bash |
| | #open SSD crypts |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98416870 ssd1 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98356f30 ssd2 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a983571d0 ssd3 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98356590 ssd4 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a0840a300 ssd5 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98356500 ssd6 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a084065d0 ssd7 --key-file /home/user/.unlock/vm.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35002538a98357220 ssd8 --key-file /home/user/.unlock/vm.key |
| | #open PLATTER crypts |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35000c500d775df03 hdd1 --key-file /home/user/.unlock/wh.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35000c500d7694517 hdd2 --key-file /home/user/.unlock/wh.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35000c500d7771943 hdd3 --key-file /home/user/.unlock/wh.key |
| | cryptsetup luksOpen /dev/disk/by-id/scsi-35000c500cb1689e3 hdd4 --key-file /home/user/.unlock/wh.key |
| | #mount the btrfs r10 pool for vm |
| | mount -o compress-force=zstd:3,noatime,autodefrag,space_cache=v2,discard=async,commit=120 /dev/mapper/ssd1 /mnt/vm |
| | #mount the btrfs r10 pool for wh |
| | mount -o compress=zstd:3,noatime,autodefrag,space_cache=v2,discard=async,commit=120 /dev/mapper/hdd1 /mnt/wh |
| | |
| | |
| | |
| |
