| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| computing:raid [2018/11/24 04:58] – oemb1905 | computing:raid [2021/04/03 02:40] (current) – oemb1905 |
|---|
| ------------------------------------------- | ------------------------------------------- |
| |
| This tutorial is based on me wanting to set up a server solution for self-hosters that can survive a remote reboot, but also encrypt the data on the server that matters most, and most importantly, be able to give me a chance to replace a hard drive if one were to fail. For those reasons, I chose RAID1 array as the wrapper, and used luks and pam_mount and once that array and debian OS were installed on my host with that arrangement - for short, I call this the 'RAID' tutorial, since that was the primary non-negotiable and all of this pivots around being able to replace failing hard drives first, and then building other 'backup' solutions on top of that later (see [[https://jonathanhaack.com/dokuwiki/doku.php?id=computing:rsyncrsnapshot|rsyncrsnapshot tutorial]]. For now, it is important to set up the RAID, luks, and pam_mount correctly, and here is what I got to work flawlessly. I used a debian stretch .iso that I ripped to a flash drive (full amd64DVD1.iso). | This tutorial is based on me wanting to set up a server solution for self-hosters that can survive a remote reboot, but also encrypt the data on the server that matters most, and most importantly, be able to give me a chance to replace a hard drive if one were to fail. For those reasons, I chose RAID1 array as the wrapper, and used luks and pam_mount and once that array and debian OS were installed on my host with that arrangement - for short, I call this the 'RAID' tutorial, since that was the primary non-negotiable and all of this pivots around being able to replace failing hard drives first, and then building other 'backup' solutions on top of that later (see [[https://wiki.haacksnetworking.com/doku.php?id=computing:rsyncrsnapshot|rsyncrsnapshot tutorial]]. For now, it is important to set up the RAID, luks, and pam_mount correctly, and here is what I got to work flawlessly. I used a debian stretch .iso that I ripped to a flash drive (full amd64DVD1.iso). |
| |
| (Optional: I recommend not using the netinst.iso but the full DVD.iso instead. I also do not use a network mirror, and do not install any other packages than the defaults, as I noticed this speeds up the install a lot. Once the system is bootable and up, you can install the packages and rebuild the the OS as needed. Debian keeps the repositories at [[https://wiki.debian.org/SourcesList|Debian Repositories]] web page and make sure that /etc/apt/sources.list is updated to those examples. //Again, do this after the final RAID syncing below.// Also, when you save time like this, you need to install sudo (yes, really). Why? The RAID1 array is suycing and slowing everything down a lot. Just wait until it finishes, reboot, and then update. And take as little time as possible during the install. That's the plan I use ... ok, moving on!) | (Optional: I recommend not using the netinst.iso but the full DVD.iso instead. I also do not use a network mirror, and do not install any other packages than the defaults, as I noticed this speeds up the install a lot. Once the system is bootable and up, you can install the packages and rebuild the the OS as needed. Debian keeps the repositories at [[https://wiki.debian.org/SourcesList|Debian Repositories]] web page and make sure that /etc/apt/sources.list is updated to those examples. //Again, do this after the final RAID syncing below.// Also, when you save time like this, you need to install sudo (yes, really). Why? The RAID1 array is suycing and slowing everything down a lot. Just wait until it finishes, reboot, and then update. And take as little time as possible during the install. That's the plan I use ... ok, moving on!) |
| md2 - 3.9TB - format it as xfs, use as "/home" | md2 - 3.9TB - format it as xfs, use as "/home" |
| |
| Once that is done (still in the ncurses/text-based installer), you can continue with the installation and the options as you desire. However, as I mentioned earlier, I do not add any additional software or utitlize the mirror during install in order to speed up the install process. Later, I will add all the packages I need and adjust sources.list, etc., but I do this ... __after__ the syncing completes. However ... since I intend to encrypt the md2 partition within the RAID1 array, when I say that in this case, I mean one should first encrypt the md2 partition, and then after that reboot, and once all is working, then you wait and let everything sync before doing any package work. Remember, if you already have your system up and need to check its RAID snync status, issue this command: | Once that is done (still in the ncurses/text-based installer), you can continue with the installation and the options as you desire. (Do not forget to install GRUB to both hard drives and to both of the reserved BIOS partitions that were made earlier.) Again, I do not add any additional software or utitlize the mirror during install in order to speed up the install process. Later, I will add all the packages I need and adjust sources.list, etc., but I do this ... __after__ the syncing completes. This tutorial will have you encrypt the md2 partition within the RAID1 array __after__ the OS installation completes - not using the installer on that piece. Once the OS installs and at any time before or after the encryption, you can use the command below to monitor the 'syncing' of the RAID1 array: |
| |
| cat /proc/mdstat | cat /proc/mdstat |
| |
| But again ... ignore the status until you complete the step below. To get your partition encrypted with luks, you must still do the following to ensure that swap and /home are encrypted. Open a tty1 (ctrl-f1 or -fx) when the system boots in and log in as root at the tty1 shell. Once logged in, perform the following on md2 (the "/home" partition on the RAID mirror). I adapted the encryption instructions that follow from my friend Jason Schaefer's blog: | But again ... __wait__ until you complete the steps below before you update or heavily customize this system. It needs to sync and it will be slow - and you still have not encrypted your /home partition. To get your /home partition encrypted with luks after the install, you need to enter tty1 (ctrl-f1 or f?) when the system boots in and log in as root at the tty1 shell. Once logged in, perform the following on md2 (the "/home" partition on the RAID mirror). I adapted the encryption instructions that [[https://jasonschaefer.com/encrypting-home-dir-decrypting-on-login-pam/|Jason Schaefer]] covered in his blog in order for encrypted servers to survive remote reboots. Here is my simplified version of what he has written there: |
| |
| apt-get install cryptsetup libpam-mount rsync | apt-get install cryptsetup libpam-mount rsync |
| su - root | su - root |
| rsync -av /home /backup | rsync -av /home /backup |
| cryptsetup luksFormat /dev/md2 | cryptsetup luksFormat /dev/md2 |
| cryptsetup luksOpen /dev/md2 home | cryptsetup luksOpen /dev/md2 home |
| mkfs.xfs -L home /dev/mapper/home | mkfs.xfs -L home /dev/mapper/home (xfs is optional, use what you want) |
| mount /dev/mapper/home /home/ | mount /dev/mapper/home /home/ |
| rsync -av /backup/home/ /home | rsync -av /backup/home/ /home |
| mdadm --readwrite /dev/mdx [restart] | mdadm --readwrite /dev/mdx [restart] |
| | |
| If a drive fails, you need to deactivate it, activate the replacement and then allow enough time for syncing. That tutorial is next and will go below. It centers around mdadm commands to deactivate the old drive, and with sfdisk to create an identical partition table on the new device. More later on that ... | Okay, the point of this is to combine RAID with pam_mount handlind the mounting of the crypt, and we also need ssh pubkey authentication. But, without some tweaks, we would never be able to access the public key, which is in the crypt on the server. So, we need to move a copy of the public keys outside the crypt. Here is how: |
| | |
| | cp -a ~/.ssh/authorized_keys /opt/authorizedkeys |
| | sudo nano /etc/ssh/sshd_config |
| | |
| | Add the following parameter in the section that pertains to it: |
| | |
| | AuthorizedKeysFile /opt/authorized_keys |
| | |
| | Now, when you reboot, even if PasswordAuthentication and ChallengeResponse are set to "no," as long as PubKeyAuthentication is set to "yes" you will be able to do the following to mount the crypt. This is because pam_mount is handling log in, and PAM is therefore allowing the password to be entered through the ssh tunnel because sshd_config also has Use PAM yes. |
| | |
| | ssh user@xx.xx.xx.xx |
| | screen |
| | su - user |
| | <authenticate> |
| | |
| | Now, press ctrl-a-d to detach from the screen. |
| | |
| | exit |
| | |
| | Okay, now you have survived reboot with a RAID array, with pam doing its magic for you on the crypt ... hardly a pain if it saves you |
| | a trip |
| |
| ------------------------------------------- | ------------------------------------------- |
| |
| The following are miscellaneous commands and sites that I am using while I add to this tutorial. T | When a drive fails, issue the commands below to remove the drive and clone the partitioning system with sfdisk to the new hard drive: |
| |
| The sfdisk command is not complete ... the proc shows kernel report on mirroring / syncing status ... | (draft) sfdisk -d | sfdisk /dev/sdg /dev/sdz |
| | (draft) cat /proc/mdstat |
| | (draft) <rebuild commands> |
| | (draft) mdadm --readwrite /dev/mdx |
| |
| sfdisk -d | sfdisk | When a hard drive fails, follow these steps with the machine still on (with failed drive as per SMART inside): |
| | |
| | mdadm --manage /dev/md0 --fail /dev/sdi1 |
| | mdadm --manage /dev/md1 --fail /dev/sdi2 |
| | mdadm --manage /dev/md2 --fail /dev/sdi3 |
| | mdadm --manage /dev/md0 --remove /dev/sdi1 |
| | mdadm --manage /dev/md1 --remove /dev/sdi2 |
| | mdadm --manage /dev/md2 --remove /dev/sdi3 |
| | sudo poweroff |
| | sfdisk -d /dev/sdj | sfdisk /dev/sdi |
| | mdadm --manage /dev/md0 --add /dev/sdi1 |
| | mdadm --manage /dev/md1 --add /dev/sdi2 |
| | mdadm --manage /dev/md2 --add /dev/sdi3 |
| | dpkg-reconfigure -plow grub-pc |
| cat /proc/mdstat | cat /proc/mdstat |
| |
| Next, I will write a tutorial on this same wikipage about how to build a RAID1 array on a running host. But for now, that's all ... here are some good resources I found while gathering my thoughts and preparing my solutions and steps above. Thanks to these guys who had great ideas: | Let syncing finish, then reboot and run: |
| | |
| | sudo grub-install /dev/sdX |
| | sudo update-grub |
| | sudo update-grub2 |
| | |
| | Next up, how to create a RAID1 array on a running host |
| | |
| | -- -- -- -- -- |
| |
| [[https://blog.sleeplessbeastie.eu/2013/10/04/how-to-configure-software-raid1-during-installation-process/]] | This tutorial is a designated "Invariant Section" of the "Technotronic" section of Haack's Wiki as described on the [[https://jonathanhaack.com/dokuwiki/doku.php?id=start|Start Page]]. |
| [[https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array]] | |
| [[https://linuxgazette.net/140/pfeiffer.html]] | |
| [[http://jasonwryan.com/blog/2012/02/11/lvm/]] | |
| |
| --- //[[netcmnd@jonathanhaack.com|oemb1905]] 2018/11/23 21:20// | --- //[[netcmnd@jonathanhaack.com|oemb1905]] 2019/01/13 12:25// |
| |
