User Tools

Site Tools


computing:rsyncrsnapshot

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
computing:rsyncrsnapshot [2018/11/22 20:02] oemb1905computing:rsyncrsnapshot [2023/04/10 21:29] (current) oemb1905
Line 7: Line 7:
 ------------------------------------------- -------------------------------------------
  
-  sudo rsync -avxHAX --progress --delete /sourcefiles/path /backup/path+Since I am backing up multiple remote hosts, I chose to create separate configs for each remote host instead of putting those destinations in the ''/etc/rsnapshot.conf'' together with the localhost/backup server. I have the config set to retain 90 days worth of backups called "alpha" and it is set to sync first. The localhost configuration is as follows:
  
-Adding //h////H////P//, will preserve hard links, give you human readable output, and show the progress (eliminating the need for --progress)+  #base config  
- +  config_version          1.2 
-  sudo rsync -avxHAXhHP --delete /sourcefiles/path /backup/path+  snapshot_root           /mnt/backups/rsnapback/localhost.domain/ 
 +  cmd_cp                  /bin/cp 
 +  cmd_rm                  /bin/rm 
 +  cmd_rsync               /usr/bin/rsync 
 +  cmd_ssh                 /usr/bin/ssh 
 +  cmd_logger              /usr/bin/logger 
 +  verbose                 2 
 +  loglevel                3 
 +  logfile                 /var/log/rsnapshot.log 
 +  sync_first              1 
 +  use_lazy_deletes        1  
 +  retain                  alpha                                           90 
 +  #localhost 
 +  backup                  /home                                         home/ 
 +  backup                  /etc/                                           etc/ 
 +  backup                  /usr/local/bin/                                 bin/ 
 +  backup                  /root/                                          root/ 
 +  backup                  /var/www/                                       www/
      
-Cloning entire drives, use:+Thento run this, I use a simple script I created. Again, this script is just for running the localhost, so it can also be used for someone's single desktop:
  
-  sudo rsync -avxHAXhHP --delete /sourcefiles/path /backup/path --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found} +  #!/bin/bash 
-   +  service="/usr/bin/rsnapshot" 
-If the drive you are cloning from has a different filesystem than the destination drive, you can remove the exclusions as //-x// will handle those exclusions in that case.  After cloning, you will need to adjust fstab on the destination partition:+  logfile="/home/logs/tempa.log" 
 +          START1="$(date +%s)" 
 +          touch $logfile 
 +          $service -V sync > $logfile 
 +          END1="$(date +%s)" 
 +          DURATION1=$[ ${END1} ${START1} ] 
 +          MINUTES=$[ ${DURATION1} 60 ] 
 +          sed -i "1s/^/Jonathanat $(date), the rsnapshot sync took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete.\n/" $logfile 
 +  if 
 +          tail -n -5 $logfile | grep "completed" 
 +  then 
 +          START1="$(date +%s)" 
 +          echo "Jonathanat $(date), the sync has completed and I am now running alpha." 
 +          $service -V alpha >> $logfile 
 +          END1="$(date +%s)" 
 +          DURATION1=$[ ${END1} - ${START1} ] 
 +          MINUTES=$[ ${DURATION1} 60 ] 
 +          sed -i "1s/^/Jonathanat $(date), the rsnapshot alpha took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete.\n/" $logfile 
 +          mail -s "[$(hostname -f)]-rsnap-success-$(date)" alerts@alerts.com < $logfile 
 +          rm $logfile 
 +          chown user:user -R /mnt/backups/rsnapback/localhost.domain/ 
 +          chmod 770 -R /mnt/backups/rsnapback/localhost.domain
 +  else 
 +          echo "Jonathan, at $(date), the sync has failed and I am now notifying you." 
 +          mail -s "[$(hostname -f)]-rsnap-failure-$(date)" alerts@alerts.com < $logfile 
 +          rm $logfile 
 +  fi
  
-  cd /etc/ +Then, in ''/etc/cron.d/rsnapshot'', I set up a cronjob to run this script on a schedule. For my particular use case, twice a day is sufficientbut this can obviously be adjusted to the user's needs:
-  sudo nano fstab +
-  <Edit the drive UUID of the new destination drive> +
-   +
-Nowmake the destination volume bootable:+
  
-  grub-install /dev/sdX +  0 0,14 * * * root /bin/bash /usr/local/bin/rsnap-alpha-netback.jonathanhaack.com.sh > /dev/null 2>&1
-   +
-If you are using this method to clone your existing server to another hard drive on the same host, then make sure to run //grub-update// on the primary OS After you do that, both the kernels for the primary OS and the cloned OS will appear in GRUB, and you can boot into the clone, specifically, to make sure that everything migrated properly and is in working order It is not sufficient to simply run rsync and hope that all went well.+
  
- --- //[[netcmnd@jonathanhaack.com|oemb1905]] 2018/11/22 12:56//+Now that the localhost/backup server is backing up it's own essential boot volume files to the backup zpool, it's time to create the configs and scripts for the remote hostsIn my case, I have over 15 remote hosts set up this way, so I will highlight the setup for one in order to demonstrate what I've done. First, create a directory for the configs such as ''/etc/rsnapshot/'' and then create a config file within it such as ''/etc/rsnapshot/rsnapshot-domain.com.conf''. The config for the remote host looks very similar to the localhost config, but it has syntax for connecting to the remote host. Of course, make sure that you have properly exchanged ssh keys and installed rsync on the remote host before proceeding:
  
 +  #base config 
 +  config_version          1.2
 +  snapshot_root           /mnt/backups/rsnapback/domain.com/
 +  cmd_cp                  /bin/cp
 +  cmd_rm                  /bin/rm
 +  cmd_rsync               /usr/bin/rsync
 +  cmd_ssh                 /usr/bin/ssh
 +  cmd_logger              /usr/bin/logger
 +  verbose                 2
 +  loglevel                3
 +  logfile                 /var/log/rsnapshot.log
 +  sync_first              1
 +  use_lazy_deletes        1
 +  retain                  alpha                                  90
 +  #directories
 +  backup                  root@domain.com:/etc/                  etc/
 +  backup                  root@domain.com:/usr/local/bin/        bin/
 +  backup                  root@domain.com:/var/www/              www/
 +  backup                  root@domain.com:/home/                 home/
 +  backup                  root@domain.com:/root/                 root/
 +
 +Now that the config is setup, you use a script very similar to the localhost script above, but note the syntax changes for sync and alpha that will specify to rsnapshot to use this particular configuration file. Additionally, just like the localhost, I specified for rsnapshot to retain a snapshot called alpha for 90 days, which suits my use case. Here's the script for the remote hosts:
 +
 +  #!/bin/bash
 +  service="/usr/bin/rsnapshot"
 +  logfile="/home/sexa/logs/domain.com.log"
 +  host="domain.com"
 +          START1="$(date +%s)"
 +          touch $logfile
 +          $service -c /etc/rsnapshot/rsnapshot-$host.conf -V sync > $logfile
 +          END1="$(date +%s)"
 +          DURATION1=$[ ${END1} - ${START1} ]
 +          MINUTES=$[ ${DURATION1} / 60 ]
 +          sed -i "1s/^/Jonathan, at $(date), the rsnapshot sync took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete.\n/" $logfile
 +  if
 +          tail -n -5 $logfile | grep "completed"
 +  then
 +          START1="$(date +%s)"
 +          echo "Jonathan, at $(date), $(hostname -f) ran a sync that completed so I am now running alpha."
 +          $service -c /etc/rsnapshot/rsnapshot-$host.conf -V alpha >> $logfile
 +          END1="$(date +%s)"
 +          DURATION1=$[ ${END1} - ${START1} ]
 +          MINUTES=$[ ${DURATION1} / 60 ]
 +          sed -i "1s/^/Jonathan, at $(date), the rsnapshot alpha took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete.\n/" $logfile
 +          #echo "Jonathan, at $(date), the rsnapshot alpha took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete." | tee -a $logfile
 +          mail -s "[${host}]-rsnap-success-$(date)" alerts@alerts.com < $logfile
 +          rm $logfile
 +          chown user:user -R /mnt/backups/rsnapback/$host/
 +          chmod 770 -R /mnt/backups/rsnapback/$host/
 +  else
 +          echo "Jonathan, at $(date), $(hostname -f) ran a sync that failed and I am now notifying you."
 +          mail -s "[${host}]-rsnap-failure-$(date)" alerts@alerts.com < $logfile
 +          rm $logfile
 +  fi
 +
 +Similarly, I need to set up a cronjob to execute this script in ''/etc/cron.d/rsnapshot'' for which I once again chose twice a day:
 +
 +  0 00,14 * * * root /bin/bash /usr/local/bin/rsnap-alpha-comain.com.sh > /dev/null 2>&1
 +
 +You would think the retain 90 would delete the old ones, but it does not seem to work for whatever reason, and so I setup a small script to delete anything past 90 which runs daily at 8am, well after the last snapshot completed. It's also a loop which will make sure to prune all the snapshots between 90-100. If the script fails, it's possible that some snapshots could run past 90, hence including a ten day buffer. Here's the backup scrip, and trigger warning, it's a kludge:
 +
 +  #!/bin/bash
 +  date=`date +"%Y%m%d-%H:%M:%S"`
 +  log="/home/logs/rsnap-delete.log"
 +  img="domain.com domain.org domain.net"
 +  for i in $img;
 +  do
 +  touch $log
 +  START0="$(date +%s)"
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.90
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.91
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.92
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.93
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.94
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.95
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.96
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.97
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.98
 +  rm -rf /mnt/backups/rsnapback/$i/alpha.99
 +  END0="$(date +%s)"
 +  DURATION0=$[ ${END0} - ${START0} ]
 +  MINUTES0=$[ ${DURATION0} / 60 ]
 +  echo "Jonathan, at $(date), the rsnapshot purge took exactly ${DURATION0} seconds which is approximately 
 +  ${MINUTES0} minutes to complete." | tee -a $log
 +  mail -s "[rsnap-delete-$i]-$(hostname -f)-$(date)" alerts@alerts.com < $log
 +  rm $log
 +  done
 +
 +So far, everything is working great. In the future, I would like to debug why rsnapshot retains more than the 90 specified in alpha and/or polish up the delete script some more. I would also like to improve the rsnap-alpha script to check for a process id rather than checking the sync log for the conditional upon which it executes the script. I already have a colleague who has a good template for doing that and it's in progress. In the mean time, however, I've already used this for 120 days or so and had to use it to restore some versions I accidentally deleted. Very pleased for now. 
 +
 +
 +----
 +
 +I've decided to keep this older information below, as it includes a more standard setup for single host, including alpha, beta, gamma, and delta versions. 
 +
 +[Begin old tutorial] Before you begin, you must already know how to add a hard drive to your system, a mount point, and an appropriate fstab entry.  This also assumes you have an MTA properly set-up in order to send email.  To create a basic configuration, do the following:
 +
 +  sudo apt install rsnapshot
 +  nano /etc/rsnapshot.conf
 +  <rsnapshot_root /enter-mount-point-here>
 +
 +Uncomment the alpha, beta, gamma, and delta sections under //backup levels// section, enable the sync-first option, and specify the directories you want to backup (remember, ''localhost'', calls whatever you have above in ''rsnapshot_root'' from above):
 +
 +  <retain  alpha   6>
 +  <retain  beta    7>
 +  <retain  gamma   4>
 +  <retain  delta   3>
 +  <sync_first      1>
 +  <backup /home/   localhost>
 +  <backup /etc/    localhost>
 +
 +Now, edit the default script inside ''/etc/cron.d/rsnapshot'' and for, simple configurations, just enable the four entries they have inside that template.  The default cron entries look like this:
 +
 +  <0 */4   * * *          root    /usr/bin/rsnapshot alpha>
 +  <30 3    * * *          root    /usr/bin/rsnapshot beta>
 +  <0  3    * * 1          root    /usr/bin/rsnapshot gamma>
 +  <30 2    1 * *          root    /usr/bin/rsnapshot delta>
      
 +Personally, I use a simple script to ensure that the sync_first command completed before I run any of the ''rsnapshot alpha'', or ''rsnapshot beta'', and so on.  My script is simple and limited to what I could get up and working.  There are reliability problems with this script in full disclosure, but for my use case, it works at present.  Here it is:
 +
 +[[https://repo.haacksnetworking.com/oemb1905/haackingclub/blob/master/rsnapshot/rsnap-alpha.sh|rsnapshot-scripts]]
 +
 +Of course, I have one of these for alpha, another for beta, and gamma, and so on until delta.  Each of these checks the verbose log for successful completion and emails me upon verification of that from the output of the ''rsnapshot sync_first'' command.  If it fails, it stops the service and emails me.  A colleague of mine has a much more advanced script that checks whether the drive is mounted and other various niceties that should be part of a more robust backup solution.  I have included a version of the script he developed and shared with me here:
 +
 +[[https://repo.haacksnetworking.com/oemb1905/haackingclub/blob/master/backups/jsbk.sh|jbsk.sh]]
 +
 +OK ... that's pretty much it for getting started with rsnapshot.  Now, let's look at rsync over ssh, which is a great way to pull an immediate mirror of a site with ease.  I have included syntax for alternate port and newer syntax without alternate port.
 +
 +From remote server to target backup localhost, with alt port:
 +
 +  rsync -av --delete -e 'ssh -p 59333 -i /home/user/.ssh/id_rsa' user@10.8.9.3:/home/user/Server/ /home/user/Server/
 +
 +From localhost to target remote backup server, with alt port: 
 +
 +  rsync -av --delete -e 'ssh -p 59333 -i /home/user/.ssh/id_rsa' /home/user/Server/ user@10.8.9.3:/home/user/Server/
 +
 +From remote server to target backup localhost, with standard port:
 +
 +  sudo rsync -avi --delete root@server.com:/home/location/of/mysqldump.sql /home/user/backup/of/mysqldump.sql 
 +
 +From localhost to target remote backup server, with standard port: 
 +
 +  sudo rsync -avi --delete /home/user/backup/of/mysqldump.sql root@server.com:/home/location/of/mysqldump.sql
 +
 +Once you master these steps, checkout remote-upgrades, so you can use these backup scripts and tools with a greater remote vpn and backup solution.
 +
 +This tutorial is a designated "Invariant Section" of the "Technotronic" section of Haack's Wiki as described on the [[https://jonathanhaack.com/dokuwiki/doku.php?id=start|Start Page]].
 +
 + --- //[[jonathan@haacksnetworking.org|oemb1905]] 2023/04/10 15:16//  
computing/rsyncrsnapshot.1542916967.txt.gz · Last modified: 2018/11/25 01:33 (external edit)