User Tools

Site Tools


computing:managingbots

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
computing:managingbots [2025/04/06 19:31] oemb1905computing:managingbots [2025/04/06 19:32] (current) oemb1905
Line 11: Line 11:
 ------------------------------------------- -------------------------------------------
  
-This tutorial is designed for Debian OS and LAMP stack users that want to track and or prohibit bot scraping (or other url requests) that might harm server performance and/or cause it to fail. In my case, I have a multi-site WordPress that includes my tech blog, poetry, and teaching blog. Additionally, I have a separate vhost on the same instance for my dokuwiki and another for file-share only Nextcloud. The same instance also runs my business’s email server with postgres and dovecot. Needless to say, I don’t want this virtual appliance to have down time. The first thing I did was create a script that would scrap and tally all the bots and how much they have done during the last day:+This tutorial is designed for Debian OS and LAMP stack users that want to track and or prohibit bot scraping (or other url requests) that might harm server performance and/or cause it to fail. In my case, I have a multi-site WordPress that includes my tech blog, poetry, and teaching blog. Additionally, I have a separate vhost on the same instance for my dokuwiki and another for file-share only Nextcloud. The same instance also runs my business’s email server with postgres and dovecot. Needless to say, I don’t want this virtual appliance to have down time. The first thing I did was create a script that would scrape and tally all the bots and how much they have done during the last day:
  
   * [[https://repo.haacksnetworking.org/haacknet/haackingclub/-/blob/main/scripts/apache/bot-scrape-daily.sh?ref_type=heads|Daily Bot Scrape]]   * [[https://repo.haacksnetworking.org/haacknet/haackingclub/-/blob/main/scripts/apache/bot-scrape-daily.sh?ref_type=heads|Daily Bot Scrape]]
computing/managingbots.txt · Last modified: 2025/04/06 19:32 by oemb1905