This is an old revision of the document!
- managingbots
- Jonathan Haack
- Haack's Networking
- webmaster@haacksnetworking.org
managingbots
This tutorial is designed for Debian OS and LAMP stack users. In my case, I have a multi-site WordPress that includes my tech blog, poetry, and teaching blog. Additionally, I have a separate vhost on the same instance that is this very dokuwiki. The first thing I did was create a script that would scrap and tally all the bots and how much they have done during the last day:
The first report is over here:
After considering my physical host of 48 threads, 384GB of RAM, and the virtual appliance that runs on it for this script was built, which is 16 vCPU cores and 16GB of RAM, I decided that the first thing to do was to tweak apache to ensure it could handle the flood long enough to take action. So, by increasing the maximum workers to 800, with each worked being able to handle approximately 1.5K - 2.5K requests per second, this means that the server could handle 95K - 145K requests. This is well-above what even the most aggressive bot did to my server, and also well-above most reports I've see most impacted servers. Here are some bird's eye averages of some of the reports we've all read about:
- Game UI Database: Bursts up to 12K / min
- SourceHut: Bursts up to 5K / min
- Triplegangers: Bursts up to 1K / min
- freeBSD Wiki: Bursts up to 10K / min
800 workers can handle ~1,600–2,400 requests/second (800 × 2 to 800 × 3), or ~96,000–144,000/minute—well above 20K/minute. Hardware Fit: 16 cores can manage 800 threads (50/core) with mpm_event’s async model; 16GB RAM supports ~1.6–4GB usage (2–5MB/thread).
— oemb1905 2025/04/06 08:57
