How to Protect a Website from Bots and Reduce Server Load

Automatic bots can send a large number of requests to a website, loading pages, images, APIs, and other resources. When the request rate is high, the load on the CPU, RAM, PHP-FPM, database, network interface increases and the disk subsystem.
However, not every bot is malicious. Search engine crawlers and monitoring services may operate legitimately, but even they can generate significant load if configured incorrectly.
In this article, we will look at how to identify the source of the load, detect suspicious traffic, limit the number of requests, and protect a website from unwanted bots without blocking regular users.

What Types of Bots Are There?

Automated requests to a website can be roughly divided into several categories:

• search engine crawlers — Googlebot, Bingbot, and other search engines;
• useful bots — monitoring services, availability checks, and other automated systems;
• scrapers — collect website content;
• scanners — search for vulnerabilities and exposed services;
• malicious bots — perform password attacks, send large numbers of requests, and carry out other unwanted actions;
• aggressive bots — send a large number of requests over a short period of time.

Not every bot poses a threat. The main problem is an excessively high request rate, especially when bots access resource-intensive pages.

How to Identify the Source of the Load

Before blocking IP addresses or User-Agents, you need to determine which traffic is actually causing the load.
The first step is to check the web server and its access.log.

Viewing access.log in Real Time

For Nginx:
tail -f /var/log/nginx/access.logIf the log is stored at a different path, specify the appropriate file.

Finding IP Addresses with the Most Requests

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | headThe command shows the IP addresses that occur most frequently in the log.

Finding the Most Frequently Requested URLs

awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | headThis helps identify the pages and endpoints that receive the most requests.

Checking a Specific IP Address

After identifying a suspicious address, you can view its requests:
grep "192.0.2.10" /var/log/nginx/access.log | tail -100When analyzing the traffic, pay attention to:

• number of requests;
• request frequency;
• IP address;
• User-Agent;
• requested URLs;
• HTTP response codes;
• number of concurrent connections;
• requests to resource-intensive pages.

Important: a large number of requests from a single IP address does not always indicate an attack. A single IP may be shared by multiple users, for example when using NAT. It may also belong to a search engine crawler or a legitimate API client.

How to Determine Whether Requests Are Generated by a Bot

One sign of automated traffic is a large number of similar requests over a short period of time.
For example, suspicious behavior may include a single source that:

• sends hundreds of requests per minute;
• sequentially browses website pages;
• requests non-existent URLs;
• repeatedly requests the same resource-intensive URL;
• attempts to access /wp-login.php or /xmlrpc.php;
• sends requests to a large number of different URLs;
• creates a large number of concurrent connections.

However, a single indicator is not enough. User-Agent, IP address, and URL should be considered together.

How to Filter Unwanted Traffic

Protection should be configured gradually. First identify the source of the load, then apply a soft restriction, and only after analyzing the results move on to blocking.

1. Blocking an IP Address

If a specific malicious IP address has been identified, it can be blocked at the Nginx level:
deny 192.0.2.10;This is a simple way to stop traffic from a specific address.
However, permanent IP blocking is not a universal solution. Bots may use multiple addresses, proxies, or frequently change their IP addresses.
Therefore, blocking is best used for clearly unwanted sources, while rate limiting should be used for large volumes of traffic.

2. Limiting the Request Rate

Rate limiting allows you to limit the number of requests from a single client.
For example:
limit_req_zone $binary_remote_addr zone=one:10m rate=5r/s;The limit can then be applied to the required section:
location / {
limit_req zone=one burst=20 nodelay;
}In this example, a client can send requests at a defined rate, while a small temporary burst is allowed by the burst parameter.
When the limit is exceeded, Nginx can delay or reject requests. This helps reduce the load without completely blocking the user.
The rate and burst values should be selected based on the characteristics of the website. Limits that are too strict may affect regular users.

3. Limiting Concurrent Connections

If a client opens a large number of parallel connections, you can limit their number:
limit_conn_zone $binary_remote_addr zone=addr:10m;Then apply the limit:
location / {
limit_conn addr 10;
}This is particularly useful for protecting against clients that create a large number of simultaneous connections.

Filtering by User-Agent

Nginx can filter requests based on the User-Agent.
For example, if a specific bot is known to be unwanted:
if ($http_user_agent ~* "BadBot") {
return 403;
}However, User-Agent should not be used as the only blocking criterion. It is easy to change, so an attacker can make a request appear to come from a regular browser or search engine crawler.
User-Agent filtering is best used as an additional layer of protection.

Does robots.txt Help?

The robots.txt file is designed to control the behavior of search engine crawlers.
For example:
User-agent: *
Crawl-delay: 5Such rules can ask a supported crawler to wait between requests.
However, robots.txt is not a security mechanism:

• malicious bots can completely ignore the file;
• not all bots support Crawl-delay;
• the file does not block network connections;
• the request can still reach the server.

Therefore, robots.txt should be treated as a recommendation for well-behaved crawlers rather than a means of protection against attacks.

How to Reduce Server Load

Bot filtering is only one part of optimization. Even with a normal number of requests, a website can generate high load due to PHP, the database, or a large amount of dynamic content.
To reduce the load, you can:

• enable caching;
• use FastCGI Cache;
• use Redis or Memcached;
• serve static content through Nginx;
• use a CDN;
• optimize PHP-FPM;
• optimize MySQL/MariaDB queries;
• restrict access to resource-intensive URLs;
• protect APIs against excessive requests;
• disable unnecessary endpoints;
• use rate limiting.

It is especially important to cache pages that are frequently requested by bots. If every request triggers PHP execution and a database query, even a relatively small stream of requests can create significant load.

How to Protect WordPress from Bots

WordPress is often targeted by automated requests. Particular attention should be paid to the following URLs:
/wp-login.php
/xmlrpc.php
/wp-admin/
/wp-json/
If the logs show a large number of requests to these URLs, you need to determine whether they are legitimate.

Limiting Requests to wp-login.php

For example:
location = /wp-login.php {
limit_req zone=one burst=5 nodelay;
}This limits the number of requests to the login page.

Disabling XML-RPC

If XML-RPC is not used, access to it can be blocked:
location = /xmlrpc.php {
deny all;
}Before disabling it, make sure that XML-RPC is not used by the website or any connected services.

When to Use a CDN or WAF

If a large amount of unwanted traffic reaches the VPS directly, blocking it with Nginx may not be sufficient.
For example:
Bot → VPS → Nginx → PHP → MySQL
In this case, the server accepts connections and processes requests before they are filtered.
With a CDN or WAF, the architecture looks different:
Bot → CDN/WAF → VPS → Nginx → PHP → MySQL
A CDN or WAF can filter some unwanted traffic before it reaches the server.
This is especially useful when dealing with a large number of requests, mass scanning, and other situations where the server is already under heavy load.

How to Avoid Blocking Regular Users

When configuring traffic filtering, it is important not to rely solely on IP addresses.
A single public IP address may be shared by multiple users due to:

• NAT;
• corporate networks;
• mobile operators;
• VPNs;
• proxies.

Therefore, an overly strict limit, such as 1 request per second per IP, may affect regular users.
You should also avoid blocking all clients with a specific User-Agent. The same User-Agent can be used by both a bot and a legitimate application.
It is better to start with soft restrictions:

1. identify the source of the load;
2. analyze its requests;
3. configure a rate limit;
4. monitor the results;
5. tighten the restriction if necessary;
6. block only clearly unwanted traffic.

This approach reduces the risk of blocking regular users.

Monitoring After Configuration

After changing the configuration, check whether the load has actually decreased.
You should monitor:

• CPU usage;
• RAM usage;
• Load Average;
• number of connections;
• requests per second;
• PHP-FPM;
• MySQL/MariaDB;
• access.log;
• number of 403 responses;
• number of 429 responses;
• number of 5xx errors.

Pay particular attention to the HTTP 429 Too Many Requests response code. An increase in 429 responses may indicate that the configured limit is working.
At the same time, make sure that a large number of 429 responses is not caused by regular users being blocked.

Step-by-Step Protection Algorithm

If a website starts experiencing increased load due to a large number of requests, you can proceed as follows:
1. Check the server load.
Determine which resource is overloaded: CPU, RAM, PHP-FPM, database, or network.
2. Check access.log.
Find the IP addresses and URLs receiving the largest number of requests.
3. Analyze the source.
Check the User-Agent, request frequency, URLs, and request patterns.
4. Start with rate limiting.
Limit the request rate without completely blocking the client.
5. Check the result.
Compare the server load before and after changing the configuration.
6. Block clearly malicious sources.
For specific IP addresses or known malicious User-Agents, blocking can be used.
7. Optimize the website.
Add caching, optimize PHP and the database, and restrict resource-intensive endpoints.
8. Use a CDN/WAF for large volumes of traffic.
If the volume of unwanted traffic is too high, it is better to filter it before it reaches the server.

Conclusion

Protection against bots starts not with blocking IP addresses, but with traffic analysis. First, you need to determine what is generating the load, which URLs are being requested, and how frequently.
After that, you can gradually apply rate limiting, concurrent connection limits, IP and User-Agent filtering, caching, and other optimization methods.
If the volume of unwanted traffic becomes too large, consider using a CDN or WAF to filter requests before they reach the server.
This approach helps reduce server load while minimizing the risk of blocking regular users.