Mitigating DoS and DDoS Attacks

LiteSpeed Web Server provides several features aimed at reducing and even eliminating the impact of HTTP-level Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks. You can either use LSWS built-in features or third party ModSecurity rules such as Owasp, Atomicorp, Comodo and CloudLinux Imunify360. The following LSWS built-in configuration settings will help mitigate such attacks.

One of the most effective methods to mitigate DoS and DDoS attacks is to enable the reCAPTCHA feature. reCAPTCHA is a free service from Google that helps protect websites from spam and abuse. A “CAPTCHA” is a turing test to tell human and bots apart. It is easy for humans to solve, but hard for bots and other malicious software to figure out. The reCAPTCHA feature is supported as of LSWS 5.4RC1 and later. Please see these instructions to enable reCAPTCHA.

Enabling LiteSpeed Cache will increase the server's capacity to handle heavy traffic.

LiteSpeed Cache is designed to improve site load speed and handle increasingly high traffic. LiteSpeed Cache cannot block any DDos attack, however, it can help the server to handle hundreds or thousands of times more requests/second. This definitely helps lessen the impact of DDoS attacks. See these instructions to learn how to enable LSCache. For a cPanel server, you can use LiteSpeed Cache manager to mass enable LSCache in one click.

LiteSpeed Web Server includes a built-in Per-Client Throttling feature which allows you to block bad IPs.

Navigate to Configuration > Server > Security configurations > Per Client Throttling to find several configuration settings that you can use to limit the request, bandwidth, and connection rate per remote IP address.

Separate controls are available for throttling requests for static files and dynamic content.

The server allows setting separate bandwidth limits for inbound and outbound traffic.

• Bandwidth numbers will be rounded up in 4KB increments.
• Set to 0 to disable throttling.
• The Outbound Bandwidth limit allows serving more unique clients and prevents limited network bandwidth from getting used up by a small number of clients with fast network connections.

These settings control concurrent connections coming from one client (IP address) and guard against DoS attacks.

• Connection Hard Limit controls how many concurrent connections are allowed from one IP address. If an IP reaches the hard connection limit, the web server will immediately close newly accepted connections from that IP address, and move on to pending connections from different IP addresses. As almost all web browsers support keep-alive/persistent connections (multiple requests pipelined through one connection), the number of connections required in normal browsing is very small. Typically, one connection is enough, but some web browsers try to establish additional connections to speed up downloading. Allowing 4 to 10 connections from one IP is recommended. Less than that will probably affect normal web services.
• Use Connection Soft Limit, Grace Period, and Banned Period to spot and mitigate abusers: An IP address that stays over the soft limit for the length of the grace period will be banned for the length of time set in Banned Period. This is a good way to identify IPs that should be added to the Denied List.

Note: The number of connections can temporarily exceed the soft limit during the grace period, as long as it is under the hard limit. After the grace period, if it is still above the soft limit, then no more connections will be allowed from that IP for duration of the banned period.

Default Settings:

Updated Settings:

Static Requests/second = 40 Dynamic Requests/second = 2 Outbound Bandwidth (bytes/sec) = 0 Inbound Bandwidth (bytes/sec) = 0 Connection Soft Limit = 15 Connection Hard Limit = 20 Block Bad Request = Yes Grace Period (sec) = 15 Banned Period (sec) = 60

Explanation: An IP that has established more than 20 connections with the web server, or has established over 15 connections of over 15 seconds (the grace period), is treated as a DoS-attacker. The server will ban the IP for 60 seconds and record a log entry in the error log file. To exclude any IP from the client throttle limits (and bypass DDoS detection), add the IP with a trailing 'T' (aka trusted) in Allowed List (WebAdmin Console > Server > Security > Access Control).

The hard limit can be adjusted based on an attacker's strategy. If the botnet is not very aggressive, you will need to lower the limit to just below their max connection per IP, to make sure it won't affect a regular user. If they only make very few connections per IP, do not use the hard limit to detect them.

The blocked IPs can be found in the real-time-stats report.

LiteSpeed Web Server version 5.0 introduces virtual host-level bandwidth throttling. This can be thought of as an extension of LSWS's Per Client Throttling settings explained as above, which allow you to control the amount of stress a single IP can put on your server. Virtual host-level bandwidth throttling allows you to customize bandwidth throttling, in Apache configs, for particular virtual hosts through MaxConnPerClient <limit_for_connections>,LargeFileLimit [Type] [Minimum Size in kilobytes] [Speed in bytes/s], BandWidth [Origin] [Speed in bytes/s] and MinBandWidth all -1. . Please check here for details.

You can take advantage of the Max Request/Response Settings to reduce memory usage.

Under Configuration > Server > Tuning:

• Try to set Max Request URL Length, Max Request Header Size, Max Request Body Size, Max Dynamic Response Header Size, and Max Dynamic Response Body Size to values that go just above what you need to run your site. Getting these settings trimmed down will help identify attackers and reduce the amount of memory used when you do get attacked.

• Set Connection Timeout to around 30 seconds and Keep-Alive Timeout to around 15 seconds or less. This will help close dead connections as soon as possible and make connections available to other clients.

Increasing the Max Connection Settings will increase capacity and allow you to mitigate attack without limiting yourself.

Default: Change to: You should adjust Max Connections and Max SSL Connections to 20K/10K or even higher as long as the server has enough free memory. The purpose of the change is to increase the capacity, not to limit yourself under DoS attack.

The number of connection on port 80 doesn't matter. As long as the service is up, you've won!

If you know an attacker's IP, you can block it. Under Configuration > Server > Security:

• Block IPs that abuse your web server by listing them in the Denied List in the Access Control table.

If your server is flooded by hundreds of requests from different IPs but to the same URL, you can set up rules to block access to that URL.

For example, in a control panel environment, to block all access to /foo/, in the /foo/.htaccess of the targeted domain virtual host, place the following:

RewriteEngine On
RewriteRule .* - [L,F]

In LSWS native mode, you can either use rewrite rules as indicated above, or native context configuration like so:

1. Create a context (Configuration > Virtual Hosts > View/Edit > Context > Add > Type = Static) to block access to that URL.
2. Set Accessible to No and the context URI to match or include the URL being attacked.

If the server is pounded with requests for /foo/bar.html, then adding a context with Accessible set to No and the URI set to /foo/bar.html will block all of those requests. You can also set the context URI to /foo/ to block requests to all URLs that start with /foo/.

LSWS is campatible with the most common ModSecurity Rules, such as Owasp, Atomicorp, Comodo and CloudLinux Imunify360 etc. You can enable one of them on LSWS.

If you need assistance configuring your site to mitigate attacks, check out LiteSpeed's Advanced Anti-DDoS Setup Service. LiteSpeed Denial of Service Packet Filter Setup Service will fine-tune your anti-DDoS configuration and set up iptables to automatically block attacking IPs detected by the web server.

This is sufficient for many common attack scenarios. In cases of extreme attacks, this service will not be sufficient, and only custom / hourly support may be appropriate.

In order to determine whether the service fits your needs, we will need to identify the type of attack your site is experiencing - for example, whether it is targeting layer 4 (IP/port) or 7 (HTTP/URL), what the scale of the attack is, how many bots are in the attacking botnet, and whether you have layer 3 protection at the firewall level for synflood attacks.

For example, LiteSpeed Advanced Anti-DDoS Setup will efficiently protect against Layer 7 HTTP and Layer 4 TCP bot attacksl, but not against Layer 3 SYN Flood attack. SYN Flood send SYN packets with spoofed source IP addresses and require Layer 3 protection at the firewall level. If a TCP connection established, it is a Layer 4 attack, but if a TCP connection is not established, it is Layer 3.

Layer 4 TCP connection floods can be detected and blocked by LiteSpeed Advanced Anti-DDos Setup Layer 4 connection hard limit settings.

For large scale attacks, server kernel level settings may need to be adjusted to handle the large amount of HTTP requests during the attack.

Generally speaking, LiteSpeed can handle a up to 1000 bots without problem. If bot number are well over 1000, while the LiteSpeed Web Server can handle the concurrent connections, typically server memory or PHP execution become bottlenecks. LiteSpeed Web Server can be configured to cache the attacked page, reducing the server/PHP resources and increasing the server overall capacity, but this is an example that is well beyond the scope of this service.

When you have a front-end proxy/CDN, the Denial of Service Packet Filter Setup may not work, since it blocks attacking robots at the IP level with iptables. When there is a front proxy, it only sees the IP of the proxy, and it cannot block the proxy IP, as all traffic is coming from that IP. If you have CloudFlare Pro or a similar service already, you may not need Denial of Service Packet Filter Setup Service since they do a similar job.

Should you need such advance support services, they may be requested through evaluation and quotation.

To order LiteSpeed Advanced Anti-DDos Setup Service, please visit our store.

To restore real visitor IPs, navigate to LiteSpeed WebAdmin Console > Configuration > General Settings and set Use Client IP in Header to Trusted IP Only, and add your CDN such as CloudFlare IPs/subnets to the trusted list. Never set Use Client IP in Header to Yes, since clients can spoof IPs with the X-Forwarded-For header that is sent to CloudFlare.

To check the number of concurrent TCP connections, run the following command:

netstat -an | grep 80 | grep ESTA | wc 

To check concurrent connections sorted by IP, run the following:

netstat -ntu | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

Please keep in mind that netstat -ntu will list TCP in TIME_WAIT state, which will inflate the number. For the correct concurrent TCP connections counting method, you should only count TCPs in ESTABLISHED state. Hence grep ESTA or grep ESTABLISHED will be required.

Bad IP's can make quick connections, and you end up with many time_waits which you won't see when just looking at established.

If you don't necessarily count concurrent connections, and just want to analyze which IPs might be attackers, you can include time_waits connections. Run the command without grep ESTABLISHED, which gives you the ability to see what IP's just connected and dropped and may need to be blocked:

 netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | awk '$1 >= 5 {print $0}'

An attacker could make a connection, send requests to an expensive URL, wait a little while, and then close the connection. If the server does not abort the process, the backend will be used up soon, as it will keep serving requests that have been abandoned. The above command will be useful during such a situation.

If an IP has been banned, but you don't know why, you can check it with SSH. Here is an example of a connection that was banned because it reached the hard limit.

Note: Your logging level must be set at least to NOTICE in order to see the reason an IP is banned.

grep BLOCKED_IP /tmp/lshttpd/.rtreport* 

BLOCKED_IP: 47.22.54.182,

Whenever the server adds an IP to the block list, it will write a log to error log:

[<IP.addr>] bot detected for vhost [<vhostname>], reason: xxxxx, close connection!

For example:

tail -f /etc/apache2/logs/error_log

 [NOTICE] [x.x.x.x reached per client hard connection limit: 1, close connection!
 [NOTICE] [x.x.x.x] bot detected for vhost [N/A], reason: OverConnHardLimit, close connection!

or

 [NOTICE] [x.x.x.x] bot detected for vhost [N/A], reason: OverConnSoftLimit, close connection

or

2018-12-05 12:18:05.440745 [NOTICE] [x.x.x.x] bot detected for vhost [APVH_example.com:443], reason: DetectByWAF, close connection!

You should be able to find out why it is added and take action accordingly.

If the IP was banned but a record was not found in error_log, it's possible that IP was dropped by mod_security.

 grep "47.22.54.182" /usr/local/apache/logs/modsec_audit.log 

If the IP address involved is in the LSWS trusted list, it shows:

2018-12-05 12:18:05.440754 [NOTICE] [x.x.x.x] trusted, ignore!

Whenever a mod_security with “drop” action is triggered, LiteSpeed will add the IP to the blacklist. If the IP is in the trusted list, it will be ignored. As with too many blocks, please review the mod_security rule and audit_log, as LSWS will follow the rules there.

If ModSecurity blocks a request and LSWS sees the IP as trusted, the request is still served with 403 response, but that IP won't be blacklisted. If an IP is blacklisted, LSWS will stop serving future requests from that IP.

Trusted IP can be either set on server level or virtual host level through .htaccess

In LSWS Admin Console Server → Security → Access Control → Allowed List, you can set Trusted IP there with trailing “T”.

Since LSWS 5.4RC1, LSWS has virtual host trusted IP support, where you may use Trusted 1.2.3.4, 5.6.7.8 in the Virtual Host document root .htaccess to unblock a blocked IP and make that IP trusted for that vhost. This is not the same as the Trusted IP configured by Admin at server level. It has no effect on bandwidth. The main effect of adding it in .htaccess is to take that IP off of the blacklist and disable WordPress Protect and reCAPTCHA when accessing that specific virtual host.

What if ModSecurity does a drop (TCP FIN) rather than deny for a trusted IP? The trusted list only has an effect on the “drop” action, but not on the “deny” action. A trusted IP won't be added to blacklist, but trust status has no effect on other actions.