[Resolved] DDoS - long requests

[IanD]

Hi,

We are currently on the receiving end of a distributed DDoS attached (luckily not very big though).

They have the clients 'POST'ing to a URL that we have disabled access to via htaccess so it points to a 403 page.

But even with this, it is seriously taking resources on our web servers so I'm looking at how we can free up the resources. On the Top50 Request sorted by Request Time, it is 100% full of this:

5.122.160.760RB542.0173862/3129436479/-1*****V2-"POST /Uxi6h HTTP/1.1"
5.162.184.1040RB441.1166686/3373023972/3972*****V2-"POST /Uxi6h HTTP/1.1"
5.21.246.1330RB341.5291606/93643220/20*****V2-"POST /Uxi6h HTTP/1.1"
188.140.140.560RB311.0297158/35148320/20*****V2-"POST /Uxi6h HTTP/1.1"
146.185.39.300RB297.0131852/13931119/19*****V2-"POST /Uxi6h HTTP/1.1"

Some requests are still processing after 500+ seconds it seems - how can stop this? this can't be normal and I can only presume an intentional part of the attack to consume server resources.

Is there is a setting in LiteSpeed to say if the request is not complete after say 30 seconds, kill it?

Thanks,

Ian

 

[NiteWave]

I think 2 related settings are:
lsws web admin -> Server -> General -> External Application Abort: Yes
lsws web admin -> Server -> Tuning -> Connection Timeout (secs): default is 300.
you can try to reduce the timeout.

 

[mistwang]

Try a rewrite rule like

RewriteRule Uxi6h - [E=blockbot:1]

 

[IanD]

Thank you for your replies.

I have modified our rewrite to for the 'blockbot' and our Connection Timeout was already only 30 seconds, External Application Abort seems to be yes by default.

But we are still seeing:

ClientKaMRIn/TotalOut/TotalVHostHandlerPRequest

5.21.227.2300RB574.6215266/4061823903/3903V2-"POST /Uxi6h HTTP/1.1"
146.185.36.1150RB414.5394318/42468316530/-1V2-"POST /Uxi6h HTTP/1.1"
212.215.162.1490RB413.8175014/1850476526/6526V2-"POST /Uxi6h HTTP/1.1"
91.217.185.850RB340.4592802/89890120/20V2-"POST /Uxi6h HTTP/1.1"
82.178.131.320RB339.6370722/3754186625/-1V2-"POST /Uxi6h HTTP/1.1"
46.152.206.140RB328.1221646/2756955226/5226V2-"POST /Uxi6h HTTP/1.1"
188.139.162.1430RB312.3274984/5329575225/5225V2-"POST /Uxi6h HTTP/1.1"

 

[mistwang]

Please check log for message like

[REWRITE] detect bad robot, block!

at INFO level, if the rewrite rule works properly, otherwise, need to make sure the rewrite rule can block those URL.

 

[IanD]

Ah yes - sorry there was an extra space in Rewrite rule, now it is working!

Around 8k IPs blocked in the 1st minute.

 

[IanD]

Even though we are now blocking all the IPs in LiteSpeed that request the URL, the request queue is still full of:

85.154.38.180RB634.2276338/7007026589/6589V2-"POST /Uxi6h HTTP/1.1"
212.126.103.1460RB361.3731366/73748320/20V2-"POST /Uxi6h HTTP/1.1"
5.162.162.350RB318.2596966/5992524112/4112V2-"POST /Uxi6h HTTP/1.1"
85.154.127.160RB311.0316398/7517446585/6585V2-"POST /Uxi6h HTTP/1.1"
82.178.131.1840RB299.2156970/35827120/20V2-"POST /Uxi6h HTTP/1.1"

should they no longer be there now?

 

[NiteWave]

how about the access log? does they exist in access log as well ?

 

[IanD]

Yes:

149.255.200.58 - - [07/May/2014:15:43:35 +0000] "POST /Uxi6h HTTP/1.1" 403 4114 "-"
94.98.101.213 - - [07/May/2014:15:43:33 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
185.16.26.25 - - [07/May/2014:15:42:26 +0000] "POST /Uxi6h HTTP/1.1" 200 3948 "-"
85.154.38.210 - - [07/May/2014:15:43:10 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
85.154.189.198 - - [07/May/2014:15:43:30 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
37.44.37.20 - - [07/May/2014:15:43:34 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
37.237.106.34 - - [07/May/2014:15:43:33 +0000] "POST /Uxi6h HTTP/1.0" 403 3983 "-"
85.154.39.247 - - [07/May/2014:15:43:30 +0000] "POST /Uxi6h HTTP/1.1" 403 20 "-"
5.162.165.162 - - [07/May/2014:15:43:24 +0000] "POST /Uxi6h HTTP/1.1" 403 3983 "-"

It looks like LiteSpeed is not blocking the request until it has finished (maybe?) and therefore the requests are using up all the resources and we have some requests lasting for 600+ seconds and the IP is not in the ban list.

I presume they are added when finally the incoming request finishes?

The amount of data they are posting is not too big, maybe 10-100kb each request but it's having a big impact.

 

[IanD]

Hi,

Sorry to bring up this old thread but we've recently disabled CloudFlare and now we have exactly the same problem as before.

Is there a way (maybe something has changed in the last 2 years!) to block a POST request to a certain URL before the POST request has finished?

At the moment it looks like the bot posts 100Kb data to the specific URL and when has finished transferring, LiteSpeed sends a 403 header.

Is it possible to send that header at the very beginning of the request so the server don't have to deal with all that wasted bandwidth and resources keeping the connection open while the data is transferred?

Blocking by IP unfortunately is not feasible due to the many many 1000s of IPs being used.

Thanks,

Ian

 

[mistwang]

You can customize a mod_security rule to detect that, use "drop" action, instead of "deny", then the connection and sub-subsequent connections from the same IP will be immediately dropped.

 

[IanD]

Works perfectly thanks - my inbound bandwidth has dropped by 50%.