Under DDoS need assistance

mistwang

LiteSpeed Staff
#5
It is possible if the botnet is large enough, for example, 1000 bots, each bots make 10 connections, that will be 10,000 connections, and it still not reach the threshold to ban a IP yet.

If it happen again, you should check your netstat output see if it is true.

What can you do if it is true? You have to lower the threshold to make LSWS able to detect those IP abusing you server, it may cause some false alarm, but it better then being taken down completely.

If your server has enough memory, you can raise the limit of "Max Connections" of LSWS under server tuning tab.
 

MikeDVB

Well-Known Member
#7
It is possible if the botnet is large enough, for example, 1000 bots, each bots make 10 connections, that will be 10,000 connections, and it still not reach the threshold to ban a IP yet.
The botnet was 15,000+

What can you do if it is true? You have to lower the threshold to make LSWS able to detect those IP abusing you server, it may cause some false alarm, but it better then being taken down completely.
Which threshold should I lower specifically?

If your server has enough memory, you can raise the limit of "Max Connections" of LSWS under server tuning tab.
It's set to the 10,000 can I Go higher?
 

auser

Super Moderator
#9
The botnet was 15,000+

Which threshold should I lower specifically?
please share what's the output of:
(1)netstat -nt|grep ESTABLISHED|wc
(2)netstat -nt|awk '{print $5;}'|awk -F ':' '{print $1;}'|sort|uniq -c|sort -r|head

to set, go web admin console:
Configuration->Server->Security->Per Client Throttling:
(for example)
Static Requests/second:10
Dynamic Requests/second:2
Outbound Bandwidth (bytes/sec):4k
Inbound Bandwidth (bytes/sec):1k
Connection Soft Limit:5
Connection Hard Limit:20
Grace Period (sec):15
Banned Period (sec):60
 

MikeDVB

Well-Known Member
#10
root@atlantis [/]# netstat -nt|grep ESTABLISHED|wc
426 2556 37914

9 88.234.114.182
93 89.77.8.199
9 222.127.223.74
9 221.126.69.111
9 203.176.146.9
9 202.176.164.237
9 190.229.83.224
9 189.194.102.96
9 189.168.54.29
9 125.26.12.233
 

MikeDVB

Well-Known Member
#11
it's around 2,000 connections total, but each one is making hundreds of requests per second.

I've blocked a lot with iptables -drop but still it's hurting
 

auser

Super Moderator
#12
it's around 2,000 connections total, but each one is making hundreds of requests per second.
in this case,the example setting
Static Requests/second:10
Dynamic Requests/second:2
is just what you want.

the 2nd command output didn't look right on your box. change to
netstat -nt|awk '{print $5;}'|awk -F ':' '{print $1;}'|sort|uniq -c|sort -nr|head

will display the top 10 IPs which have most tcp connections with your host.
 

mistwang

LiteSpeed Staff
#13
You can check /usr/local/apache/log/error_log, there will be many entries like a IP hit the connection hard limit or the soft limit.
You can write a script to parse the list of IP that hits the hard limit, then block them with iptables.
 

felosi

Well-Known Member
#14
Install CSF and use the connection tracking feature, set ct_limit to 30 at 30 second interval. Litespeed cant do it all on its own. Also SYND is another helpful script but use only after csf has ran with ct for a while as it uses netstat and can lag your system

SYND - http://nix101.com/2007/07/21/syn-deflate/
csf - http://configserver.com/cp/csf.html

Also in lsws admin ? security

static requests -10
dynamic - 1

connection soft limit - 5
connection hard limit - 15

in my experience its best to disable smart keep alive, set keep alive timeout to 15, connection timeout to 30

Then you need to tune your tcp stack depending on how much ram you have.

Here is config I sue for servers with 8gb ram, its probably too much but ips should be getting blocked before using all this:
net.core.rmem_max = 184217728
net.ipv4.tcp_rmem = 4096 33554432 184217728
net.core.wmem_max = 184217728
net.ipv4.tcp_wmem = 4096 33554432 184217728
net.ipv4.tcp_mem = 8388608 16777216 184217728
net.core.optmem_max = 107108864
net.core.rmem_default = 10097152
net.core.rmem_default = 10097152
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 30
kernel.panic=5
net.netfilter.nf_conntrack_max = 5048576
net.nf_conntrack_max = 5048576
net.ipv4.netfilter.ip_conntrack_max = 5048576
 
Last edited:

MikeDVB

Well-Known Member
#15
Install CSF and use the connection tracking feature, set ct_limit to 30 at 30 second interval. Litespeed cant do it all on its own. Also SYND is another helpful script but use only after csf has ran with ct for a while as it uses netstat and can lag your system

SYND - http://nix101.com/2007/07/21/syn-deflate/
csf - http://configserver.com/cp/csf.html

Also in lsws admin ? security

static requests -10
dynamic - 1

connection soft limit - 5
connection hard limit - 15

in my experience its best to disable smart keep alive, set keep alive timeout to 15, connection timeout to 30

Then you need to tune your tcp stack depending on how much ram you have.

Here is config I sue for servers with 8gb ram, its probably too much but ips should be getting blocked before using all this:
net.core.rmem_max = 184217728
net.ipv4.tcp_rmem = 4096 33554432 184217728
net.core.wmem_max = 184217728
net.ipv4.tcp_wmem = 4096 33554432 184217728
net.ipv4.tcp_mem = 8388608 16777216 184217728
net.core.optmem_max = 107108864
net.core.rmem_default = 10097152
net.core.rmem_default = 10097152
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 30
kernel.panic=5
net.netfilter.nf_conntrack_max = 5048576
net.nf_conntrack_max = 5048576
net.ipv4.netfilter.ip_conntrack_max = 5048576
Very helpful, the server actulaly has 4gb so I will see what I can do with this to tweak it for the server.

My biggest issue is that after an attack is so large I get tossed onto CiscoGuard automatically so then there are only a few hand full of IPs that are making lots of connections (CiscoGuard Proxy IPs) - so if I start limiting connections by IP I run into issues because there is no way to differentiate between the attacker and legitimate traffic from CiscoGuard.

Sort of a rock and a hard place, the system that is supposed to protect me makes it nearly impossible for me to handle a DDoS attack that simulates real traffic. The flow of packets is enough to trigger CiscoGuard but then CiscoGuard does nothing against a syn flood for example.
 
Last edited:

MikeDVB

Well-Known Member
#16
The server is seeing upwards of 100,000+ HTTP requests per second. Although LiteSpeed is very fast and efficient I don't know that it can handle this sort of load.

I've done everything I can to optimize it however 10+megabits of pure HTTP requests seems to just bring it down in a hurry.
 

MikeDVB

Well-Known Member
#18
Wow, never thought litespeed can scale this much. :D
It was handling upwards of 1500~2,000 requests per second pretty well but beyond that the server just begins to choke.

I changed the site being requested to a static page and that reduced the load substantially but I am only running a 1-cpu license so I'm not sure if that is really limiting my abilities.

I would think on a 2 or 4 cpu license it would simply cause more Disk I/O wait and more problems.
 

MikeDVB

Well-Known Member
#20
George have said that a 2-cpu license handles ddos better than a 1-cpu license. It's better for the I/O.
Yeah, but is 2cpu going to make a difference over 1cpu with 100,000 requests per second?

I would think at best doubling the CPU/Process would double it's ability to handle requests but there is the additional overhead of having a secondary process and then I/O blocking to look into.

I am curious about AIO, but not found anything useful about it.
 
Top