Under DDoS need assistance

[mistwang]

Usually, the dummy bots only request one URL, so the page is cached in memory by kernel, multiple CPU license wont cause higher disk I/O.
AIO will help when the I/O wait of your server is high.
I don't think CiscoGuard really help on this, so it is better off not using it.

 

[anewday]

How is your server coping with the attacks now?

 

[Bono]

It would be cool if LiteSpeed could work with CSF in banning clients that are establishing too many connections.

 

[MikeDVB]

How is your server coping with the attacks now?

We ended up null-routing one IP that was coming under about 15gbps of attack and we moved another client to a litespeed powered dedicated server that was receiving about 50mbps worth of SYN requests that Cisco Guard was failing to filter.

 

[anewday]

omg What was the load during the 50mbps sync attack and how many cpu license? I take it that lsws was able to fend it off.

 

[MikeDVB]

omg What was the load during the 50mbps sync attack and how many cpu license? I take it that lsws was able to fend it off.

Load stayed low as I changed the site to 100% static but the issue was the kernel was ending up with a huge amount of orphaned tcp_sockets and eventually was running out of available connections.

 

[Bono]

Install CSF and use the connection tracking feature, set ct_limit to 30 at 30 second interval. Litespeed cant do it all on its own. Also SYND is another helpful script but use only after csf has ran with ct for a while as it uses netstat and can lag your system

SYND - http://nix101.com/2007/07/21/syn-deflate/
csf - http://configserver.com/cp/csf.html

Also in lsws admin ? security

static requests -10
dynamic - 1

connection soft limit - 5
connection hard limit - 15

in my experience its best to disable smart keep alive, set keep alive timeout to 15, connection timeout to 30

Then you need to tune your tcp stack depending on how much ram you have.

Here is config I sue for servers with 8gb ram, its probably too much but ips should be getting blocked before using all this:
net.core.rmem_max = 184217728
net.ipv4.tcp_rmem = 4096 33554432 184217728
net.core.wmem_max = 184217728
net.ipv4.tcp_wmem = 4096 33554432 184217728
net.ipv4.tcp_mem = 8388608 16777216 184217728
net.core.optmem_max = 107108864
net.core.rmem_default = 10097152
net.core.rmem_default = 10097152
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 30
kernel.panic=5
net.netfilter.nf_conntrack_max = 5048576
net.nf_conntrack_max = 5048576
net.ipv4.netfilter.ip_conntrack_max = 5048576

From your experience why we should turn off smart keep alive? When i'm under attack I see i'm not getting many syn requests so I cannot kill it with your tool, my main problem is TIME_WAIT, so i'm looking into to somehow fix that.

Usually my server load is around .30 but today i'm under attack so my load is pretty high.

1 CLOSING
14 ESTABLISHED
3 FIN_WAIT1
17 FIN_WAIT2
1 LAST_ACK
66 LISTEN
623 TIME_WAIT

01:05:15 up 227 days, 21:29, 1 user, load average: 2.71, 2.31, 2.14
So TIME_WAIT connections are causing load of web server.
How do you handle this kind of situations?
I have already limited dynamic requests, soft and hard limit but that doesn't help with many connections, I got 20-30 ips that are according to netstat connected more than 20-80 times in TIME_WAIT state.

 

[felosi]

Adjusting your time_wait timeouts in systctl as I listed above will help. But Time wait is not important, you can just set the timeout lower for that. You dont need to track it.

I am also currently soliciting developers to work on a new idea I have, what we came up with was BARF - Block Apache Request Floods - This is for http get style attacks - http://nix101.com/2009/09/04/new-barf-update-script-for-get-floods/
or just www/nix101.com/barf for direct link.

Also
1 CLOSING
14 ESTABLISHED
3 FIN_WAIT1
17 FIN_WAIT2
1 LAST_ACK

Does not look like attack to me, even with the amount of time waits you have. If your server goes down udner those conditions something is very wrong, check dmesg for packet drop errors and such

 

[Bono]

Adjusting your time_wait timeouts in systctl as I listed above will help. But Time wait is not important, you can just set the timeout lower for that. You dont need to track it.

I am also currently soliciting developers to work on a new idea I have, what we came up with was BARF - Block Apache Request Floods - This is for http get style attacks - http://nix101.com/2009/09/04/new-barf-update-script-for-get-floods/
or just www/nix101.com/barf for direct link.

Also
1 CLOSING
14 ESTABLISHED
3 FIN_WAIT1
17 FIN_WAIT2
1 LAST_ACK

Does not look like attack to me, even with the amount of time waits you have. If your server goes down udner those conditions something is very wrong, check dmesg for packet drop errors and such

It doesn't go down, just load is higher than usual, but when my sites are busy load peaks at 12 and this is how it looks like.

49 ESTABLISHED
25 FIN_WAIT1
17 FIN_WAIT2
3 LAST_ACK
66 LISTEN
2 SYN_RECV
1058 TIME_WAIT

Later on:
1 CLOSE_WAIT
1 CLOSING
76 ESTABLISHED
8 FIN_WAIT1
48 FIN_WAIT2
3 LAST_ACK
66 LISTEN
12 SYN_RECV
1736 TIME_WAIT
Is BARF replacement for your synd script?

 

[MikeDVB]

In a (long awaited) update - LiteSpeed was able to keep up however it wasn't able to do 100k requests/second + the normal daily requests that would ordinarily hit the server.

I moved this one site onto it's own dedicated server with LiteSpeed and it stayed online and very responsive although it was using around 2tb of bandwidth/day

 

[anewday]

What are the specs of the quad core server?

 

[felosi]

If it is getting GET attacks you should really try BARF. It works good, you just have to manually specify the requests the attackers are made. Sometimes they will change them when they realize whats going on but I handle 5 servers with sites getting ddosed all the time and I am able to keep up with them on at least 2 sites I can think of now.

Eventually when i find the right programmer I am going to get it programmed where it detects and blocks on ANY reptitive GET. Now that will be cool. I think you would still have to specify domain to watch for because it would just be too much for it to be checking all domlogs at once.

With BARF, SYND, Csf w/connection tracking and litespeed with proper settings you can pretty much handle anything that gets by your network filters as long as it doesnt consume your pipe.

But there is one bad thing about litespeed I noticed. For example for a lot of these attacking bots they was not giving user agent so we added some mod sec and rewrite rules to stop this. Now all attacking bots get a 403 yet it is totally draining my bandwidth as if it was really getting the image. I think Ill make a post about this here somewhere. Has anyone else noticed this? Also even though attacking bots are getting 403 the lsphp and lshttpd processes are going crazy. I guess where its all based on php - the error pages and all.

 

[anewday]

What's SYND? Could you share those modsec and rewrite rules?

 

[MikeDVB]

What are the specs of the quad core server?

Just a 2.4ghz Quad with 4gb ram but the storage was network attached SCSI (so it could keep up).

 

[Bono]

If it is getting GET attacks you should really try BARF. It works good, you just have to manually specify the requests the attackers are made. Sometimes they will change them when they realize whats going on but I handle 5 servers with sites getting ddosed all the time and I am able to keep up with them on at least 2 sites I can think of now.

Eventually when i find the right programmer I am going to get it programmed where it detects and blocks on ANY reptitive GET. Now that will be cool. I think you would still have to specify domain to watch for because it would just be too much for it to be checking all domlogs at once.

With BARF, SYND, Csf w/connection tracking and litespeed with proper settings you can pretty much handle anything that gets by your network filters as long as it doesnt consume your pipe.

But there is one bad thing about litespeed I noticed. For example for a lot of these attacking bots they was not giving user agent so we added some mod sec and rewrite rules to stop this. Now all attacking bots get a 403 yet it is totally draining my bandwidth as if it was really getting the image. I think Ill make a post about this here somewhere. Has anyone else noticed this? Also even though attacking bots are getting 403 the lsphp and lshttpd processes are going crazy. I guess where its all based on php - the error pages and all.

This is probably related to my post
http://www.litespeedtech.com/support/forum/showthread.php?t=3387

Please check which process is overloading your server, on my server system cannot find files and it takes most of the resources and overloads the server.