What settings for maximum speed on a large configuration ?
- Thread starter Monarobase
- Start date
Sorry for the delay.
Due to lack of time I installed Xcache 2.0.1 and have disabled the variable cache, hopeing that someone might debug and fix the issue without me reporting it…
I've got one last (quite big) problem.
I run Atomicorp ASL T-WAF in front of litespeed and sometimes I get an error saying from the T-WAF with the following message :
---
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /XXX.html.
Reason: Error reading from remote server
Apache Server at www.XXXXXXXX.tld Port 30001
---
ASL seem to be saying it's an issue with litespeed but litespeed doesn't log anything when the error happens and ASL T-WAF doesn't log much either (except that Litespeed gave an invalid response).
Do you have any suggestions about how I could determine if it rearly is Litespeed giving a bad response or if it's an issue with the T-WAF.
The error message only happens about 20 times a day so it's very difficult to catch and almost impossible to determine if Litespeed is actually giving a blank page or a strange result or if it's the T-WAF that's not responding fast enough for litespeed or the other way around.
When I searched for these issues, alot of topics about Litespeed came up… with various other proxies, so it seems that litespeed could be for something in this issue.
Due to lack of time I installed Xcache 2.0.1 and have disabled the variable cache, hopeing that someone might debug and fix the issue without me reporting it…
I've got one last (quite big) problem.
I run Atomicorp ASL T-WAF in front of litespeed and sometimes I get an error saying from the T-WAF with the following message :
---
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /XXX.html.
Reason: Error reading from remote server
Apache Server at www.XXXXXXXX.tld Port 30001
---
ASL seem to be saying it's an issue with litespeed but litespeed doesn't log anything when the error happens and ASL T-WAF doesn't log much either (except that Litespeed gave an invalid response).
Do you have any suggestions about how I could determine if it rearly is Litespeed giving a bad response or if it's an issue with the T-WAF.
The error message only happens about 20 times a day so it's very difficult to catch and almost impossible to determine if Litespeed is actually giving a blank page or a strange result or if it's the T-WAF that's not responding fast enough for litespeed or the other way around.
When I searched for these issues, alot of topics about Litespeed came up… with various other proxies, so it seems that litespeed could be for something in this issue.
since 4.2.2 is out, can you try 4.2.2 see if issue still there ?
in 4.2.2 release log:
http://www.litespeedtech.com/litespeed-web-server-release-log.html
"Improved mod_security compatibility with gotroot ruleset"
in 4.2.2 release log:
http://www.litespeedtech.com/litespeed-web-server-release-log.html
"Improved mod_security compatibility with gotroot ruleset"
Thanks I will give it a try 
We don't run the rules in litespeed but have ASL's proxy in front of litespeed.
Running their hourly updated rules seems a bit dangerous directly in litespeed as the rules are only tested in ASL's modsecurity before being released and it seems that it's almost impossible for litespeed to follow these updates.
ASL is however being helpfull on this issue. They helped me set up full modsecurity logging to try and see if there's something reported when this issue happens.
I'll let you know if/when I get some more information.
We don't run the rules in litespeed but have ASL's proxy in front of litespeed.
Running their hourly updated rules seems a bit dangerous directly in litespeed as the rules are only tested in ASL's modsecurity before being released and it seems that it's almost impossible for litespeed to follow these updates.
ASL is however being helpfull on this issue. They helped me set up full modsecurity logging to try and see if there's something reported when this issue happens.
I'll let you know if/when I get some more information.
I've got an update from ASL.
ASL T-WAF runs an optimised version of Apache with Mod Security and Mod_proxy in front of litespeed. (note : this hardly adds any latency on our server, we get 15ms responses even with mod_security in T-WAF proxy).
This is what is happening :
I'm getting the following error from mod_proxy : "proxy: error reading status line from remote server"
1) Client connects to the T-WAF (local reverse proxy)
2) T-WAF checks connection is live
3) Litespeed goes away
4) T-WAF proxy doesn't get the data from litespeed
Do you have any suggestions about how I could get litespeed to play well running behind a reverse proxy ?
Thanks
ASL T-WAF runs an optimised version of Apache with Mod Security and Mod_proxy in front of litespeed. (note : this hardly adds any latency on our server, we get 15ms responses even with mod_security in T-WAF proxy).
This is what is happening :
I'm getting the following error from mod_proxy : "proxy: error reading status line from remote server"
1) Client connects to the T-WAF (local reverse proxy)
2) T-WAF checks connection is live
3) Litespeed goes away
4) T-WAF proxy doesn't get the data from litespeed
Do you have any suggestions about how I could get litespeed to play well running behind a reverse proxy ?
Thanks
while no environment to experiment with this, it looks litespeed is behind apache and handle the requests passed from apache, but connection between apache and litespeed closed by litespeed in some cases.
need tune
1)lsws admin console -> Server -> Tuning -> Connection
not sure if disable keep-alive will be a solution
Max Keep-Alive Requests:0
may need increase
Connection Timeout (secs)
as well
2)lsws admin console -> Server -> Security:
disable "Per Client Throttling"(the default value)
since now only one client IP.
need tune
1)lsws admin console -> Server -> Tuning -> Connection
not sure if disable keep-alive will be a solution
Max Keep-Alive Requests:0
may need increase
Connection Timeout (secs)
as well
2)lsws admin console -> Server -> Security:
disable "Per Client Throttling"(the default value)
since now only one client IP.
Thanks,
I'm not sure how to completly disable per client throttling.
In Server > Security I've got 5 sections :
File Access
Per Client Throttling
External Application Ressource
Access Denied Directories
Access Control
Inside Per Client Throttling I've got :
Static Requests/second (number >= 0)
Dynamic Requests/second (number >= 0)
Outbound Bandwidth (bytes/sec)
Inbound Bandwidth (bytes/sec)
Connection Soft Limit
Connection Hard Limit
Block Bad Request
Grace Period (sec)
Banned Period (sec)
Is it one of these settings I need to change to completly disable it ?
Thanks !
I'm not sure how to completly disable per client throttling.
In Server > Security I've got 5 sections :
File Access
Per Client Throttling
External Application Ressource
Access Denied Directories
Access Control
Inside Per Client Throttling I've got :
Static Requests/second (number >= 0)
Dynamic Requests/second (number >= 0)
Outbound Bandwidth (bytes/sec)
Inbound Bandwidth (bytes/sec)
Connection Soft Limit
Connection Hard Limit
Block Bad Request
Grace Period (sec)
Banned Period (sec)
Is it one of these settings I need to change to completly disable it ?
Thanks !
1)any improvement regarding the error rate before and after the change?
2)any URL pattern or method(GET or POST) which is easier to trigger the error or just random ?
3)will the same error happen between T-WAF proxy and other web server(apache, nginx etc)?
again, it'll be much helpful if can work out a test case which can easily reproduce the error.
2)any URL pattern or method(GET or POST) which is easier to trigger the error or just random ?
3)will the same error happen between T-WAF proxy and other web server(apache, nginx etc)?
again, it'll be much helpful if can work out a test case which can easily reproduce the error.
No yesterday I got 10 errors between 0am and 7 pm
Today I got 9 errors from 0am and 7 pm
2)any URL pattern or method(GET or POST) which is easier to trigger the error or just random ?
It seems random, I don't have many POST requests so it's not likley to happen with a POST request. It happens randomly on images / html / PHP requests. (about 1 for every 1000 requests)
3)will the same error happen between T-WAF proxy and other web server(apache, nginx etc)?
An ASL developper has just told me that they have never seen this issue before and protect nginx servers without issues. Apache servers generaly integrate mod_security directly so they don't need the T-WAF.
Yes it would, but I've even tried benchmarking (siege) it without managing to create an error. It doesn't seem to happen so much when the server is under more stress.
Apache docs mention a race condition :
http://httpd.apache.org/docs/2.2/en/mod/mod_proxy_http.html
Today I got 9 errors from 0am and 7 pm
2)any URL pattern or method(GET or POST) which is easier to trigger the error or just random ?
It seems random, I don't have many POST requests so it's not likley to happen with a POST request. It happens randomly on images / html / PHP requests. (about 1 for every 1000 requests)
3)will the same error happen between T-WAF proxy and other web server(apache, nginx etc)?
An ASL developper has just told me that they have never seen this issue before and protect nginx servers without issues. Apache servers generaly integrate mod_security directly so they don't need the T-WAF.
Yes it would, but I've even tried benchmarking (siege) it without managing to create an error. It doesn't seem to happen so much when the server is under more stress.
Apache docs mention a race condition :
http://httpd.apache.org/docs/2.2/en/mod/mod_proxy_http.html
Last edited:
I'm giving it a try to see if it makes any difference. I will let you know if it removes the errors.
ASL said I should set the T-WAF (mod_security) to enforce non compressed data because this could cause issues and also prevent mod_security from analysing the data transmitted.
I've given this (disabling compression) a try but it doesn't help either (I still get errors). I think I will activate gzip compression in the T-WAF so it can analyze the data given by litespeed and see what the performance impact is.
I wil reactive keep-alive once I've finished the test with
ASL said I should set the T-WAF (mod_security) to enforce non compressed data because this could cause issues and also prevent mod_security from analysing the data transmitted.
I've given this (disabling compression) a try but it doesn't help either (I still get errors). I think I will activate gzip compression in the T-WAF so it can analyze the data given by litespeed and see what the performance impact is.
I wil reactive keep-alive once I've finished the test with
Code:
setEnv proxy-initial-not-pooled 1
Last edited: