1-2 minute no response from server

I

inxile

Member
#1
#1
I'm trying to debug an issue with our server (CentOS 5.4, LiteSpeed 4.0.12 and PHP 5.3.1). Every day, as often as every 10-15 minutes, we see an unresponsive request for a web page lasting between 1-2 minutes. After that the page loads as normal. Sometimes the server is slow and continues to be for a few minutes, other times it is as fast as normal after that.

Logs show dozens, sometimes hundreds of 'lsphp5' processes. It appears as if either the memory or processing is nearly maxed out.

I could use some help trying to find out what exactly is causing this issue. Is it a configuration issue, or a known issue with v4.0.12?

Any help in determining where the issue could be would be very much appreciated!
 
I

inxile

Member
#3
#3
EAProc Idle

In the console I show nearly 400 EAProc Idle. What determines this number and what settings can I adjust to keep this lower? I'm assuming this should be much lower. I can't imagine 400 idle processes is optimal?
 
N

NiteWave

Administrator
#4
#4
yes, it's not normal. something wrong of your php.

please check if there are too many php session files in /tmp. and refer the blog post I posted last time. a system bottle neck(for example, slow /tmp partition) can cause php process fork and fork, until system resource is used up.
 
C

cmanns

Well-Known Member
#5
#5
me too

I get this randomly too.

My system never swaps and tmp never gets more then 40% or so. However our software RAID1 is maxed, going to RAID10 w/more ram soon.

The issue seems LiteSpeed closes the PHP Processes before XCache can even cache, so our cpu gets ate up by dozens of 5% PHP processes?
 
I

inxile

Member
#7
#7
Still having issues. Can't seem to find any outside sources causing the issues. We've upgraded the OS, LiteSpeed and PHP all to current versions.

Is there a place that recommends config settings for LiteSpeed in a high volume PHP production environment?
 
I

inxile

Member
#8
#8
NiteWave said:
yes, it's not normal. something wrong of your php.

please check if there are too many php session files in /tmp. and refer the blog post I posted last time. a system bottle neck(for example, slow /tmp partition) can cause php process fork and fork, until system resource is used up.
Click to expand...
There were about 3500. For a server with 4G ram, what would be the optimal settings for memory usage, connections, etc?
 
M

mistwang

LiteSpeed Staff
#9
#9
when it happens, check server memory usage, CPU usage and I/O wait.
If you have spare memory left, put php sessions to /dev/shm.
It could be a MySQL issue, check mysql status with

mysqladmin processlist
 
I

inxile

Member
#10
#10
Yeah, MySQL has always been a minimal impact and shows no sign of being taxed.

This is really becoming an issue and our hosting provider has been unable to find any solutions or even causes. Unfortunately this was all setup three or four people before me so I have no idea what they did that might have been different or not suggested/default.

I noticed you offer 1 hour support, 15 hours of support and an anti-DDoS/tuning. Which or these would be best for taking a look at our setup and fine tuning it?
 
I

inxile

Member
#12
#12
Here is a set of stats when an issue arose:


Here is the monitoring stats:

kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
01:00:01 PM 3109568 1038472 25.04 23304 635420 1995456 101016 4.82 23360
01:05:01 PM 3289428 858612 20.70 28836 532984 1995500 100972 4.82 23688
01:10:09 PM 152268 3995772 96.33 10612 1402236 1499320 597152 28.48 46508
01:15:08 PM 175720 3972320 95.76 14028 1198712 796212 1300260 62.02 338488
01:20:08 PM 410212 3737828 90.11 14292 1149120 813132 1283340 61.21 283320

What config settings would be best to adjust when in this kind of scenario?
 
I

inxile

Member
#13
#13
high EAProc WaitQ

We've been making a number of corrections and tweaks to the system and things have gotten better.

However, we continue to experience a complete hang of the system from time to time... about 15-20 minutes or half hour or so. The website times out and is unresponsive for about a minute, then all is fine.

During this time we see the following:

EAProc WaitQ is in the mid 100's
EAProc In Use is less than 10
Req in Processing is slightly higher than EAProc WaitQ, mid 100's

It appears settings are throttled a little to tight, what settings should I be looking to to adjust to relieve the EAProc WaitQ?

Follow Up: Logs are stating the PHP suEXEC Max Conn 512 is invalid and are defaulting to 5. What are valid values?
 
Last edited:
W

webizen

Well-Known Member
#14
#14
PHP suEXEC Max Conn sets to 5-10 normally.

You should adjust Max Connections in lsphp5 (in External App): bump it up by 50 at a time (environment variable PHP_LSAPI_CHILDREN needs to be set the same as Max Connections).
 
W

webizen

Well-Known Member
#16
#16
change to following to begin with:

General >>
PHP suEXEC Max Conn: 10

lsphp5 >>
Max Connections: 50
PHP_LSAPI_MAX_REQUESTS=512
PHP_LSAPI_CHILDREN=50
 
I

inxile

Member
#17
#17
After about 10 minutes it happened again... a spike in WaitQ and site was unresponsive. I upped the max conn and PHP_LSAPI_CHILDREN to 100 and restarted. Took a minute for the WaitQ to die down again. Now 5 minutes later its tapped out again.

We average about 10 million hits a month.
 
I

inxile

Member
#19
#19
2 7200 RPM 1 TB hard disks in a RAID-1 array, with a 3Ware 9650SE-2LP RAID controller
8 GB of RAM
Intel Core 2 Quad Q6600

Litespeed Web Server Enterprise v4.0.20
CentOS and PHP were recently updated to current
 
M

mistwang

LiteSpeed Staff
#20
#20
It is likely a slow SQL query or something else blocked all PHP processed, increasing "max connections" wont work.
You need to find out what happened from command line, check "top" output for CPU utilization, I/O wait, load, memory usage, etc.
Check mysql slow query cache, check "mysqladmin processlist" .
Also, check "dmesg" output.
 
You must log in or register to reply here.
Top