Huge lsphp5 problem

A

alex95bg

Member
#1
#1
I seem to be unlucky.. litespeed worked great
I upgraded to the final version and recompiled php 5.2.9 with eAccelerator and Suhosin
Everything seems to be normal.. until something very strange happens
A huge amount of lsphp5 processes are spawned and the server load spikes to 80+

Code: 
top - 22:48:08 up  4:36,  1 user,  load average: 71.27, 40.58, 19.89
Tasks: 163 total,  81 running,  82 sleeping,   0 stopped,   0 zombie
Cpu(s): 97.4%us,  1.6%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.3%hi,  0.7%si,  0.0%st
Mem:    183412k total,   180520k used,     2892k free,     4948k buffers
Swap:   345356k total,    65448k used,   279908k free,    75828k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                      
16924 nobody    20   0 66200 3164 1560 R  2.0  1.7   0:04.70 lsphp5                       16564 nobody     1 -19  7484 4016 1328 S  1.3  2.2   0:30.88 lshttpd                      16617 root      20   0  2340 1172  852 R  1.3  0.6   0:12.90 top                          
16854 nobody    20   0 66412 5220 3452 R  1.3  2.8   0:06.76 lsphp5                       16888 nobody    20   0 66600 7580 5628 R  1.3  4.1   0:09.90 lsphp5                       16891 nobody    20   0 66460 7160 5204 R  1.3  3.9   0:08.80 lsphp5                       
16894 nobody    20   0 66368 4948 3220 R  1.3  2.7   0:05.32 lsphp5                       16897 nobody    20   0 66332 4676 2964 R  1.3  2.5   0:05.22 lsphp5                       16899 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.94 lsphp5                       
16900 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.92 lsphp5                       16901 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.90 lsphp5                       16902 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.88 lsphp5                       
16903 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.86 lsphp5                       16904 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.86 lsphp5                       16905 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.82 lsphp5                       
16906 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.80 lsphp5                       16907 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.78 lsphp5                       16908 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.80 lsphp5                       
16909 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.78 lsphp5                       16911 nobody    20   0 66200 3176 1572 R  1.3  1.7   0:04.76 lsphp5                       16912 nobody    20   0 66200 3164 1560 R  1.3  1.7   0:04.74 lsphp5
I rebooted the PC, with no result
So this is literally stopping the server and it spits out 503 errors after a huge delay
I have to do this:
Code: 
dragonheart:/usr/local/lsws/bin# ./lswsctrl stop
[OK] lshttpd: stopped.
dragonheart:/usr/local/lsws/bin# killall lsphp5
dragonheart:/usr/local/lsws/bin# ./lswsctrl start
[OK] lshttpd: pid=17156.
dragonheart:/usr/local/lsws/bin#
A restart doesn't make any effect
I havent seen this preblem never before, the code is the same, lsws has been running on this machine for morew than seven months, and almost two with this kernel (2.6.24-etchnhalf; debian 4.0r3 etch)
 
M

mistwang

LiteSpeed Staff
#2
#2
You should to do one upgrade at a time, at this point it is hard tell it is a PHP issue or LSWS issue.

What is your "Max Connections" for lsphp5? maybe you should tune it down.
 
A

alex95bg

Member
#3
#3
I upgraded only litespeed, i used the same setup (PHP 5.2.9, suhosin, eAccelerator) before, just now eAccelerator and suhosin are different versions
Code: 
PHP_LSAPI_MAX_REQUESTS=500
PHP_LSAPI_CHILDREN=35
And Max Connections is 35 :)

The strange thing is, i tried very hard to replicate this, but i didn't succeed
I think this happens only when the machine is very low on ram
Yesterday there was a traffic spike, but on previous traffic spikes nothing similar happened

Also, i can't turn on the eAccelerator optimizer, i try from control.php but it stays on no
Code: 
Optimizer enabled  	no
It's also enabled in php.ini
--
Other than this, and the solved php problem LiteSpeed is working great. I would like to thank you, lsws saved my website. It survived on apache for about a week after it's startup and apache started throwing timeouts. So when i switch to the apache backup, the machine almost dies :)

NEW edit:
It happened again, i killed all the processes and they spawned again
It's ok now, let's see for how long
 
Last edited:
M

mistwang

LiteSpeed Staff
#5
#5
You should downgrade eaccelerator and suhosin to the older version.
eAcc 0.9.6-rc1 is in beta, and optimizer is removed if you read the release log.
 
A

alex95bg

Member
#6
#6
How do i downgrade from the Compile PHP section? :)

edit:
I now looked at the server log and this doesn't seem ordinary:

2009-08-27 23:15:17.347 INFO Pid: 16335 associated with [lsphp5]
2009-08-27 23:15:17.347 INFO ProcInfo Pid: 16335, socket: /tmp/lshttpd/lsphp5.sock
2009-08-27 23:15:17.347 INFO [lsphp5] pid list size: 0, pid stop list size: 0
2009-08-27 23:15:17.347 INFO [90.154.236.27:3252-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.347 NOTICE [90.154.236.27:3252-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.347 INFO [77.85.229.13:3650-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.347 NOTICE [77.85.229.13:3650-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.348 INFO [212.25.53.81:2610-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.348 NOTICE [212.25.53.81:2610-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.348 INFO [91.134.62.117:4458-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.348 NOTICE [91.134.62.117:4458-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.348 INFO [77.78.168.80:57504-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.348 NOTICE [77.78.168.80:57504-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.348 INFO [93.123.6.16:4513-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.348 NOTICE [93.123.6.16:4513-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.348 INFO [213.91.144.42:1119-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.348 NOTICE [213.91.144.42:1119-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.348 INFO [87.120.54.230:13599-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.348 NOTICE [87.120.54.230:13599-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.348 INFO [87.126.196.229:52308-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.348 NOTICE [87.126.196.229:52308-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.348 INFO [77.85.91.23:2162-0#cntstats] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
2009-08-27 23:15:17.348 NOTICE [77.85.91.23:2162-0#cntstats] Max retries has been reached, 503!
2009-08-27 23:15:17.348 NOTICE [77.85.91.23:2162-0#cntstats] oops! 503 Service Unavailable
2009-08-27 23:15:17.348 NOTICE [77.85.91.23:2162-0#cntstats] Content len: 0, Request line:
GET /?s=121&rnd=125475287&img=h&resolut=800x600&ref=http%3A//99mp3bg.net/cat.php%3Fcat%3Doriental-music HTTP/1.1
2009-08-27 23:15:17.349 NOTICE [87.126.196.229:52308-0#cntstats] oops! 503 Service Unavailable
2009-08-27 23:15:17.349 NOTICE [87.126.196.229:52308-0#cntstats] Content len: 0, Request line:
GET /?s=121&rnd=974597684&img=h&resolut=1366x768&ref=http%3A//99mp3bg.net/search.php%3Fs%3Dpixie%20lott%26btn%3Dmp3 HTTP/1.1
2009-08-27 23:15:17.350 NOTICE [90.154.236.27:3252-0#cntstats] oops! 503 Service Unavailable
2009-08-27 23:15:17.350 NOTICE [90.154.236.27:3252-0#cntstats] Content len: 0, Request line:
GET /?s=121&rnd=823395602&img=h&resolut=1280x1024&ref= HTTP/1.1
2009-08-27 23:15:17.350 NOTICE [77.85.229.13:3650-0#cntstats] oops! 503 Service Unavailable
2009-08-27 23:15:17.350 NOTICE [77.85.229.13:3650-0#cntstats] Content len: 0, Request line:
GET /?s=121&rnd=1222735824&img=h&resolut=1024x768&ref=http%3A//99mp3bg.net/song.php%3Fs%3Dplay%26id%3D28039 HTTP/1.1
2009-08-27 23:15:17.350 NOTICE [212.25.53.81:2610-0#cntstats] oops! 503 Service Unavailable
2009-08-27 23:15:17.351 NOTICE [212.25.53.81:2610-0#cntstats] Content len: 0, Request line:
GET /?s=121&rnd=752332234&img=h&resolut=1440x900&ref= HTTP/1.1
2009-08-27 23:15:17.351 NOTICE [91.134.62.117:4458-0#cntstats] oops! 503 Service Unavailable
2009-08-27 23:15:17.351 NOTICE [91.134.62.117:4458-0#cntstats] Content len: 0, Request line:
GET /?s=121&rnd=1282873319&img=h&resolut=1152x864&ref=http%3A//99mp3bg.net/albums.php%3Fview%3D862 HTTP/1.1
2009-08-27 23:15:17.351 NOTICE [77.78.168.80:57504-0#cntstats] oops! 503 Service Unavailable
2009-08-27 23:15:17.351 NOTICE [77.78.168.80:57504-0#cntstats] Content len: 0, Request line:
GET /?s=121&rnd=461096347&img=h&resolut=1280x720&ref= HTTP/1.1
2009-08-27 23:15:17.351 NOTICE [93.123.6.16:4513-0#cntstats] oops! 503 Service Unavailable
2009-08-27 23:15:17.351 NOTICE [93.123.6.16:4513-0#cntstats] Content len: 0, Request line:
GET /?s=121&rnd=240882222&img=h&resolut=1280x800&ref=http%3A//99mp3bg.net/index.php HTTP/1.1
2009-08-27 23:15:17.351 NOTICE [213.91.144.42:1119-0#cntstats] oops! 503 Service Unavailable
2009-08-27 23:15:17.351 NOTICE [213.91.144.42:1119-0#cntstats] Content len: 0, Request line:
GET /?s=121&rnd=1516600785&img=h&resolut=1024x768&ref=http%3A//99mp3bg.net/cat.php%3Fcat%3Dbulgarian-music%26orderby%3Ddownload HTTP/1.1
2009-08-27 23:15:17.351 NOTICE [87.120.54.230:13599-0#cntstats] oops! 503 Service Unavailable
Click to expand...
new edit:
I setted Max Idle Time to 5
I hope that this will kill the phantom processes
 
Last edited:
A

alex95bg

Member
#13
#13
No, i haven't but i really think that it is the problem. eAccelerator gives a huge performance boost in my site and i use the eaccelerator_get and eaccelerator_put functions so i can't try it out on the live environment :(
Though i will try with litespeed 4.0.6 because (i hope) it doesn't use the rc version of eAccelerator
Also it will be good if we can select eAccelerator and Suhosin versions from the Compile PHP tab, having rc or beta in a production environment isn't the best thing to do :)
 
A

alex95bg

Member
#14
#14
same thing when i downgraded to .6 but with the old php
Now when i try to recompile php i get
100 389 100 389 0 0 347 0 0:00:01 0:00:01 --:--:-- 347
100 389 100 389 0 0 346 0 0:00:01 0:00:01 --:--:-- 0

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error exit delayed from previous errors
On Extracting LSAPI

i really hope that someone will help me
 
A

alex95bg

Member
#16
#16
Is there a work around? I didn't find one
Maybe i will do a os change, migrating from debian to centos soon, let's hope that the issue will go away
And now i fixed it with (a very stupid way) 1 minute cron job which counts the lsphp5 processes and if they are > 6 kills them all
 
A

alex95bg

Member
#17
#17
I switched over to Ubuntu 8.10 Server because debian broke very bad in the middle of a traffic spike
I watched the server for a week, everything is working great. :) i only had a few problems with the rc eaccelerator so i compiled myself the stable version and everything worked great
i suggest you don't download the RC version on php build, it has bugs and it's a very very bad idea to pud unstable software on production servers
 
A

art182

Member
#18
#18
i think i have this same problem ...

version 4.0.9, 4.0.11 and on 4.0.6 also...

php 5.2.11 and APC 3.1.3

4850 nobody 21 1 82132 47m 9608 R 49.3 4.8 0:04.34 lsphp5
4675 nobody 21 1 86304 57m 16m R 48.9 5.8 1:20.31 lsphp5 4671 nobody 20 0 8668 4908 4000 S 0.7 0.5 0:00.74 lshttpd
4672 nobody 21 1 44496 4408 3040 S 0.0 0.4 0:00.04 lsphp5 4674 nobody 20 0 7944 5756 1564 S 0.0 0.6 0:00.14 admin_php
4715 nobody 21 1 46668 15m 13m S 0.0 1.6 0:01.52 lsphp5 4842 nobody 21 1 44892 10m 8708 S 0.0 1.0 0:00.12 lsphp5
4844 nobody 21 1 44892 10m 8688 S 0.0 1.0 0:00.10 lsphp5 4845 nobody 21 1 44892 10m 8688 S 0.0 1.0 0:00.16 lsphp5
4848 nobody 21 1 44892 10m 8688 S 0.0 1.0 0:00.14 lsphp5
4852 nobody 21 1 44892 9996 8276 S 0.0 1.0 0:00.08 lsphp5
Click to expand...

what's going on ? ( everything was good for about 1 year and now something f### )
maybe this is problem with LSAPI or PHP ?
 
M

mistwang

LiteSpeed Staff
#19
#19
It is likely a PHP/APC internal problem.
Try turning off APC, or try eAccelerator, or upgrade APC.
Also, if it does not solve the problem, do
"strace -tt -T -p <pid_of_lsphp5>" and "strace -tt -T -p <pid_of_lshttpd>"

and send us the output.
 
A

art182

Member
#20
#20
oh i know whats going on now...
there was problem with one of virtual hosts ( wordpress + spam bot + php error )

everything ok ( 4.0.12 )
sorry for trouble
 
You must log in or register to reply here.
Top