High cpu usage problem

C

cumhuronat

Member
#1
#1
Hi, I'm currently using litespeed enterprise edition,
my server is a core2duo with 4GB of RAM.
There seems to be a problem with lsphp as it utilizes more than %90 of the cpu most of the time. I use APC for caching and adodb for mysql connections.
Here is a screenshot from a top command.



Does anyone have any suggestions to fix this issue, if required I can give any further information.
 
M

mistwang

LiteSpeed Staff
#2
#2
Does it happen all the time, or once for a while?
You can strace the lsphp processes, see what they are doing. If something really is wrong, it will be a PHP bug, maybe in a PHP module.
Please show us the strace result of one of the lsphp process.
 
C

cumhuronat

Member
#3
#3
yes it happens all the time, I'm using the php provided by litespeed installation.
I straced a lsphp process using %20 of total cpu.
The result was nothing:

shel:/opt/lsws # strace -p 32308
Process 32308 attached - interrupt to quit

My configure string is as following:
./configure --with-litespeed --with-config-file-path=../php --with-mysql=../mysql --with-zlib --with-zlib-dir=.. --with-gd --with-jpeg-dir=.. --with-png-dir=.. --enable-shmop --enable-track-vars --enable-sockets --enable-sysvsem --enable-sysvshm --enable-magic-quotes
 
M

mistwang

LiteSpeed Staff
#4
#4
Looks like there is a infinite loop there. Please try gdb on one of lsphp process.
Code: 
gdb
attach <pid_of_lsphp_process>
bt
q
[/gdb]
It will show the location where lsphp stuck. 
You can also try build your own lsphp binary, the pre-built binary does not have anything like adodb builtin.
 
C

cumhuronat

Member
#5
#5
seems like there is an issue with apc right?

Code: 
(gdb) attach 2606
Attaching to process 2606
Reading symbols from /opt/lsws/fcgi-bin/lsphp...(no debugging symbols found)...done.
Using host libthread_db library "/lib/libthread_db.so.1".
Error while mapping shared library sections:
./../lib/apc.so: No such file or directory.
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libm.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Error while reading shared library symbols:
./../lib/apc.so: No such file or directory.
Reading symbols from /lib/libpthread.so.0...
(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread -1210100032 (LWP 2606)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Error while reading shared library symbols:
./../lib/apc.so: No such file or directory.
Failed to read a valid object file image from memory.
0x08110989 in ap_php_vsnprintf ()
(gdb) bt
#0  0x08110989 in ap_php_vsnprintf ()
#1  0x081110ee in vspprintf ()
#2  0x08111138 in spprintf ()
#3  0x0810cea1 in php_html_puts ()
#4  0x081324e2 in zend_error ()
#5  0x0813714f in zend_fetch_resource ()
#6  0x080c1246 in zif_feof ()
#7  0x0814521f in execute ()
#8  0x081453c3 in execute ()
#9  0x081453c3 in execute ()
#10 0x081453c3 in execute ()
#11 0x081453c3 in execute ()
#12 0x081453c3 in execute ()
#13 0x0814997d in execute ()
#14 0x08132836 in zend_execute_scripts ()
#15 0x0810e9b2 in php_execute_script ()
#16 0x0814bf16 in execute ()
#17 0x0814c042 in execute ()
#18 0x0814c775 in main ()
 
M

mistwang

LiteSpeed Staff
#6
#6
No, it is not a problem with APC.
Looks like something wrong with the php_html_puts().
Is this a fresh install of LSWS 3.2.4? or upgrade from earlier version?
What OS are you using? Can you try building your own lsphp binary?
 
C

cumhuronat

Member
#7
#7
uname -a:
Linux zulot 2.6.18.2-34-bigsmp #1 SMP Mon Nov 27 11:46:27 UTC 2006 i686 i686 i386 GNU/Linux

its an upgrade from the free version, I can compile the php myself if it will fix the issue
 
M

mistwang

LiteSpeed Staff
#8
#8
Please do

diff lsws/admin/fcgi-bin/admin_php lsws/fcgi-bin/lsphp

if they are different, then overwrite lsws/fcgi-bin/lsphp with admin_php, lsphp is a earlier build. If they are identical, you have to try building your own lsphp binary.
 
C

cumhuronat

Member
#9
#9
the diff showed no difference, so I made a fresh compile,

When i straced again a lsphp process:

zulot:/usr/local/lib/php/extensions/no-debug-non-zts-20060613 # strace -p 12586
Process 12586 attached - interrupt to quit
select(2, [1], NULL, NULL, {0, 936000}) = 0 (Timeout)
kill(12585, SIG_0) = 0
select(2, [1], NULL, NULL, {1, 0}) = 0 (Timeout)
kill(12585, SIG_0) = 0
select(2, [1], NULL, NULL, {1, 0}) = 0 (Timeout)
kill(12585, SIG_0) = 0
select(2, [1], NULL, NULL, {1, 0}) = 0 (Timeout)
kill(12585, SIG_0) = 0
select(2, [1], NULL, NULL, {1, 0}) = 0 (Timeout)
kill(12585, SIG_0) = 0
select(2, [1], NULL, NULL, {1, 0}) = 0 (Timeout)
kill(12585, SIG_0) = 0
select(2, [1], NULL, NULL, {1, 0}) = 0 (Timeout)
kill(12585, SIG_0) = 0
select(2, [1], NULL, NULL, {1, 0}) = 0 (Timeout)
kill(12585, SIG_0) = 0
 
Last edited:
C

cumhuronat

Member
#10
#10
another strace result:

writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\4\0\224 \0\0", 8}, {"<br />\n<b>Warning</b>: feof(): "..., 8332}], 2) = 8340
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2 = 128
 
C

cumhuronat

Member
#11
#11
The latest gdb result:

This GDB was configured as "i586-suse-linux".
(gdb) attach 12586
Attaching to process 12586
Reading symbols from /opt/lsws/fcgi-bin/lsphp...done.
Using host libthread_db library "/lib/libthread_db.so.1".
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libmysqlclient.so.15...done.
Loaded symbols for /usr/lib/libmysqlclient.so.15
Reading symbols from /usr/lib/libpng12.so.0...done.
Loaded symbols for /usr/lib/libpng12.so.0
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/libjpeg.so.62...done.
Loaded symbols for /usr/lib/libjpeg.so.62
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread -1213348160 (LWP 12586)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-zts-20060613/eaccelerator.so...done.
Loaded symbols for ./eaccelerator.so
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Failed to read a valid object file image from memory.
0x0825ffdc in vspprintf (pbuf=0xbffd5358, max_len=1024, format=0x83de5b2 "%s%s(): supplied argument is not a valid %s resource",
ap=0xbffd53ec "
 
C

cumhuronat

Member
#12
#12
i upgraded to version 3.3 and the issue seems to remain..
I will make a cronjob that will kilall lsphp every minute :)
Lsphp processes are going well for some 10 seconds and then the cpu usage tops. Sometimes just one of them, some times all of them..
By the way I'm very pleased with litespeed server and I have 3 servers with it, and this occured for the first time.
 
Last edited:
M

mistwang

LiteSpeed Staff
#13
#13
Looks like PHP internal is looping when feof() is received when reading a request body. We will dig into that when we get a chance.

cumhuronat said:
another strace result:

writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\4\0\224 \0\0", 8}, {"<br />\n<b>Warning</b>: feof(): "..., 8332}], 2) = 8340
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2) = 129
writev(1, [{"LS\6\0\200\0\0\0", 8}, {"PHP Warning: feof(): supplied a"..., 120}], 2) = 128
writev(1, [{"LS\6\0\201\0\0\0", 8}, {"PHP Warning: fgets(): supplied "..., 121}], 2 = 128
Click to expand...
 
You must log in or register to reply here.
Top