Performance

Stefan Soellner · Dec 2, 2007

Hi all,

recently I've upgraded my dedicated server from a single core CPU to a dual core system. After doing some performance tests I ask myself if LSWS ENT #1 CPU is the bottleneck? And what do you think about failed requests?

If you need more detail about the configs please advise.

Thank you.

System:
o AMD 5600 X2, 2GB RAM
o Debian Etch x64

Software:
o nginx 0.5.33 ; 2 worker instances ; php-fcgi 5.2.5 via spawn-fcgi (5 fcgi children), xcache 1.2.1
o LSWS 3.3.1 ENT #1 CPU ; php 5.2.5 via lsapi, default lsapi/php config, xcache 1.2.1

Code:

PHP/Info 
ab -n 10000 -c [concurrent level] http://[ip]:[port]/phpinfo.php 

<HTML>
<BODY>
   <?php phpinfo(); ?>
</BODY>
</HTML>

Server Software:        nginx/0.5.33                                       
Server Hostname:        <del>
Server Port:            80

Document Path:          /phpinfo.php
Document Length:        34572 bytes

Concurrency Level:      100
Time taken for tests:   7.341 seconds
Complete requests:      10000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      346940000 bytes
HTML transferred:       345720000 bytes
Requests per second:    1362.21 [#/sec] (mean)
Time per request:       73.41 [ms] (mean)
Time per request:       0.73 [ms] (mean, across all concurrent requests)
Transfer rate:          47260.59 [Kbytes/sec] received

Code:

Server Software:        LiteSpeed                                          
Server Hostname:        <del>
Server Port:            81

Document Path:          /phpinfo.php
Document Length:        32245 bytes

Concurrency Level:      100
Time taken for tests:   12.660 seconds
Complete requests:      10000
Failed requests:        3
   (Connect: 0, Length: 3, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      323685462 bytes
HTML transferred:       322353265 bytes
Requests per second:    789.89 [#/sec] (mean)
Time per request:       126.60 [ms] (mean)
Time per request:       1.27 [ms] (mean, across all concurrent requests)
Transfer rate:          25567.57 [Kbytes/sec] received

I have tested Apache 2.2.6 / mod_fcgid also. But the load factor was unacceptable.

mistwang · Dec 2, 2007

The failed requests in LiteSpeed test is due the difference in response body size, the request should be successful. I have been seeing that for long time during our internal tests.

The performance difference between LiteSpeed result and nginx result is mainly because LiteSpeed start lsphp dynamically, while nginx start FastCGI PHP processes statically. Starting new PHP processes is expensive, if it happens during the test, it will affect the test results.

Please post your nginx configuration and litespeed PHP configuration, I need to make sure you have compared apple to apple. Are you using the latest 3.3.1 release?

Stefan Soellner · Dec 2, 2007

Hi,

nginx.conf

Code:

user nginx;
worker_processes 2;
error_log logs/nginx.log;
pid logs/nginx.pid;

events {
  worker_connections  1024;
}

http {
        include       conf/mime.types;
        default_type  application/octet-stream;

        sendfile        on;

        tcp_nopush      on;
        tcp_nodelay     off;

        keepalive_timeout  65;

        gzip  on;
        gzip_http_version 1.0;
        gzip_comp_level 1;
        gzip_proxied any;
        gzip_types      text/plain text/html text/css application/x-javascript text/xml application/xml
                        application/xml+rss text/javascript;

        server {
                listen       <del>:80;
                server_name  <del>;
                                                   
        access_log  off;

        location / {
                root   /home/soellner/www/public_html/;
                index  index.html index.php;
        }

        #error_page  404              /404.html;


        location ~ \.php$ {
                include conf/fastcgi.conf;
                fastcgi_pass  127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param  SCRIPT_FILENAME  /home/soellner/www/public_html$fastcgi_script_name;
        }

    }
}

fastcgi.conf

Code:

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

Extract of FCGI Startup

Code:

#!/bin/sh

SPAWNFCGI="/home/soellner/www/bin/spawn-fcgi"
FCGIPROGRAM="/home/soellner/www/bin/php-fcgi"
## FCGISOCKET="/tmp/soellner-fcgi.sock"
FCGIIP=127.0.0.1
FCGIPORT=9000
PHPRC="/home/soellner/www/conf/"

PHP_FCGI_CHILDREN=4
PHP_FCGI_MAX_REQUESTS=1000
FCGI_WEB_SERVER_ADDRS="127.0.0.1"
ALLOWED_ENV="PATH USER"

USERID=stesoell
GROUPID=soellner

LSWS PHP LSAPI

Code:

Name	 	        soellner_php
Address	        uds://home/soellner/www/tmp/lsphp.sock
Notes	 	        Not Set
Max Connections	20

Environment
PHP_LSAPI_MAX_REQUESTS =500
PHP_LSAPI_CHILDREN         =20

Initial Request Timeout (secs) 180
Retry Timeout (secs) 0
Persistent Connection Not Set
Connection Keepalive Timeout Not Set
Response Buffering No
Auto Start	Yes
Command $VH_ROOT/bin/php-lsws -c $VH_ROOT/conf/php.ini
Back Log 100
Instances 1
Run On Start Up Not Set
Max Idle Time Not Set
Priority 0
Memory Soft Limit (bytes) 250M
Memory Hard Limit (bytes) 300M
Process Soft Limit 200
Process Hard Limit 200

mistwang · Dec 2, 2007

OK, I recommend the following changes to soellner_php configurations:

Code:

Max Connections    4

Environment
PHP_LSAPI_MAX_REQUESTS =1000
PHP_LSAPI_CHILDREN     =4
LSAPI_AVOID_FORK = 1

Run On Start Up  Yes

Make sure the php.ini used for litespeed and nginux has the same value.
You can retry your test. LiteSpeed may still need to start children process on demand for the first run, so, run it a couple times.

Stefan Soellner · Dec 2, 2007

Ok. New results. LSWS now performs better.

changed LSAPI PHP as suggested ...
Max Connections 4
Environment
PHP_LSAPI_MAX_REQUESTS1000
PHP_LSAPI_CHILDREN=4
LSAPI_AVOID_FORK=1
Run On Start Up Yes

Code:

Server Software:        LiteSpeed                                          
Server Hostname:        <del>
Server Port:            81

Document Path:          /test/phpinfo.php
Document Length:        34309 bytes

Concurrency Level:      100
Time taken for tests:   9.665 seconds
Complete requests:      10000
Failed requests:        18
   (Connect: 0, Length: 18, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      344223049 bytes
HTML transferred:       342815478 bytes
Requests per second:    1034.66 [#/sec] (mean)
Time per request:       96.65 [ms] (mean)
Time per request:       0.97 [ms] (mean, across all concurrent requests)
Transfer rate:          35615.42 [Kbytes/sec] received

(Load >1)

nginx: no change in config ...

Code:

Server Software:        nginx/0.5.33                                       
Server Hostname:        <del>
Server Port:            80

Document Path:          /test/phpinfo.php
Document Length:        36575 bytes

Concurrency Level:      100
Time taken for tests:   7.976 seconds
Complete requests:      10000
Failed requests:        9
   (Connect: 0, Length: 9, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      366969955 bytes
HTML transferred:       365749955 bytes
Requests per second:    1253.76 [#/sec] (mean)
Time per request:       79.76 [ms] (mean)
Time per request:       0.80 [ms] (mean, across all concurrent requests)
Transfer rate:          46009.27 [Kbytes/sec] received

(Load >2)

mistwang · Dec 2, 2007

We will setup this in our lab to compare those two.
Which version of PHP are you using for the test?
What is the configuration for building fastcgi PHP binary for nginx?
You run "ab" from the same server?

If there is anything else you would like us know about you setup, please let us know.

Stefan Soellner · Dec 3, 2007

php 5.2.5 for nginx with xcache 1.2.1

Code:

#! /bin/sh
#
# Created by configure

CFLAGS='-O3 -fno-exceptions -fomit-frame-pointer -march=k8 -msse3 -pipe' \
CXXFLAGS='-O3 -felide-constructors -fno-exceptions -fno-rtti -fomit-frame-pointer -march=k8 -msse3 -pipe' \
'./configure' \
'--prefix=/usr/local/php' \
'--with-php-config=../conf' \
'--enable-fastcgi' \
'--enable-force-cgi-redirect' \
'--enable-discard-path' \
'--enable-sysvmsg' \
'--enable-sysvsem' \
'--enable-sysvshm' \
'--enable-mbstring' \
'--disable-pdo' \
'--with-mysqli=/usr/local/mysql/bin/mysql_config' \
'--without-sqlite' \
'--with-pear' \
"$@"

php 5.2.5 for lsws lsapi (LSWS 3.3.1 Ent #1CPU) with xcache 1.2.1

Code:

#! /bin/sh
#
# Created by configure

CFLAGS='-O3 -fno-exceptions -fomit-frame-pointer -march=k8 -msse3 -pipe' \
CXXFLAGS='-O3 -felide-constructors -fno-exceptions -fno-rtti -fomit-frame-pointer -march=k8 -msse3 -pipe' \
'./configure' \
'--prefix=/usr/local/php' \
'--with-litespeed' \
'--with-php-config=../conf' \
'--enable-fastcgi' \
'--enable-force-cgi-redirect' \
'--enable-discard-path' \
'--enable-sysvmsg' \
'--enable-sysvsem' \
'--enable-sysvshm' \
'--enable-mbstring' \
'--disable-pdo' \
'--with-mysqli=/usr/local/mysql/bin/mysql_config' \
'--without-sqlite' \
'--with-pear' \
"$@"

php.ini

http://soellner.info/test/php.ini.txt

ApacheBench

ab Version 1.3d <$Revision: 1.73 $> run on the same server.

mistwang · Dec 3, 2007

Thanks for the information.
Can you try nginx with "worker_processes 1;"

It will be a much closer match to the 1-CPU license.

Stefan Soellner · Dec 3, 2007

Damn. I had to re-bench nginx/1worker cause CPU was in powersafe mode.

nginx, 1 worker

Stefan Soellner · Dec 3, 2007

New results above. Sorry for trouble

ffeingol · Dec 3, 2007

What does the LSAPI_AVOID_FORK=1 do?

Frank

mistwang · Dec 3, 2007

Can please double check if the same php.ini is used? LiteSpeed will use $VH_ROOT/conf/php.ini if available.

Stefan Soellner · Dec 3, 2007

Yep, same php.ini is used.

mistwang · Dec 3, 2007

OK, here is our test results,

Hardware: dual opteron 250
OS: OpenSuse 10.3 x86_64
kernel: 2.6.22.12-0.1-default

LSWS 3.3.1 Enterprise with 1-CPU license

typical LiteSpeed result

typical nginx result:

I have done multiple run for each server, those are the typical results. Sometime, the result is worse when, somehow, CPU idle are high during the test, it happens to both nginux and LSWS. I think it is a kernel scheduler issue.

I found that even with "LSAPI_AVOID_FORK" set, lsphp still kills idle children process, so, I added another environment variable "LSAPI_MAX_IDLE_CHILDREN=4" to lsphp external app configuration.

Another issue I encountered is the confliction of xcache between lsphp and fastcgi php if I keep both running at the same time, both try to lock /tmp/xcache, got some errors, so I have to make sure only have one PHP backend running. Please check if there is any error message in lsws/logs/stderr.log, lsws/logs/error.log.

Maybe you can try the test without xcache in the mix. it would not affect the result much for such a simple script.

LSAPI PHP has a few advantage over nginx + fcgi php:

optimized IPC protocol, LSAPI has the minimum overhead in handling the IPC procotol.
Persistent connection between web server and backend app server. nginx + fcgi php does not use persistent connections.
Unix domain socket vs TCP socket, unix domain socket is faster.

Due to above advantages, is not likely LSAPI lose to a fcgi php backend. However, I do notice that not all CPU cycles has been fully used during a test which result in bad benchmark results. Maybe that happen more frequently on your test environment.

The test do expose one weakness of LSWS when I switch LSWS to use fcgi PHP backend, the result is not as good, and consistently, only half the CPU cycles has been used during the test. I guess it is about time to fine tune our FCGI support as we have been focused on LSAPI for a while.

Stefan Soellner · Dec 4, 2007

Thanks for your effort, George.

I did a re-bench without xcache and using an unix socket for external fcgi.
I added also max_idle_children to lsapi environment.

Code:

Server Software:        LiteSpeed                                          
Server Hostname:        <del>
Server Port:            81

Document Path:          /test/phpinfo.php
Document Length:        31540 bytes

Concurrency Level:      100
Time taken for tests:   9.724 seconds
Complete requests:      10000
Failed requests:        26
   (Connect: 0, Length: 26, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      316266494 bytes
HTML transferred:       314888418 bytes
Requests per second:    1028.38 [#/sec] (mean)
Time per request:       97.24 [ms] (mean)
Time per request:       0.97 [ms] (mean, across all concurrent requests)
Transfer rate:          32524.32 [Kbytes/sec] received

Code:

Server Software:        nginx/0.5.33                                       
Server Hostname:        <del>
Server Port:            80

Document Path:          /test/phpinfo.php
Document Length:        33877 bytes

Concurrency Level:      100
Time taken for tests:   7.104 seconds
Complete requests:      10000
Failed requests:        11
   (Connect: 0, Length: 11, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      339989945 bytes
HTML transferred:       338769945 bytes
Requests per second:    1407.66 [#/sec] (mean)
Time per request:       71.04 [ms] (mean)
Time per request:       0.71 [ms] (mean, across all concurrent requests)
Transfer rate:          47858.95 [Kbytes/sec] received

There are some entries in the error.log I don't know :
http://soellner.info/test/lsws-error.txt

mistwang · Dec 4, 2007

Using unix domain socket with nginx is not reliable. better not using it.

I think it is related to the kernel, somehow, your Linux kernel scheduler assign most process to one CPU when persistent connection is used. Please check the CPU idle time during the test.
You can try increase "max connection" and "PHP_LSAPI_CHILDREN" to 10, see if it helps.

You can apply a trial license, it is equivalent to a 2-CPU license.

Which version of kernel are you using? Linux distribution?

We will test it a little bit more.

Stefan Soellner · Dec 4, 2007

Debian Etch stable; Kernel 2.6.18-5-amd64 #1 SMP

ts77 · Dec 4, 2007

mistwang said:

Yes please

.
Only using fcgi here for being able to use the distributions packages instead of building my own.

mistwang · Dec 5, 2007

ts77 said:

I think it should be a better idea to convert to LSAPI PHP, even with your load balancing setup, LSAPI PHP has everything you need.

ts77 · Dec 5, 2007

I agree, it would be a better idea but with too many servers the manual processes take up a lot of time - and yes, I know that I could do this by now through the webinterface.

Performance

Member

LiteSpeed Staff

Member

LiteSpeed Staff

Member

LiteSpeed Staff

Member

LiteSpeed Staff

Member

Member

Well-Known Member

LiteSpeed Staff

Member

LiteSpeed Staff

Member

LiteSpeed Staff

Member

Well-Known Member

LiteSpeed Staff

Well-Known Member