Optimization request

MentaL

Well-Known Member
#1
Hi,

I'm running a large VB3 forum (possibly migrating to VB4) and looking to tweak Litespeed as much as possible to reduce load. Can you please take a look over my configurations and offer as much advice as possible. I'm using XCACHE for opcode and Memcache(d) for the forum DB.

top - 11:14:42 up 3 days, 13:00, 1 user, load average: 6.26, 7.13, 6.92
Tasks: 142 total, 5 running, 137 sleeping, 0 stopped, 0 zombie
Cpu(s): 59.2%us, 3.5%sy, 0.0%ni, 36.6%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 8181048k total, 7147468k used, 1033580k free, 205656k buffers
Swap: 16771576k total, 168k used, 16771408k free, 5431996k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20069 website 16 0 267m 106m 75m R 64.2 1.3 1:29.07 lsphp5
20059 website 15 0 269m 106m 73m R 39.3 1.3 1:32.17 lsphp5
29025 mysql 10 -5 835m 639m 4084 S 30.9 8.0 345:43.36 mysqld
20060 website 15 0 270m 108m 74m R 28.3 1.4 1:23.39 lsphp5
20279 website 16 0 271m 71m 36m S 27.3 0.9 0:19.56 lsphp5
20067 website 16 0 271m 112m 77m S 22.3 1.4 1:25.14 lsphp5
20338 website 16 0 266m 57m 24m R 19.0 0.7 0:04.68 lsphp5
20062 website 15 0 281m 117m 72m S 17.6 1.5 1:19.76 lsphp5
4016 nobody 15 0 98000 30m 780 S 2.0 0.4 94:43.51 memcached
Litespeed version: 4.1.2
PHP Version: 5.1.17

Memory I/O Buffer ; 60m
Max connections; 2000
Connection timeout; 60
Max Keep-Alive Requests; 1000
smart keepalive; yes
Keep-Alive Timeout (secs); 7

Lsphp5

Max connections; 100
PHP_LSAPI_MAX_REQUESTS=2000
PHP_LSAPI_CHILDREN=100
Initial request timeout; 60
connection keepalive timeout; 30
max idle time; 10
priority; 0
Memory Soft Limit (bytes); 640m;
Memory Hard Limit (bytes); 720m
Process Soft Limit; 800m
Process Hard Limit; 1000m


Thanks guys
 

MentaL

Well-Known Member
#5
Having trouble;
*path /tmp/diskcache is not readable

Regards.

/edit
chmod -c 0775 diskcache

fixed.

/edit2

my virtual hosts are blank?
 
Last edited:

webizen

Well-Known Member
#6
http://www.litespeedtech.com/suppor...litespeed:cache#how_to_enable_litespeed_cache


3. For imported Apache vhost: Cache can be turned on/off with “CacheEnable” , “CacheDisable” directives in Apache config file. Nothing needs to be done for the cache policy. It will assume the same default settings at LSWS Server level. Rewrite rules can be placed in httpd.conf or in-directory .htaccess file as shown in step 5 below. Note: Apache mod_cache directives CacheIgnoreCacheControl, CacheMaxExpire also can be used in the Apache config file as well as .htaccess to fine tune cache policy.
 

cmanns

Well-Known Member
#7
And there is no conflicts with Xcache and Memcache correct?
However Memcached's useful for PHP clustered enviorments. Ever seen a forum run off over 5 dual/quad core PHP (Only) systems?

Grab XCache, update your PHP and get to tuning. Paying for a webserver doesn't mean performance.

If you need it done quick and reliable feel free to hit me up. I work with forums and blogs doing dozens of php req/sec, if you let me know your sites work load like if theres lots of guests- as NiteWave said enable LSCache properly.

If theres lots of members online, well might be your best bet to hit me up or pay LiteSpeed for some support :)

You can send me details of traffic stats and the domain and I can give you a quote of what all I can do. Already appears your PHP's eating wayy too much cpu for vb3. We specialize in mostly vb/ipb/wordpress high performance tuning and hosting for over 5 years, so yeah I know what I'm talking about.

http://uploadpla.net/files/6740_7bp6n/Screen shot 2011-07-16 at 5.40.14 AM.png

Theres a screenie I took of our huge shared box with over 16gB of SQL databases, mixed forum types. The 26/10% cpu used are old messed up WordPress install's that'll be fixed by the customers soon. Theres a lot you can do when it comes to PHP tuning.

Good luck m8
 

MentaL

Well-Known Member
#8
Well the DB has millions of posts, and raw text is around 6gb in size with attachments its allot more. Online users is over 2000. I'll look at my options shortly. I'll more than likely reconfigure allot.

Regards.
 

cmanns

Well-Known Member
#9
Well the DB has millions of posts, and raw text is around 6gb in size with attachments its allot more. Online users is over 2000. I'll look at my options shortly. I'll more than likely reconfigure allot.

Regards.
LSCache won't much at all help if it's members online is like that.

You'll need atleast 8 or so cores and properly tuned php. If php is using 2% cpu or so avg for your site, that leaves almost 400 php processes possible. If your php eating 30-50%, lets say 50% on avg maybe able to do 15 or so processes. This is considering if MySQL is on separate system, or you had 8 cores.

for a ipb forum which php runs alil slower cpu wise when tuned right then vb3 back then, for 1,500-2,500 or so users we had to setup a php cluster for client. There was about 3-4 php servers to accomidate that load, this was a few years ago though you can prob get by with 1 spare php server, set the main one for the uploads, rsync the data, profit :)

However by your connection/in processing graph I see a major lack of tuning for your site, prob lacking on cores for php to scale on too :p
 

cmanns

Well-Known Member
#11
Running on 4 cores can upgrade to 8 if required (sponsored). VBSeo seems to like to eat resources also.
Shouldn't.

Had a vb3 with seo and such with few million posts/thousands users, mostly like 100-500 online tops I'd say, but guests would get the users in the thousands. PHP was normally around 6%-18% roughly.

I'd not touch eaccelerator with a 10 foot pole, enable xcache in vb config might help (vb approves of xcache) I think we tried memcached, it sometimes adds a few 100ms overhead like MySQL over lan adds 100-150 or so normally. The longer php takes to execute, the more cputime used, the less you'll beable to push.

I've worked with php content for over 5 years in high demand envs...

You may also want to run LSPHP as child and such

Since the admin panel was handy try XCache, enable it in config, maybe disable MySQL persistent, disable suexec, add realpath cache larger in php.ini, and try this php config in lsws admin panel.

I do help people free, mostly in IRC's but since I happen to be in the lsws admin panel this may be of use, I've modified it slightly to your needs though we run something similar in a high demand shared enviorment with maybe 1 minute downtime a month while pushing a few million requests a day mostly php, so you may find it works nice compared to now settings. Also take the no suexec in consideration to XCache functions 100%

It's in external app -> lsphp5 -> Environment
Code:
PHP_LSAPI_MAX_REQUESTS=2000
PHP_LSAPI_CHILDREN=65
LSAPI_ACCEPT_NOTIFY=1
LSAPI_AVOID_FORK=1
LSAPI_CHILDREN=0
LSAPI_MAX_REQS=2000
LSAPI_EXTRA_CHILDREN=25
If you have memory leaks in PHP make sure to lower max requests on both lines, if you can get PHP down to roughly 10%, slowly raise lsapi_children till cpu's all being used good.

So with 400% total, you prob got mysql untuned and what not doing 100-150%, so factor in the cpu your php is using and thats how many children you can push without overloading, maybe set backlog to 100-150 in that same page.

Give XCache about 32mb to 16mb shm. Since so many processes spawned it's sorta useless to give it too much, it's quick on the fly. Also look into giving php/mysql /dev/shm for tmp's if you can afford the ram. If your disks are bottlenecked and mysql bottlenecked, theres your issue php wont run right.

Almost every site we pump executes in under .02 seconds and delivers in 100ms-500ms on avg depending on your latency of course but in general around that low, if your vb's executing like .1 and delivering around 800-1s+ you def got tuning issues and what not.

I've found vB3 to use very little resources, sometimes .6-2% cpu with that 1000+ user site and more. Sadly the vb and other forum's benchmarks I'm looking to do is in the future not already done or I'd show you that these scripts can run so low utilization.

LSPHP doesn't run multiple requests per process so doing like 65% and what not is just totally eating your shit up, hell strace the php process old trick, sometimes you'll find it just looking for directories or finding missing files. pro hint, the realpath cache and high php req's helps that, we use 1000 for our shared to avoid random client's having a memory leak, then again 12gb ram and the ability to add more cores on the go helps with our setup.

Good luck:)
 

cmanns

Well-Known Member
#12
Also another thing when posting for optimization help which LS charges for (I've been turned done for some rather simple stuff) atleast give the community who donates their time to help good info

Try Munin/Cacti statistical graphs, vmstat, top (-* you have many different options to show info), iostat, mysql statistics.

and to go back on your MySQL
29025 mysql 10 -5 835m 639m 4084 S 30.9 8.0 345:43.36 mysqld
I'd def say PHP's screwing you up then, def look into stracing it then if MySQL isn't eating excessive cpu.
 

MentaL

Well-Known Member
#13
Vbseo is being cached via xcache and the VB datastore engine uses Memcached. I'll look into your suggestions shortly.

Thanks.

MY.CNF
Code:
[mysqld]

local-infile=0
back_log = 75
skip-innodb
skip-name-resolve
max_connections = 2000
key_buffer = 384M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1800
thread_cache_size = 384
wait_timeout = 180
connect_timeout = 10
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 64M
max_connect_errors = 1000
read_rnd_buffer_size = 524288
bulk_insert_buffer_size = 8M
query_cache_limit = 4M
query_cache_size =128M
query_cache_type = 1
query_prealloc_size = 65536
query_alloc_block_size = 131072
default-storage-engine = MyISAM

[mysqld_safe]
nice = -5
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
Xcache

Code:
xcache.shm_scheme = "mmap"
xcache.size = 128M
xcache.count = 8
xcache.slots = 8K
xcache.ttl = 7200
xcache.gc_interval = 300

xcache.var_size = 32M
xcache.var_count = 8
xcache.var_slots = 8K
xcache.var_ttl = 3600
xcache.var_maxttl = 14400
xcache.var_gc_interval = 300

xcache.test = Off
xcache.readonly_protection = Off
xcache.mmap_path = "/dev/zero"
xcache.coredump_directory = ""
xcache.cacher = On
xcache.stat = On
xcache.optimizer = Off

;[xcache.coverager]
xcache.coverager = Off
xcache.coveragedump_directory =
Regards.

#Update

It's more than likely also one of the plugins I use causing leaks hence why a VB4 upgrade maybe beneficial to reduce some unneeded plugins.
 
Last edited:

cmanns

Well-Known Member
#14
Vbseo is being cached via xcache and the VB datastore engine uses Memcached. I'll look into your suggestions shortly.

Thanks.
tbh I'd ditch memcached as your not clustering php and go with more mem for XCache, use it for vb datastore if it works right. If I recall it didn't work too great for us but wasn't needed when php's tuned.


However look into /dev/shm mainly and strace which will lead you up the horses poop to the solution. It might take a bit, it can tell you when MySQL is lagging up, etc - It's how we find fubar'd wordpress's which happens naturally :D

Like maybe /dev/shm for your temp dir if not shared, boom sessions and such not being wrote to disk...lsws's default tmp will be cached too. We *may* utilize something like that for more performance :rolleyes:

In most cases when you can tell memcached is not needed is disable it, mysql not much higher? Then your just wasting tcp cons to a similar daemon besides MySQL per each connection. If XCache functions for you, that's built into php. In most cases I'd say the on the go opcode benefits of XCache and the other php tunes help more then anything else cpu wise after years of finding the best.

In your case of php being the hog you want it doing as little work as possible :p
 

cmanns

Well-Known Member
#15
Vbseo is being cached via xcache and the VB datastore engine uses Memcached. I'll look into your suggestions shortly.

Thanks.

MY.CNF
Code:
[mysqld]

local-infile=0
back_log = 75
skip-innodb
skip-name-resolve
max_connections = 1000
key_buffer = 384M
myisam_sort_buffer_size = 64M
join_buffer_size = 16M
read_buffer_size = 2M
sort_buffer_size = 8M
table_cache = 2000
thread_cache_size = 512
wait_timeout = 120
connect_timeout = 5
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 64M
max_connect_errors = 1000
read_rnd_buffer_size = 524288
bulk_insert_buffer_size = 8M
query_cache_limit = 3M
query_cache_size = 128M
query_cache_type = 2 #try cache as 2, with graphs yes it'll make barely any #cache's but tbh we switch it 1-2 and notice not much of anything, maybe #less random spikes when set to 2 or off.
query_prealloc_size = 65536
query_alloc_block_size = 131072
default-storage-engine = MyISAM

[mysqld_safe]
nice = -5
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
Xcache

Code:
xcache.shm_scheme = "mmap"
xcache.size = 64M
xcache.count = 8
xcache.slots = 16K
xcache.ttl = 7200
xcache.gc_interval = 1440

xcache.var_size = 64M
xcache.var_count = 8
xcache.var_slots = 16K
xcache.var_ttl = 3600
xcache.var_maxttl = 14400
xcache.var_gc_interval = 800

xcache.test = Off
xcache.readonly_protection = Off
xcache.mmap_path = "/dev/shm" #try /dev/shm if php can write
xcache.coredump_directory = ""
xcache.cacher = On
xcache.stat = Off #try with this off, may benefit
xcache.optimizer = Off

;[xcache.coverager]
xcache.coverager = Off
xcache.coveragedump_directory =
Regards.

#Update

It's more than likely also one of the plugins I use causing leaks hence why a VB4 upgrade maybe beneficial to reduce some unneeded plugins.
without any mysql stats or anything can't say much but try those configs (read comments if you're a copy paster)
 
Top