Terrible Lag

M

MisterNinja

Member
#1
#1
Ever since I've installed LiteSpeed I've been trying to optimize it.



I've gotten to a point where I think found the problem: Is the number of requests vs the number being processed off or something? Shouldn't more requests be processed, what could be the reason for so few requests actually being handled?

I have a 2 core server with 24 threads and 32 gigabytes of ram, the response is TOO slow for this server. I even enabled caching for guests and the lag spikes still show up.
 
M

MisterNinja

Member
#3
#3
With 24 threads I should theoretically be permitted to have a load of 24. That is also what is bugging me, the load is only 6 right now, yet the site loads extremely slow. It isn't even utilizing the full server.

Code: 
top - 22:47:20 up 176 days, 21:02,  2 users,  load average: 6.00, 6.61, 6.95
Tasks: 466 total,   6 running, 383 sleeping,  74 stopped,   3 zombie
Cpu(s): 22.4%us,  0.9%sy,  0.0%ni, 72.1%id,  4.4%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  32955440k total, 15059180k used, 17896260k free,   282628k buffers
Swap: 34996216k total,  1735756k used, 33260460k free,  4220196k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
14118 mysql     10  -5 15.9g 8.0g 6288 S 145.1 25.3 219:18.02 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysql/error.log --open-files-limit=16384 --pid-
10051 nobody     6 -10  108m  34m 4376 S 58.0  0.1   1:22.32 lsphp5
10182 nobody     6 -10  123m  48m 4300 R 38.1  0.2   1:03.39 lsphp5:/xxxxxxxxxxxx/vbseo.php
10585 nobody     6 -10  123m  48m 4216 R 29.5  0.2   0:13.00 lsphp5:/xxxxxxxxxxxx/vbseo.php
10540 nobody     6 -10  115m  40m 4308 R 25.2  0.1   0:27.89 lsphp5:/xxxxxxxxxxxx/vbseo.php
10677 nobody     5 -10  114m  43m 3548 S 24.2  0.1   0:00.73 lsphp5
10680 nobody     6 -10  110m  38m 3560 S 23.2  0.1   0:00.70 lsphp5
10678 nobody     6 -10  106m  35m 3560 S 19.9  0.1   0:00.60 lsphp5
10624 nobody     5 -10  110m  39m 3828 S 18.6  0.1   0:05.32 lsphp5
10682 nobody     5 -10  108m  36m 3544 S 18.6  0.1   0:00.56 lsphp5
10545 nobody     5 -10  122m  48m 4188 S 17.9  0.1   0:28.89 lsphp5
10676 nobody     5 -10  105m  33m 3544 S 17.2  0.1   0:00.52 lsphp5
10686 nobody     6 -10  126m  54m 3552 R 14.6  0.2   0:00.44 lsphp5:/xxxxxxxxxxxx/vbseo.php
10656 nobody     5 -10  105m  34m 3780 S 14.2  0.1   0:00.91 lsphp5
10672 nobody     5 -10  115m  43m 3776 S 14.2  0.1   0:00.43 lsphp5
10675 nobody     7 -10  103m  31m 3560 S 14.2  0.1   0:00.43 lsphp5
10679 nobody     5 -10  104m  32m 3568 S 12.3  0.1   0:00.37 lsphp5
10588 nobody     6 -10  118m  45m 4148 R 11.6  0.1   0:18.26 lsphp5:/xxxxxxxxxxxx/vbseo.php
10532 nobody     5 -10  111m  36m 4292 S  8.9  0.1   0:14.93 lsphp5:/xxxxxxxxxxxx/vbseo.php
10681 nobody     5 -10  100m  28m 3528 S  7.0  0.1   0:00.21 lsphp5
10601 nobody     5 -10  109m  36m 4084 S  5.0  0.1   0:11.09 lsphp5
10625 nobody     5 -10  108m  35m 4080 S  4.6  0.1   0:06.22 lsphp5
10662 nobody     5 -10  116m  41m 3992 S  4.3  0.1   0:03.64 lsphp5:/xxxxxxxxxxxx/vbseo.php
10673 nobody     5 -10 98.1m  26m 3488 S  3.3  0.1   0:00.10 lsphp5
 2880 nobody    15   0 1280m 270m  252 S  3.0  0.8   4550:41 memcached -d -p 11211 -u memcached -m 8192 -c 2048 -P /var/run/memcached/memcached.pid -l 127.0.0.1 -u nobody
 7388 nobody     0 -19 78056  34m  22m S  1.0  0.1   0:12.24 litespeed (lshttpd)
 7389 nobody     0 -19 88872  46m  30m S  1.0  0.1   0:21.84 litespeed (lshttpd)
10674 nobody     5 -10 82448 8036 2536 S  1.0  0.0   0:00.03 lsphp5
 1491 root      10  -5     0    0    0 S  0.3  0.0 524:14.54 [kjournald]
    1 root      15   0 10372  552  520 S  0.0  0.0   2:08.25 init [3]
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:01.25 [migration/0]
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.20 [ksoftirqd/0]
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/0]
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:02.13 [migration/1]
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.35 [ksoftirqd/1]
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/1]
    8 root      RT  -5     0    0    0 S  0.0  0.0   2:27.24 [migration/2]
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.30 [ksoftirqd/2]
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/2]
   11 root      RT  -5     0    0    0 S  0.0  0.0   3:07.76 [migration/3]
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.40 [ksoftirqd/3]
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/3]
   14 root      RT  -5     0    0    0 S  0.0  0.0   4:22.44 [migration/4]
   15 root      34  19     0    0    0 S  0.0  0.0   0:00.35 [ksoftirqd/4]
   16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/4]
   17 root      RT  -5     0    0    0 S  0.0  0.0   2:57.49 [migration/5]
   18 root      34  19     0    0    0 S  0.0  0.0   0:00.40 [ksoftirqd/5]
   19 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/5]
   20 root      RT  -5     0    0    0 S  0.0  0.0   2:33.48 [migration/6]
   21 root      34  19     0    0    0 S  0.0  0.0   0:00.35 [ksoftirqd/6]
   22 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/6]
   23 root      RT  -5     0    0    0 S  0.0  0.0   2:44.55 [migration/7]
   24 root      34  19     0    0    0 S  0.0  0.0   0:00.53 [ksoftirqd/7]
   25 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/7]
   26 root      RT  -5     0    0    0 S  0.0  0.0   4:45.06 [migration/8]
Seconds since request size is also an issue, especially with static files. 60 seconds is crazy:
 
Last edited:
N

NiteWave

Administrator
#4
#4
still can't find other problems other than CPU --- and the mysqld occupied CPU much. install XCache can reduce lsphp5 load a bit. has it installed?

wait: Swap: 34996216k total, 1735756k used
with 32G memory, why still swap use 1.7G ? swap will cause very slow response.

since you still have a lot of free memory, so advice for your current configuration: allocate more memory to mysqld; monitor and try to reduce used swap --- it should be 0 in most time. I believe you'll feel much faster when used swap=0.
 
M

MisterNinja

Member
#5
#5
NiteWave said:
still can't find other problems other than CPU --- and the mysqld occupied CPU much. install XCache can reduce lsphp5 load a bit. has it installed?

wait: Swap: 34996216k total, 1735756k used
with 32G memory, why still swap use 1.7G ? swap will cause very slow response.

since you still have a lot of free memory, so advice for your current configuration: allocate more memory to mysqld; monitor and try to reduce used swap --- it should be 0 in most time. I believe you'll feel much faster when used swap=0.
Click to expand...
I am using memcache instead of XCache. My previous tests with XCache didn't show much of a performance boost compared tests with memcache. I don't know which is quicker tbh, everyone has a different opinion.

I think the swap is being used with litespeed. I enabled the caching of pages for guests - which should speed stuff up and prevent stuff from being regenerated. I have been getting DDOS'ed lately so I enabled that to prevent php from completely overloading the server.

MySQL was customized by George of vBulletin for the website, and he thinks the load/lag issue is with litespeed/php:





EDIT:
I don't think swap will ever be 0:
 
Last edited:
eva2000

eva2000

Well-Known Member
#6
#6
Hey guys :)

Just noticed your very large small/medium file cache total and size limits in 1.9GB and 5M sizes respectively, maybe that has some bearing ? Tried reducing them back to defaults ? Also lsphp5 app max idle time you have it set to 10, not good for xcache usage at least, try setting it back to default of -1

I'd drop static file compression level from 6 to 2, i did tests and difference between level 2 vs 3+ is negligible
 
M

MisterNinja

Member
#7
#7
eva2000 said:
Hey guys :)

Just noticed your very large small/medium file cache total and size limits in 1.9GB and 5M sizes respectively, maybe that has some bearing ? Tried reducing them back to defaults ? Also lsphp5 app max idle time you have it set to 10, not good for xcache usage at least, try setting it back to default of -1

I'd drop static file compression level from 6 to 2, i did tests and difference between level 2 vs 3+ is negligible
Click to expand...
I changed that to -1 and changed the compression. I thought the small/medium file cache is memory based (that is the reason for me setting it so high).

The pages seem to be loading quicker, however the server is no longer optimized for DDOS attacks which seem to have started out of the blue. I find that the DDOS protection with litespeed doesn't seem to stop mysql's cpu usage from skyrocketing even with caching for guests. I woke up this morning to a website that was down for 8 hours. It seems that with the endless restarts for 503 errors and a session table that is constantly locked, the server's load increases until the site is down.


Code: 
| 12513 | forumvb4             | localhost | forumvb4 | Query   |    9 | Table lock           | SELECT *
                                FROM session
                                WHERE userid = 0
                                        AND host = 'xxxxxxxxxx'
                                        AND idhash  |    0.000 |
| 12514 | forumvb4             | localhost | forumvb4 | Query   |    9 | Table lock           | SELECT *
                                FROM session
                                WHERE userid = 0
                                        AND host = 'xxxxxxxxxx'
                                        AND idhash = |    0.000 |
| 12516 | forumvb4             | localhost | forumvb4 | Query   |    9 | Table lock           | SELECT *
                                FROM session
                                WHERE userid = 0
                                        AND host = 'xxxxxxxxxx'
                                        AND idhash  |    0.000 |
Code: 
 8297 mysql     10  -5 16.1g 4.2g 6056 S 1192.3 13.4  30:35.64 mysqld
					   ^ CPU USAGE
 
Last edited:
M

MisterNinja

Member
#8
#8
MisterNinja said:
I changed that to -1 and changed the compression. I thought the small/medium file cache is memory based (that is the reason for me setting it so high).

The pages seem to be loading quicker, however the server is no longer optimized for DDOS attacks which seem to have started out of the blue. I find that the DDOS protection with litespeed doesn't seem to stop mysql's cpu usage from skyrocketing even with caching for guests. I woke up this morning to a website that was down for 8 hours. It seems that with the endless restarts for 503 errors and a session table that is constantly locked, the server's load increases until the site is down.

Code: 
| 12513 | forumvb4             | localhost | forumvb4 | Query   |    9 | Table lock           | SELECT *
                                FROM session
                                WHERE userid = 0
                                        AND host = 'xxxxxxxxxx'
                                        AND idhash  |    0.000 |
| 12514 | forumvb4             | localhost | forumvb4 | Query   |    9 | Table lock           | SELECT *
                                FROM session
                                WHERE userid = 0
                                        AND host = 'xxxxxxxxxx'
                                        AND idhash = |    0.000 |
| 12516 | forumvb4             | localhost | forumvb4 | Query   |    9 | Table lock           | SELECT *
                                FROM session
                                WHERE userid = 0
                                        AND host = 'xxxxxxxxxx'
                                        AND idhash  |    0.000 |
Code: 
 8297 mysql     10  -5 16.1g 4.2g 6056 S 1192.3 13.4  30:35.64 mysqld
					   ^ CPU USAGE
Click to expand...
It turns out that the issue with load has to do with a plugin and the lack of cache for queries.

Whenever someone attacks vbulletin's index.php, I see this spammed about 200 times:

Code: 
| 20126 | forumvb4 | localhost | forumvb4 | Query   |    2 | Sending data         | SELECT forum.forumid, forum.lastpost, forum.lastposter, forum.lastposterid, forum.lastthread, forum. |    0.000 |
 
Last edited:
eva2000

eva2000

Well-Known Member
#11
#11
Yeah you have to be careful with plugins which call/alter vB tables such as post, thread, user, session especially for large forums with large associated tables. I've seen plugins as such which when enable immediately crash large forums as there's no proper indexing on the queries coded within the plugins.
 
You must log in or register to reply here.
Top