Optimization Request

bhanuprasad1981 · Jan 9, 2012

i am running server with 2 cpu license past some days load is too high tries a lot of configs but fail, please help me to find issue , let me know what details you need .

webizen · Jan 9, 2012

run 'top' first and see what process consumes the most resources/cpu cycles.

bhanuprasad1981 · Jan 10, 2012

here is top result :-

Code:

Tasks: 191 total,   1 running, 190 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  1.0%sy,  0.0%ni, 86.5%id, 10.7%wa,  0.5%hi,  0.2%si,  0.0%st
Mem:  12289824k total, 12075560k used,   214264k free,   409056k buffers
Swap: 12594940k total,        0k used, 12594940k free,  9524700k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6641 mysql     10  -5 10.5g 1.5g 4728 S  5.7 12.8  26:02.59 /usr/sbin/mysqld --
11590 nobody    15   0  551m  31m  16m S  4.7  0.3   0:00.14 lsphp5
11588 nobody    15   0  544m  22m  14m S  2.0  0.2   0:00.06 lsphp5
11491 nobody    15   0  548m  87m  78m S  1.0  0.7   0:01.27 lsphp5
11589 nobody    15   0  542m  14m 8796 S  0.7  0.1   0:00.02 lsphp5
 2490 root      10  -5     0    0    0 D  0.3  0.0   0:23.56 [kjournald]
    1 root      15   0 10368  688  572 S  0.0  0.0   0:01.37 init [3]
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [migration/0]
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/0]
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/0]
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [migration/1]
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/1]
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/1]
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [migration/2]
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/2]
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/2]
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [migration/3]

webizen · Jan 10, 2012

you missing the first line (contains 'load average'). pls add back.

bhanuprasad1981 · Jan 11, 2012

top -c result during normal time not peak* during peak load touches 20+ and sites start to crawl.

Code:

root@ecat [~]# top -c
top - 05:34:02 up 17:19,  1 user,  load average: 5.77, 6.10, 6.44
Tasks: 194 total,   1 running, 193 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.7%us,  1.2%sy,  0.0%ni, 82.5%id, 13.3%wa,  0.2%hi,  0.2%si,  0.0%st
Mem:  12289824k total, 11967808k used,   322016k free,   462640k buffers
Swap: 12594940k total,   102876k used, 12492064k free,  8291120k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20436 nobody    15   0  550m  98m  87m S  4.0  0.8   0:13.11 lsphp5
16045 nobody    18   0  553m 131m 117m D  3.3  1.1   0:52.27 lsphp5:/home/cluefo
20488 nobody    15   0  553m  97m  79m S  3.3  0.8   0:01.41 lsphp5
20589 nobody    15   0  543m  39m  33m S  3.3  0.3   0:00.26 lsphp5
 6641 mysql     10  -5 10.5g 2.2g 4872 S  3.0 19.1 483:48.96 /usr/sbin/mysqld --
20588 nobody    15   0  545m  51m  43m S  2.0  0.4   0:00.34 lsphp5
20591 nobody    15   0  544m  31m  24m S  1.3  0.3   0:00.19 lsphp5
20594 nobody    15   0  541m  19m  15m S  1.3  0.2   0:00.07 lsphp5
20521 nobody    15   0  550m  91m  81m S  1.0  0.8   0:00.87 lsphp5
20596 nobody    15   0  544m  18m  10m S  1.0  0.2   0:00.06 lsphp5
 2490 root      10  -5     0    0    0 D  0.7  0.0  11:55.34 [kjournald]
20404 root      15   0 12760 1192  824 R  0.7  0.0   0:00.30 top -c
20437 nobody    15   0  562m 121m 100m S  0.7  1.0   0:01.84 lsphp5
20516 nobody    15   0  556m  82m  65m S  0.7  0.7   0:00.86 lsphp5
13377 nobody     4 -19 63908  22m 9.9m D  0.3  0.2   0:23.16 litespeed (lshttpd)
20593 nobody    15   0  543m  19m  12m S  0.3  0.2   0:00.07 lsphp5
    1 root      15   0 10368  688  572 S  0.0  0.0   0:01.40 init [3]

bhanuprasad1981 · Jan 11, 2012

cpanel daily process logs show :-

mysql 73.0 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --log-error/var/lib/mysql/ecat.10gbin.com.err --open-files-limit8192 --pid-file/var/lib/mysql/ecat.10gbin.com.pid
mysql 64.1 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --log-error/var/lib/mysql/ecat.10gbin.com.err --open-files-limit8192 --pid-file/var/lib/mysql/ecat.10gbin.com.pid

another top :-

Code:

root@ecat [~]# top -c
top - 06:27:28 up 18:12,  1 user,  load average: 10.74, 11.85, 10.50
Tasks: 198 total,   1 running, 197 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.4%us,  1.7%sy,  0.0%ni, 79.6%id, 12.9%wa,  0.1%hi,  0.3%si,  0.0%st
Mem:  12289824k total, 11967816k used,   322008k free,   517812k buffers
Swap: 12594940k total,      188k used, 12594752k free,  8252116k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6641 mysql     10  -5 10.5g 2.3g 4924 S 50.9 19.5 499:53.38 /usr/sbin/mysqld --
25090 nobody    15   0  549m  24m  13m S  1.7  0.2   0:00.16 lsphp5
  795 root      10  -5     0    0    0 D  0.7  0.0   1:03.64 [kjournald]
25083 nobody    15   0  547m  34m  23m S  0.7  0.3   0:00.20 lsphp5
    1 root      15   0 10368  688  572 S  0.0  0.0   0:01.40 init [3]
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.01 [migration/0]
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.29 [ksoftirqd/0]
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/0]
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.06 [migration/1]
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/1]
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/1]
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.04 [migration/2]
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/2]
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/2]
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:00.01 [migration/3]
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/3]
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/3]

NiteWave · Jan 11, 2012

looks like the bottle-neck at io(12.9%wa) and mysql.
how many CPU/cores of this server ?

"top -c" is most helpful when the load is high.

also lsws console->Real-time Stats when high-load is helpful.

bhanuprasad1981 · Jan 11, 2012

this is 8 physical core server with raid 0 running php 5.3.x and xcache issue raised past 5 days back only i cant understand whats going on

bhanuprasad1981 · Jan 12, 2012

running apache with latest 5.3.8 php mpm prefork,without any opcode cache server load is around 7-10 now, sites are running ok not fast as litespeed but better than 1 min page load time of litespeed here is top :-

Code:

top - 13:16:54 up 2 days,  1:02,  1 user,  load average: 6.91, 8.34, 10.24
Tasks: 213 total,   8 running, 205 sleeping,   0 stopped,   0 zombie
Cpu(s): 27.3%us,  6.8%sy,  0.0%ni, 50.3%id, 14.6%wa,  0.2%hi,  0.8%si,  0.0%st
Mem:  12289824k total, 11957784k used,   332040k free,   340720k buffers
Swap: 12594940k total,      188k used, 12594752k free,  8660456k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7309 mysql     10  -5 10.5g 2.0g 4824 S 46.9 17.1 273:28.81 mysqld
27702 nobody    17   0  116m  26m 5984 R  3.7  0.2   0:00.11 php
27703 nobody    18   0  108m  19m 6136 R  3.7  0.2   0:00.11 php
27704 nobody    18   0  106m  15m 5848 R  2.0  0.1   0:00.06 php
22071 nobody    16   0  328m  11m 1872 S  1.7  0.1   0:01.85 httpd
 8004 nobody    16   0 71768 3348  900 S  1.3  0.0   0:24.95 httpd
22147 nobody    15   0  326m  10m 1832 S  1.3  0.1   0:01.49 httpd
22270 nobody    15   0  370m  10m 1828 S  1.0  0.1   0:01.54 httpd
22296 nobody    15   0  296m  10m 1832 S  1.0  0.1   0:01.49 httpd
24805 nobody    15   0  275m 9380 1828 S  1.0  0.1   0:00.83 httpd
24032 nobody    16   0  326m  10m 1832 S  0.7  0.1   0:01.13 httpd
27333 nobody    15   0  262m 6656 1800 S  0.7  0.1   0:00.11 httpd
27705 nobody    18   0  101m  10m 5472 R  0.7  0.1   0:00.02 php
27706 nobody    18   0  100m  10m 5464 R  0.7  0.1   0:00.02 php
32203 nobody    18   0  121m  31m 6136 D  0.7  0.3   0:11.16 php
 2490 root      10  -5     0    0    0 S  0.3  0.0  15:08.80 kjournald
 9317 root      18   0  126m  25m 1568 S  0.3  0.2   0:28.40 lfd

NiteWave · Jan 12, 2012

yes, this is not acceptable.

can you post the real-time stats?

maybe you can increase
Server->General->PHP suEXEC Max Conn

load time should be less than apache, otherwise abnormal.

bhanuprasad1981 · Jan 13, 2012

yes it is strange , i am not phpsuexec realtime stats are normal except one account which was showing some 20+ process everytime, even suspending that account doesnt increase page load time on litespeed , i suspect settings responsible for it, what you gusy think ? i am using latest version of litespeed.

bhanuprasad1981 · Jan 16, 2012

thanks for no reply i think i may need stay with apache on one server

webizen · Jan 16, 2012

sorry for missing this. pls pm your server temporary root access for us to look at further if you would like.

bhanuprasad1981 · Jan 17, 2012

not possible atm for privacy issues , i can attach screens what ever details you need

NiteWave · Jan 17, 2012

please give screenshot of real-time stats when high-load

Optimization Request

bhanuprasad1981

Well-Known Member

webizen

Well-Known Member

bhanuprasad1981

Well-Known Member

webizen

Well-Known Member

bhanuprasad1981

Well-Known Member

bhanuprasad1981

Well-Known Member

NiteWave

Administrator

bhanuprasad1981

Well-Known Member

bhanuprasad1981

Well-Known Member

NiteWave

Administrator

bhanuprasad1981

Well-Known Member

bhanuprasad1981

Well-Known Member

webizen

Well-Known Member

bhanuprasad1981

Well-Known Member

NiteWave

Administrator