Help with high load server

#1
Hi,

I've been using litespeed for some time now, but since about 2-3 days ago, one of the servers started having loading problems. At first I thought it was being atacked, but then I noticed that there were a lot of http processes used(around 300-350) and the load was at a constant 10-12. Normally I wouldn't get alarmed because it's not such a high load, but clients started reporting that the websites backends(wp-admin for example) took forever to load.

Could you please advise what I can tweak so admin backends start loading normally?(I'm guessing it's because the load exceeds the normal value of smaller than 8). MySQL is not a bottleneck, it has 2000 available connections and the average load is 1-1,5.

In peak times, the server has around 350-400 processes:

Code:
ps -ef|wc
    358    3092   25232
This is a load average from today:

Code:
07:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
09:00:03 AM         6      1201      6.33      7.41      7.55
09:10:01 AM        16      1210      6.97      7.33      7.49
09:20:02 AM        17      1188      9.76      9.22      8.36
09:30:01 AM        39      1243      9.51      8.37      8.21
09:40:01 AM         9      1182      7.27      8.08      8.18
09:50:01 AM         4      1207     11.87      9.68      8.90
10:00:01 AM        24      1207      8.66      9.88      9.44
10:10:01 AM         4      1216     10.31     10.75     10.14
10:20:03 AM        29      1249     11.44     11.28     10.64
10:30:02 AM        11      1245     11.18     10.96     10.78
10:40:02 AM        19      1201      8.88      9.93     10.31
10:50:01 AM        18      1202     12.02     11.41     10.71
11:00:01 AM         8      1205     11.44      9.89     10.02
11:10:01 AM        14      1188      8.34      9.49      9.83
11:20:02 AM        25      1217     10.42      9.51      9.65
11:30:01 AM         6      1214      6.99      8.35      9.07
11:40:01 AM         6      1210      8.40      8.59      8.76
11:50:02 AM        19      1257      9.30     10.58      9.86
12:00:01 PM        13      1233      9.34      9.37      9.43
12:10:01 PM        13      1233      7.44      9.34      9.62
12:20:01 PM         6      1247      5.84      7.28      8.56
12:30:02 PM        13      1258      7.87      7.46      7.94
Average:           13      1177      6.34      6.57      6.59
The hardware setup is as follows:

1.Http server(Cpanel)
1 x Intel Xeon(quad-core)
8 Gb RAM
4 x 1 TB HDD in RAID 10

2. MySQL server
1 x Intel Xeon(quad-core)
16 Gb RAM
2 x 150 Gb HDD(SAS) in RAID 1

My litespeed config is as follows:

Code:
<?xml version="1.0" encoding="UTF-8"?>
<httpServerConfig>
  <serverName>$HOSTNAME</serverName>
  <user>nobody</user>
  <group>nobody</group>
  <priority>10</priority>
  <chrootPath>/</chrootPath>
  <enableChroot>0</enableChroot>
  <enableLVE>2</enableLVE>
  <inMemBufSize>2548M</inMemBufSize>
  <swappingDir>/tmp/lshttpd/swap</swappingDir>
  <autoRestart>1</autoRestart>
  <autoFix503>1</autoFix503>
  <gracefulRestartTimeout></gracefulRestartTimeout>
  <loadApacheConf>1</loadApacheConf>
  <autoReloadApacheConf>2</autoReloadApacheConf>
  <apacheBinPath>/usr/local/apache/bin/httpd</apacheBinPath>
  <apacheConfFile>/usr/local/apache/conf/httpd.conf</apacheConfFile>
  <apachePortOffset>0</apachePortOffset>
  <apacheIpOffset>0</apacheIpOffset>
  <apacheHandledContent></apacheHandledContent>
  <apacheIgnoredModules></apacheIgnoredModules>
  <phpSuExec>1</phpSuExec>
  <phpSuExecMaxConn>15</phpSuExecMaxConn>
  <apacheEnableFP></apacheEnableFP>
  <apacheEnvVars></apacheEnvVars>
  <mime>$SERVER_ROOT/conf/mime.properties</mime>
  <disableInitLogRotation>1</disableInitLogRotation>
  <showVersionNumber>0</showVersionNumber>
  <enableIpGeo></enableIpGeo>
  <useIpInProxyHeader>0</useIpInProxyHeader>
  <autoUpdateInterval>86400</autoUpdateInterval>
  <autoUpdateDownloadPkg>1</autoUpdateDownloadPkg>
  <adminEmails>admin@keyhost.ro</adminEmails>
  <adminRoot>$SERVER_ROOT/admin/</adminRoot>
  <logging>
    <log>
      <fileName>/usr/local/apache/logs/error_log</fileName>
      <logLevel>INFO</logLevel>
<debugLevel>0</debugLevel>
      <rollingSize>2000M</rollingSize>
      <enableStderrLog>1</enableStderrLog>
    </log>
    <accessLog>
      <fileName>$SERVER_ROOT/logs/access.log</fileName>
      <rollingSize>2000M</rollingSize>
      <keepDays>30</keepDays>
      <compressArchive>0</compressArchive>
    </accessLog>
  </logging>
  <indexFiles>index.html, index.php, index.php5, index.htm</indexFiles>
  <autoIndex>1</autoIndex>
  <autoIndexURI>/_autoindex/default.php</autoIndexURI>
  <htAccess>
    <allowOverride>31</allowOverride>
    <accessFileName>.htaccess</accessFileName>
  </htAccess>
  <expires>
    <enableExpires>1</enableExpires>
    <expiresByType>image/*=A604800, text/css=A604800, application/x-javascript=A604800, application/javascript=A604800</expiresByType>
  </expires>
  <tuning>
    <maxConnections>1000</maxConnections>
    <maxSSLConnections>100</maxSSLConnections>
    <connTimeout>120</connTimeout>
    <maxKeepAliveReq>1</maxKeepAliveReq>
    <smartKeepAlive>0</smartKeepAlive>
    <keepAliveTimeout>3</keepAliveTimeout>
    <sndBufSize>0</sndBufSize>
    <rcvBufSize>0</rcvBufSize>
    <eventDispatcher>best</eventDispatcher>
    <maxCachedFileSize>16k</maxCachedFileSize>
    <totalInMemCacheSize>128M</totalInMemCacheSize>
    <maxMMapFileSize>2M</maxMMapFileSize>
    <totalMMapCacheSize>1024M</totalMMapCacheSize>
    <useSendfile>1</useSendfile>
<useAIO>2</useAIO>
    <AIOBlockSize>7</AIOBlockSize>
    <fileETag>0</fileETag>
    <SSLCryptoDevice>null</SSLCryptoDevice>
    <maxReqURLLen>8K</maxReqURLLen>
    <maxReqHeaderSize>4096</maxReqHeaderSize>
    <maxReqBodySize>500M</maxReqBodySize>
    <maxDynRespHeaderSize>8K</maxDynRespHeaderSize>
    <maxDynRespSize>500M</maxDynRespSize>
    <enableGzipCompress>1</enableGzipCompress>
    <enableDynGzipCompress>1</enableDynGzipCompress>
    <gzipCompressLevel>2</gzipCompressLevel>
    <compressibleTypes>text/*, application/x-javascript, application/xml, application/javascript</compressibleTypes>
    <gzipAutoUpdateStatic>1</gzipAutoUpdateStatic>
    <gzipCacheDir>/home/loguri/gzip</gzipCacheDir>
    <gzipStaticCompressLevel>6</gzipStaticCompressLevel>
    <gzipMaxFileSize>1M</gzipMaxFileSize>
    <gzipMinFileSize>32K</gzipMinFileSize>
  </tuning>
  <security>
    <fileAccessControl>
      <followSymbolLink>1</followSymbolLink>
      <checkSymbolLink>0</checkSymbolLink>
      <requiredPermissionMask>000</requiredPermissionMask>
      <restrictedPermissionMask>000</restrictedPermissionMask>
    </fileAccessControl>
    <perClientConnLimit>
      <staticReqPerSec></staticReqPerSec>
      <dynReqPerSec></dynReqPerSec>
      <outBandwidth>0</outBandwidth>
      <inBandwidth>0</inBandwidth>
      <softLimit>20</softLimit>
      <hardLimit>50</hardLimit>
      <blockBadReq>1</blockBadReq>
      <gracePeriod>30</gracePeriod>
      <banPeriod>5</banPeriod>
    </perClientConnLimit>
</perClientConnLimit>
    <CGIRLimit>
      <cgidSock></cgidSock>
      <maxCGIInstances>20</maxCGIInstances>
      <minUID>11</minUID>
      <minGID>10</minGID>
      <forceGID></forceGID>
      <priority>-15</priority>
      <CPUSoftLimit>0</CPUSoftLimit>
      <CPUHardLimit>0</CPUHardLimit>
      <memSoftLimit></memSoftLimit>
      <memHardLimit></memHardLimit>
      <procSoftLimit>20</procSoftLimit>
      <procHardLimit>25</procHardLimit>
    </CGIRLimit>
    <censorshipControl>
      <enableCensorship>1</enableCensorship>
      <logLevel>0</logLevel>
      <defaultAction>deny,log,status:403</defaultAction>
      <scanPOST>0</scanPOST>
      <disableSecHtaccess></disableSecHtaccess>
      <secAuditLogEngine></secAuditLogEngine>
      <secAuditLog>/home/loguri/sec_audit.log</secAuditLog>
    </censorshipControl>
    <accessDenyDir>
      <dir>/</dir>
      <dir>/etc/*</dir>
      <dir>/dev/*</dir>
      <dir>$SERVER_ROOT/conf/*</dir>
      <dir>$SERVER_ROOT/admin/conf/*</dir>
    </accessDenyDir>
    <accessControl>
      <allow>ALL</allow>
      <deny></deny>
    </accessControl>
  </security>
  <extProcessorList>
    <extProcessor>
 <type>lsapi</type>
      <name>lsphp5</name>
      <address>uds://tmp/lshttpd/lsphp5.sock</address>
      <note></note>
      <maxConns>300</maxConns>
      <env>PHP_LSAPI_MAX_REQUESTS=100</env>
      <env>PHP_LSAPI_CHILDREN=300</env>
      <initTimeout>60</initTimeout>
      <retryTimeout>0</retryTimeout>
      <persistConn>0</persistConn>
      <pcKeepAliveTimeout>5</pcKeepAliveTimeout>
      <respBuffer>0</respBuffer>
      <autoStart>1</autoStart>
      <path>$SERVER_ROOT/fcgi-bin/lsphp5</path>
      <backlog>100</backlog>
      <instances>1</instances>
      <extUser></extUser>
      <extGroup></extGroup>
      <runOnStartUp>0</runOnStartUp>
      <extMaxIdleTime>2</extMaxIdleTime>
      <priority>10</priority>
      <memSoftLimit>256M</memSoftLimit>
      <memHardLimit>312M</memHardLimit>
      <procSoftLimit>200</procSoftLimit>
      <procHardLimit>200</procHardLimit>
    </extProcessor>
I cut it short because that's all that I need to tune. I don't use Ruby.
 

webizen

Well-Known Member
#2
1. paste 'top -c' from command line so we can see which process takes more cpu cycles.

2. check Real-Time Stats in LSWS Admin Console => Actions and see if anything in the WaitQ column.

3. also, try request static URL instead of dynamic (requires db access) and if still takes long time.
 
#3
Hi,

I have 2 top -c outputs. Most of the heavy sites have cache scripts(most of the are wordpress and I've installed supercache).

Code:
top - 11:27:00 up 85 days,  4:25,  3 users,  load average: 7.78, 6.90, 6.10
Tasks: 252 total,   7 running, 245 sleeping,   0 stopped,   0 zombie
Cpu(s): 57.0%us, 27.4%sy,  0.0%ni,  8.6%id,  1.3%wa,  0.0%hi,  3.7%si,  2.0%st
Mem:   7340032k total,  6611876k used,   728156k free,   519160k buffers
Swap:  4194296k total,    35164k used,  4159132k free,  3153480k cached

    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 706820 co***asa  16   0  166m  62m  22m S 57.0  0.9   0:01.12 lsphp5
 706829 pi***ex   17   0  157m  48m  18m R 54.4  0.7   0:01.07 lsphp5:/home/pi***ex/public_html/index.php
 706781 m***tord  16   0  195m  34m  19m S 50.9  0.5   0:01.40 lsphp5
 706828 s***elib  15   0  174m  70m  22m S 50.4  1.0   0:00.99 lsphp5
 706836 u****kid  15   0  167m  64m  22m S 38.2  0.9   0:00.75 lsphp5
 706825 su**t     18   0  159m  55m  22m S 37.1  0.8   0:00.73 lsphp5
 706830 g***pele  15   0  168m  64m  22m S 35.1  0.9   0:00.69 lsphp5:/home/gh****ele/public_html/index.php
 706818 f***ec    16   0  169m  64m  22m S 33.1  0.9   0:00.65 lsphp5
 598709 nobody    18   0  325m 154m  17m D 30.0  2.1   6:00.30 litespeed (lshttpd)
 706823 o****tiv  15   0  158m  52m  22m S 29.5  0.7   0:00.58 lsphp5:/home/o****tiv/public_html/index.php
 706819 gh****le  15   0  152m  49m  22m S 28.5  0.7   0:00.56 lsphp5
 706840 c*****se  17   0  146m  42m  22m S 25.4  0.6   0:00.50 lsphp5
 706826 d****20   18   0  158m  54m  22m S 24.4  0.8   0:00.48 lsphp5
 706842 s****et   18   0  163m  58m  22m R 21.4  0.8   0:00.42 lsphp5:/home/s***et/public_html/index.php
 706827 mo***ord  16   0  202m  39m  18m R 15.8  0.6   0:00.31 lsphp5:/home/m****rd/public_html/index.php
 706816 mo***ord  15   0  201m  40m  19m S 15.3  0.6   0:00.71 lsphp5
 706822 r****re   15   0  139m  31m  17m S 15.3  0.4   0:00.30 lsphp5
 706844 i***za    16   0  152m  44m  19m R 13.7  0.6   0:00.27 lsphp5:/home/i****za/public_html/index.php
 706841 im***me   15   0  156m  35m 9252 S 12.7  0.5   0:00.25 lsphp5:/home/i****me/public_html/index.php
 598708 nobody    17   0  299m 139m  14m D 12.2  1.9   4:43.17 litespeed (lshttpd)
 706766 o****tiv  15   0  159m  54m  22m S 10.7  0.8   0:00.74 lsphp5:/home/ob***tiv/public_html/index.php
 706787 p***ob    18   0  153m  50m  22m S 10.2  0.7   0:01.01 lsphp5
 706848 s***ic    15   0  146m  35m  15m R 10.2  0.5   0:00.20 lsphp5:/home/s***ic/public_html/index.php
 706843 v****nin  17   0  137m  20m 9.9m D  6.1  0.3   0:00.12 lsphp5:/home/v****unin/public_html/index.php
 706837 f***ec    18   0  132m  13m 7652 S  5.1  0.2   0:00.10 lsphp5
 706765 ob**ctiv  15   0  159m  54m  22m S  4.6  0.8   0:00.49 lsphp5
 706838 fi**ec    18   0  132m  13m 7652 S  4.6  0.2   0:00.09 lsphp5
 706839 z***o     18   0  134m  13m 7732 S  4.6  0.2   0:00.09 lsphp5
 706833 o*****iv  18   0  132m  12m 6544 S  4.1  0.2   0:00.08 lsphp5
 706835 i****e    20   0  131m  10m 5628 S  4.1  0.1   0:00.08 lsphp5
 706824 p****67   18   0  131m  10m 5820 S  3.6  0.1   0:00.07 lsphp5
 706834 i****e    18   0  131m  10m 5628 S  3.6  0.1   0:00.07 lsphp5
 706821 an***om   18   0  132m  12m 7232 S  3.1  0.2   0:00.06 lsphp5
 706832 rb***re   18   0  132m  10m 6052 D  3.1  0.2   0:00.06 lsphp5:hemes/branfordmagazine-4/scripts/timthumb.php
  41344 named     18   0  373m  61m 2052 S  1.0  0.9 135:11.51 /usr/sbin/named -u named
 459133 root      15   0 12864 1276  828 S  1.0  0.0   0:32.46 top
 706809 gi***81   18   0  171m  66m  23m S  1.0  0.9   0:00.54 lsphp5:home/g***1/public_html/wp-comments-post.php
Code:
top - 11:08:57 up 85 days,  4:07,  3 users,  load average: 6.43, 6.09, 5.33
Tasks: 236 total,   8 running, 227 sleeping,   0 stopped,   1 zombie
Cpu(s): 31.6%us, 10.7%sy,  0.0%ni, 53.9%id,  1.0%wa,  0.0%hi,  2.4%si,  0.3%st
Mem:   7340032k total,  6429340k used,   910692k free,   542788k buffers
Swap:  4194296k total,    35164k used,  4159132k free,  3204008k cached

    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 686202 q***n  15   0  171m  68m  22m R 36.2  1.0   0:02.28 lsphp5:/home/q***zin/public_html/index.php
 686252 s***elib  16   0  189m  85m  22m R 28.6  1.2   0:00.86 lsphp5:/home/s***lib/public_html/index.php
 686250 r***ire   16   0  162m  58m  22m R 23.6  0.8   0:00.71 lsphp5:/home/rb***re/public_html/index.php
 686180 r***ire   15   0  161m  55m  22m S 21.6  0.8   0:01.84 lsphp5
 686251 g***pele  15   0  168m  64m  22m S 20.9  0.9   0:00.63 lsphp5:/home/gh***ele/public_html/index.php
 686256 m***ncri  17   0  179m  74m  22m R 19.9  1.0   0:00.60 lsphp5:/home/ma***cri/public_html/index.php
 686248 f***ec    15   0  169m  65m  22m S 16.6  0.9   0:00.50 lsphp5
 686249 g***881   15   0  166m  62m  22m S 15.9  0.9   0:00.48 lsphp5
 686242 ***iaza   15   0  193m  28m  14m S 14.3  0.4   0:00.43 lsphp5
 686228 urb***id  15   0  176m  71m  22m R 13.3  1.0   0:01.36 lsphp5:/home/urb***id/public_html/index.php
 686258 ja***and  15   0  155m  50m  22m R 12.9  0.7   0:00.39 lsphp5:/home/jan***and/public_html/index.php
 686246 pe***ana  20   0  155m  49m  22m S 11.0  0.7   0:00.33 lsphp5
 686253 g***pele  16   0  154m  49m  22m R 10.0  0.7   0:00.30 lsphp5:/home/ghi***le/public_html/index.php
 686255 m***tord  15   0  200m  36m  16m S 10.0  0.5   0:00.30 lsphp5:/home/mo***rd/public_html/index.php
 686233 n***s     17   0  197m  30m  15m S  8.6  0.4   0:00.34 lsphp5
 598709 nobody    15   0  293m 116m  20m S  7.0  1.6   4:24.77 litespeed (lshttpd)
 598708 nobody    18   0  280m 113m  14m S  4.3  1.6   3:25.37 litespeed (lshttpd)
 686257 p***oana  18   0  131m  11m 6272 S  2.0  0.2   0:00.06 lsphp5
 686254 pr***eri  17   0  187m  11m 6408 S  1.7  0.2   0:00.05 lsphp5
  41344 named     18   0  373m  61m 2052 S  1.3  0.9 135:08.96 /usr/sbin/named -u named
 598664 root      15   0  107m  72m 2272 S  1.0  1.0   0:23.13 litespeed (lshttpd)
 395667 root      18   0 13412 1768  780 S  0.3  0.0  84:04.01 top
 657463 root      15   0 12888 1288  832 R  0.3  0.0   0:04.66 top
      1 root      16   1 10372  648  552 S  0.0  0.0   0:39.80 init [3]
      3 root      39  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/0]
      5 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/1]
      7 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/2]
      9 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/3]
     11 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/4]
     13 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/5]
     15 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/6]
     17 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/7]
     18 root      10  -5     0    0    0 S  0.0  0.0   1:11.82 [events/0]
     19 root      10  -5     0    0    0 S  0.0  0.0   0:40.95 [events/1]
     20 root      10  -5     0    0    0 S  0.0  0.0   0:38.57 [events/2]
     21 root      10  -5     0    0    0 S  0.0  0.0   0:38.38 [events/3]
     22 root      10  -5     0    0    0 S  0.0  0.0   0:38.35 [events/4]
 

webizen

Well-Known Member
#4
for quad-core svr, 10-12 cpu load avg isn't very high.

have you ever tested any static pages and see if they are slow?

most cpu cycles on PHP. improve PHP code or put in opcode cache (APC, xcache, eAccelerator) to speed up. Also, recommend followings for per client throttling settings to prevent any http level ddos attack (excessive requests from individual remote IP).

<perClientConnLimit>
<staticReqPerSec>50</staticReqPerSec>
<dynReqPerSec>3</dynReqPerSec>
<outBandwidth>0</outBandwidth>
<inBandwidth>0</inBandwidth>
<softLimit>20</softLimit>
<hardLimit>30</hardLimit>
<blockBadReq>1</blockBadReq>
<gracePeriod>15</gracePeriod>
<banPeriod>300</banPeriod>
</perClientConnLimit>
 
#5
Hi webizen,

Thanks for the suggestions. I have eAccelerator installed but it's not wanting to work.

Here's an output of the disk space the cache folder occupies.

root@s1 [/home/l***ri]# du -sh eaccelerator
1.2M eaccelerator

Also, permissions:
root@s1 [/home/l***ri]# ll -h|grep eaccelerator
drwxrwxrwx 18 root root 4.0K Jun 8 05:16 eaccelerator/

And php.ini config:

Code:
extension="eaccelerator.so"
eaccelerator.cache_dir="/home/lo***i/eaccelerator"
eaccelerator.check_mtime="1"
eaccelerator.compress="1"
eaccelerator.compress_level="9"
eaccelerator.debug="0"
eaccelerator.enable="1"
eaccelerator.filter=""
eaccelerator.optimizer="1"
eaccelerator.shm_max="0"
eaccelerator.shm_only="0"
eaccelerator.shm_prune_period="0"
eaccelerator.shm_size="16"
eaccelerator.shm_ttl="0"
 

webizen

Well-Known Member
#6
since you are shared hosting, to leverage opcode cache(EA), you need to run lsphp5 in suEXEC Daemon mode:
<type>lsapi</type>
<name>lsphp5</name>
<address>uds://tmp/lshttpd/lsphp5.sock</address>
...
<runOnStartUp>2</runOnStartUp>
EA cache_dir should point to somewhere that anyone can access like
eaccelerator.cache_dir="/tmp/eaccelerator"
 
#7
Hi,

I implemented all the changes you indicated in your last reply, but still EA does not want to cache.

root@s1 [/tmp]# du -s eaccelerator
1216 eaccelerator

It just creates the cache folders and that's it.
 

anewday

Well-Known Member
#9
What's the owner of that eaccelerator cache folder? It has to be the same as for lsws.

Here's mine:

drwxr-xr-x 18 nobody nobody
 
Top