performances issue with x86_64 and cpu cores

[IrPr]

Hi there,
I was on a 32bit system, hosting large major php/mysql cms and many static file size up to 10M, more than 2K concurrent connections with no any lag even at traffic peak, that was awesome

Now i've migrated to new server with more 4G RAM and better CPU, changes are RAM, CPU, RAID controller PCI and system arch and now im using cPanel
but its getting IO bottleneck and high iowait even at medium traffic
I guess odd raid controller, RocketRaid 174x which seems to be fakeraid caused this issue but i doubt in 2 things with LS

one is system arch that changed from previous case, now i've switched to 64bit and this is my 1st xperience with x86_64
i know 64bit system increases system abilities well such as big files spport but dont know about performances
is it possible that 64bit caused this lag problem?

another one is cpu cores, in previus configuration it was added listeners and VH in LSWS manually, assigned cores to listeners, but now i'm on cPanel and i dont find how to set cores for listeners, its loaded from httpd.conf automatically and i tought its using 1 core

here is mpstat in at a very low traffic

Linux 2.6.18-8.el5 (hdbox)  08/18/2008      _x86_64_

07:21:47 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:48 AM  all    0.51    0.51    2.53    7.07    0.51    1.01    0.00    0.00   87.88
07:21:48 AM    0    0.00    1.02    0.00    0.00    0.00    0.00    0.00    0.00   98.98
07:21:48 AM    1    0.00    0.00    5.00   15.00    1.00    3.00    0.00    0.00   76.00

07:21:48 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:49 AM  all    0.51    3.55    2.03   13.20    0.51    1.02    0.00    0.00   79.19
07:21:49 AM    0    1.01    1.01    1.01    2.02    0.00    0.00    0.00    0.00   94.95
07:21:49 AM    1    0.00    5.10    4.08   24.49    0.00    2.04    0.00    0.00   64.29

07:21:49 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:50 AM  all    0.51    4.04    2.53   14.14    0.00    1.01    0.00    0.00   77.78
07:21:50 AM    0    1.00    0.00    1.00    3.00    0.00    0.00    0.00    0.00   95.00
07:21:50 AM    1    0.00    8.25    3.09   24.74    1.03    2.06    0.00    0.00   60.82

07:21:50 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:51 AM  all    0.00    0.00    2.04    4.08    0.51    1.02    0.00    0.00   92.35
07:21:51 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
07:21:51 AM    1    1.01    0.00    4.04    9.09    0.00    2.02    0.00    0.00   83.84

07:21:51 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:52 AM  all    0.00    0.00    2.02   11.62    0.00    1.01    0.00    0.00   85.35
07:21:52 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
07:21:52 AM    1    0.00    0.00    4.04   22.22    1.01    2.02    0.00    0.00   70.71

07:21:52 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:53 AM  all    0.00    0.00    1.53    9.18    0.51    1.02    0.00    0.00   87.76
07:21:53 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
07:21:53 AM    1    0.00    0.00    4.04   19.19    1.01    2.02    0.00    0.00   73.74

07:21:53 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:54 AM  all    0.51    1.52    2.02    9.09    0.51    1.01    0.00    0.00   85.35
07:21:54 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
07:21:54 AM    1    0.00    4.08    3.06   17.35    0.00    2.04    0.00    0.00   73.47

07:21:54 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:55 AM  all    0.00    0.00    1.54    6.67    0.51    0.51    0.00    0.00   90.77
07:21:55 AM    0    0.00    0.00    0.00    5.10    0.00    0.00    0.00    0.00   94.90
07:21:55 AM    1    1.01    0.00    4.04    8.08    1.01    1.01    0.00    0.00   84.85

07:21:55 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:56 AM  all    0.00    0.00    2.03    6.09    0.00    1.02    0.00    0.00   90.86
07:21:56 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
07:21:56 AM    1    0.00    0.00    4.00   13.00    1.00    2.00    0.00    0.00   80.00

07:21:56 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:57 AM  all    3.00    6.00    4.50    8.00    0.50    1.00    0.00    0.00   77.00
07:21:57 AM    0    4.08    2.04    2.04    2.04    0.00    0.00    0.00    0.00   89.80
07:21:57 AM    1    1.04    9.38    6.25   13.54    0.00    2.08    0.00    0.00   67.71

07:21:57 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:58 AM  all    1.53    6.63    3.06   10.20    0.00    1.02    0.00    0.00   77.55
07:21:58 AM    0    3.03    6.06    1.01    0.00    0.00    0.00    0.00    0.00   89.90
07:21:58 AM    1    0.00    8.00    6.00   20.00    1.00    2.00    0.00    0.00   63.00

07:21:58 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:21:59 AM  all    2.53    8.59    4.04    3.54    0.51    1.01    0.00    0.00   79.80
07:21:59 AM    0    5.00    7.00    2.00    2.00    0.00    0.00    0.00    0.00   84.00
07:21:59 AM    1    1.00   10.00    6.00    6.00    0.00    2.00    0.00    0.00   75.00

07:21:59 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:22:00 AM  all    7.04    8.54    5.53    8.54    0.00    1.01    0.00    0.00   69.35
07:22:00 AM    0   12.12    8.08    3.03    1.01    0.00    0.00    0.00    0.00   75.76
07:22:00 AM    1    1.03    8.25    8.25   16.49    0.00    2.06    0.00    0.00   63.92

07:22:00 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:22:01 AM  all    0.00    0.00    2.55    9.69    0.00    0.51    0.00    0.00   87.24
07:22:01 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
07:22:01 AM    1    1.01    0.00    4.04   19.19    1.01    1.01    0.00    0.00   73.74

07:22:01 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
07:22:02 AM  all    2.51    5.03    2.51   16.58    0.50    1.01    0.00    0.00   71.86
07:22:02 AM    0    4.00    9.00    2.00    0.00    0.00    0.00    0.00    0.00   85.00
07:22:02 AM    1    0.00    2.02    3.03   33.33    1.01    2.02    0.00    0.00   58.59

approximately core 0 has no iowait while core 1 has whole system's iowait

Any commen would be appreciated,
Thanks

 

[IrPr]

PS: another major difference is io event dispatcher
due to raid controller drive issue with 2.6 kernels i'm currently on 2.4 and therefore no epoll IO event dispatcher available
in previus case it was set to epoll while its forced to be poll in new box

 

[mistwang]

You can create the listener manually with IPort and assign the binding. LSWS will use it together with configuration from httpd.conf. Using multiple CPU to handle I/O intensive task is important.

epoll may have some impact as well, but should not be as significant.

 

[IrPr]

Thanks for reply,
any idea about arch ?

 

[mistwang]

I don't know how good is the 64bit support in 2.4 kernel, hope it is good enough. I think the high I/O wait may related to the RAID card driver. Hope you can get good support for the RAID card in latest 2.6 kernel.

Not sure what kind RAID is used. RAID0 should be the best, then RAID10, RAID5, etc.

 

[IrPr]

I don't know how good is the 64bit support in 2.4 kernel, hope it is good enough. I think the high I/O wait may related to the RAID card driver. Hope you can get good support for the RAID card in latest 2.6 kernel.

Not sure what kind RAID is used. RAID0 should be the best, then RAID10, RAID5, etc.

Thanks a lot Mr Wang,
I know its not LSWS related question and maybe an offtopic thread, but it would be more appreciated if you suggest me more,

i've planned to upgrade Raid controller to 3WARE which i'v experienced with
it was workin like a charm pushed upto 350Mbit/s with RAID1 Array over Litespeed Ent on 32bit system with 2.6 kernel using epoll as IO event dispatcher, while now on 64bit 2.4 kernel same files and same RAID1 Array but on RocketRaid fakeraid controller even couldnt push 100Mbit/s
I host large files upper than 100MB with a lot of concurrent connections up to 2-3k

What would u recommend for my download server? Which RAID Array, 0 or 1 or 5 ? and 32bit or 64 over 2.6 kernel?

again, Special thanks

 

[mistwang]

3WARE raid card with 2.6 kernel should work better.
Go with either RAID0 or RAID10 or RAID1, depends on what level of data protection do you need. 32bit or 64bit should not matter much, if you need to serve file larger than 2GB, you have to use 64bit.

 

[IrPr]

Thanks for your advice,
I did setup 3 SATA II in RAID0 ARRAY using 3WARE 6xxx-7xxx controller on 64bit CentOS 5.1 release with 2.6 kernel

no better performances yet
im hosting large files ~100MB and about ~k concurrent connections using epoll as event dispatcher
my iowait value is still too high and seems just one core is used by lshttpd process:

root@mybox[~]# mpstat -P ALL 1
Linux 2.6.18-92.1.10.el5 (mybox) 09/01/2008 _x86_64_

03:45:58 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:45:59 PM all 0.51 0.00 1.01 44.44 1.52 3.03 0.00 0.00 49.49
03:45:59 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:45:59 PM 1 1.02 0.00 1.02 89.80 3.06 5.10 0.00 0.00 0.00

03:45:59 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:00 PM all 0.51 0.00 1.52 37.56 1.52 3.05 0.00 0.00 55.84
03:46:00 PM 0 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.99
03:46:00 PM 1 1.01 0.00 3.03 73.74 3.03 7.07 0.00 0.00 12.12

03:46:00 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:01 PM all 0.00 0.00 1.03 44.62 1.54 2.56 0.00 0.00 50.26
03:46:01 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:01 PM 1 0.00 0.00 2.06 89.69 3.09 5.15 0.00 0.00 0.00

03:46:01 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:02 PM all 1.02 0.00 1.52 43.15 1.52 9.14 0.00 0.00 43.65
03:46:02 PM 0 0.00 0.00 1.02 0.00 0.00 11.22 0.00 0.00 87.76
03:46:02 PM 1 1.02 0.00 3.06 86.73 3.06 6.12 0.00 0.00 0.00

03:46:02 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:03 PM all 0.00 0.00 1.52 44.67 1.52 2.54 0.00 0.00 49.75
03:46:03 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:03 PM 1 1.01 0.00 2.02 88.89 2.02 6.06 0.00 0.00 0.00

03:46:03 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:04 PM all 0.51 0.00 1.03 43.59 1.54 3.08 0.00 0.00 50.26
03:46:04 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:04 PM 1 0.00 0.00 2.04 87.76 4.08 6.12 0.00 0.00 0.00

03:46:04 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:05 PM all 0.51 0.00 1.52 40.10 2.03 3.55 0.00 0.00 52.28
03:46:05 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:05 PM 1 1.01 0.00 3.03 79.80 4.04 6.06 0.00 0.00 6.06

03:46:05 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:06 PM all 0.00 0.00 1.02 44.16 1.52 3.05 0.00 0.00 50.25
03:46:06 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:06 PM 1 1.02 0.00 2.04 87.76 3.06 6.12 0.00 0.00 0.00

03:46:06 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:07 PM all 0.51 0.00 0.51 44.90 1.02 3.06 0.00 0.00 50.00
03:46:07 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:07 PM 1 0.00 0.00 1.03 90.72 2.06 6.19 0.00 0.00 0.00

03:46:07 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:08 PM all 0.00 0.00 1.03 44.10 1.54 3.08 0.00 0.00 50.26
03:46:08 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:08 PM 1 0.00 0.00 3.06 87.76 3.06 6.12 0.00 0.00 0.00

03:46:08 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:09 PM all 0.00 0.00 1.02 43.88 1.53 3.57 0.00 0.00 50.00
03:46:09 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:09 PM 1 1.00 0.00 2.00 87.00 2.00 8.00 0.00 0.00 0.00

03:46:09 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:10 PM all 0.51 0.00 1.53 43.88 1.53 3.06 0.00 0.00 49.49
03:46:10 PM 0 0.00 0.00 1.02 0.00 0.00 0.00 0.00 0.00 98.98
03:46:10 PM 1 0.00 0.00 2.08 88.54 4.17 5.21 0.00 0.00 0.00

03:46:10 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:11 PM all 0.00 0.00 1.02 44.16 1.52 3.55 0.00 0.00 49.75
03:46:11 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:11 PM 1 0.00 0.00 1.02 88.78 3.06 7.14 0.00 0.00 0.00

03:46:11 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:12 PM all 0.00 0.00 1.02 45.41 1.02 2.55 0.00 0.00 50.00
03:46:12 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:12 PM 1 0.00 0.00 2.04 90.82 2.04 5.10 0.00 0.00 0.00

03:46:12 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:13 PM all 0.51 0.00 1.01 43.94 2.02 2.53 0.00 0.00 50.00
03:46:13 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:13 PM 1 1.02 0.00 2.04 88.78 3.06 5.10 0.00 0.00 0.00

03:46:13 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:14 PM all 0.00 0.00 1.03 39.69 0.52 2.58 0.00 0.00 56.19
03:46:14 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:14 PM 1 0.00 0.00 2.02 77.78 2.02 6.06 0.00 0.00 12.12

03:46:14 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:15 PM all 0.51 0.00 1.52 41.62 1.52 2.54 0.00 0.00 52.28
03:46:15 PM 0 0.00 0.00 1.01 0.00 0.00 0.00 0.00 0.00 98.99
03:46:15 PM 1 1.01 0.00 3.03 83.84 3.03 4.04 0.00 0.00 5.05

03:46:15 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:16 PM all 0.00 0.00 1.02 44.67 1.52 3.05 0.00 0.00 49.75
03:46:16 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:16 PM 1 0.00 0.00 2.04 88.78 3.06 6.12 0.00 0.00 0.00

03:46:16 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:46:17 PM all 0.00 0.00 1.03 43.59 2.05 3.08 0.00 0.00 50.26
03:46:17 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:46:17 PM 1 0.00 0.00 2.04 87.76 4.08 6.12 0.00 0.00 0.00

root@mybox[~]#

thats odd
im binding to all cpu cores but core 0 is actually idle while core 1 uses more than 70% iowait
i just host large static files right now on this box with no php/cgi/rubby/mysql process
maybe something buggy with LSWS when hosting large static contents on SMP systems?

any suggestion would be appreciated

Thanks

 

[mistwang]

How about the CPU usage of lshttpd processes, if two lshttpd are configured properly to serve that site, there should be two lshttpd processes on top of "top" output.

 

[IrPr]

How about the CPU usage of lshttpd processes, if two lshttpd are configured properly to serve that site, there should be two lshttpd processes on top of "top" output.

Right, there are 2 lshttpd process in top result

root@mybox [~]# ps axu | grep httpd
root 3004 0.0 0.0 18112 2424 ? S Aug31 0:16 lshttpd
root 3005 0.0 0.0 3780 384 ? S Aug31 0:00 httpd (lscgid)
nobody 3006 1.5 0.2 31780 17684 ? D Aug31 34:05 lshttpd
nobody 3007 1.6 0.2 32504 18348 ? S Aug31 36:48 lshttpd
root 20105 0.0 0.0 61156 724 pts/0 S+ 11:43 0:00 grep httpd
root@mybox [~]#

top - 11:43:52 up 1 day, 13:50, 1 user, load average: 1.96, 2.03, 1.86
Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 1.2%sy, 0.0%ni, 49.4%id, 45.0%wa, 1.2%hi, 2.9%si, 0.0%st
Mem: 8166024k total, 8116624k used, 49400k free, 19884k buffers
Swap: 8193140k total, 10076k used, 8183064k free, 7730104k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3006 nobody 16 0 31780 17m 1436 D 1 0.2 34:05.49 lshttpd
3007 nobody 15 0 32504 17m 1420 D 1 0.2 36:48.97 lshttpd
297 root 10 -5 0 0 0 S 0 0.0 3:02.39 kswapd0
8573 backupdl 15 0 43484 2492 1212 S 0 0.0 1:24.65 proftpd
1 root 15 0 10328 700 588 S 0 0.0 0:00.50 init

top - 11:44:29 up 1 day, 13:51, 1 user, load average: 1.96, 2.03, 1.86
Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 1.3%sy, 0.0%ni, 53.6%id, 40.1%wa, 1.5%hi, 3.1%si, 0.0%st
Mem: 8166024k total, 8119996k used, 46028k free, 19864k buffers
Swap: 8193140k total, 10076k used, 8183064k free, 7734052k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3006 nobody 15 0 31780 17m 1436 D 2 0.2 34:06.02 lshttpd
3007 nobody 16 0 32504 17m 1420 D 2 0.2 36:49.40 lshttpd
1 root 15 0 10328 700 588 S 0 0.0 0:00.50 init
2 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.02 ksoftirqd/0

top - 11:44:44 up 1 day, 13:51, 1 user, load average: 1.82, 1.99, 1.85
Tasks: 111 total, 2 running, 109 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 1.2%sy, 0.0%ni, 53.1%id, 41.1%wa, 1.7%hi, 2.5%si, 0.0%st
Mem: 8166024k total, 8116820k used, 49204k free, 19828k buffers
Swap: 8193140k total, 10076k used, 8183064k free, 7736444k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3007 nobody 15 0 32504 17m 1420 D 2 0.2 36:49.58 lshttpd
3006 nobody 16 0 31780 17m 1436 D 1 0.2 34:06.21 lshttpd
8573 backupdl 15 0 43484 2492 1212 S 0 0.0 1:24.85 proftpd
20196 root 15 0 12584 1068 800 R 0 0.0 0:00.02 top
1 root 15 0 10328 700 588 S 0 0.0 0:00.50 init

[root@mybox ~]# pstree | grep httpd
|-lshttpd-+-httpd
| `-2*[lshttpd]
[root@ns354712 ~]#

root@mybox [~]# mpstat -P ALL 1
Linux 2.6.18-92.1.10.el5 (mybox) 09/02/2008 _x86_64_

11:44:59 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
11:45:00 AM all 0.00 0.00 0.51 44.62 1.54 3.08 0.00 0.00 50.26
11:45:00 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
11:45:00 AM 1 1.02 0.00 2.04 87.76 3.06 6.12 0.00 0.00 0.00

11:45:00 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
11:45:01 AM all 0.51 0.00 1.01 44.44 1.52 2.53 0.00 0.00 50.00
11:45:01 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
11:45:01 AM 1 0.00 0.00 2.02 89.90 3.03 5.05 0.00 0.00 0.00

11:45:01 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
11:45:02 AM all 14.65 0.00 8.08 33.84 1.01 14.14 0.00 0.00 28.28
11:45:02 AM 0 15.00 0.00 7.00 13.00 0.00 8.00 0.00 0.00 57.00
11:45:02 AM 1 14.14 0.00 9.09 53.54 2.02 21.21 0.00 0.00 0.00

11:45:02 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
11:45:03 AM all 1.53 0.00 2.04 42.86 1.02 2.55 0.00 0.00 50.00
11:45:03 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
11:45:03 AM 1 4.04 0.00 3.03 85.86 2.02 5.05 0.00 0.00 0.00

11:45:03 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
11:45:04 AM all 0.51 0.00 1.01 44.44 1.52 2.53 0.00 0.00 50.00
11:45:04 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
11:45:04 AM 1 0.00 0.00 2.06 89.69 3.09 5.15 0.00 0.00 0.00

 

[mistwang]

From the CPU cycles consumed by the two lshttpd processes, two processes has been well balanced, so the configuration should be all right.

And it does look like the kernel allocate I/O operations to only one CPU core only, it is up to the kernel I/O scheduler, not much we can do about in user land applications.

It might be at the peak I/O throughput for the disk array, since those are random read accesses. there are a few things we can improve if have not done yet.

Make sure NCQ has been enabled for the RAID card and hard disks. try a different I/O scheduler in the kernel.

We can try a 4-CPU or 8-CPU license on this server, see if it get any better when more I/O worker processes are used. It may help a little with NCQ enabled I think.

 

[IrPr]

From the CPU cycles consumed by the two lshttpd processes, two processes has been well balanced, so the configuration should be all right.

And it does look like the kernel allocate I/O operations to only one CPU core only, it is up to the kernel I/O scheduler, not much we can do about in user land applications.

It might be at the peak I/O throughput for the disk array, since those are random read accesses. there are a few things we can improve if have not done yet.

Make sure NCQ has been enabled for the RAID card and hard disks. try a different I/O scheduler in the kernel.

We can try a 4-CPU or 8-CPU license on this server, see if it get any better when more I/O worker processes are used. It may help a little with NCQ enabled I think.

WOW! AWESOME!

switched IO scheduler to deadline and some other tips with kernel, now server performances increased dramatically

this mrtg graph indicates that:

Look output is about 3x now

Thanks George, you are the man!

 

[mistwang]

Glad it work out beautifully.

What is the most important factor on this, I/O scheduler? or other kernel tuning tips?

 

[IrPr]

Glad it work out beautifully.

What is the most important factor on this, I/O scheduler? or other kernel tuning tips?

• Switching to deadline I/O Scheduler has the most efficient
• set READAHEAD to much higher value, changed 256 to 16M
• increased VM dirty ratio 20 values

 

[mistwang]

Thanks!
We will add a wiki entry for this.