[solve] cpu usage high

#1
hi guys

i'm hosting a site that gets 1M+ visits a day in litespeed right now.
the site is a combination of php and cgi and i noticed that my cpu's hover around 20-30 percent w/ load avg around 8-15. server has 32 cpu cores of intel romley btw. litespeed is configured with lsphp + suexec.

the server has a ton of ram (64gb), but it is only utilizing 2-3 percent of it and i was wondering if i haven't configured litespeed correctly for this. disk is SSD w/ disk i/o less than 2 percent;. mysql is hosted on separate server.

is there some setting that i'm not aware of?

i've upped all the sysctl settings, like this:

net.ipv4.netfilter.ip_conntrack_max=524288
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65023
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.ipv4.tcp_synack_retries = 3
net.core.somaxconn = 10000

not sure if it's too low or too high.

can i please get some assistance in optimizing our current situation?
 
Last edited by a moderator:
#2
here is a snapshot of the top:

top - 13:55:25 up 41 days, 1:55, 2 users, load average: 11.67, 11.76, 11.85
Tasks: 469 total, 12 running, 457 sleeping, 0 stopped, 0 zombie
Cpu0 : 46.7%us, 3.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 16.7%hi, 33.3%si, 0.0%st
Cpu1 : 50.0%us, 15.6%sy, 0.0%ni, 31.2%id, 0.0%wa, 0.0%hi, 3.1%si, 0.0%st
Cpu2 : 63.6%us, 12.1%sy, 0.0%ni, 21.2%id, 0.0%wa, 0.0%hi, 3.0%si, 0.0%st
Cpu3 : 68.8%us, 12.5%sy, 0.0%ni, 18.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 70.0%us, 13.3%sy, 0.0%ni, 16.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 78.1%us, 12.5%sy, 0.0%ni, 9.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 74.2%us, 12.9%sy, 0.0%ni, 12.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 75.0%us, 12.5%sy, 0.0%ni, 9.4%id, 0.0%wa, 0.0%hi, 3.1%si, 0.0%st
Cpu8 : 62.5%us, 12.5%sy, 0.0%ni, 25.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 67.7%us, 9.7%sy, 0.0%ni, 22.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 61.3%us, 12.9%sy, 0.0%ni, 25.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 40.6%us, 6.2%sy, 0.0%ni, 53.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 : 35.5%us, 3.2%sy, 0.0%ni, 61.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 : 12.9%us, 3.2%sy, 0.0%ni, 83.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 : 15.6%us, 3.1%sy, 0.0%ni, 81.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu16 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu17 : 16.7%us, 3.3%sy, 0.0%ni, 80.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu18 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu19 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu20 : 9.7%us, 0.0%sy, 0.0%ni, 90.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu21 : 6.7%us, 0.0%sy, 0.0%ni, 93.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu22 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu23 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu24 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu25 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu26 : 12.5%us, 9.4%sy, 0.0%ni, 78.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu27 : 75.0%us, 15.6%sy, 0.0%ni, 9.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 51532768k total, 35681456k used, 15851312k free, 554420k buffers
Swap: 7124816k total, 0k used, 7124816k free, 32931908k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4852 someuser 25 0 73480 11m 2328 R 28.8 0.0 0:00.09 index_dl.cgi
4861 someuser 25 0 73472 11m 2328 R 22.4 0.0 0:00.07 index_dl.cgi
5736 nobody 0 -19 124m 21m 6872 S 22.4 0.0 31:23.28 litespeed
4863 someuser 25 0 73424 11m 2328 R 19.2 0.0 0:00.06 index_dl.cgi
4864 someuser 25 0 73376 11m 2332 R 16.0 0.0 0:00.05 index_dl.cgi
4866 someuser 25 0 42720 9816 1484 R 16.0 0.0 0:00.05 index_dl.cgi
4865 someuser 25 0 69200 10m 2304 S 12.8 0.0 0:00.04 index_dl.cgi
4870 someuser 24 0 38708 7568 1452 R 9.6 0.0 0:00.03 index_dl.cgi
4869 someuser 25 0 27400 4548 1424 R 6.4 0.0 0:00.02 index_dl.cgi
4874 someuser 24 0 27380 4528 1424 R 6.4 0.0 0:00.02 index_dl.cgi
2417 someuser 15 0 138m 9868 6284 S 3.2 0.0 0:00.16 lsphp5
2653 root 15 0 13020 1424 828 R 3.2 0.0 0:00.27 top
4701 someuser 15 0 138m 9552 5900 S 3.2 0.0 0:00.04 lsphp5
4791 someuser 18 0 69196 10m 2288 S 3.2 0.0 0:00.05 index_dl.cgi
4875 someuser 24 0 21348 2748 1356 R 3.2 0.0 0:00.01 index_dl.cgi
4878 someuser 24 0 21240 2636 1352 R 3.2 0.0 0:00.01 index_dl.cgi
5718 root 4 -19 5892 440 352 S 3.2 0.0 1:16.52 httpd
1 root 15 0 10364 692 580 S 0.0 0.0 0:06.43 init
2 root RT -5 0 0 0 S 0.0 0.0 0:15.52 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:03.30 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:17.37 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.26 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:21.21 migration/2
9 root 34 19 0 0 0 S 0.0 0.0 0:00.24 ksoftirqd/2
10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
11 root RT -5 0 0 0 S 0.0 0.0 0:14.93 migration/3
12 root 34 19 0 0 0 S 0.0 0.0 0:00.43 ksoftirqd/3
13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
14 root RT -5 0 0 0 S 0.0 0.0 0:28.79 migration/4
15 root 34 19 0 0 0 S 0.0 0.0 0:00.21 ksoftirqd/4
16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4
17 root RT -5 0 0 0 S 0.0 0.0 0:29.10 migration/5
18 root 34 19 0 0 0 S 0.0 0.0 0:00.27 ksoftirqd/5
 
#4
yes but when load avg starts to reach 15+ (indicating more traffic), the site starts to slow down and response time of site starts to become greater than 4+ secs, when normally it's 1 second or so.
could i be hitting some max connections settings (either litespeed and/or php, kernel, etc) and it is placed in queue?
 

NiteWave

Administrator
#5
so when slow down, capture the screen shot of real-time stats and paste here. it provides many useful info there. most likely can identify where the bottle-neck is.
 
#6
Hi Nitewave, site is taking a while to load now. here is a screen paste of w, top, and iostat:


// w

-bash-3.2# w
14:30:52 up 6 days, 22:05, 2 users, load average: 20.58, 22.98, 24.17

// top by cpus

top - 14:31:41 up 6 days, 22:06, 2 users, load average: 21.18, 22.81, 24.05
Tasks: 567 total, 17 running, 550 sleeping, 0 stopped, 0 zombie
Cpu0 : 7.7%us, 7.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 30.8%hi, 53.8%si, 0.0%st
Cpu1 : 53.3%us, 6.7%sy, 0.0%ni, 33.3%id, 0.0%wa, 0.0%hi, 6.7%si, 0.0%st
Cpu2 : 71.4%us, 14.3%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 50.0%us, 14.3%sy, 0.0%ni, 28.6%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Cpu4 : 71.4%us, 14.3%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 61.5%us, 7.7%sy, 0.0%ni, 30.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 61.5%us, 7.7%sy, 0.0%ni, 30.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 78.6%us, 14.3%sy, 0.0%ni, 7.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 50.0%us, 7.1%sy, 0.0%ni, 42.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 71.4%us, 14.3%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 66.7%us, 8.3%sy, 0.0%ni, 25.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 64.3%us, 14.3%sy, 0.0%ni, 21.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 : 78.6%us, 14.3%sy, 0.0%ni, 7.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 : 64.3%us, 7.1%sy, 0.0%ni, 28.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 28.6%us, 7.1%sy, 0.0%ni, 57.1%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Cpu15 : 64.3%us, 7.1%sy, 0.0%ni, 28.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu16 : 21.4%us, 7.1%sy, 0.0%ni, 64.3%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Cpu17 : 76.9%us, 15.4%sy, 0.0%ni, 7.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu18 : 21.4%us, 14.3%sy, 0.0%ni, 42.9%id, 0.0%wa, 0.0%hi, 21.4%si, 0.0%st
Cpu19 : 50.0%us, 7.1%sy, 0.0%ni, 42.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu20 : 14.3%us, 14.3%sy, 0.0%ni, 71.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu21 : 61.5%us, 7.7%sy, 0.0%ni, 23.1%id, 0.0%wa, 0.0%hi, 7.7%si, 0.0%st
Cpu22 : 46.2%us, 7.7%sy, 0.0%ni, 46.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu23 : 71.4%us, 7.1%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Cpu24 : 50.0%us, 7.1%sy, 0.0%ni, 42.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu25 : 46.2%us, 7.7%sy, 0.0%ni, 38.5%id, 0.0%wa, 0.0%hi, 7.7%si, 0.0%st
Cpu26 : 76.9%us, 15.4%sy, 0.0%ni, 7.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu27 : 53.8%us, 7.7%sy, 0.0%ni, 38.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu28 : 46.2%us, 7.7%sy, 0.0%ni, 46.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu29 : 64.3%us, 14.3%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Mem: 51532700k total, 48579012k used, 2953688k free, 550444k buffers
Swap: 7124816k total, 0k used, 7124816k free, 44709260k cached



// top

top - 14:31:09 up 6 days, 22:06, 2 users, load average: 21.34, 23.01, 24.16
Tasks: 567 total, 20 running, 547 sleeping, 0 stopped, 0 zombie
Cpu(s): 63.0%us, 12.0%sy, 0.0%ni, 20.8%id, 0.0%wa, 1.0%hi, 3.1%si, 0.0%st
Mem: 51532700k total, 48567912k used, 2964788k free, 550444k buffers
Swap: 7124816k total, 0k used, 7124816k free, 44704736k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15041 nobody 0 -19 79736 30m 6952 S 53.1 0.1 8:09.29 litespeed
20394 someuser 25 0 69064 10m 2076 S 9.0 0.0 0:00.27 index_dl.cgi
20756 someuser 25 0 73624 11m 2360 R 8.6 0.0 0:00.26 index_dl.cgi
16314 someuser 15 0 138m 9876 6292 S 4.6 0.0 0:00.48 lsphp5
20634 someuser 25 0 69248 10m 2288 S 4.3 0.0 0:00.13 index_dl.cgi
20800 someuser 25 0 73580 11m 2360 R 4.0 0.0 0:00.12 index_dl.cgi
4757 someuser 15 0 138m 10m 6616 S 3.7 0.0 0:01.70 lsphp5
7801 someuser 15 0 138m 13m 8968 S 3.7 0.0 0:01.49 lsphp5
20808 someuser 25 0 42720 9816 1484 R 3.7 0.0 0:00.11 index_dl.cgi
20803 someuser 25 0 79836 12m 2396 S 3.3 0.0 0:00.10 index_dl.cgi
20807 someuser 18 0 42716 9800 1484 R 3.3 0.0 0:00.10 index_dl.cgi
20812 someuser 25 0 73580 11m 2360 S 3.3 0.0 0:00.10 index_dl.cgi
11907 someuser 15 0 139m 13m 8904 S 3.0 0.0 0:00.85 lsphp5
12113 someuser 16 0 139m 13m 8968 S 3.0 0.0 0:00.90 lsphp5
19864 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
19865 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
19882 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20115 someuser 25 0 69068 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20291 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20354 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20474 someuser 23 0 69200 10m 2272 S 3.0 0.0 0:00.09 index_dl.cgi
20530 someuser 17 0 69196 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
20567 someuser 25 0 69068 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20621 someuser 25 0 69248 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
20639 someuser 25 0 69248 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
20665 someuser 25 0 69196 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
20714 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20717 someuser 25 0 69200 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
19868 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19915 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19916 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19923 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19937 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20047 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20048 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20072 someuser 25 0 69064 10m 2080 S 2.7 0.0 0:00.08 index_dl.cgi
20078 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20292 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20294 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20301 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20353 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20361 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20410 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20415 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20416 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20468 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20469 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20471 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20819 someuser 19 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19944 someuser 25 0 69068 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20131 someuser 25 0 69068 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20133 someuser 25 0 69064 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20171 someuser 25 0 69068 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20296 someuser 25 0 69064 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20366 someuser 25 0 69064 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20460 someuser 25 0 69200 10m 2288 S 2.3 0.0 0:00.07 index_dl.cgi
20585 someuser 25 0 69064 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20748 someuser 25 0 69248 10m 2288 S 2.3 0.0 0:00.07 index_dl.cgi
20805 someuser 25 0 69200 10m 2288 S 2.3 0.0 0:00.07 index_dl.cgi
20815 someuser 25 0 69204 10m 2248 S 2.3 0.0 0:00.07 index_dl.cgi
20823 someuser 25 0 42720 9812 1484 R 2.3 0.0 0:00.07 index_dl.cgi
20824 someuser 25 0 42716 9812 1484 R 2.3 0.0 0:00.07 index_dl.cgi
15011 root 6 -19 5892 436 352 S 2.0 0.0 0:19.76 httpd
20239 someuser 25 0 69064 10m 2076 S 2.0 0.0 0:00.06 index_dl.cgi
20271 someuser 25 0 69064 10m 2076 S 2.0 0.0 0:00.06 index_dl.cgi
20588 someuser 25 0 69068 10m 2080 S 2.0 0.0 0:00.06 index_dl.cgi



// iostat
-bash-3.2# iostat
Linux 2.6.18-308.11.1.el5 10/18/2012

avg-cpu: %user %nice %system %iowait %steal %idle
19.63 0.03 5.86 0.01 0.00 74.47

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
hda 10.52 9.09 709.88 5434707 424570329
hda1 0.00 0.00 0.00 2707 71
hda2 5.21 0.27 126.23 161635 75498586
hda3 0.00 0.00 0.00 1459 0
hda4 0.00 0.00 0.00 8 0
hda5 5.30 8.81 583.65 5268306 349071672



Please let me know if there is something I can do to optimize.
 

webizen

Well-Known Member
#7
it seems your cgi scripts take most cpu cycles. so your application is more of cpu bound.

take a look at real-time stats in LiteSpeed Admin Console (Actions=> Real-Time Stats) and paste a screen shot here (as NiteWave requested). It should tell us whether your application processes are piling up and where bottleneck if any, etc.
 

webizen

Well-Known Member
#9
Is this for busy time? lsws seems to be ok. NO processes in WaitQ which means no delay in processing PHP or CGI. BTW, is your up link 100Mb/s or 1Gb/s?
 
#10
i don't think we really have a busy time, traffic is constant throughout as we also get lot of international traffic.
uplink is 100mb. do you think we are hitting a bottleneck in network?
i see we use 30mb down and 30-50mb up.
 

webizen

Well-Known Member
#11
for 100Mb/s uplink, outbound (serving end user) can go 50Mb/s-90Mb/s depending on your traffic pattern. if you have many small downloads, 50-60Mb/s is probably the limit due to tcp overhead.
 
Top