High APC Misses Rate

semprot

Well-Known Member
#1
On last 2 days, sometimes my APC misses rate is almost 100%, with "hits" only 1, and "misses" more than 1000 (i can't remember the exact "misses" value, but the "hits" is only 1 for sure).
When i restarted the httpd, everything went fine again.
Here are some lines from stderr.log, any useful information there?

Code:
2013-11-11 04:27:23.315 [STDERR] Killing runaway process PID: 11835 with SIGTERM
2013-11-11 04:27:23.316 [STDERR] Child process with pid: 11835 was killed by signal: 15, core dump: 0
2013-11-11 04:48:07.003 [STDERR] Child process with pid: 19554 was killed by signal: 15, core dump: 0
2013-11-11 05:31:32.005 [STDERR] Child process with pid: 1532 was killed by signal: 15, core dump: 0
2013-11-11 06:05:23.003 [STDERR] Child process with pid: 15136 was killed by signal: 15, core dump: 0
2013-11-11 06:05:23.007 [STDERR] Child process with pid: 14926 was killed by signal: 15, core dump: 0
2013-11-11 06:05:25.009 [STDERR] Child process with pid: 14108 was killed by signal: 15, core dump: 0
...
and lot of "Child process with pid" lines (same as above)
And there is 1 unique line there
Code:
2013-11-11 07:37:26.098 [STDERR] lscgid: execve():/usr/local/cpanel/cgi-sys/index.html: Permission denied
Additional information:
I just found that someone scanned my server with malicious purpose and he/she generates lot of 404 error on my error log.
Maybe the 404 not found causes lot of misses? (because the file does not exist so APC can't cache it?)
 
Last edited:

NiteWave

Administrator
#2
looks php crashed.
keep monitoring it, if high frequency, need review what changes has made recently, and try to revert back the changes, then monitor ...
 

semprot

Well-Known Member
#3
looks php crashed.
keep monitoring it, if high frequency, need review what changes has made recently, and try to revert back the changes, then monitor ...
On my error log, lot of lines like these on recent days.
Website is still very fast anyway, except if APC misses rate explodes again.

Last change was APC but that was days before error started to happen.

Code:
2013-11-12 04:20:40.057 [NOTICE] [46.119.114.140:57627-0#APVH_xxx.com] No request delivery notification has been received from LSAPI process group [-1], possible run away process.
2013-11-12 04:20:40.057 [NOTICE] [46.119.114.140:57627-0#APVH_xxx.com] Retry with new process group.
2013-11-12 04:20:50.091 [NOTICE] [46.119.114.140:57627-0#APVH_xxx.com] No request delivery notification has been received from LSAPI process group [-1], possible run away process.
2013-11-12 04:20:50.091 [NOTICE] [46.119.114.140:57627-0#APVH_xxx.com] Retry with new process group.
Code:
2013-11-12 04:19:25.821 [INFO] [173.245.51.234:56823-0#APVH_xxx.com] Abort request processing by PID:21179, kill: 1, begin time: 10, sent time: 0, req processed: 0
2013-11-12 04:19:26.242 [INFO] [CLEANUP] Send signal: 10 to process: 21179
2013-11-12 04:19:38.808 [INFO] [141.0.9.4:37279-0#APVH_xxx.com] Abort request processing by PID:14313, kill: 1, begin time: 2, sent time: 2, req processed: 1989
2013-11-12 04:19:39.242 [INFO] [CLEANUP] Send signal: 10 to process: 14313
2013-11-12 04:19:46.016 [INFO] [54.244.60.133:34675-0#APVH_xxx.com] Abort request processing by PID:20868, kill: 1, begin time: 1, sent time: 1, req processed: 138
Code:
2013-11-12 04:09:32.378 [INFO] [180.254.214.81:15402-0#APVH_xxx.com] connection to [uds://tmp/lshttpd/lsphp5.sock.14670] on request #2500, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer!
 
Last edited:

semprot

Well-Known Member
#5
I still get this line

Code:
2013-11-13 00:34:22.203 [INFO] [180.241.221.232:27770-0] Connection idle time: 121 while in state: 1 watching for event: 25,close!

2013-11-13 00:40:32.425 [INFO] [39.222.96.93:8078-0#APVH_xxx.com] Abort request processing by PID:22574, kill: 0, begin time: 0, sent time: 0, req processed: 39
Maybe it's time for hardware upgrade.
 

NiteWave

Administrator
#6
since lsapi 6.4 has upgraded a few times, you can Build Match PHP again to ensure it's built with latest lsapi 6.4. then do a force-reinstall 4.2.5.

2013-11-13 00:34:22.203 [INFO] [180.241.221.232:27770-0] Connection idle time: 121 while in state: 1 watching for event: 25,close!
for this error message, you have to increase
Server->Tuning->Connection Timeout (secs):120
as we discussed before.
 

mistwang

LiteSpeed Staff
#7
I still get this line

Code:
2013-11-13 00:34:22.203 [INFO] [180.241.221.232:27770-0] Connection idle time: 121 while in state: 1 watching for event: 25,close!

2013-11-13 00:40:32.425 [INFO] [39.222.96.93:8078-0#APVH_xxx.com] Abort request processing by PID:22574, kill: 0, begin time: 0, sent time: 0, req processed: 39
Maybe it's time for hardware upgrade.
Likely something wrong with the PHP or MySQL code if the request should not take more than 120 seconds to finish.
The second log entry tells that the request was not aborted.
 

semprot

Well-Known Member
#8
since lsapi 6.4 has upgraded a few times, you can Build Match PHP again to ensure it's built with latest lsapi 6.4. then do a force-reinstall 4.2.5.


for this error message, you have to increase
Server->Tuning->Connection Timeout (secs):120
as we discussed before.
I still use 4.2.4. Is 4.2.5 stable yet? Will do lsapi 6.4 recompiling later.

Likely something wrong with the PHP or MySQL code if the request should not take more than 120 seconds to finish.
The second log entry tells that the request was not aborted.
Probably i overtuned the APC, but i will try nitewave's suggestion first.
 

semprot

Well-Known Member
#9
I did 4.2.5 upgrade & LSAPI 6.4 & SSD drive at the same time.
After the upgrade everything is slowed down. Not sure which one caused it, whether LSWS / lsapi / ssd upgrade.
Is 4.2.5 stable?
 
Top