User C complained of 503 errors in their shared servers. Looking in one server, we found many entries like the following in their stderr.log:
2014-01-29 22:11:43.214 [STDERR] Child process with pid: 6259 was killed by signal: 11, core dump: 128 2014-01-29 22:11:44.374 [STDERR] Child process with pid: 6263 was killed by signal: 11, core dump: 128 2014-01-29 22:11:54.171 [STDERR] Child process with pid: 6277 was killed by signal: 11, core dump: 128 2014-01-29 22:11:54.581 [STDERR] Child process with pid: 6276 was killed by signal: 11, core dump: 128 2014-01-29 22:11:55.373 [STDERR] Child process with pid: 6284 was killed by signal: 11, core dump: 128 2014-01-29 22:11:55.941 [STDERR] zend_mm_heap corrupted 2014-01-29 22:12:03.113 [STDERR] Child process with pid: 6320 was killed by signal: 11, core dump: 128 2014-01-29 22:12:03.943 [STDERR] Child process with pid: 6322 was killed by signal: 11, core dump: 128 2014-01-29 22:12:05.759 [STDERR] Child process with pid: 6324 was killed by signal: 11, core dump: 128 2014-01-29 22:12:12.723 [STDERR] Child process with pid: 6326 was killed by signal: 11, core dump: 128
Seeing that PHP was crashing (as demonstrated by the killed by signal: 11
), we guessed that the problem might be the opcode cache. We turned off APC and the errors stopped. Opcode cache is a frequent cause of PHP crashes.