If your PHP script need to communicate with remote server, you need to make sure DNS is fast. A local forward DNS server which can cache DNS result is highly recommended.
Your server load is not high, no mysql, so I guess most PHP are just waiting for the remote server, blocking on sockets. That's kind of typical result when server talks to remote server.
Try "strace -tt -T -p <pid_of_php>", you can have an idea what slow down PHP.