Apache config mode + vhost-level External App handlers

[jdr2]

I'm currently experimenting with migrating an existing Apache + PHP-FPM setup to LSWS, and ran into an issue involving External App definitions. This is using LSWS 6.0.8.

For storage latency reasons, we prefer to have our PHP processes always running, to ensure that opcaches are maximally kept alive and thus minimize the need to hit the disk (even at the cost of consuming more memory). For that reason, we currently have a bunch of separate always-on PHP-FPM processes, one per user, with each vhost configured to talk to its own instance over a UNIX socket.

To run this under LSWS, ProcessGroup mode seems pretty close, but there doesn't seem to be a way to get around the finite lifetime of the lsphp processes it spawns. Instead, I found this article titled "Set up LSPHP in Command Line Mode" that describes how to run LSPHP as separate instances, and then how to get LSWS to talk to them as External Apps.

That works great and matches our existing setup closely, but there's one problem: using a server-level External App definition as described in that article, I can't get individual vhosts (as parsed from their Apache configs) to talk to individual UNIX sockets. I would've expected that using a $VH_NAME or $VH_USER variable in the External App's address line would've worked for this, but it turns out these variables are empty at runtime.



Trying to run a PHP script under such a vhost discovered through Apache config results in a 503 with this in the error.log:

2022-08-16 14:21:32.398793 [INFO] [1313881] [T0] [192.168.56.1:44716:HTTP2-15#_AdminVHost] add child process pid: 1313896, procinfo: 0x28ee480
2022-08-16 14:21:37.215833 [INFO] [1313881] [T0] [192.168.56.1:51406-1#APVH_acc1.be:81] connection to [uds://tmp/lsphp-.sock] on request #0, confirmed, 0, associated process: 0, running: 0, error: No such file or directory!
2022-08-16 14:21:37.215838 [INFO] [1313881] [T0] [uds://tmp/lsphp-.sock] Connection refused, restart!
2022-08-16 14:21:37.215852 [INFO] [1313881] [T0] [192.168.56.1:51406-1#APVH_acc1.be:81] connection to [uds://tmp/lsphp-.sock] on request #0, confirmed, 0, associated process: 0, running: 0, error: No such file or directory!
2022-08-16 14:21:37.215853 [INFO] [1313881] [T0] [uds://tmp/lsphp-.sock] Connection refused, restart!
2022-08-16 14:21:37.215860 [INFO] [1313881] [T0] [192.168.56.1:51406-1#APVH_acc1.be:81] connection to [uds://tmp/lsphp-.sock] on request #0, confirmed, 0, associated process: 0, running: 0, error: No such file or directory!
2022-08-16 14:21:37.215861 [INFO] [1313881] [T0] [uds://tmp/lsphp-.sock] Connection refused, restart!
2022-08-16 14:21:37.215862 [NOTICE] [1313881] [T0] [192.168.56.1:51406-1#APVH_acc1.be:81] Max retries has been reached, 503!
2022-08-16 14:21:37.215872 [NOTICE] [1313881] [T0] [192.168.56.1:51406-1#APVH_acc1.be:81] oops! 503 Service Unavailable

Which indeed shows that the variable is not getting filled in.

I figured this might have something to do with the fact that the Handler Name field in the Script Handler section mentions "Server Level". Indeed, if I replace the Apache-loaded configs with LSWS-native vhosts (i.e. disable loading of Apache configs and configure the vhosts in XML instead), and then add the same External App and Script Handler at the Virtual Host level, then the variable does get substituted and everything works ok.

That kind of makes sense, since at the server-level there is no individual user/vhost name to speak of yet, but I would've expected the definition to get inherited by vhosts and have its variables substituted only there.

Is there a way to get vhosts loaded from Apache to use vhost-level External App definitions like this, in which $VH_USER and $VH_NAME would work? Or some other method of pointing each one to a vhost-specific UNIX socket for external LSPHP processes? For ProcessGroup mode there are some LSWS-specific Apache directives listed, but I couldn't find any for defining external apps this way.

 

[mistwang]

Just use detached ProcessGroup with very long "Idle timeout" should be very close to your desired setup.
It saves a lot of headache for you to manage those manually.

 

[jdr2]

Just use detached ProcessGroup with very long "Idle timeout" should be very close to your desired setup.
It saves a lot of headache for you to manage those manually.

I've considered that as well, but there's the additional complication that I need the PHP processes to run with certain mount namespaces applied to ensure isolation between users, i.e. make certain paths on disk invisible to PHP processes by mounting empty tmpfs namespaces over them. Regular chmod file permissions are insufficient for this, they really must be invisible.

This requires a wrapper script to be run as root, apply the mount namespace, and only then switch UID/GID and exec lsphp. This is easily done with separate processes started in advance, e.g. through systemd units, but I don't think I can do this through ProcessGroup mode, since LSWS already applies the chroot and user/group switching before whatever executable you have configured gets exec'd.

I've looked into the recent bubblewrap integration, but I haven't been able to get that to work yet, and doesn't seem to be particularly straightforward to inspect/troubleshoot.
So still hoping for a way to achieve the original idea ..

 

[mistwang]

I see. You can look into bubblewrap more.

 

[jdr2]

So I did look into bubblewrap some more, and it turns out that it cannot work in combination with <enableChroot>1</enableChroot> in httpd_config.xml. The reason is that the Linux kernel will not allow a process that's chroot'ed to create a new user namespace, as that would grant it capabilities to escape the chroot within the new user namespace.

As documented on https://www.man7.org/linux/man-pages/man2/clone.2.html:

EPERM (since Linux 3.9)
              CLONE_NEWUSER was specified in the flags mask and the
              caller is in a chroot environment (i.e., the caller's root
              directory does not match the root directory of the mount
              namespace in which it resides).

Indeed, when both chroot and bubblewrap are enabled in the LSWS config and a PHP process is spawned, it first applies the chroot, then switches UID/GID, and then execve()'s bubblewrap. At that point, bubblewrap is no longer able to create a new user namespace because the calling process is chroot'ed at that point.

(Found no mention of this on the Bubblewrap documentation page, might be good to include this information there?)

I could disable chrooting in httpd_config.xml, but then the litespeed (lshttpd - #0x) processes are no longer chroot'ed either.