LSAPI PHP output streaming

[foxyfred]

Hello! First off, let me thank you for a great server; we replaced Apache with Litespeed about 6 months ago and it's been rock solid.

I'm running into a problem right now with PHP scripts who output more than around 120MB of data. When I use curl to request the file, it works fine:
> GET dl.php HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3
> Host: localhost
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 29 Jul 2011 17:56:43 GMT
< Connection: close
< Content-Type: application/octet-stream
< Content-Description: File Transfer
< Content-Disposition: attachment; filename="tutorial.mov";
< Content-Transfer-Encoding: binary
< Content-Length: 152466240
< … 150 MB of binary data

However, when hitting the same script from a browser, this results in a zero-byte download. I traced that down to the browser requesting transfer body compression with an "Accept-Encoding: gzip, deflate" header.

When I supply that header via curl, I see the exact same behavior as the browser:
> GET dl.php HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3
> Host: localhost
> Accept: */*
< HTTP/1.1 200 OK
< Date: Fri, 29 Jul 2011 17:56:43 GMT
< Connection: close
< Content-Type: application/octet-stream
< Content-Description: File Transfer
< Content-Disposition: attachment; filename="tutorial.mov";
< Content-Transfer-Encoding: binary
< Content-Length: 152466240
* transfer closed with 152466240 bytes remaining to read

The response is sent using PHP's readfile() mechanism, which writes a complete file directly to PHP's output. I don't know how it buffers the file internally, but it does work with smaller files.

Here's what I've done:
- Make sure that PHP's output compression is disabled. (Not only is it disabled via .htaccess, but it's explicitly disabled in this script using ini_set)
- Try disabling PHP's output buffering before sending the response. Same behavior.
- Disable Litespeed's dynamic output compression. Didn't make a difference (it shouldn't have mattered, anyway, because the mime-types enabled it only for CSS, Javascript, and text/* output).
- Made sure that the lsphp5 external application settings have Response Buffering disabled.
- Change the "Max Dynamic Response Size" setting from 500M to 1000M. Same behavior.

I've checked the Litespeed SAPI source code, and it does not do any kind of output compression, and appears to send the response to Litespeed in 16K packets.

The lsphp5 EA settings mention next to the "response buffering" setting something about enabling Apache's non-parsed headers. I can't find any setting that does this, but that Apache feature does appear to do what we want — send the response directly from the script to the browser without buffering.

Any thoughts? The most telling symptom, to me, is that passing in the Accept-Encoding header immediately triggers the bug with no warnings or errors in the output, or the PHP error log, or the Litespeed error logs. Thanks in advance!

 

[webizen]

what's the PHP memory_limit?

/usr/local/lsws/fcgi-bin/lsphp5 -i | grep memory_limit

 

[foxyfred]

what's the PHP memory_limit?

/usr/local/lsws/fcgi-bin/lsphp5 -i | grep memory_limit 
memory_limit => 200M => 200M

That reminds me, I bumped the memory_limit from 128M to 200M when I started having trouble. I may try bumping it up really high to see if that solves the problem.

I looked at the source to PHP's

readfile()

and it does an 8K-buffered memory-mapped stream of the file directly to the SAPI output using PHP's streams library.

 

[foxyfred]

OK, just tried bumping PHP's memory limit to 500M, same result.

Also note that Litespeed's external app memory limits for lsphp5 are:
- Soft limit: 600M
- Hard limit: 1000M

I assume that the since the PHP interpreter's limit is set lower that will govern the actual memory limits.

 

[foxyfred]

OK, after some more testing, it looks like a this bug with PHP's memory-mapped IO. When I do a loop in the script to stream the file contents out 8K at a time, it works, although it eats up crazy CPU. I'm going to work more to see if I can find why this is triggered by the gzip/deflate encoding request.

 

[foxyfred]

So, I fixed the problem, but the cause is still a mystery.

It turns out that the PHP zlib.output_compression setting WAS turned on for this script, which is surprising because I explicitly disabled in the site's .htaccess, and it's not in the "Apache-style settings" in the Litespeed control panel.

Disabling it in the script with ini_set() wasn't enough, I had to explicitly disable it for the script with this .htaccess entry:

<Location /dl.php>
php_flag zlib.output_compression Off
</Location>

The mystery is this: the setting's disabled in php.ini, it's not enabled by Litespeed settings, and unless I explicitly disable it for this script PHP tries to compress the output.

Any ideas why the flag would get turned on? I can't find anything in the Litespeed PHP SAPI that does this, so I wonder if there's some environment variable passed by Litespeed that causes this.

 

[mistwang]

must be triggered by request header "Accept-Encoding: gzip, deflate" .

 

[foxyfred]

I agree, but I can't find where in the PHP engine this happens.

I find that it specially disables "zlib.output_compression" for image/* response MIME types, but can't find anything that implicitly enables "zlib.output_compression" based on that header's presence.

 

[cmanns]

Is it a static file?

Use X-Sendfile.

 

[foxyfred]

Good idea, thanks. However, in this particular case it won't help — the script needs to stay "on the line" to make sure the browser completely received the download before doing some accounting and recording the completed download.

Think of where you sell a customer an eBook, you only want them to be able to download the file 5 times: you want to make sure they really have the file before counting it in order to avoid support headaches.