Out of swapping space while reading request body!

[J.T.]

Got this "out of swapping space while reading request body!" just now on an Enterprise license whilst trying to use a webmail kind of script, sending out an email with two attachments. One just short of 2Mb and one not even 1Mb. PHP upload limit is 12Mb.

# df -m
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda2 19840 4708 14109 26% /
/dev/sda5 209559 37940 160802 20% /home
/dev/sda1 99 24 71 25% /boot
tmpfs 1976 0 1976 0% /dev/shm

Plenty of disk space and with 4Gb of ram top normally says something like:

Mem: 4046812k total, 3514924k used, 531888k free, 324524k buffers
Swap: 1052248k total, 84k used, 1052164k free, 1970004k cached

Swap is hardly ever needed.

Restarting LSWS reenabled me to send out this email. So problem solved in that sense but I'd like to understand this issue better and see if I can prevent it from happening again.

What space exactly did it run out of?

Memory I/O Buffer is 120M and we don't have big downloads or uploads going on at any time of day. A 4Mb PDF may be about the biggest download happening and I doubt ever concurrently.

So given the server ram and hd spec, how did it think it runs out of any space? /tmp/lshttpd/swap filles up with numbered and lettered folders as expected. Restarting lsws doesn't seem to flush old swap folders as those dated before today remain on restart.

This is on v4.0.14 by the way.

Until this issue, things were running flawlessly.

Any insights you can offer?

 

[NiteWave]

please upgrade to 4.0.15 because of security hole.

where's the error message from? error.log ?

also check
Max Request Body Size
in http://www.litespeedtech.com/docs/webserver/config/tuning/#maxReqBodySize

 

[J.T.]

The error came from the admin UI home page. I logged in after bumping in the 500 server error. I'll have a look in the error logs too. Yes:

./error.log.2010_07_20:2010-07-20 16:11:00.040 [ERROR] [214.41.153.118:4237-0#help.domain.com] out of swapping space while reading request body!
./error.log.2010_07_20:2010-07-20 16:11:05.835 [ERROR] [214.41.153.118:4243-0#help.domain.com] out of swapping space while reading request body!
./error.log.2010_07_20:2010-07-20 16:11:10.219 [ERROR] [214.41.153.118:4245-0#help.domain.com] out of swapping space while reading request body!
./error.log.2010_07_20:2010-07-20 16:11:19.892 [ERROR] [214.41.153.118:4248-0#help.domain.com] out of swapping space while reading request body!

I noticed the exploit and subsequent prompt fix. I have the mod sec rule applied and indeed plan to upgrade later today. Thanks.

At the server level I have Max Request Body Size (bytes) 500M

In my mind, with limited understanding of the intricacies, that should be plenty to email 2MB worth of data whilst elsewhere not much is happening.

 

[mistwang]

Can it be reliably reproduced? If yes, we can take a closer look at it.

 

[J.T.]

It happened a few weeks after the last restart so it seems I need to wait a few weeks again and see if it recurs on the new version. I'll keep an eye on it and report back if anything comes up.

 

[mistwang]

Maybe it is a bug of internal swapping memory usage tracking, lshttpd thought swapping memory has been used up, but actually not. we will add more logging when it happens

 

[J.T.]

Perhaps the real time statistics can show how much free swapping memory it *thinks* it has available. As part of preventative warnings, it can say stuff like:

- 80% of swapping space used
- XYZ VHost 90% of concurrent connections used
- etc.

That helps us troubleshoot before it even happens.