Script Timeout Causes Retry & Multiple SQL Inserts

[robfrew]

I'm having a weird problem and I think it may have to do with our Litespeed settings. We have a custom php script that submits an online mass email to all of our users into our mysql database. It used to work just fine without any problems and recently, we started messing around with our litespeed settings to make things faster and more efficient. Since making a few changes, this script now inserts 10 copies of the same email to each user as if it were in a loop.

My initial thought is that the script times out so litespeed sees this and retries submitting the script again which causes it to insert the sql statements again from the beginning. This ends up in an infinite loop and the only way to stop it is to wait a while or restart mysql. What settings do I need to change within the web server to prevent this from happening?

 

[robfrew]

Here is this php code for the insert so I know it's not the code...

$sql = "SELECT memberid FROM members WHERE disabled = '0'";

$result = mysql_query($sql) or die(mysql_error());

while($row = mysql_fetch_row($result)){

   $memberid = $row[0];

   /* INSERT INTO MESSAGE TABLE */

   mysql_query("INSERT INTO message (sender, receiver, subject, time, folderid, status, sender_status, filename) VALUES ('$senderid', '$memberid', '$subject', '".time()."', 'inbox', '0', '3', '$filename')");

   $messageid = mysql_insert_id();
				
   /* INSERT INTO MESSAGE BODY TABLE */

   mysql_query("INSERT INTO messagebody (messageid, message) VALUES ('$messageid', '$message')");

}

 

[mistwang]

Maybe it is the memory limit for PHP processes, you can try increase the memory soft/hard limit for lsphp5 external application.

 

[robfrew]

Tried that and it did not help. As a matter of fact, we are now noticing the same problem with large video uploads that go through ffmpeg to convert to flv files. FFMPEG is called from a php script to convert the video to FLV and after watching top during the process, you will see FFMPEG restart after several minutes, over and over until the php script times out. Then looking in the folder that the video is saved to, there are 8-10 copies of the converted video.

We are using a PHP external app (lsphp) for this site. What settings could be affecting this?

 

[mistwang]

http://www.litespeedtech.com/docs/webserver/config/extapps/#initTimeout

 

[robfrew]

After doing some research, here is what I have found. If you are using php as external app (lsapi) (lsphp), if your script does any type of external processing, such as converting videos or inserting a lot of data into a database, it will eventually time out based on the max_execution_time in your php.ini. It also looks like LSWS will restart that php process once it hits that max time and start it all over again until eventually LSWS times out the connection. That is why we were seeing duplicates as stated above.

So what we have done is set our max_execution_time in our php.ini to 180:

max_execution_time=180

and for individual scripts that need more time, we have added the following php code to the beginning of the script:

ini_set( "max_execution_time", "3600" ); // sets the maximum execution time of this script to 1 hour.

The reason we set this number individually to the necessary scripts is to avoid any other security issues that may arise from scripts that do not need that extra time to process.

Hope this helps others.

 

[robfrew]

I guess I spoke too soon. I am still having the same problems. I have increased the Initial Request Timeout and the php max_execution_time to 300, 600 etc... and it hasn't helped. I'm looking at top and ffmpeg dies at around 154 seconds then restarts.

Any other ideas?

 

[robfrew]

After changing more settings within LSWS, I finally decided to try changing Connection Timeout in the main Server Configuration under Tuning. It was currently set at 60. I changed it to 600 just to see if it would help and tada... it solved the problem.

So now, my question is, how is Connection Timeout causing this issue? Does this mean I have to set this setting high now so I can allow large videos to be converted by a php script? That doesn't sound to efficient. Or is there a way to set this setting temporarily by .htaccess or something? Or is this a bug?

 

[mistwang]

Specifies the maximum allowed connection idle time during processing one request. The connection will be closed whenever it is idle, no i/o activity, for this period of time.

If the file takes more than 60 seconds to be processed and there is no I/O on the socket, the connection will be closed. So, you need to increase it under server tuning configuration.

 

[robfrew]

But isn't it dangerous to have such a high Connection Timeout setting? Is there a way to temporarily set it to a higher number depending on the file or script?

 

[mistwang]

It is not that dangerous, you can set the per client throttling limits to limit the number of concurrent connection each client can have. It is a global configuration. cannot be overridden at this point.

 

[robfrew]

What we have found is that for what we are doing, video processing, we should actually unload that from the actual user experience and put it into a batch processing script that is run from a cron job. This will enable us to take as much time as we need to convert video without having to adjust our connection timeout settings to a higher value. This is a recommended practice for anyone who plans on doing video conversion with php for files that are large.

 

[mistwang]

Yes, I knew other video sharing sites do that as well. way to go.

 

[Tony]

I've been seeing this problem as well from time to time. I don't think the server should be redoing the request this is very problematic.

With PHP max execution time is suppose to stop the requests from continuing to go. It may very well be doing this but lsws re does the request over and over again. If you're doing something financial this could be big trouble. I experienced this on the actual litespeed site while paying via credit card. My request kept looping and I ended up with something like 4 charges with 3 of them needing to be voided. The best part in my browser all I saw as page loading but litespeed kept calling the php script again I imagine.


I have also encountered this with other scripts and page. Some user has a perl script and it times out. Rather than actually timing out it keeps making the request over and over again. The user sees a loading page but on LiteSpeed itself its sending the request to the perl script again.


So I do believe this is a bug and a big problem depending on the types of applications. It should be abiding by timeouts and not re-sending the request to the external application. What good is that other then piling up requests when the script is broken.

 

[anewday]

I agree with Tony.

 

[mistwang]

I think there are two sources for this problem.

One is that user only send one request, LiteSpeed tries it multiple times due to back-end (PHP or backend app server) failure, like the charging credit card problem Tony had, for this part, it can be improved from LSWS internal.

Another one is that user get an blank page most likely due to script problem, each request only being processed once by LiteSpeed. User refresh the browser again and again. This kind of problem cannot be address by LSWS internally.

 

[Tony]

One is that user only send one request, LiteSpeed tries it multiple times due to back-end (PHP or backend app server) failure, like the charging credit card problem Tony had, for this part, it can be improved from LSWS internal.

What version do you estimate improvements to this portion? It continues to be a problem for us where poorly made shared hosting user scripts which used to do 30 seconds then timeout with the php execution timeout error now continue to run over and over again making a huge headache for us.

 

[mistwang]

Please download the latest 4.0rc1 release package. it should have been addressed, at least for PHP, it wont happen.