[solved] Retry with new instance.

I

IanD

Well-Known Member
#1
#1
Hi,

We have a PHP script that runs some MySql queries, it is for reporting and these queries can take up to a minute. That is not a problem for us.

The problem is after x seconds, LiteSpeed restarts the process with:

[NOTICE] Retry with new instance.

In the log file.

If I'm watching 'mytop' - I can see the first MySql process running and then I can see an identical query running from the 2nd process when a new instance is started.. and then another one sometimes from a 3rd retry.

All these new instances are making original query take longer.

I'm sure this is pretty basic, but how can I stop this? I presume there is a timeout setting I can change? Or just turn off the retries?

Many thanks,

Ian
 
Last edited by a moderator:
I

IanD

Well-Known Member
#3
#3
It is currently 30 seconds.

The problem is, I want to keep this as low as possible as we get a some DDoS problems. The server is getting over 1 million visits a day so I wasn't keen to increase this for every connection.

I can try increasing it to 60 though and see what happens.

Is there any way to turn off the 'retrying' feature? Just have it fail if it hits 60 seconds?

The restarting of a long process just makes things worse as the problem just escalates more with each restart.

Many thanks,

Ian
 
N

NiteWave

Administrator
#4
#4
Is there any way to turn off the 'retrying' feature? Just have it fail if it hits 60 seconds?
Click to expand...
not sure if set "Retry Timeout (secs)" of the lsphp5 ext app to a big number will help or not --- I've never tried it, just FYI.

for long running php script, not sure if you can run it in command line, i.e., lsphp5 abc.php, if so, the connection time out between lsphp5 and lsws won't take effect, you can run abc.php as long as possible.
 
I

IanD

Well-Known Member
#6
#6
Unfortunately the settings don't seem to make any difference, this still happens after 60 seconds:

2011-01-16 23:50:29.702 [NOTICE] [86.20.198.21:42054-0#APVH] Content len: 0, Request line: 'GET /test.php HTTP/1.1'
2011-01-16 23:50:39.004 [NOTICE] [205.213.216.254:10736-0#APVH] No request delivery notification has been received from LSAPI process:27784, possible run away process.
2011-01-16 23:50:39.004 [NOTICE] [205.213.216.254:10736-0#APVH] Retry with new instance.

My settings are:

Initial Request Timeout (secs) 600
Retry Timeout (secs) 600

Under External App - lsphp5

Any more ideas? Should this work, is it a bug?

Thanks.
 
N

NiteWave

Administrator
#7
#7
can you try other 2 suggestions in my previous posts? to try them first anyway if possible. this is to identify where the problem is. if one workaround can resolve the issue, it'll help a lot for the final solution, and tell where the correct direction is.
 
I

IanD

Well-Known Member
#8
#8
please try increase: http://www.litespeedtech.com/docs/we...g/#connTimeout
to over 60 seconds
Click to expand...
I've increased in to 60 seconds. I really don't want to increase it more than that.. It's a very busy server and these problem requests must only be ~ 200 / 10M+ requests a day.

Running the script on the command would work in regards to not having the 'retry new instance'.

Updating all legacy systems that are run so infrequently to integrate with a command line php script.. it's a lot of work I'd rather not do.

Much rather LiteSpeed just didn't retry long running scripts :)
 
M

mistwang

LiteSpeed Staff
#10
#10
I think you may have a really old build of lsphp5, if yes, try rebuilding PHP again with latest PHP LSAPI. lsphp5 should send back a request received notification for each request, then LSWS will wait till the request finish, not timeout and try again.

If you use latest LSAPI, and set "LSAPI_ACCEPT_NOTIFY" environment variable for lsphp external app, should remove it.
2011-01-16 23:50:39.004 [NOTICE] [205.213.216.254:10736-0#APVH] No request delivery notification has been received from LSAPI process:27784, possible run away process.
Click to expand...
It is the root cause.
 
I

IanD

Well-Known Member
#11
#11
That would be ideal.

I see now it's not so much the 'retrying' that is the problem, because even if LiteSpeed didn't retry the process - the process would have stopped at the connection timeout.

If Retry Timeout (secs) worked as well though, that would be great.

Because at times mysql is under too much pressure and normal queries can back up fast. If LiteSpeed is then retrying these scripts (which run these queries) things just melt down quicker.. This might just be unique to me though and yes, I need upgrade / work on the server!

I've been using LiteSpeed since October 2010 - would this be classed as old?

I've got:

Found LiteSpeed php binary at /usr/local/lsws/fcgi-bin/lsphp5. Version is 5.3.2

If you use latest LSAPI, and set "LSAPI_ACCEPT_NOTIFY" environment variable for lsphp external app, should remove it.
Click to expand...
Not 100% sure what you mean here, I've got:

Code: 
PHP_LSAPI_MAX_REQUESTS=500
PHP_LSAPI_CHILDREN=35
currently in my Environment. Do I add something like:

Code: 
PHP_LSAPI_MAX_REQUESTS=500
PHP_LSAPI_CHILDREN=35
LSAPI_ACCEPT_NOTIFY = 1
Many thanks for your help.
 
N

NiteWave

Administrator
#14
#14
we just added new feature on 4.0.19 to address this timeout issue:
"added another special rewrite environment variable "noconntimeout", this will disable connection timeout, it will keep the connection open forever till the request complete."

for example:

RewriteRule abc.php - [E=noconntimeout:1]

please test at your end, see if it 100% resolves your issue.
 
N

NiteWave

Administrator
#16
#16
manually upgrade:

download lsws...4.0.19..tar.gz
tar zxvf lsws...4.0.19..tar.gz
cd lsws-4.0.19
./install.sh

install to same directory of past installations. select U(upgrade)
in cPanel, it's /usr/local/lsws
 
I

IanD

Well-Known Member
#17
#17
The 'no connection timeout' is great for one problem but I'm still looking a solution to the

retry with new instance

problem. It don't want it to retry, I want it to hit the connection timeout and die. If it's hitting the connection timeout it means there is a problem and just retrying it only makes things worse..

In my situation with long running mysql queries anyway! I look at mytop and see 4 or 5 identical long queries running because they are from the same user and their process is just trying and retrying.

Any ideas?
 
N

NiteWave

Administrator
#18
#18
I tried to set
LSAPI_MAX_PROCESS_TIME (default value: 300 seconds)
"In Self Managed Mode, LSAPI_MAX_PROCESS_TIME controls the maximum processing time allowed when processing a request. If a child process can not finish processing of a request in the given time period, it will be killed by the parent process. This option can help getting rid of dead or runaway child process."
it looks working in test environment.

your php in suExec mode or not? if not, above setting will give you an option.

update: I re-read your post:
I look at mytop and see 4 or 5 identical long queries running because they are from the same user and their process is just trying and retrying.
Click to expand...
it may not result of "re-try". rather, there are 4 or 5 independent requests to same page, and this page need access mysql. for each php request, lsws has to launch a lsphp process to handle it. I think the root cause is slow mysql. if you use memcache to cache the mysql query result(just for example), mysql and the whole site's performance may improve a lot.
 
Last edited:
M

mistwang

LiteSpeed Staff
#19
#19
IanD said:
The 'no connection timeout' is great for one problem but I'm still looking a solution to the

retry with new instance

problem. It don't want it to retry, I want it to hit the connection timeout and die. If it's hitting the connection timeout it means there is a problem and just retrying it only makes things worse..

In my situation with long running mysql queries anyway! I look at mytop and see 4 or 5 identical long queries running because they are from the same user and their process is just trying and retrying.

Any ideas?
Click to expand...
I am not sure those mysql queries are caused by "retry with new instance", maybe user keep trying to load that page.

There are many cases will trigger retry with new instance, mainly due to PHP crashes or exits after finishing "PHP_MAX_REQUESTS".

"retry with new instance" is to reduce 503 error as much as possible. Will see anything can be done to avoid "retry with new instance" when possible.
 
I

IanD

Well-Known Member
#20
#20
No it's not the user refreshing the page that causes it - because I can replicate it and I'm not refreshing the page.

Yes, I agree to the core problem is the mysql slow query.. but at the moment is unavoidable. I just don't want it to keep retrying it.

This is what I got from the log file at the time:

Code: 
2011-02-21 20:15:23.007 [NOTICE] [89.238.173.**:1399-0#APVH_] Content len: 0, Request line: 'GET /test.php HTTP/1.1'
2011-02-21 20:15:33.006 [NOTICE] [94.250.17.**:21751-0#APVH_] No request delivery notification has been received from LSAPI process:8249, possible run away process.
2011-02-21 20:15:33.006 [NOTICE] [94.250.17.**:21751-0#APVH_] Retry with new instance.
2011-02-21 20:15:54.654 [NOTICE] [89.238.173.**:1409-0#APVH_] Content len: 0, Request line: 'GET /test.php HTTP/1.1'
test.php is the script that contained my test slow mysql query.

89.238.173.** is my IP - not sure where 94.250.17.** came from. Maybe it was a coincidence (there is other traffic on the server) but it certainly looks like when it says "Retry with new instance" it's talking me and my script.

My php is in SuExec mode.

I think something like 'LSAPI_MAX_PROCESS_TIME' is what I'm looking for, I want to set that to 60 seconds and then I want the process to die (along with the mysql query).

Thanks for all your help.
 
Last edited:
You must log in or register to reply here.
Top