killed with SIGTERM, 3-4 times a day

[fantasydreaming]

Lsws seems to be sending a sigterm to my processes during processing occassionally.

I'm using 3.3.4. I believe the latest lsapi as well.

My backtraces show it happens sometimes during a template render, sometimes during a memcached call, and sometimes during a database requests. None of these things should take very long, so it just seems to be random.

Load has been very low, under 0.4.

Any ideas for debugging? I haven't gotten any complaints from people (besides the occassional "it logged me out" complaint, which I'm still trying to figure out, perhaps some cookie problem).

Thanks for a great product!

 

[mistwang]

Have you set "Max Idle time" under Ruby Rails tab?

http://www.litespeedtech.com/docs/webserver/config/rails/#extMaxIdleTime

 

[fantasydreaming]

Yes, but the backtrace and logged sigterm exceptions are showing that this is happening while the request is in procesing, and that it was a very fast request (i.e. it hasn't hung, from what I can tell). So it shouldn't be considered 'idle' since it was just given a new request 10ms previously...

 

[mistwang]

OK, we will investigate. Maybe the watchdog ruby process think the child process is idle while it is not.

 

[mistwang]

Please try adding the following environment variables under ruby tab

LSAPI_MAX_PROCESS_TIME=10000000
LSAPI_AVOID_FORK=1
LSAPI_MAX_IDLE=1000000

If those variable helps, try removing them one by one and find out which one affect this problem. Which this hint, we can locate the problem and fix it accordingly.

 

[iwarshak]

I am having the exact same issue. Processes getting SIGTERMed on a very fast page. Did you have any luck with this?

 

[mistwang]

Please try above environment variables, see if it helps.
Also, please check the PID of "ruby" processes, see if the killed process is a child process. On a busy server, there should be a parent ruby process keep running.
At least, we can narrow down which process send the SIGTERM signal, lshttpd, or the parent ruby process.

 

[iwarshak]

I will try that.

I also see quite a few of these:

[idle] connection to [uds://tmp/lshttpd/mydomain.com:_.sock] on request #81, error: Connection reset by peer!

Could this be related to the above problem?

 

[pbengtson]

Specifying all three options:

LSAPI_MAX_PROCESS_TIME=10000000
LSAPI_AVOID_FORK=1
LSAPI_MAX_IDLE=1000000

makes our server crash, hard, after a while (server starts swapping then goes firmly into a downward spiral). Removing LSAPI_AVOID_FORK=1, keeping only the LSAPI_MAX_PROCESS_TIME and LSAPI_MAX_IDLE alleviates the problem for us - only a tenth of the requests result in SIGTERMs - but it doesn't remove it. Which is unacceptable; management is already talking about switching to another webserver. And we have a purchased license.

Please advise.

 

[mistwang]

We are working on a new ruby lsapi release to address this.

 

[mistwang]

Everyone,

Please upgrade ruby-lsapi 3.1, see if the problem is fixed or not.

 

[pbengtson]

Your 3.1 gem distro is b0rkinated (it is empty). Please fix ASAP.

 

[pbengtson]

Thanks for fixing the gem so quickly. I will post again when we have verified that it removes the SIGTERM problem.

 

[mistwang]

Your 3.1 gem distro is b0rkinated (it is empty). Please fix ASAP.

It takes a while for rubyforge to update all its mirrors.

 

[pbengtson]

The new ruby-lsapi 3.1 gem fixes the SIGTERM problem, at least for us (with servers under a heavy load).

(The gem was there on rubyforge, but it was empty. Zero bytes file length.)

 

[mistwang]

Thanks for the confirmation.

 

[fantasydreaming]

Working great for me as well, thanks!

 

[mistwang]

Thanks for the update.