Getting Lots of Ruby Processes

jp_n9 · Sep 11, 2006

I am testing LS on a VPS running RH9. It is not beng accessed by more than 3 people. Probably none of them concurrently.

When I restart the server I have 2 ruby instances. In the 6 or so hours since I restarted this morning, I now have 16 ruby instances. Eventually I end up with about 25 - 30 of them and the Rails application stops responding.

UPDATE:

2 hours later - Now up to 22 ruby instances.

Any idea what might be going on here?

mistwang · Sep 12, 2006

What is your "max connection" setting? which version of LSAPI has been installed?
Is that possible for me to login and take a look at the server?

amalik · Sep 12, 2006

This is funny, I am getting the same kind of issue, I have set the max connections for each of my VHosts to 3, but if this morning I noticed I had 16 processes.. at 20mb each, it eats up my half gig pretty easily..

I also wonder why LS starts a new RailsRunner when I click on a link in my RoR generated page, even though it already has 6 processes started (this is after I did a graceful restart)

I have NO users. None. Nada. Just me. and I don't have aborted connections or anything like that at all.

I'm using LSWS 2.2 and the latest version of the ruby-lsapi 1.6 gem. The backend database is postgres.

mistwang · Sep 12, 2006

When LSWS being started gracefully, all associated ruby processes should be killed after all pending requests has been served. The new LSWS instance will start it own group of ruby processes.
The problem is that ruby processes won't die after being killed by LSWS. We spent some time on work around on this issue.
Please upgrade to ruby-lsapi 1.7, if the problem persist, I need to take a look.

jp_n9 · Sep 12, 2006

mistwang said:

Max connection is set to 300. I believe it is LSAPI version 1.6.

I'll send you the login in a private message.

There are now 36 ruby processes that have been started.

amalik · Sep 12, 2006

mistwang said:

I updated to 1.7 and it seems to be holding much better when I use "ab". The process count doesn't increase.

lsapi 1.7 seems about 3-5% slower (vs 1.6) on the same machine though, but that's just a rough calc. database intensive pages seem to be around 10-15% slower, but that is just off of one reading so it may be way off.

btw, I didn't have any problem with the ruby processes not dying when I stopped LSWS, it is just that even though I had 6 already up and running with 0 requests, if I clicked on a link, it would fire up a new one, which doesn't really make sense to me..

Is there some tech spec as to how many HTTP connections are pooled per RailsRunner (ruby-lsapi) process?

thx,

-Amr

thijs · Sep 12, 2006

I've had the same issues, it sometimes builds up a huge number of processes and eats up all the available memory.

I did notice another thing, I'm running a few Rails 1.0 apps and they broke down with the new 2.2.1 release.

/home/***/app/current/../vendor/rails/activerecord/lib/active_record/base.rb:991:`method_missing': undefined method `clear_active_connections!' for ActiveRecord::Base:Class (NoMethodError)
from /usr/local/lsws/fcgi-bin/RailsRunner.rb:9

The 1.1 apps work great. I haven't seen the processes get out of hand since I moved the 1.0 apps back to Mongrel. Maybe this has something to do with it?

jp_n9 · Sep 12, 2006

Overnight it grew to 50 processes, causing the app to stop responding. Now I just get the 503 Service Unavailable error.

Did the "Graceful Restart", and that killed all the processes and restarted 2. Here we go again...

mistwang · Sep 12, 2006

jp_n9:

Please upgrade ruby LSAPI to 1.7. Your 'Max connection' is way too high, your VPS does not have enough memory to start 300 ruby processes. your server could be DoS attacked by a simple request flood. please try '10' first. I I will take a look at your server if the problem persist with 1.7 release.

Amr,
LSWS 2.2.1 + LSAPI 1.7 will close the DB connection established during initialization, so the first request will be slow down a little, but the subsequent requests should be processed as fast as before.

Thijs,

Looks like 'clear_active_connections!' is a new function for rails 1.1, we need to implement something similar for 1.0 to deal with the DB connections.

jp_n9 · Sep 12, 2006

mistwang said:

I didn't understand that "Max connection" == "Max Ruby Processes", otherwise I would have set it much lower.

Even still, it would seem that something was wrong if it keeps spawning new processes when at most two users are ever accessing it at once (most likely only one at a time).

mistwang · Sep 12, 2006

Yes, that true. LSAPI 1.7 should do much better job.

mistwang · Sep 13, 2006

We improved the process manager in LSAPI, the spawning extra processes problem should be fixed completely.
You can try LSAPI 1.8 at http://www.litespeedtech.com/packages/lsapi/ruby-lsapi-1.8.tar.gz .

jp_n9 · Sep 13, 2006

mistwang said:

Ok. I downloaded and installed the patch.

In the Rails tab, the Environment setting includes the value:

LSAPI_MAX_REQS=1000

Does that need to be lowered too, or is that OK?

Also, is there any way to verify the version of LiteSpeed or LSAPI installed on a system - just to make sure everything installed correctly? Thanks!

Hopefully the 1.8 will do the trick, because the system had spawned crazy processes again last night.

mistwang · Sep 13, 2006

No need to lower it.
Web admin interface or the error log file has the version number of LSWS.
"gem list" will show the LSAPI version if installed through RubyGem, if installed manually, you should know it. ;-)

jp_n9 · Sep 13, 2006

mistwang said:

When will the ruby-lsapi gem be online for auto installation?

Also, any word on when the Capistrano wiki How-To will be available?

mistwang · Sep 13, 2006

After we belive that the too much ruby process problem has been truely fixed.

amalik · Sep 13, 2006

mistwang said:

1.7 barfed all over itself last night. I woke up this morning and it had been spawning until I think it ran out of memory. I have 18 rubly lsap and 3 lsphp (didn't have this prob before) sitting there. 1.21 maybe contributing to it?

I dunno.

I'll try the 1.8 lsapi and see how that goes. Thanks.

amalik · Sep 13, 2006

Regarding 1.8, do you recommend building a gem and then installing or just use setup.rb? I'm wondering if this will interfere with a gem-only strategy I've been following.

thanks,

mistwang · Sep 13, 2006

The problem should be mainly in
gem package will be available when we officially announce this release.
Yes, packages installed via gem and setup.rb may interfere with each other, remove the other one when switch the package. I think the gem one may has lower priority, not 100% sure.

Be careful not to over configure your rails application like setting "max connections" to 100 when you hardware cannot take 100 ruby processes.

jp_n9 · Sep 14, 2006

Ok. I downloaded and installed LSAPI 1.8 as instructed in the README:

ruby setup.rb config
ruby setup.rb setup
ruby setup.rb install

Is that it?

It would be nice to be able to run something like "lsws -v" and have LiteSpeed verify what versions are being used of both the server and LSAPI.top

I don't recall seeing anything in the instructions, but is there any need to modify any files within the individual Rails applications?

Getting Lots of Ruby Processes

Active Member

LiteSpeed Staff

Member

LiteSpeed Staff

Active Member

Member

Member

Active Member

LiteSpeed Staff

Active Member

LiteSpeed Staff

LiteSpeed Staff

Active Member

LiteSpeed Staff

Active Member

LiteSpeed Staff

Member

Member

LiteSpeed Staff

Active Member