Error messeges on restart with Rails 2.3.2

[paratiger]

Hi,

We have a server running Litespeed v4.0.9 and some Rails 2.3 and earlier sites. We have the latest RailsRunner.rb installed.

When the we do a graceful restart we get several errors from the 2.3 site.

MySQL errors:
Mysql::Error: Lost connection to MySQL server during query: xxx

Rails errors:
These can be as fundamental as 'undefined method find_by_id for xxx' or missing attributes, (where we know they exist)

These errors seem to happen straight after a restart.

Any ideas

Thanks

 

[subBlue]

Graceful restart not so graceful

Just to clarify a little further (I work with Paratiger above):

Often when doing a graceful restart the server load shoots up to 40-50 and we then start getting lots MySQL of errors such as described above plus others about too many connections to the MySQL database. We have about 25 vhosts on the server. The rails apps aren't set to start on restart and this issue has only started happening since we added a couple of Rails 2.3.2 sites (all the others are rails 2.0.5). The normal running load for the server is about 0.5 We are using LS Ent 4.0.9 on an 8GB 64bit CentOS 5.2 box.

It almost seems that if a request for a site comes in during a server restart then the app tries to serve that request before it is properly loaded and ready. Some of the errors we are getting are from missing methods added by plugins that are perfectly valid once the app is properly initialised.

Running the Rails 2.3.2 sites using locally with an Apache mongrel or passenger setup causes no problems. Apart from the version of Rails these sites are very similar to all the other Rails sites we've been hosting for years with LS.

Finally, restarting LS early in the morning when there is no activity on the site works fine without any errors.

Could there still be some problems with the LS RailsRunner.rb?

Unfortunately this is the first significant issue we have had using LS in almost 3 years of hosting Rails sites and if we can't resolve it very soon we're going to be forced to move to Apache/Passenger

 

[subBlue]

A further update: When restarting LS it doesn't seem to be stopping the current Ruby processes before spawning new ones, which is causing some of our memory problems.
Even after stopping LS there are lots of ruby processes still listed in memory. I'm having to manually killall ruby before starting LS back up again. After doing this the memory usage settles at about 4 gigs whereas after a 'graceful' restart it maxes out at 8GB and starts chewing into the swap drive.

If there are ruby processes left in memory after a restart and LS is still trying to use them then it could explain many of the odd errors we have been seeing.

 

[mistwang]

There is what happened during GRACEFUL restart, while start a new instance of LSWS, the old LSWS tries to finish the current pending requests, so it need to keep the old Ruby processes alive, so, the old LSWS instance and ruby processes may not terminate immediately even the new LSWS instance is ready to serve new requests.
when pending request finished, old LSWS and ruby will stop.

It is the design of LSWS graceful restart. For your setup, you have to down tune rails "Max Connections" configuration to allow two sets of rails apps run in the memory. That's why you do not have problem during non-peak hour when less ruby processes are needed. You can experiment with "Max Connections" configuration.

Have you tried Ruby Enterprise? I think it should save memory for lsruby as well as we all do fork(), we have a chance to try it yet. Maybe need to add some special code to enable the copy-on-write optimization.

 

[subBlue]

We are using Ruby Enterprise.

I did some more testing this morning when the server was quiet and it seems that with LS 4.0.9 the ruby processes aren't getting shut down at all on restart - or even server stop - they were still there 5 mins later!

I've switched back to v4.0.6 which is killing off the old processes as expected on restart and for the moment the server seems a little happier.

 

[mistwang]

Have you noticed change in memory usage? I remember the last time I tried, seems it does not help as expected, maybe still missing something to turn it on.
have added code

if GC.respond_to?copy_on_write_friendly=)
GC.copy_on_write_friendly = true
end

Yes, you are right, a graceful restart bug was introduced in 4.0.9, it has been fixed in our 4.0.10 build.