503 troubleshooting

[djsodom]

Hi,

I've got a fresh install (2.6) on a Fedora 6 box (actually ec2 image). Used the EasyRailsTemplate to create a vhost. The listener seems looks right. There are no notices or erros on restart.

However, when I try to access my rails site it times out and gives me a 503 Service Unavailable message.

Nothing shows up in my rails dev log. No errors show up in the litespeed log either.

I know its going to be something silly I'm overlooking.

Where should I begin troubleshooting this?

Steve

 

[mistwang]

Here you go.

Make sure the rails application works well under WEBrick or mongrel running as the user who own the application.
Double check file permissions for your rails application directory.
Make sure Memory and process limit has been set high enough.
Turn on LSWS debug logging by change "Debug Level" to "High".
Check stderr.log in addition to error.log.

 

[djsodom]

• Make sure the rails application works well under WEBrick or mongrel running as the user who own the application.

webbrick starts up fine with no errors. The console starts up too and I can access my models.

• Double check file permissions for your rails application directory.

Permissions look fine for my rails directory. Its owned by steveodom and group www. Litespeed is running under user lsws, who also belongs to group www. All the group permissions are set to allow rw. I can start up webbrick as lsws user with no problem.

• Make sure Memory and process limit has been set high enough.

I'll leave this to check last, because I think my problem lies below. Everything is the default right now.

• Turn on LSWS debug logging by change "Debug Level" to "High".
• Check stderr.log in addition to error.log.

Here is the recurring error in my stderr.log:

from /opt/lsws/fcgi-bin/RailsRunner.rb:6
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require': no such file to load -- lsapi (MissingSourceFile)
from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
from /home/steveodom/quizical/config/../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:364:in `require'
from /opt/lsws/fcgi-bin/RailsRunner.rb:6

I previously installed the ruby-lsapi gem and got the message "Successfully installed ruby-lsapi-1.11". My permissions for my opt/lsws directory are:

drwxr-xr-x 7 lsws www 4096 Dec 31 12:44 DEFAULT
-rw-r--r-- 1 root root 4014 Jan 2 06:56 LICENSE
-rw-r--r-- 1 root root 2214 Jan 2 06:56 LICENSE.OpenLDAP
-rw-r--r-- 1 root root 6279 Jan 2 06:56 LICENSE.OpenSSL
-rw-r--r-- 1 root root 3208 Jan 2 06:56 LICENSE.PHP
-rw-r--r-- 1 root root 6 Jan 2 06:56 VERSION
drwxr-xr-x 4 root root 4096 Dec 31 12:44 add-ons
drwxr-xr-x 8 root root 4096 Jan 2 06:56 admin
drwxr-xr-x 2 lsws www 4096 Jan 2 06:56 autoupdate
drwxr-xr-x 2 root root 4096 Jan 2 06:56 bin
drwx------ 4 lsws www 4096 Jan 2 06:57 conf
drwxr-xr-x 2 root root 4096 Dec 31 12:44 docs
drwxr-xr-x 2 root root 4096 Dec 31 12:44 fcgi-bin
drwxr-xr-x 2 root root 4096 Dec 31 12:44 lib
drwxr-xr-x 2 root root 4096 Jan 2 21:42 logs
drwxr-xr-x 3 root root 4096 Dec 31 12:44 share

I'm running litespeed with user lsws.

Here is output from my error.log:

2007-01-02 21:45:22.944 [DEBUG] Remove pid: 12094
2007-01-02 21:46:06.186 [NOTICE] [AutoUpdate] Failed, updateagent exit code: 3
2007-01-02 21:46:10.046 [NOTICE] [66.90.167.160:62038-0#quizical:lsapi] ExtConn timed out while processing.
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] connection to [uds://tmp/lshttpd/quizical:_.sock] on request #0, error: Connection timed out!
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] [ExtConn] close()
2007-01-02 21:46:10.046 [DEBUG] [uds://tmp/lshttpd/quizical:_.sock] No Request has been processed successfully through this connection, the maximum connections allowed will be reduced!
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] HttpExtConnector::tryRecover()...
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] HttpIOLink::continueWrite()...
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] write resumed!
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] HttpIOLink::handleEvents() events=4!
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] HttpIOLink::suspendWrite()...
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] HttpConnection::sendHttpError(),code=503 Service Unavailable
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] HttpConnection::flush()!
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] Written to client: 621
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] HttpConnection::nextRequest()!
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] HttpExtConnector::cleanUp() ...
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] abort request...
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] release ExtProcessor!
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] Non-KeepAlive, CLOSING!
2007-01-02 21:46:10.046 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] Shutting down out-bound socket ...
2007-01-02 21:46:10.181 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] HttpIOLink::handleEvents() events=17!
2007-01-02 21:46:10.182 [DEBUG] [66.90.167.160:62038-0#quizical:lsapi] Close socket ...
2007-01-02 21:48:02.076 [DEBUG] [idle] Idle connection timed out, close!
2007-01-02 21:48:02.076 [DEBUG] [idle] [ExtConn] close()
2007-01-02 21:48:02.076 [DEBUG] Remove pid: 11637

thanks for your help,

Steve

 

[mistwang]

Looks like you need to double check your lsapi installation.

 

[djsodom]

Looks like you need to double check your lsapi installation.

Yes, I didn't have the make package installed and installing my ruby-lsapi gem, though it says successfully install, wasn't complaining loud enough for me.

I installed the make package and it seems like ruby-lsapi installs correctly, though I can't 100% tell. Here is what is generated:

-bash-3.1# gem install ruby-lsapi
Building native extensions. This could take a while...
make: i386-redhat-linux-gcc: Command not found
make: *** [lsapilib.o] Error 127
make: i386-redhat-linux-gcc: Command not found
make: *** [lsapilib.o] Error 127
ruby extconf.rb install ruby-lsapi
checking for main() in -lsocket... no
creating Makefile

make
i386-redhat-linux-gcc -I. -I. -I/usr/lib/ruby/1.8/i386-linux -I. -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -Wall -fPIC -c lsapilib.c

make install
mkdir -p /usr/lib/ruby/gems/1.8/gems/ruby-lsapi-1.11/lib
i386-redhat-linux-gcc -I. -I. -I/usr/lib/ruby/1.8/i386-linux -I. -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -Wall -fPIC -c lsapilib.c

make clean
Successfully installed ruby-lsapi-1.11

I'm still getting the same error messages in stderror.log as detailed in my previous post.

I deleted my existing virtual host and started over using the EasyRails template and as per step 4 of: http://www.litespeedtech.com/support/wiki/doku.php?id=litespeed_wiki:ruby_rails_easy

I'm not doing these steps as per this post::
Copy dispatch.lsapi to Rails app's public/ directory
- Change shebang line to reference /opt/local/bin/ruby
- Change rewrite rule in public/.htaccess as described in Ruby LSAPI README
- Add same rewrite rule in Litespeed admin console (just in case)
- Server Configuration > External Apps > Add, Type: LSAPI APP

I'm under the impression I don't have to do those steps if I use the template.

What else should I try?

Thanks,

Steve

 

[mistwang]

It complains "i386-redhat-linux-gcc: Command not foun", do you have gcc installed on your system.

Maybe you should try installing ruby-lsapi package manually.

 

[djsodom]

It complains "i386-redhat-linux-gcc: Command not foun", do you have gcc installed on your system.

Maybe you should try installing ruby-lsapi package manually.

I installed it manually and get this:

-bash-3.1# ruby setup.rb config
---> ext
---> ext/lsapi
/usr/bin/ruby /home/steveodom/ruby-lsapi-1.11/ext/lsapi/extconf.rb
checking for main() in -lsocket... no
creating Makefile
<--- ext/lsapi
<--- ext
-bash-3.1# ruby setup.rb setup
---> ext
---> ext/lsapi
make
i386-redhat-linux-gcc -I. -I. -I/usr/lib/ruby/1.8/i386-linux -I/home/steveodom/ruby-lsapi-1.11/ext/lsapi -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -Wall -fPIC -c lsruby.c
lsruby.c: In function 'lsapi_ruby_setenv':
lsruby.c:60: warning: implicit declaration of function 'ruby_setenv'
lsruby.c:62: warning: implicit declaration of function 'ruby_unsetenv'
lsruby.c: In function 'lsapi_fork_child':
lsruby.c:216: warning: implicit declaration of function 'setsid'
lsruby.c:242: warning: implicit declaration of function 'usleep'
lsruby.c:284: warning: implicit declaration of function 'fork'
lsruby.c:288: warning: implicit declaration of function 'getppid'
lsruby.c:306: warning: implicit declaration of function 'close'
lsruby.c:318: warning: implicit declaration of function 'getpgrp'
lsruby.c: In function 'chdir_file':
lsruby.c:401: warning: implicit declaration of function 'chdir'
lsruby.c: In function 'lsapi_write':
lsruby.c:448: warning: unused variable 'data'
lsruby.c: At top level:
lsruby.c:32: warning: 'lsapi_objrefs' defined but not used
lsruby.c:56: warning: 'lsapi_ruby_setenv' defined but not used
lsruby.c:73: warning: 'lsapi_free_data' defined but not used
lsruby.c:395: warning: 'chdir_file' defined but not used
lsruby.c:671: warning: 'lsapi_isatty' defined but not used
i386-redhat-linux-gcc -I. -I. -I/usr/lib/ruby/1.8/i386-linux -I/home/steveodom/ruby-lsapi-1.11/ext/lsapi -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -Wall -fPIC -c lsapilib.c
i386-redhat-linux-gcc -shared -L"/usr/lib" -o lsapi.so lsruby.o lsapilib.o -lruby -lpthread -ldl -lcrypt -lm -lc
<--- ext/lsapi
<--- ext
-bash-3.1# ruby setup.rb install
rm -f InstalledFiles
---> ext
---> ext/lsapi
mkdir -p /usr/lib/ruby/site_ruby/1.8/i386-linux
install lsapi.so /usr/lib/ruby/site_ruby/1.8/i386-linux/.
<--- ext/lsapi
<--- ext
-bash-3.1#

Blah.

 

[mistwang]

It should work by now. looks like i386-redhat-linux-gcc is there, don't know why it does not work with the gem installation.

Just want to remind you that you need to manually upgrade ruby-lsapi later on if there is a new release, the manually installation has higher priority than the gem installation.

 

[djsodom]

It should work by now. looks like i386-redhat-linux-gcc is there, don't know why it does not work with the gem installation.

I'm getting this now:

Application error (Apache)

Change this error message for exceptions thrown outside of an action (like in Dispatcher setups or broken Ruby code) in public/500.html

My stderror.log gives this..

/usr/lib/ruby/site_ruby/1.8/i386-linux/lsapi.so: warning: already initialized constant ENV

...which I read is okay.

There are no messages in my rails logs.

I increased my hard & soft memory to 500M each.

I determined to make this happen.

Steve

 

[mistwang]

I have no clue what causing it. it should be a ruby/rails internal error.
Maybe you should check the DB configuration for the mode that rails running in. try changing it from "production" to "development" or vice versa, see if it helps.

 

[djsodom]

Bingo. My app was set to production in its virtual host tab. Changing it to development - where a populated db exists - solved the problem.

thanks very much for your help.

Steve

I have no clue what causing it. it should be a ruby/rails internal error.
Maybe you should check the DB configuration for the mode that rails running in. try changing it from "production" to "development" or vice versa, see if it helps.