Intermittant 404 errors.

[TheColonial]

Hi guys,

I'm having an interesting problem with my LiteSpeed install, and I'm not sure if it's a configuration problem or not.

The server hosts a site called 'shiftperception.com'. This site has a fair bit of content, including two instances of wordpress:
* www.shiftperception.com/blog/
* www.shiftperception.com/uts/webmedia1/

Both require mod_rewrite functionality, and each of them have their own .htaccess files. They look like this:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /blog/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)$ /blog/index.php [L,QSA]
</IfModule>

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /uts/webmedia1/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)$ /uts/webmedia1/index.php [L,QSA]
</IfModule>

OK, so the problem is this. Both of these sites suffer from intermittant 404 errors. You can be browsing the site, and you might browse to a url like: http://www.shiftperception.com/uts/webmedia1/week-by-week-summary/week-3/html-tags/

and it'll return with a 404 error. If you refresh immediately, or hit 'back' in the browser and click the link again, it works perfectly!?

I turned on debug logging and have taken a snapshot of what happens. When the first request fails (and the 404 is returned), the debug log shows the following:

2007-04-09 22:17:37.708	DEBUG	[121.45.245.177:4992-0] HttpIOLink::handleEvents() events=1!
2007-04-09 22:17:37.708	DEBUG	[121.45.245.177:4992-0] HttpConnection::onReadEx(), state: 0!
2007-04-09 22:17:37.708	DEBUG	[121.45.245.177:4992-0] readToHeaderBuf().
2007-04-09 22:17:37.708	DEBUG	[121.45.245.177:4992-0] Read from client: 876
2007-04-09 22:17:37.708	DEBUG	[121.45.245.177:4992-0] read 876 bytes to header buffer
2007-04-09 22:17:37.708	DEBUG	[121.45.245.177:4992-0] processHeader() return 0, header state: 3.
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0] readToHeaderBuf() return 0.
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0#shiftperception.com] New request:
Method=[GET], URI=[/uts/webmedia1/week-by-week-summary/week-3/html-tags/],
QueryString=[]
Content Length=0
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0#shiftperception.com] Find context with URI: [/], location: [/home/sites/dan/shiftperception.com/public_html/]
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0#shiftperception.com] File not found [/home/sites/dan/shiftperception.com/public_html/uts/webmedia1/week-by-week-summary/week-3/html-tags/]
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0#shiftperception.com] processContextPath() return 25
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0#shiftperception.com] processNewReq() return 25.
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0#shiftperception.com] HttpConnection::sendHttpError(),code=404 Not Found
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0#shiftperception.com] HttpConnection::flush()!
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0#shiftperception.com] Written to client: 581
2007-04-09 22:17:37.709	DEBUG	[121.45.245.177:4992-0#shiftperception.com] HttpConnection::nextRequest()!

If I hit refresh immediately, the log shows the following:

2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0] HttpIOLink::handleEvents() events=1!
2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0] HttpConnection::onReadEx(), state: 0!
2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0] readToHeaderBuf().
2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0] Read from client: 902
2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0] read 902 bytes to header buffer
2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0] processHeader() return 0, header state: 3.
2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0] readToHeaderBuf() return 0.
2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0#shiftperception.com] New request:
Method=[GET], URI=[/uts/webmedia1/week-by-week-summary/week-3/html-tags/],
QueryString=[]
Content Length=0
2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0#shiftperception.com] Find context with URI: [/], location: [/home/sites/dan/shiftperception.com/public_html/]
2007-04-09 22:18:48.834	DEBUG	[HTAccess] Updating configuration file [/home/sites/dan/shiftperception.com/public_html/.htaccess]
2007-04-09 22:18:48.834	DEBUG	[HTAccess] Updating configuration file [/home/sites/dan/shiftperception.com/public_html/uts/.htaccess]
2007-04-09 22:18:48.834	DEBUG	[HTAccess] Updating configuration file [/home/sites/dan/shiftperception.com/public_html/uts/webmedia1/.htaccess]
2007-04-09 22:18:48.834	DEBUG	[HTAccess] No change in configuration file [/home/sites/dan/shiftperception.com/public_html/uts/webmedia1/.htaccess].
2007-04-09 22:18:48.834	DEBUG	[121.45.245.177:4997-0#shiftperception.com] Find .htaccess context with URI: [/uts/webmedia1/], location: [/home/sites/dan/shiftperception.com/public_html/uts/webmedia1/]
2007-04-09 22:18:48.834	INFO	[121.45.245.177:4997-0#shiftperception.com] [REWRITE] strip rewrite base: '/uts/webmedia1/' from URI: '/uts/webmedia1/week-by-week-summary/week-3/html-tags/'
2007-04-09 22:18:48.834	INFO	[121.45.245.177:4997-0#shiftperception.com] [REWRITE] Rule: Match 'week-by-week-summary/week-3/html-tags/' with pattern '^(.+)$', result: 2
2007-04-09 22:18:48.835	INFO	[121.45.245.177:4997-0#shiftperception.com] [REWRITE] stat( /home/sites/dan/shiftperception.com/public_html/uts/webmedia1/week-by-week-summary/week-3/html-tags/ ) failed
2007-04-09 22:18:48.835	INFO	[121.45.245.177:4997-0#shiftperception.com] [REWRITE] stat( /home/sites/dan/shiftperception.com/public_html/uts/webmedia1/week-by-week-summary/week-3/html-tags/ ) failed
2007-04-09 22:18:48.835	INFO	[121.45.245.177:4997-0#shiftperception.com] [REWRITE] Source URI: 'week-by-week-summary/week-3/html-tags/' => Result URI: '/uts/webmedia1/index.php'
2007-04-09 22:18:48.835	INFO	[121.45.245.177:4997-0#shiftperception.com] [REWRITE] Last Rule, stop!
2007-04-09 22:18:48.835	DEBUG	[121.45.245.177:4997-0#shiftperception.com] Find handler [shiftperception.com_lsphp] for [.php]
2007-04-09 22:18:48.835	DEBUG	[121.45.245.177:4997-0#shiftperception.com] processContextPath() return 0
2007-04-09 22:18:48.835	DEBUG	[121.45.245.177:4997-0#shiftperception.com] run lsapi processor.
2007-04-09 22:18:48.835	DEBUG	[uds://tmp/lshttpd/shiftperception.com_lsphp.sock] request [121.45.245.177:4997-0#shiftperception.com:lsapi] is assigned with connection!
.
.
.
 (this carries on while the rest of the page is loaded)

So it works the second time. Can someone tell me what's going on here please? I have no idea what could be causing this problem.

Many thanks for your help.
OJ.

 

[TheColonial]

Hi,

The problem is strange, and it's as if the rewrite handler doesn't get called.

A few more points:

* I have two other Wordpress blogs running on the same machine (different domain) and they work fine with the same rewrite rules.
* If I hit refresh (F5) constantly or quickly, I get nothing bot 404 errors.
* Sometimes the site doesn't load for ages even after a few attempts to refresh. Then it'll start to work all of a sudden
* If the rewrite functionality is turned off in wordpress, the site works fine.

So it's definitely dodgey handling of rewrites, but I'm not sure why.

I look forward to your response.

OJ

 

[mistwang]

Looks like it relates to LSWS .htaccess cache, for some reason, htaccess has been ignored until the cache checks if there is any change in .htaccess file. We will investigate.

When does it start to happen? How frequently? Does .htaccess under public_html, uts or webmedia1 get updated frequently?

 

[TheColonial]

Hello,

No the .htaccess file doesn't change often, but I was mucking around with it when the issue first started to happen. That .htaccess file will be static as soon as the issue goes away. It's exactly the same as the other .htaccess files that seem to work fine.

When does it start to happen?
Immediately. As soon as the litespeed server is started. It doesn't happen on other domains, only this one.

How frequently?
Very. If you browse to the site right now you'll see it happen as you cruise around. Hitting refresh very quickly almost guarantees a 404.

Does .htaccess under public_html, uts or webmedia1 get updated frequently?
Nope. It's static. The only reason it's been modified at all is to try and get this problem to go away.

Thank you!
OJ

 

[TheColonial]

Hi again,

Before I posted, I attempted to move the rewrite rules to the vhost rather than having them stored in .htaccess files. But unfortunately I couldn't get it to work because I specified two RewriteBase commands (one for blog and one for uts/webmedia1) and that didn't behave itself at all.

How would I go about setting up the rewrite rules listed in my first post using the VHOST rewrite rather than .htaccess?

Thanks again
OJ

 

[mistwang]

There is no rewrite base at vhost level, you have to use the full URI for matching.
However, your rewrite rule can be used without any change at context level, just add two static context "/blog/" and "/uts/webmedia1/", and add those rules there, no need to specify the rewrite base with used with LSWS.

 

[TheColonial]

Hello,

I'm sorry, but I don't know what you mean. Do you mean I add two different entries into the "Rewrite Map" section? I have no idea how to use that section, would it be possible for you to give me an example please?

If that's not what you're talking about, can you please give me a few more details? What do you mean by 'static context'?

I appreciate your help.
Regards
OJ.

 

[TheColonial]

Hi!

Right, I've figured out what you're talking about now, and i've added the contexts. It works fine now! Thanks very much.

It seems that the .htaccess support in this area might have an issue, but when set up with the contexts inside LSWS it's fine.

Cheers.
OJ

 

[aww]

I'm having a similar problem, could you show me the
final replacement code you used in .htaccess to prevent the 404 issues?

 

[TheColonial]

Hello,

I removed all the .htaccess files and I used the litespeed rewrite feature instead. For each URI that requires rewrite rules I set up a new static context, and added the rules to the context.

Hope that helps
OJ

 

[mistwang]

OJ,

I think we had identified and fixed your 404 problem in ealier release. It will be great if you can confirm this.

aww,
Your 404 problem might be different, please let me know the detail.

 

[TheColonial]

Sorry I took so long to get back to you on this. I had no idea you'd asked the question!

By the looks of it the latest version of Litespeed as of today (29 Dec 07) works fine Hope that helps!

OJ