Two php crashes since our last update of litespeed and php

M

Monarobase

Well-Known Member
#1
#1
Hello,

We recently updated to litespeed 4.2.3 and LSAPI 6.2, running PHP 5.3.25 where PHP crashes and litespeed continues to serve static files but doesn't restart or detect a 503 error.

All PHP scripts were down until I manually restarted litespeed.

I've got log entries and although core dumps are deactivated for php in the litespeed config customer accounts that appear un the error logs have multimple core dumps, each between 1 and 2 GB each.

Luckly memory isn't compressed and compressing 3GB of files brings them down to 204MB for 3GB of files.

How can I send you these core dumps and log extract ?

This has happened twice so far and the last one was at 5am so I had to wake up to fix it. The previous one was around 7PM two days ago. The crashes aren't regular, they just seem to be happening quite often whereas they never happened when I was running litespeed 4.2.2 with LSAPI 6.1 and PHP 5.3.22

Thanks
 
M

Monarobase

Well-Known Member
#2
#2
Just happened again !

I've just changed the new option from "No Abort" to "Abort for all" to go back to how this setting is by default, hopefully this might help !
 
M

mistwang

LiteSpeed Staff
#3
#3
How can I send you these core dumps and log extract ?
Click to expand...
can you make it available for us to download? please send URL to bug@litespeed...

Or, let our staff login and take a look?

Change that option may not have any effect.
 
M

Monarobase

Well-Known Member
#8
#8
Just happened again although the option was deactivated…

But this time I got an e-mail 10 minutes before it stopped responding :

At [22/Jun/2013:23:22:41 +200], web server with pid=212521 recieved unexpected signal=9, no core file created. A new instance of web server will be started automatically!
Click to expand...
 
M

Monarobase

Well-Known Member
#9
#9
Not sure if it's related but I've just sent another log extact to bug AT litespeed…

This log extract happened when there wasn't a crash but I think it might be related as I getting a few entries like this one every day.
 
M

Monarobase

Well-Known Member
#10
#10
Any news ?

After some research I might have found the culprit…

A while ago I tested xCache 3.0.1 with variable cache and got alot of crashes so I reverted to xCache 2 with variable cache deactivated.

Since then the server has been stable and we have been gradually adding sites.

My theroy is that for some reason xCache can not cope with 1GB of cache.

When I put more that 1GB xCache would not start but even though it would run with 1GB of cache it caused PHP to crash when it went over a certain amount of memory.

My guess is that when I had variable cache activated it hit the 1GB faster and this caused my crashes.

I've set it to 600MB (16 * 37,5 MB) and it seems fine. Before I had it at 1024MB with 32 * 32MB.

This server's got alot of available memory (over 150GB free at the moment).

Touch wood, I have't had any problems since but I didn't have problems every day before so it's a bit early to be shure.

If this is the case, any idea why xCache can't cope with much memory ? Or is it maybe I was using to may caches (32 threads but 16 real cores). ?

I will try and ask around to understand why this is happening. I would rearly like to have 20GB of ram dediaceted to variable cache… 600MB is enough for oopcode caching but not for variable cacheing.
 
M

mistwang

LiteSpeed Staff
#11
#11
You need to increase the memory limit of lsphp5 external app to take memory used by xcache into consideration, that's the only configure related.

Other than that, it is completely a xCache internal issue.
 
M

Monarobase

Well-Known Member
#12
#12
I don't think it's this as I believe our limits are high enough :

Memoy Soft Limit is set to : 30720M
Memory Hardd Limit is set to : 32768M

My guess is more something to do with xCache hitting the 1024MB limit any trying to empty all this memory at one time, taking a few seconds to do this and causing PHP to crash…

Maybe xCache isn't meant for large servers…

--

Off topic, but any idea when I post a new answer to litespeedtech.com's forums my answers are always truncated, I have to hit return to copy my answer then go and edit my answer ? This is a real pain !
 
M

Monarobase

Well-Known Member
#13
#13
With xCache set to 600MB of cache, we've had no crashes since then, but we've got alot of memory that's not being used. If you have any ideas how to increase the amount of memory xCache can use for it's without PHP crashing that would be great ! :)

Whats strange is I've seen some people allowing 2GB of cache and it working for them… There isn't supposed to be a limit !

I don't think this is a litespeed issue but more something to do with PHP or Xcache or maybe the way mmap works.
 
M

Monarobase

Well-Known Member
#15
#15
Just crashed again with 600MB. I've reducted it to 500M to see but am thinking about giving eaccelerator a try.

This is a pain, I wanted to allow 30GB originaly for opcode cache and varcache.

I had some strange crashing with PHP cr
 
M

Monarobase

Well-Known Member
#16
#16
Test to see if It will save this time, your forum keeps cutting my messages !

Just crashed again with 600MB. I've reducted it to 500M to see but am thinking about giving eaccelerator a try.

This is a pain, I wanted to allow 30GB originaly for opcode cache and varcache.

I had some strange crashing with PHP cr
 
M

Monarobase

Well-Known Member
#18
#18
Here's the text I wanted to post last night (hope it works this time, other wise I'll host the text and post a download link untill your forum is fixed.

Just crashed again with 600MB. I've reducted it to 500M to see but am thinking about giving eaccelerator a try.

This is a pain, I wanted to allow 30GB originaly for opcode cache and varcache.

I had some strange crashing with PHP cr
 
M

Monarobase

Well-Known Member
#19
#19
Last test :

I think I've found the culprit, a single non acsii character… still a huge bug if you can't post uft8 characters…

Here's the message :

Just crashed again with 600MB. I've reducted it to 500M to see but am thinking about giving eaccelerator a try.

This is a pain, I wanted to allow 30GB originaly for opcode cache and varcache.

I had some strange crashing with PHP creating core dumps of 2GB (maximum memoy CloudLinux is set to allow for accounts. I suppose it's something to do with processes being shared accross accounts but each time PHP has made litespeed crash there have been quite a few coredumps before. Could there be a more global memory issue ? It's normal to see xcache in any core dump as scripts are cached…

Dam, can't seem to get my full post to save, only happens on litespeed's forum, I guess could be litespeed cache issue ?
 
M

Monarobase

Well-Known Member
#20
#20
I might have worked something out, are CloudLinux virtual memory limits compatible with Litespeed LSAPI + deamon mode ?

I had limits of 2GB per account and before PHP crashed I think PHP's processes were using up 2GB (currently using 1.3GB of virtual and 30m of residual memory).

My guess is that xCache's memory is included in the Virtual memory and reducing xCache size was just preventing it from hitting a user's maximum virtual memory size.

When changing the value in php.ini, all PHP processes had a fresh start, thus reducing the amount of virtual memory used and preventing and accounts from hitting the Cloudlinux user's 2GB limit.

If this is the case then it's the second incompatibility with CloudLinux. It would be great, concidering that most of your customers have cPanel + Cloudlinux + litespeed if you could work with them to get it fully compatible.

We're able to remove certain limits because we've got alot of memory but we're now down to disk and cpu limits knowing that whenever an account goes over his limit too much Litespeed doesn't detect the error correctly and forces a restart…

I've upped each accounts virtual memory limit to 4GB… I'll see if it happens again.
 
You must log in or register to reply here.
Top