Switched to LSWS, server not using free memory

[Balerion]

Hi all,

Recently switched to LSWS after failing to get OLS to work for my use set. It's a Mediawiki site, and LS Cache is enabled and I was able to confirm that it was fine. I could use ab to get thousands of request responses per second locally, and was able to use loader.io to do stress testing. But, as soon as I switched DNS records to make the site go live, top showed increasing CPU usage -- hitting 30.00 -- and everything bogged down.

Notably, for some reason free RAM refused to budge whatsoever -- I thought all the stuff in cache/buffer would be used by the lsphp tasks that were spawned with the visitors, but RAM usage just did not move at all. The server is using 8GB of 64GB worth of RAM, half of it Redis, another chunk MySQL, but even as scores or hundreds of visitors showed up, and lsphp was spawned, no RAM was used for these tasks. It seems really strange.

Any suggestions of where to look to fix this? I feel like I'm on the verge of this site being production-ready after struggling for weeks trying to get OLS to work.

 

[serpent_driver]

Any suggestions of where to look to fix this?

Try this, but don't use root to identify which script is causing the load.

https://forum.openlitespeed.org/threads/performance-problems-on-dedicated-server.5894/post-15421


**********************************************************************************************************************************************************************************
Join https://www.cachecrawler.com - Lightning fast Cache Warmup Crawler for Wordpress, OpenCart, PrestaShop, W3 Total Cache, WP Rocket, Shopware and Magento.

 

[Pong]

Do you host other site other than Mediawiki? Did high CPU load come from mediawiki site lsphp or from other lsphp? Normally once you enabled cache for mediawiki, it should reduce number of PHPs. When pages are serviced from cache, they won't be any PHP processes or RAM/CPU usages for such requests. You'd better log a ticket with us with root access so that we can check what happened on your server.

 

[Balerion]

Try this, but don't use root to identify which script is causing the load.

https://forum.openlitespeed.org/threads/performance-problems-on-dedicated-server.5894/post-15421

Thank you, that was helpful!

Do you host other site other than Mediawiki? Did high CPU load come from mediawiki site lsphp or from other lsphp? Normally once you enabled cache for mediawiki, it should reduce number of PHPs. When pages are serviced from cache, they won't be any PHP processes or RAM/CPU usages for such requests. You'd better log a ticket with us with root access so that we can check what happened on your server.

I managed to sort it out with the advice from serpent_driver, and various tips I found at the Litespeed knowledgebase and so on. The main issue, though, simply seemed to be getting the cache filled. As more and more of the site was cached, load fell progressively. Now it's generally under 1 (on a 4 core/8 thread server), and as low as 0.5, with the web crawler running and steady traffic.

 

[serpent_driver]

I managed to sort it out with the advice from serpent_driver, and various tips I found at the Litespeed knowledgebase and so on. The main issue, though, simply seemed to be getting the cache filled. As more and more of the site was cached, load fell progressively. Now it's generally under 1 (on a 4 core/8 thread server), and as low as 0.5, with the web crawler running and steady traffic.

Has the issue dissolved itself?

**********************************************************************************************************************************************************************************
Join https://www.cachecrawler.com - Lightning fast Cache Warmup Crawler for Wordpress, OpenCart, PrestaShop, W3 Total Cache, WP Rocket, Shopware and Magento.

 

[Balerion]

Has the issue dissolved itself?

Mostly! We felt comfortable enough that we moved two other sites (CMSs) to the server, and at the moment those aren't cached so load is a bit higher than I'd like ( averaging about 2.0, slightly higher than the same traffic on our old server but we had Varnish in place for the CMSs) because I need to figure out the rewrite rules for it.

My main concern is I'd like people with the login cookie to skip the cache (there's only a couple, and they need to be able to see content immediately in case there are errors that need fixing) while everyone else gets the cached version of the site, and I've not quite figured it out yet. And since LSCache is around, I'd rather try to use that rather than figuring out how to get Varnish in place again with LSWS.

 

[serpent_driver]

Unfortunately, the cache plugin for MediaWiki only offers very few options, although this plugin should be fundamentally revised. Especially since the cache function only has an on/off function. If you want to exclude logged in users from caching, use this rewrite rule.

RewriteCond %{HTTP_COOKIE} [cookie_name] [NC]
RewriteRule .* - [E=Cache-Control:no-cache]

**********************************************************************************************************************************************************************************
Join https://www.cachecrawler.com - Lightning fast Cache Warmup Crawler for Wordpress, OpenCart, PrestaShop, W3 Total Cache, WP Rocket, Shopware and Magento.

 

[Balerion]

Unfortunately, the cache plugin for MediaWiki only offers very few options, although this plugin should be fundamentally revised. Especially since the cache function only has an on/off function. If you want to exclude logged in users from caching, use this rewrite rule.

RewriteCond %{HTTP_COOKIE} [cookie_name] [NC]
RewriteRule .* - [E=Cache-Control:no-cache]

Very useful! It's not the Mediawiki that i'm concerned with, since private users get private cache and also the plugin clears out cache when pages are changed, as I understand it. It's the CMS sites (which run Expression Engine) but I think that rule should work for them as well.

 

[Balerion]

Erk. This made me check the Mediawiki plugin's claim that it purges cache when a page is updated... but it seems that's only the private cache, the public cache doesn't get cleared. That's a problem.

 

[serpent_driver]

Erk. This made me check the Mediawiki plugin's claim that it purges cache when a page is updated... but it seems that's only the private cache, the public cache doesn't get cleared. That's a problem.

Please check this aiagn because the plugin code says otherwise. Only the cache of the respective article is purged. This purge function lacks the possibility of differentiation, so that, like other LScache plugins, you have the option to choose which cache from which pages should be purged for which action.

**********************************************************************************************************************************************************************************
Join https://www.cachecrawler.com - Lightning fast Cache Warmup Crawler for Wordpress, OpenCart, PrestaShop, W3 Total Cache, WP Rocket, Shopware and Magento.

 

[Balerion]

Please check this aiagn because the plugin code says otherwise. Only the cache of the respective article is purged. This purge function lacks the possibility of differentiation, so that, like other LScache plugins, you have the option to choose which cache from which pages should be purged for which action.

You're right, I forgot that now that LSCache is working, our cloudflare caching is also intervening. Once I turned it off I could see it working properly. Maybe I need to now change cloudflare rules so it only takes cares of images and stuff and not the actual pages, leaving that to LSCache.

 

[serpent_driver]

You're right, I forgot that now that LSCache is working, our cloudflare caching is also intervening. Once I turned it off I could see it working properly. Maybe I need to now change cloudflare rules so it only takes cares of images and stuff and not the actual pages, leaving that to LSCache.

Tip of the day! Never use more than 1 page cache for dynamically generated documents, because

1 page cache + 1 page cache != 2 page cache != double speed

btw. Where does the most vistors of your MediaWiki site come from? Your own country or other countries?

Configure CF to use headers from origin host!!!!!!!!!!!!!!!!!!

**********************************************************************************************************************************************************************************
Join https://www.cachecrawler.com - Lightning fast Cache Warmup Crawler for Wordpress, OpenCart, PrestaShop, W3 Total Cache, WP Rocket, Shopware and Magento.

 

[Balerion]

Tip of the day! Never use more than 1 page cache for dynamically generated documents, because

1 page cache + 1 page cache != 2 page cache != double speed

btw. Where does the most vistors of your MediaWiki site come from? Your own country or other countries?

Configure CF to use headers from origin host!!!!!!!!!!!!!!!!!!

Most visitors are from the US by far. I think I already have CF configured that way but will double check.

The main thing is Cloudflare's caching of images and such helps a _lot_ with bandwidth usage. Certain times of the year, we easily hit big overages, which depending on host could get very expensive indeed (we previously hosted the wiki on IBM's service, and overage was _very_ expensive).

 

[serpent_driver]

Most visitors are from the US by far.

The basic idea of a CDN is: "Bring content closer to the user". This idea is based on that all of the content of your page is cached and stored at CDN node. For dynamically generated documents this idea makes no sense if the origin host uses a page cache as well, because the cache status of both cache function is not synchronized. Using the page cache with CDN causes much trouble, so again, don't use the page cache of CF with CDN.

If most of the visitors come from your own country a CDN doesn't makes sense as well and indepentendly which document type is requested. Without using the page cache of the CDN it is only about nameserver resolution for THE FIRST REQUEST. In this case the origin host is faster than any CDN. That means you don't need CF. Traffic doesn't matter or do you pay for it or is it limited?

There is only 1 reason to use CF as CDN and this is security, but in your case nothing else.

Read this to better understand what a CDN is good for:
https://blog.litespeedtech.com/2021/07/26/cdns-with-kevin-fwu/

**********************************************************************************************************************************************************************************
Join https://www.cachecrawler.com - Lightning fast Cache Warmup Crawler for Wordpress, OpenCart, PrestaShop, W3 Total Cache, WP Rocket, Shopware and Magento.

 

[Balerion]

Well, when I say "most", I mean like 50%. But there's substantial amount of visitors from outside the US, particularly Western and Central Europe, where I think a CDN like cloudflare does work great.

And there's still the issue of traffic. When traffic surges, it surges BIG, and with incredible overages. But as it is, Cloudflare takes care of like 80% of that traffic by delivering images and such itself, and not charging for it (for now, anyways). I'm not sure of another solution for that, unless there's CDNs out there that are easier to tell to clear content they have cached?

 

[serpent_driver]

Well, when I say "most", I mean like 50%. But there's substantial amount of visitors from outside the US, particularly Western and Central Europe, where I think a CDN like cloudflare does work great.

"Most" should mean more than 50%

When traffic surges, it surges BIG, and with incredible overages

Who cares about the traffic if the traffic doesn't matter?! Anyway, use CF for static sources if you think that you get any advantages of using it.

FYI: In your case, the loading of static sources is not faster with a CDN, as they are only loaded after the main document has been loaded.

**********************************************************************************************************************************************************************************
Join https://www.cachecrawler.com - Lightning fast Cache Warmup Crawler for Wordpress, OpenCart, PrestaShop, W3 Total Cache, WP Rocket, Shopware and Magento.

 

[Balerion]

Who cares about the traffic if the traffic doesn't matter?!

Well, most of the time it doesn't, but sometimes it does. I suppose I can leave off caching from CF fo now, but down the road I'm definitely going to have to use it or our bandwidth charges are going to be over the moon. And to be honest, we were thinking of going Enterprise to use their full caching (instead of LSCache) because some tests we've done shows it can handle peak load easily (we've had tens of thousands of requests per second at certain points, and it's hard for any server to handle that even with the best possible caching and tuning.)

FYI: In your case, the loading of static sources is not faster with a CDN, as they are only loaded after the main document has been loaded.

That, I did not know. Interesting.

 

[serpent_driver]

You should think about changing your hosting. There are often traffic limits, but these limits are very tolerant, so I can only imagine you exceeding these limits to a limited extent.

If you notice several tens of thousands of requests per second, that may initially sound unbelievable or difficult to imagine. Do you run a porn site? I don't want to question the interest in the content of your site, but such a high number of requests cannot be generated by natural users. You should therefore think about regulating your site's traffic and filtering or blocking natural requests from "unnaturally" generated ones. CF offers you the corresponding options and functions, which you can also set up locally on your server. I would be happy to support you.

 

[Balerion]

Do you run a porn site? I don't want to question the interest in the content of your site, but such a high number of requests cannot be generated by natural users.

No, not porn... but hey, it's HBO, so...

No, we run one of the top sites in the world for a globally popular series and TV show. When the show is on, millions of people who Google details about characters or events are likely to see us as the #1 result and rush over to our site. It's a huge amount of traffic.

 

[serpent_driver]

No, not porn... but hey, it's HBO, so...

Does HBO make porn?

It's a huge amount of traffic.

Again, if you think that caching of static sources reduces whatever, then use a CDN. If the load goes to high for dynamically generated content keep the cache always warmed.

However, the cache crawlers offered by LiteSpeed are lame ducks and cause a relatively high server load. You should therefore look for alternative solutions. With LiteSpeed Web ADC you also have the option of distributing the high load, but this only makes sense if you operate more than 1 server.

**********************************************************************************************************************************************************************************
Join https://www.cachecrawler.com - Lightning fast Cache Warmup Crawler for Wordpress, OpenCart, PrestaShop, W3 Total Cache, WP Rocket, Shopware and Magento.