Opencart rebuild all cache too slow

[vacancy]

Hi

We have a site in Opencart version 3.0.3.2, we have about 14 thousand products. We use Litespeed opencart cache, when we want to do all rebuild after purge cache, the process continues very slowly. We have increased the php.ini and my.cnf limits on the server, there are no restrictions on the litespeed side. We are using the latest version litespeed and lscache. We have Cpanel control panel.

Why can this be caused?

 

[serpent_driver]

You need a different method to rebuild or to warmup the cache. The current method with internal cache crawler works serial. That means each URL must must be requestet first before the next URL will be requested. This way takes time, but maybe too much time for a higher number of URLs.

Solution 1: Find one who is able to develop a crawler that works parallel to request a defined number of URLs at the same time.
Solution 2: Reduce the number or URLs to the most wanded product pages, because why to warmup the cache for product pages that nobody is interested in?

 

[AndreyPopov]

Hi

We have a site in Opencart version 3.0.3.2, we have about 14 thousand products. We use Litespeed opencart cache, when we want to do all rebuild after purge cache, the process continues very slowly. We have increased the php.ini and my.cnf limits on the server, there are no restrictions on the litespeed side. We are using the latest version litespeed and lscache. We have Cpanel control panel.

Why can this be caused?

same problem. 8k products on store.

at that's why I made "advanced crawler"

you can try ocmod module

you can choose what recache:
- products
- categories
- manufacturers
- catalog
- or combination of above

store 14k urls in memory (for standart crawler) also may exceed php memory limit,
also I have php script time limit 3600s on my hoster - advanced crawler also solve these.



some tricks:
1. use cli command by curl for recache (cron tasks - best solution)
2. to increase crawl speed you also can comment in catalog/controller/extension/module/lscache.php line

usleep(round($diff));

(crawler calculate time for one request and sleep)

 

[Germont]

Installing object cache can spare some resources.
You can also add a second simultaneous cronjob, and speed crawling is increased. No need to edit the files.

 

[AndreyPopov]

You can also add a second simultaneous cronjob, and speed crawling is increased. No need to edit the files.

my hoster:
- block same cron jobs execution
- php script execution limit to 3600 secs

by default crawler recache
- "standard" pages (contacts.sitemap,information etc)
- custom added by GUI pages
- categories (with &pages)
- manufacturer (with &pages)
- and only after products


on my hoster by default crawler recache maximum 800 urls (not recached before) per hour
already recached urls crawler check ~1000urls per 10 minutes

when in my store was 3500 products. default crawler make recache 5-6 hours
but after products became 4100 - recache process never ends
products from 3900 to 4100 recached by 10 urls per hour

and these only for one User Agent!

P.S. parallel execution two, three or more cron jobs by standard crawler NOT increase speed, because standard crawler always start from beginning: "standard" pages -> custom pages -> categories -> manufacturers -> products.
if be more accuracy - standard crawler always build new url's lists for crawling.
parallel execution two, three or more cron jobs only heavy load CPU and increase site response time
some reduce time of recache (not increase speed) is start cron jobs with time delays (~30-40 mins) from each other.
only if you have muticore CPU and more RAM for php, you can:
- prevent php memory exceed limit. in my example more than 4000 urls exceed 1MB php memory
- execute more than one crawler job

 

[Germont]

@AndreyPopov
I agree, you're right on target.
I would rather try another hosting.
Usually a responsive design is preferred, but with so many pages is A MUST.
Have you ever tried manual preloading with two instances simultaneously?
If second it is started when first already crawled 500 urls, then the second doesn't create more database calls until position ~500 is reached. And it happens in very little time, 25 seconds on my VPS. Then both processes continue at the same speed. Cronjob doesn't wait a browser response, so it recovers the delayed start even faster.
I also mapped cache folder to a TMPFS partition, as the VPS is on a mechanical HDD. Made a dramatic difference.

 

[AndreyPopov]

I also mapped cache folder to a TMPFS partition, as the VPS is on a mechanical HDD. Made a dramatic difference.

I use Shared Hosting Server. Hoster provide NVMe SSD for my tariff plan.
problem not in where cache folder or site stored.

Have you ever tried manual preloading with two instances simultaneously?
If second it is started when first already crawled 500 urls, then the second doesn't create more database calls until position ~500 is reached. And it happens in very little time, 25 seconds on my VPS.

I try many-many variants until products became more than 4000.

after 5000 products only separate recache for categories, manufacturer, products solve problem.

also I use Journal Theme and it's feature webp image format:
Desktop View webp
Mobile View webp
Desktop View no-webp(Safari- iMac)
Mobile View no-webp(Safari - iPhone,iPad)

and also Journal Theme use QuickView Product PopUp and Product Options Price PopUp.

no other way to recache all or these - only advanced crawler

any external crawler NOT cache Journal Theme PopUps!

 

[serpent_driver]

@Germont

To prevent any issues caused by hosting limits you always should use a dedicated server or at least a virtual private server (VPS) that allows to set your own custom (script) limits and enough RAM. A shared hosting is never be able to handle a shop with than 2000 items or more. A HTTP cache like Litespeed makes your shop flying, but it flys first if URLs are cached, but flying itself costs more ressources a shared hosting can have, if you have more than 2000 items. This is a wellknown handicap and not a secret. If you or @AndreyPopov have trouble with cache warmup open your mind and create the minimum of conditions to run a shop system with a higher number of items. You will get all benefits of a HTTP cache first if your shop runs proper without a HTTP cache like Litespeed. Litespeed or any other HTTP cache can't fix issues you already have without a HTTP cache, because (and this is a loop) you have to warmup the cache first.

 

[AndreyPopov]

A shared hosting is never be able to handle a shop with than 2000 items or more. This is a wellknown handicap and not a secret.

may be you right, but I never find recommendations about quantity of product items and what server model use.

may be when number of orders become 200 (two hundreds) per day than VPS or dedicate server will best solution.
now 20-30 orders per day on autumn-winter season and max 10 per day on spring-summer.

I'm admin, not owner. I must work in budget limits
owner not ready pay me for VPS/dedicate server support

To prevent any issues caused by hosting limits you always should use a dedicated server or at least a virtual private server (VPS) that allows to set your own custom (script) limits and enough RAM.

now shared hosting model provide good variants. now I have 3 core / 6GB RAM (4GB for php)
but year ago (2 core/ 4GB RAM) I can use only 1GB for php

and I think crawler algorithm must be work on all possible variants and not stop on exceed php memory limit or php script time execution limit.

 

[serpent_driver]

may be you right, but I never find recommendations about quantity of product items and what server model use.

You can trust me if I say it. I have more than 20 years experience. If you buy a car, you also won't find any documentation that a car can only be used for driving on streets and not on arables.

may be when number of orders become 200 (two hundreds) per day than VPS or dedicate server will best solution.
now 20-30 orders per day on autumn-winter season and max 10 per day on spring-summer.

The number of orders doesn't matter for calculating which hosting could be the best choice. If you have only 20 to 30 orders a day, but 8000 items in your shop there is a disproportion and you (your owner) don't seem to know what your customers really want. If you sell clothes, but not each clothing can be selled in every season, only offer such clothes that is meant for a certain season. This would dramatically reduce the number of items and also would reduce the need for system resources without any business disadvantage. Less sometimes can mean more.

now shared hosting model provide good variants. now I have 3 core / 6GB RAM (4GB for php)

In VPS these are only virtual resources and can't be compared with resources in a dedicated server.

and I think crawler algorithm must be work on all possible variants and not stop on exceed php memory limit or php script time execution limit.

Okay, you want a car that should can drive, fly, swim, diving, transport a whole school class of kids and much more and all together at the same time. And everything for free of charge! Hm, good idea, but reality looks a bit different. For example if you pay $20 a month for hosting, you only get resources and performance for what $20 costs. Not more not less.

If the owner of your shop isn't willing to pay more, it is your job to convince him that he can't expect to get the most. Otherwise you must learn how to perform magic....

 

[AndreyPopov]

The number of orders doesn't matter for calculating which hosting could be the best choice. If you have only 20 to 30 orders a day, but 8000 items in your shop there is a disproportion and you (your owner) don't seem to know what your customers really want.

you are wrong!

if summer yarns not requested now(winter), I must remove they from site? or not cache they?
some yarns for pre-order or sale only in pack. pack buy one-two orders per week - not cache they? remove from site?

 

[serpent_driver]

you are wrong!

Why?

if summer yarns not requested now(winter), I must remove they from site? or not cache they?
some yarns for pre-order or sale only in pack. pack buy one-two orders per week - not cache they? remove from site?

Anyway, temporary remove (if possible set items inactive) all items that are not meant for the current season.

 

[AndreyPopov]

Anyway, temporary remove (if possible set items inactive) all items that are not meant for the current season.

really?!?!?!?!?!?!
even if some buyers want buy summer (cotton) yarn in winter?
for example Amigurumi yarn to make toys!

 

[serpent_driver]

Okay, I am not an expert fpr selling products in your shop. It was only a suggestion to find new ideas to reduce the load because reducing the load is the only way to solve most of your issues.

 

[Germont]

@AndreyPopov
I hope you don't mind an offtopic suggestion. I've seen your shop and maybe offer should be more diverse.
Women who do home tailoring use more than balls of thread and wool.
There are also textiles for skirts, dresses, and then the client can buy everything he needs from one shop.
That's when the owner can afford a VPS and management for it.

 

[AndreyPopov]

Okay, I am not an expert fpr selling products in your shop. It was only a suggestion to find new ideas to reduce the load because reducing the load is the only way to solve most of your issues.

I already say to you: reduce number of product links NOT way to solve problem because:
a) number of products only grow - new manufacturers(brand), yarn's manufacturer add new collections and new (trend of season) colors
b) disable "not-season" yarn with HIGH search rating and wait until new or season yarn receive same search rating - way to dead end

 

[AndreyPopov]

@AndreyPopov
There are also textiles for skirts, dresses, and then the client can buy everything he needs from one shop.

I have my own business (computers and network).
Yarn - I only help to admin. I not decide what and how sale.

 

[serpent_driver]

@AndreyPopov

You always know everything better than me, so why do you ask for solution for your issues if you already know it?

 

[AndreyPopov]

so why do you ask for solution for your issues if you already know it?

I finding solution and ask because may be anyone already solved same problem.

You always know everything

no, but I want to know everything

better than me

you always try say like: if you want drive faster then buy new more powerful car or leave spare wheel, tools and passenger seats in garage

 

[serpent_driver]

I finding solution and ask because may be anyone already solved same problem.

You know that this forum is for LSWS and not OLS?

no, but I want to know everything

No, you are resistant against knowledge.

you always try say like: if you want drive faster then buy new more powerful car or leave spare wheel, tools and passenger seats in garage

This is why you always fail. You either can't understand or you don't want to understand. What I try to make you understand is, that you can't transport 100 people if you have only 5 seats in your car. You understand what the different is?