Advanced crawler for recache. some ideas and code

AndreyPopov · Aug 31, 2021

LSCache Advanced Crawler for Opencart 3.x ocmod

(for https://github.com/litespeedtech/ls.../package/lscache-opencart3.0-latest.ocmod.zip from 18th June 2021)

!!!ATTENTION!!!!!
not work as expected if session driver is file
!!!ATTENTION!!!!!

with session driver is file you must:
- click Rebuild Part of LiteSpeed Cache once and wait until file cache expired (this check need settings and create them)
- click Rebuild Part of LiteSpeed Cache again and again wait until file cache expired (build urls list and save it to DB)
- only after that clicking Rebuild Part of LiteSpeed Cache starting recache process.

on my host I use Redis driver for cache and for session and all work perfect.

Lee · Aug 31, 2021

OK, what's a session driver?

AndreyPopov · Aug 31, 2021

Lee said:

you must change it by hand in
/system/config/default.php
/system/config/admin.php
/system/config/catalog.php

in Opencart 2.x default session driver is db
why in Opencart 3.x default session driver is file - I don't know and why with session driver file not work correctly also cannot understanding.

I add in code

PHP:

        $this->cache->delete('lscache_pages');
        $this->cache->delete('lscache_modules');
        $this->cache->delete('lscache_esi_modules');
        $this->cache->delete('lscache_itemes');

but it not work for file session driver

Lee · Aug 31, 2021

Is this what you're talking about?
// Cache
$_['cache_engine'] = 'memcached'; // apc, file, mem or memcached
$_['cache_expire'] = 3600;

AndreyPopov · Aug 31, 2021

Lee said:

this is cache driver.

session is:
$_['session_engine']

https://github.com/opencart/opencart/blob/master/upload/system/config/default.php

PHP:

// Session
$_['session_autostart']    = false;
$_['session_engine']       = 'file'; // db or file
$_['session_name']         = 'OCSESSID';

Lee · Aug 31, 2021

My session-engine is db

AndreyPopov · Aug 31, 2021

Lee said:

with db session driver must work.
try.

doxis · Mar 25, 2022

Hello,
I have a problem with the creation of recache. I use version 2.3.0.2. Journal theme 3 to both eshops, but they can’ t be completed and they stop, because I have 40.000 products. Also, despite the fact that I have 384 categories, it completes only 84 and after that it moves to the products’ update. Could you do an updated version for the 2.3.0.2, which includes all of them and it can be stored in the database? Because right know it runs with Php and the package Web Host Elite is activated, but the sites delay. Moreover I had to deactivate another app to make Lite Speed work.
Thank you.

AndreyPopov · Mar 25, 2022

now my country Ukraine under russian putler attack

ask me again when war ends

serpent_driver · Mar 26, 2022

doxis said:

Try this:

For me on my test shared hosting server it works and is supported by LiteSpeed webserver. PHP scripts never stops and run for hours, but handle with care, if you don't have permissions to control running processes on your server. If so, you can't stop it!!!!! With this code script will only stop by session timeout if OpenCart crawler is binded to session.

AndreyPopov · Aug 23, 2022

temporary post whole plugin (not extension mode)

last added features:

- for cli:
1. &startnum= and &endnum=
can use separate or combine

startnum - from what number in builded DB urls list start
endnum - at what number in builded DB urls list stop

for example:
curl -N "https://www.yourdomain.com/index.ph...e&from=cli&what=1&startnum=38760&endnum=40800"

2. &mode=restart
same as after press Save in GUI. clean saved already recached parameters, build new urls list

3. some others like &botsua= and &renew= describe later

- if enabled SEO, algorithm check same urls and exclude

AndreyPopov · May 2, 2023

- use more _lscache_vary cookies
- use curl_multi calls
- mod for Journal3 webp detection

rewrite rules fot .htaccess

Code:

### LITESPEED_CACHE_START - Do not remove this line
<IfModule LiteSpeed>
CacheLookup on
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "bot|compatible|images|cfnetwork|favicon|facebook|crawler|spider|addthis" [NC]
RewriteCond %{HTTP_USER_AGENT} !Chrome [NC]
RewriteCond %{HTTP_USER_AGENT} !Mobile [NC]
RewriteRule .* - [E=Cache-Control:vary=isBot]
RewriteCond %{HTTP_USER_AGENT} Bot [NC]
RewriteCond %{HTTP_USER_AGENT} Mobile [NC]
RewriteCond %{HTTP_USER_AGENT} !Chrome [NC]
RewriteRule .* - [E=Cache-Control:vary=ismobilebot]
RewriteCond %{HTTP_USER_AGENT} "Android|iPhone|iPad" [NC]
RewriteRule .* - [E=Cache-Control:vary=ismobile]
</IfModule>
### LITESPEED_CACHE_END

serpent_driver · May 2, 2023

AndreyPopov said:

- use more _lscache_vary cookies
- use curl_multi calls
- mod for Journal3 webp detection

rewrite rules fot .htaccess

Code:

### LITESPEED_CACHE_START - Do not remove this line
<IfModule LiteSpeed>
CacheLookup on
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "bot|compatible|images|cfnetwork|favicon|facebook|crawler|spider|addthis" [NC]
RewriteCond %{HTTP_USER_AGENT} !Chrome [NC]
RewriteCond %{HTTP_USER_AGENT} !Mobile [NC]
RewriteRule .* - [E=Cache-Control:vary=isBot]
RewriteCond %{HTTP_USER_AGENT} Bot [NC]
RewriteCond %{HTTP_USER_AGENT} Mobile [NC]
RewriteCond %{HTTP_USER_AGENT} !Chrome [NC]
RewriteRule .* - [E=Cache-Control:vary=ismobilebot]
RewriteCond %{HTTP_USER_AGENT} "Android|iPhone|iPad" [NC]
RewriteRule .* - [E=Cache-Control:vary=ismobile]
</IfModule>
### LITESPEED_CACHE_END

Far too fault-tolerant and therefore not usable.

AndreyPopov · May 2, 2023

serpent_driver said:

you still believe in Santa Claus?

but my rewrite rules and crawler work perfectly!

serpent_driver · May 2, 2023

They can't neither work nor perfectly. There are hundreds of bots that you can't catch with these primitive rules. Using User-Agent is highly unsafe.

AndreyPopov · May 2, 2023

serpent_driver said:

Santa Claus never die!

serpent_driver · May 3, 2023

AndreyPopov said:

Let me be your Santa Clause so you finally believe in me again.

You should really think about it, that an exclusive detection via the user agent cannot be a viable solution and you know it. Putting aside the insecure use of the user-agent, it takes far too long and costs unnecessary resources to search through a long user-agent string.

Try that. This solution is 99.99% secure. From where I know this? I've been tracking not only the users of my pages for about 3 years, but also bots. This allows me, unlike you, to qualify my code.

Code:

RewriteCond %{HTTP_USER_AGENT} "!safari|MAC OS X|android|opera|SamsungBrowser" [NC]
RewriteCond %{HTTP_USER_AGENT} "!chrome" [NC]
RewriteCond %{HTTP:Sec-Fetch-Dest} ^$ [NC]
RewriteCond %{HTTP:Sec-Fetch-Mode} ^$ [NC]
RewriteCond %{HTTP:Sec-Fetch-Site} ^$ [NC]
RewriteCond %{HTTP:Sec-Fetch-User} ^$ [NC]
RewriteRule .* - [E=Cache-Control:vary=isBot]

RewriteCond %{HTTP_USER_AGENT} "Applebot" [NC]
RewriteRule .* - [E=Cache-Control:vary=isBot]

AndreyPopov · May 3, 2023

serpent_driver said:

NO!!!

man that only hear himself cannot be believed Santa Claus

serpent_driver · May 3, 2023

I can't listen to you. You always talk nonsense.

AndreyPopov · May 3, 2023

serpent_driver said:

nonsense talk/say/write here always only you!!!

Advanced crawler for recache. some ideas and code

Well-Known Member

Attachments

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

New Member

Well-Known Member

Well-Known Member

Well-Known Member

Attachments

Well-Known Member

Attachments

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member