Magic curl option for fast recache and small lscache size

[serpent_driver]

.... and that's why request method GET has explicit to be set.

 

[AndreyPopov]

.... and that's why request method GET has explicit to be set.

CURLOPT_NOBODY on true - SET request method of curl session to HEAD

 

[serpent_driver]

Again yes, but this is why request method has explicit to set to override HEAD requuest to GET method.

 

[AndreyPopov]

Again yes, but this is why request method has explicit to set to override HEAD requuest to GET method.

first - by default NOT CURLOPT_CUSTOMREQUEST present
second - not sure that (CURLOPT_CUSTOMREQUEST, "GET") override method or vice versa set CURLOPT_NOBODY to false

 

[serpent_driver]

Don't think complicate, use it and enjoy to save data that has not to be unnecessary downloaded. All, but not LScache extension for OC use this method.

 

[AndreyPopov]

some statistics about "magic" option

same!!!! server load - max three core(three processes) from four

recache time:
- without "magic" option full recache for 3(three) UAs for 9000 products: more than 3 days
- with "magic" option full recache for 12(twelve) UAs for 9000 products: less than 24 hours

cache storage:
- without "magic" option full recache for 5(five) UAs for 9000 products: 60GB
- with "magic" option full recache for 18(eighteen) UAs for 9000 products: 22 GB

 

[serpent_driver]

Install https://phpsysinfo.github.io/phpsysinfo/ to check the real load. It also works on shared hosting

Define "recache". What is recache? A warmup with completely purged cached before?

Again, there is no magic option. Documentation about curl is plublic, not secret. Only LS forgot to add this option in cache extension for OC.

 

[AndreyPopov]

A warmup with completely purged cached before?

yes

Install https://phpsysinfo.github.io/phpsysinfo/ to check the real load. It also works on shared hosting

hosting have own statistics

 

[serpent_driver]

To check the load of a server there is a common technic from Linux. Checking the number of used processes gives no information about the load. You can simple check the load if you run "uptime" in command line interface. The result you get displayed gives you more information about the load, but on shared hosting it is difficult to measure the load, because you are not the only one who causes load.

If you don't have CLI access you can also get information about the load with sys_getloadavg() PHP function. Use the example code of php.net how to get this information: https://www.php.net/manual/en/function.sys-getloadavg.php

FYI: Your Hosting also uses this method.

 

[AndreyPopov]

The result you get displayed gives you more information about the load, but on shared hosting it is difficult to measure the load, because you are not the only one who causes load.

update my post. 100 is one core.

 

[serpent_driver]

some statistics about "magic" option
cache storage:
- without "magic" option full recache for 5(five) UAs for 9000 products: 60GB
- with "magic" option full recache for 18(eighteen) UAs for 9000 products: 22 GB

To explain what seems to be "magic" related to the disk space, here are some additional information. First of all what @AndreyPopov found out is not magic. LiteSpeed only forgot to add a default curl option in code. This option has affect on how cache files are stored. If this option is set it will compress the cache files and as everybody knows compressing will reduce the file and the disk space. This simple public option reduces the disk space dramatically.

 

[serpent_driver]

update my post. 100 is one core.

This is not an accepted method. It differs from server to server and depends on many facts. Value "100" can be everything, but gives no information about if a server is overloaded or in idle mode. Why don't you accept what is "law"?

 

[AndreyPopov]

LiteSpeed only forgot to add a default curl option in code.

but I find it. forgot or bug or anything else - not matter.
for LSCache for Opencart - this is magic!

This is not an accepted method.

why?
I have 4 core in exclusive use according to my tariff plan.

 

[serpent_driver]

for LSCache for Opencart - this is magic!

Yes, Mr. Magic reduced on this it is maaaaaaagic! You've earned an award for it

why?
I have 4 core in exclusive use according to my tariff plan.

Because nobody can accept it for his own use. Your load is related on your server but each server has different ressources. That is why Linux offers a method to check it really, but always related to the used server with different power.

 

[serpent_driver]

Yes, Mr. Magic reduced on this it is maaaaaaagic! You've earned an award for it[/QUOTE}

But there are a lot of really magic "tricks" you don't know to make crawling and the warmup of the cache really fast. 24 hours for 9000 URLs? No problem, I do it in 1 hour with less load.

 

[AndreyPopov]

Your load is related on your server but each server has different ressources.

I compare on:
- same server
- same configuration
- same conditions

24 hours for 9000 URLs? No problem, I do it in 1 hour with less load.

you are really stupid man
hear only yourself.
never read what others wrote.

- 9000 product links + 9000 Journal QuickView popup links + 9000 Journal option popup links = 27000 links
- 12 (twelve) UAs: 27000 * 12 = 324000 links (13500 links per hour)

again and again: open your eyes, free your mind and break edges of your knowledges

 

[serpent_driver]

I just crawled your page within 1 hour., 16.000 Urls with 11 different cache profiles. Who is the stupid man? It's not just me.

 

[AndreyPopov]

YI just crawled your page within 1 hour., 16.000 Urls with 11 different cache profiles.

you crawled already crawled(cached) pages? really?!?!?!?!?!?!?!? and proud?

open your eyes, free your mind and break edges of your knowledges

P.S. another two fixes (after my tickets) that LS "forgot"
https://github.com/litespeedtech/lscache-opencart/commit/d96d45cc814aca8cdc009bd9086345ae263db4c4
https://github.com/litespeedtech/lscache-opencart/commit/cd1c069b42502c4dc39dba8c8d6b7fd5d8a66155

 

[serpent_driver]

As long you don't acceppt other advices you will never be an expert for caching and for improvements for faster cache wamup. For your information I developed a lot of applications for LScache and for warming the cache. In comparison to me your are a Newbie without knowing what he is doing.

 

[AndreyPopov]

For your information I developed a lot of applications for LScache and for warming the cache.

this is NOTHING matter.
because you hear only yourself.
all our discussions here you only can say: you are wrong, you are wrong.
but ...... LS developers have another point of view and ADD all my ideas and founded problems to code.
and all your "help" only bla-bla-bla-bla: I'm great crawler developer

In comparison to me your are a Newbie without knowing what he is doing.

but I try find and solve problems and enhance features of LSCache plugin for Opencart
and you only bla-bla-bla-bla: I'm great crawler developer