M2-crawler issues with some pages

[Hedloff]

Hello,

Testing the M2 crawler for a customer, but have some issues.
On one page it's showing this:

username@server [~]# sh M2-crawler.sh -v -d https://domain.tld/category/test.html
https://domain.tld/category/test.html ->
-------Debug curl start-------
URL: 'https://domain.tld/category/test.html'
AGENTDESKTOP: 'User-Agent: lscache_runner'
COOKIE: ''
HTTP/1.1 201 Created
Content-Length: 0
Date: Mon, 29 Apr 2019 09:33:29 GMT
Server: LiteSpeed
Strict-Transport-Security: max-age=15768000
X-UA-Compatible: IE=edge
Alt-Svc: quic=":443"; ma=2592000; v="35,39,43,44"
Connection: Keep-Alive

-------Debug curl end-------
Header Match: 'HTTP/1.1 201 Created'
Already cached

And when trying on a another page many times it just gives this:

username@server [~]# sh M2-crawler.sh -v -d https://domain.tld/category/test2.html
https://domain.tld/category/test2.html ->
-------Debug curl start-------
URL: 'https://domain.tld/category/test2.html'
AGENTDESKTOP: 'User-Agent: lscache_runner'
COOKIE: ''
HTTP/1.1 200 OK
Set-Cookie: PHPSESSID=fb76e9fec83f017b4463e74e214975bc; expires=Mon, 29-Apr-2019 10:33:33 GMT; Max-Age=3600; path=/; domain=domain.tld; HttpOnly
Set-Cookie: searchReport-log=0; path=/; secure; HttpOnly
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
X-LiteSpeed-Tag: store,cms_b,cms_b.B_header_top_links,cms_b.B_header_top_left,cms_b.B_header_top_right,cms_b.B_footer_links,cms_b.B_footer_links2,cms_b.B_footer_primary_bottom_left,cms_b.B_footer_primary_bottom_right,cms_b.B_footer_column1,cms_b.B_footer_column2,cms_b.B_footer_column3,cms_b.B_footer_column4,cms_b.B_footer_column5,cms_b.B_footer_column6,cms_b.B_footer_payment,cat_c,store_group,cms_b.B_nav_links,cms_b.B_nav_dropdown,cat_c_130,cat_c_p_130,cat_p_3718,cat_p,cat_p_3716,cat_p_3715,cat_p_3714,cat_p_3713,cat_p_3522,cat_p_3505,cat_p_3502,cat_p_3501,cat_p_3486,cat_p_3485,cat_p_3484,mobile,cms_b_
X-LiteSpeed-Cache-Control: public,max-age=86400
Pragma: no-cache
Cache-Control: max-age=0, must-revalidate, no-cache, no-store
Expires: Sun, 29 Apr 2018 09:33:34 GMT
Content-Type: text/html; charset=UTF-8
Etag: "22353-1556530414;;;"
X-Litespeed-Cache: miss
Transfer-Encoding: chunked
Date: Mon, 29 Apr 2019 09:33:34 GMT
Server: LiteSpeed
Strict-Transport-Security: max-age=15768000
X-UA-Compatible: IE=edge
Alt-Svc: quic=":443"; ma=2592000; v="35,39,43,44"
Connection: Keep-Alive

-------Debug curl end-------
Header Match: 'X-Litespeed-Cache: miss'
Caching

Not quite sure where I should start to find the cause of this. Any ideas?

 

[Hedloff]

Tested another page now, and it still gives a miss.
But after visiting the site in my own browser and then run the M2-script it's showing "Already cached".
So it's a issue with the craweler somewhere.....

Customer is using
Magento ver. 2.3.1

 

[Unique_Eric]

Hi @Hedloff ,

Please try sh M2-crawler.sh with `-c` parameter see if it helps.

e.g. verify method:
Do twice this command
sh M2-crawler.sh -c -d https://domain.tld/category/test2.html?10
We expect first time return `Caching` and second time should be `Already cached` in the end of the strings.

 

[Hedloff]

That didn't work
Only works after I have visited the site trough Chrome or some other type of browser.

Any other suggestions? Does crawler not work with Magento v.2.3.1 ?

 

[Unique_Eric]

Maybe need to reproduce first on my end. Could you submit a ticket to LiteSpeed with site/sitemap URL and append this post link?