Do Not Cache User Agents - for Googleboot

B

bobby47

New Member
#1
#1
my sites pages are not being indexed by google, after waiting a while compared to other sites there is something not quite right & after going through a process of elimination i think it might be googlebot being served up the cached pages


so under "Do Not Cache User Agents" under "excludes" under "LiteSpeed Cache Settings" what do you put in the box for Googlebot ?

so that googlebot is not served up the cached pages & gets to cache all the new pages.
 
B

bobby47

New Member
#4
#4
Thanks very much for that, its started indesxing the pages, but much slower than normal, & that was after i purged the cache, i would have thouht this would be one of the most important & sought after answers because if google is'nt caching the new pages then everything else is irrelevant .

i was adding loads of new pages, and eventually it has started caching & indexing all of them, but only after i purged the cache, so i dont know if its that that done it.
 
Unique_Eric

Unique_Eric

Administrator
Staff member
#5
#5
Hi,

Basically page cache will be purged if you add/update/delete anything on it. So I am not sure why index not working until you purge it or make google bot agent as no cache.

You can test it by:
1. check page "A" is a cache miss/hit
2. edit page "A" and update
3. Revisit page "A" and it should be a miss

Best
 
B

Behinesazy

Member
#6
#6
Hello, I'm having the same problem while excluding these agents in the LiteSpeed plugin. They are indeed in the .htaccess file, but Googlebot reports a hit.

### marker NOCACHE USER AGENTS start ###
RewriteCond %{HTTP_USER_AGENT} APIs\-Google|Googlebot\ smartphone|GoogleOther\-Video/1\.0|Google\-Extended|Googlebot|GoogleOther|AdsBot\-Google\-Mobile|GoogleOther\-Image|Google\-InspectionTool|Storebot\-Google|Googlebot\-News|Googlebot\-Image|AdsBot\-Google|Mediapartners\-Google|Google\-Read\-Aloud|google\-speakr|Google\ Inspection\ Tool\ smartphone|Googlebot\ News|Googlebot\ Video|Googlebot\ Desktop|Google\ Favicon|Storebot\-Google/1\.0|Googlebot/2\.1|Googlebot\-Image/1\.0|Google\-InspectionTool/1\.0|Google\ ADT\-2|bingbot|Google\-Safety|FeedFetcher\-Google|GoogleOther\-Video|Google\-CloudVertexBot|http\://www\.google\.com/bot\.html|Google\|Googlebot [NC]
RewriteRule .* - [E=Cache-Control:no-cache]
### marker NOCACHE USER AGENTS end ###

is there any fix for this? ty
 

Attachments

B

Behinesazy

Member
#7
#7
Now I do a live URL test in Google Search Console or using Google Rich Results:

As you see its working good, but I don't know why for Googlebot smartphone it doesn't work.


1735117749011.png
 
Last edited:
B

Behinesazy

Member
#10
#10
1735132095247.png
1735132108482.png

It won't work for Google Smartphone or Desktop bot. You can test it here:

https://httpstatus.io/

Also, I sent a screenshot from Google Search Console itself (your screenshot is probably from Chrome Developer Tools or Firefox). Search Console is more trusted and shows what Google sees.
 
serpent_driver

serpent_driver

Well-Known Member
#12
#12
The inspection tool doesn't use Googlebot User-Agent. If you want to check if it works use any browser extension or plugin to change the User-Agent of your browser. There is no need to use any external services.

User-Agent for Googlebot (Desktop):
Code: 
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
User-Agent for Googlebot (Mobile):
Code: 
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.204 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
 
B

Behinesazy

Member
#13
#13
Sir, the inspection tool in Google Search Console is indeed Googlebot. I don't change anything with extensions or from the developer; I only care about what Google says officially in their Search Console.

1735133257082.png
 
B

Behinesazy

Member
#15
#15
1735133749939.png

here my settings :

### marker NOCACHE USER AGENTS start ###
RewriteCond %{HTTP_USER_AGENT} APIs\-Google|Googlebot\ smartphone|GoogleOther\-Video/1\.0|Google\-Extended|Googlebot|GoogleOther|AdsBot\-Google\-Mobile|GoogleOther\-Image|Google\-InspectionTool|Storebot\-Google|Googlebot\-News|Googlebot\-Image|AdsBot\-Google|Mediapartners\-Google|Google\-Read\-Aloud|google\-speakr|Google\ Inspection\ Tool\ smartphone|Googlebot\ News|Googlebot\ Video|Googlebot\ Desktop|Google\ Favicon|Storebot\-Google/1\.0|Googlebot/2\.1|Googlebot\-Image/1\.0|Google\-InspectionTool/1\.0|Google\ ADT\-2|bingbot|Google\-Safety|FeedFetcher\-Google|GoogleOther\-Video|Google\-CloudVertexBot|http\://www\.google\.com/bot\.html|Google\|Googlebot [NC]
RewriteRule .* - [E=Cache-Control:no-cache]
### marker NOCACHE USER AGENTS end ###

And as you see, I have no Mozilla/5.0 (compatible; Google-InspectionTool/1.0;), just Google-InspectionTool and Google-InspectionTool/1.0. I did the same for the smartphone, but it's not working.

LiteSpeed needs to fix it ASAP. It prevents pages from being indexed. I don't know why no one ever noticed, only this article from 2019.
 
serpent_driver

serpent_driver

Well-Known Member
#16
#16
There is nothing to fix. Settings to exclude certain User-Agents are correct and the x-litespeed-cache-control header is correct as well: "no-cache" and this is what you want to achieve.
 
You must log in or register to reply here.
Top