[solved] Give option robots.txt for each vhosts

[andych]

I would like to suggest that if LiteSpeed have an option for robots.txt builtin as feature just like .htaccess

So we can adjust robots.txt for each vhosts without having one robots.txt reflects all vhosts.

Example:
We had 2 domains,
We only welcome the crawlers to crawls from domain A
We blocks crawlers to crawls domain B

Thank you and I hope I will find solution for my case and this suggestions makes LiteSpeed even perfect web server.

 

[NiteWave]

can archive this by rewriterule easily(in .htaccess for example):

RewriteCond %{HTTP_HOST} domainB
RewriteRule robots.txt /robots_for_domainB.txt

create a text file robots_for_domainB.txt under document root to block all requests; another robots.txt is for other domains.

 

[andych]

Below are my current existing .htaccess

In which line I should add the rules above?

root@www:~# cat /var/www/.htaccess
order allow,deny
allow from all

deny from 1.2.3.4 #bad-ips

Options +FollowSymlinks
RewriteEngine On
RewriteBase /
# 4 lines below is to force visitors to use www subdom.
RewriteCond %{http_host} ^main.dom
RewriteRule ^(.*) http://www.main.dom/$1 [R=301,L]
RewriteCond %{http_host} ^secondary.dom
RewriteRule ^(.*) http://www.secondary.dom/$1 [R=301,L]

#LSWS Cache:
RewriteCond %{HTTP_COOKIE} !if_imloggedin=yes
RewriteCond %{HTTP_COOKIE} !if_userid=
RewriteCond %{HTTP_COOKIE} !if_password=
RewriteCond %{REQUEST_METHOD} ^HEAD|GET$
RewriteCond %{QUERY_STRING} !s=[a-fA-F0-9]{32}
RewriteCond %{QUERY_STRING} !product=vbnexus
RewriteCond %{REQUEST_URI} !^/(login|register|usercp|private|profile|cron|image)\.php$
RewriteCond %{REQUEST_URI} !^/admincp
RewriteCond %{REQUEST_URI} !^/modcp
RewriteRule (.*) - [E=Cache-Control:max-age=300]

RewriteRule ^home/$ forum.php [L,R=301]
RewriteRule ^f([0-9]+)/$ forumdisplay.php?f=$1 [L,R=301]
RewriteRule ^f([0-9]+)-([0-9]+)/$ forumdisplay.php?f=$1&page=$2 [L,R=301]
RewriteRule ^t([0-9]+)/$ showthread.php?t=$1 [L,R=301]
RewriteRule ^t([0-9]+)-([0-9]+)/$ showthread.php?t=$1&page=$2 [L,R=301]
RewriteRule ^t([0-9]+)-print/$ showthread.php?t=$1 [L,R=301]
RewriteRule ^blogs/u([0-9]+)-e([0-9]+)/$ entry.php?b=$2 [L,R=301]
root@www:~#

 

[NiteWave]

can add the rules before
#LSWS Cache

 

[andych]

can add the rules before
#LSWS Cache

Hi NiteWave, thank you for confirming