A fast site crawler
There is a simple, but very efficient application available called “Xenu's Link Sleuth”. It can crawl site fast, but may not able to simulate different clients though.
Notice We're building a new documentation site. Check it out at docs.litespeedtech.com . litespeed_wiki:cache:thirdparty-crawlers
There is a simple, but very efficient application available called “Xenu's Link Sleuth”. It can crawl site fast, but may not able to simulate different clients though.