Block Bad Search Bots and Spiders Using mod_rewrite
There are many search bots, spiders, and content scrapers out in the internet. Some are completely legitimate like Google and Bing; however, there are also many that are overly aggressive or are simply out to scrape content or email addresses from your web site. Analyzing my own web site access log files led me to discover that there are several extremely aggressive bots that are hitting my web site frequently all day, that seem to have no true origin or search engine use and they were causing me some concern.
[ Read More ]