02-06-14, 03:57 AM
(31-05-14, 04:43 PM)Buzz link Wrote: These are web crawlers, there are others in the user list like Baidu (a massive Chinese search engine) that will be eating up the site's bandwidth. Depending on the server access the Admins should be able to block these. More info here.
yes, you can of course block them in your robots.txt file (i assume thats what your article is about), but you need to be selective about what you block or you will muller your search rankings in google et al if you block too much. bit of a balancing act.