old - Fazer Owners Club - old
General => General => Topic started by: slimwilly on 31 May 2014, 07:53:48 am
-
Anyone else noticed that at the foot of the home page ,,it shows logged on users,,
well there always seems to be an "unknown robot" logged on :eek
wonder if he rides a Fazer
-
There is 10 of the buggers logged on now
-
These are web crawlers, there are others in the user list like Baidu (a massive Chinese search engine) that will be eating up the site's bandwidth. Depending on the server access the Admins should be able to block these. More info here (http://searchenginewatch.com/article/2067357/Bye-bye-Crawler-Blocking-the-Parasites).
-
These are web crawlers, there are others in the user list like Baidu (a massive Chinese search engine) that will be eating up the site's bandwidth. Depending on the server access the Admins should be able to block these. More info here ([url]http://searchenginewatch.com/article/2067357/Bye-bye-Crawler-Blocking-the-Parasites[/url]).
yes, you can of course block them in your robots.txt file (i assume thats what your article is about), but you need to be selective about what you block or you will muller your search rankings in google et al if you block too much. bit of a balancing act.