Spiders

Spiders list:

Yahoo – Yahoo Slurp Google – Googlebot The web archive (IA Archiver) Alexa (IA Archiver) Netcraft

Spider with lots of trafic :

Spajderi koji prave veliki saobracaj :

Twisted PageGetter Twisted PageGetter IP Address is “207.0.19.182?. Ukoliko zelite da ga blokirate ubacite sledeci kod u .htaccess na vasem serveru If you want to block it, you can put this code in your .htaccess file: 1 2 3 RewriteEngine On RewriteCond %{HTTP_USER_AGENT} “Twisted PageGetter” [NC] RewriteRule ^.*$ – [F,L] That’ll give it a 403 Forbidden Error when it tries to access your site. Ovo ce mu dati 403 Zabranjen greska ukoliko pokusa da pristupi vasem sajtu

Related posts