Make your robots.txt, hide files or directories like this Example: "http://www.yourdomain.com/cgi-bin/" (Excludes the /cgi-bin/ directory) Napravi svoj robots.txt , sakrij fajlove ili direktorijume kao u ovom primeru: Sajt adresa: "http://www.yourdomain.com/cgi-bin/" Robots.txt zabrana: Excludes the /cgi-bin/ directory)
Tags: AltaVista, directory, Excite, generator, google, HotBot, msn, robots.txt, search crawler, seo