Tag: robotstxt

Google Technical guidelines

Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.

Read more →

Related posts

Chinese (Traditional)EnglishFrenchGermanHindiRussianSerbianSpanish