Tag: spiders

Sitemap for novice – up to 500 pages

If you have small website with up to 500 pages this describe best method to get your sitemap in following formats : sitemap.xml, urllist.txt, sitemap.html, sitemap.xml.gz , ror.xml.

Read more →

Related posts

Spiders

Spiders list:

Yahoo – Yahoo Slurp Google – Googlebot The web archive (IA Archiver) Alexa (IA Archiver) Netcraft

Read more →

Related posts

Google Technical guidelines

Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.

Read more →

Related posts

Chinese (Traditional)EnglishFrenchGermanHindiRussianSerbianSpanish