User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /help/ Disallow: /images/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ Disallow: /cgi-bin/ Disallow: /icons1/ Disallow: /icons1/volley/ Disallow: /icons1/agap/ # don't let search engines see the RSS feed, it's just confusing. User-agent: Googlebot Disallow: / # User-agent: InfoNaviRobot # User-agent: TV33_Mercator # User-agent: AVSearch # User-agent: Mercator # User-agent: Scooter # User-agent: Slurp # User-agent: SearchengineLicenceSheep # User-agent: shadow # User-agent: MultiText # User-agent: FAST-WebCrawler # User-agent: Lycos_Spider # User-agent: Atomz # User-agent: htdig # User-agent: spider00.logika.net # User-agent: NetMechanic # User-agent: libwww-perl # User-agent: Teleport Pro # Disallow: /searchtools-rss.xml # updated 2002-03-22 (disallow rtestprob links) # updated 2002-06-25 (disallow info/slides links, info/robots/) # updated 2002-07-25 (disallow /searchtools/ which is an alias)