User-agent: *
Disallow: /cgi-bin/ 
Spider Name                      Search Engine
AbachoBOT                        Abacho 
Acoon                                Acoon  
AESOP_com_SpiderMan        Aesop 
ah-ha.com crawler                Ah-ha  
appie                                   Walhello 
Arachnoidea                          Euroseek
ArchitextSpider                      Excite  
Atomz                                   Atomz
DeepIndex                              DeepIndex
ESISmartSpider                      Ttravel Finder 
EZResult                                  EZResults 
FAST-WebCrawler                    AlltheWeb
Fido                                          PlanetSearch 
Fluffy the spider                         SearchHippo
Googlebot                                  Google 
Gigabot                                      Gigablast
Gulliver                                      Northernlight
Gulper                                       Yuntis
HenryTheMiragoRobot                  Mirago 
ia_archiver                                  Alexa
KIT-Fireball                                 Fireball 
LNSpiderguy                                Lexis-Nexis - 
Lycos_Spider_(T-Rex)                  Lycos
MantraAgent                               LookSmart
MSN                                           Microsoft Prototype Crawler
NationalDirectory-SuperSpider       National Directory - 
Nazilla                                         Websmostlinked - 
Openbot                                      Openfind - 
Openfind  piranha,Shark                Openfind - 
Scooter                                      AltaVista
Scrubby                                      Scrub The Web 
Slurp.so/1.0                                Inktomi
Slurp/2.0j                                   Inktomi
Slurp/2.0                                    Inktomi
Slurp/3.0                                    Inktomi
Tarantula                                  AltaVista
Teoma_agent1                         Teoma 
UK Searcher Spider                  UKSearcher - 
WebCrawler                            WebCrawler - 
Winona                                  WhatUSeek
ZyBorg                                  Wisenut