# Robot User-agent: Ad Muncher User-agent: ADSAComponent User-agent: ADSARobot User-agent: Albert Indexer User-agent: AnswerChase User-agent: ASPSeek User-agent: ATA-Translation-Service User-agent: AVSearch User-agent: Avant Browser User-agent: Beamer User-agent: beholder or User-agent: e-sense User-agent: BravoBrian User-agent: bumblebee@relevare.com User-agent: Checkbot User-agent: contype User-agent: Convera Internet Spider User-agent: ConveraCrawler User-agent: ccubee User-agent: Custo User-agent: CyberNavi_WebGet User-agent: DaviesBot User-agent: deepweb User-agent: EbiNess User-agent: EmailWolf User-agent: Excalibur Internet Spider User-agent: Expired Domain Sleuth User-agent: Everest-Vulcan Inc./ User-agent: GigaBaz User-agent: GigaBazVStheWeb User-agent: crawler@brainbot.com User-agent: Giskard User-agent: grub-client User-agent: heritrix User-agent: htdig User-agent: http://webwarper.net User-agent: infoGIST User-agent: InterGO User-agent: InternetArchive User-agent: Internet Ninja User-agent: InternetSeer User-agent: ipiumBot User-agent: InternetAmi IOR User-agent: InsumaScout/ User-agent: Katriona User-agent: larbin User-agent: LEIA User-agent: LexiBot User-agent: LimeBot User-agent: logikabot User-agent: Mata Hari User-agent: metabot User-agent: Mister Pix II User-agent: MOSES 2.0 Spider User-agent: MonkeyCrawl User-agent: NetCruiser User-agent: NPBot User-agent: NetZippy User-agent: NutchCVS User-agent: NZBot User-agent: Opencola User-agent: ORA_checksite User-agent: Onekit.com - PAD File Get. User-agent: Oxxbot1 User-agent: Pansophica User-agent: Phoaks User-agent: phpMySearch-Crawler User-agent: PICgrabber User-agent: PictureOfInternet User-agent: erik@malfunction.org User-agent: PicSpider User-agent: PintaSpider User-agent: Pita (Chub.Stanford.EDU) User-agent: PitSpyder Thread0 User-agent: psbot User-agent: PolyBot User-agent: PureSight User-agent: Rumours-Agent User-agent: RepoMonkey Bait & Tackle User-agent: Robot2.0(PingSoft) User-agent: SilentSurf User-agent: SlySearch User-agent: slysearch@slysearch.com User-agent: SpaceBison User-agent: CrawlWave User-agent: SpotOn User-agent: SQ Webscanner User-agent: Sqworm User-agent: Steganos Internet Anonym User-agent: SurfControl User-agent: Tagword User-agent: TaWWWantula User-agent: Tcl http client package User-agent: TeraCrawl User-agent: TurnitinBot User-agent: UCmore User-agent: UdmSearch User-agent: unchaos_crawler User-agent: unlostBot User-agent: unlostBot@unlost.com User-agent: URLBlaze User-agent: utopy User-agent: crawler@utopy.com User-agent: UtilMind HTTPGet User-agent: UrlScope User-agent: Vagabondo User-agent: VCI WebViewer User-agent: vspider User-agent: WAVETools User-agent: Webbandit User-agent: Webclipping.com User-agent: webcollage User-agent: WebCompass User-agent: WebGenie User-agent: Web Hound User-agent: Web Magnet User-agent: WebMiner User-agent: WebPix User-agent: Webpush User-agent: WebSymmetrix User-agent: webrank User-agent: webwasher User-agent: WhosTalking User-agent: www.MacroX.de User-agent: XupiterToolbar User-agent: yacy User-agent: YottaShopping_Bot User-agent: BizBot04 kirk.overleaf.com User-agent: HappyBot (gserver.kw.net) User-agent: CaliforniaBrownSpider User-agent: EI*Net/0.1 libwww/0.1 User-agent: Ibot/1.0 libwww-perl/0.40 User-agent: Merritt/1.0 User-agent: StatFetcher/1.0 User-agent: TeacherSoft/1.0 libwww/2.17 User-agent: WWW Collector User-agent: processor/0.0ALPHA libwww-perl/0.20 User-agent: wobot/1.0 from 206.214.202.45 User-agent: Libertech-Rover www.libertech.com? User-agent: WhoWhere Robot User-agent: ITI Spider User-agent: w3index User-agent: MyCNNSpider User-agent: SummyCrawler User-agent: OGspider User-agent: linklooker User-agent: CyberSpyder (amant@www.cyberspyder.com) User-agent: SlowBot User-agent: heraSpider User-agent: Surfbot User-agent: Bizbot003 User-agent: WebWalker User-agent: SandBot User-agent: EnigmaBot User-agent: spyder3.microsys.com User-agent: www.freeloader.com. User-agent: Robot identifier User-agent: AbachoBOT User-agent: abcdatos_botlink User-agent: http://www.abcdatos.com/botlink/ User-agent: AESOP_com_SpiderMan User-agent: ah-ha.com crawler (crawler@ah-ha.com) User-agent: ia_archiver User-agent: Scooter User-agent: Mercator User-agent: Scooter2_Mercator_3-1.0 User-agent: roach.smo.av.com-1.0 User-agent: Tv_Merc_resh_26_1_D-1.0 User-agent: AltaVista-Intranet User-agent: jan.gelin@av.com User-agent: FAST-WebCrawler User-agent: crawler@fast.no User-agent: www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html User-agent: Wget User-agent: Acoon Robot User-agent: antibot User-agent: Atomz User-agent: AxmoRobot User-agent: Buscaplus Robi User-agent: http://www.buscaplus.com/robi/ User-agent: CanSeek/ User-agent: support@canseek.ca User-agent: ChristCRAWLER User-agent: http://www.christcrawler.com/ User-agent: Clushbot User-agent: http://www.clush.com/bot.html User-agent: Crawler User-agent: admin@crawler.de User-agent: DaAdLe.com ROBOT/ User-agent: RaBot User-agent: Agent-admin/ phortse@hanmail.net User-agent: contact/jylee@kies.co.kr User-agent: Agent-admin/ webmaster@kisco.go.kr User-agent: DeepIndex User-agent: DittoSpyder User-agent: Jack User-agent: EARTHCOM.info User-agent: Speedy Spider User-agent: ArchitextSpider User-agent: ArchitectSpider User-agent: EuripBot User-agent: Arachnoidea User-agent: arachnoidea@euroseek.net User-agent: EZResult User-agent: Fast PartnerSite Crawler User-agent: FAST Data Search Crawler User-agent: FAST Data Search Document Retriever User-agent: KIT-Fireball User-agent: france.misesajour.com User-agent: FyberSearch User-agent: GalaxyBot User-agent: http://www.galaxy.com/galaxybot.html User-agent: geckobot User-agent: GenCrawler User-agent: GeonaBot User-agent: getRAX User-agent: googlebot@googlebot.com User-agent: http://googlebot.com/ User-agent: moget/2.0 User-agent: moget@goo.ne.jp User-agent: Aranha User-agent: Slurp.so/1.0 User-agent: slurp@inktomi.com User-agent: Slurp/2.0j User-agent: www.inktomisearch.com User-agent: Slurp/2.0-KiteHourly User-agent: slurp@inktomi.com; User-agent: www.inktomi.com/slurp.html User-agent: Slurp/2.0-OwlWeekly User-agent: spider@aeneid.com User-agent: Slurp/3.0-AU User-agent: Toutatis 2.5-2 User-agent: Hubater User-agent: http://www.almaden.ibm.com/cs/crawler User-agent: IlTrovatore-Setaccio User-agent: IncyWincy User-agent: UltraSeek User-agent: InfoSeek Sidewinder User-agent: Mole2/1.0 User-agent: webmaster@intags.de User-agent: MP3Bot User-agent: C-PBWF-ip3000.com-crawler User-agent: ip3000.com-crawler User-agent: http://www.istarthere.com User-agent: spider@istarthere.com User-agent: Knowledge.com/ User-agent: kuloko-bot/0.2 User-agent: LNSpiderguy User-agent: Linknzbot User-agent: lookbot User-agent: MantraAgent User-agent: NetResearchServer User-agent: www.loopimprovements.com/robot.html User-agent: Lycos_Spider_(T-Rex) User-agent: JoocerBot User-agent: HenryTheMiragoRobot User-agent: MojeekBot User-agent: mozDex/ User-agent: MSNBOT/0.1 User-agent: http://search.msn.com/msnbot.htm) User-agent: Navadoo Crawler User-agent: Gulliver User-agent: ObjectsSearch/0.01 User-agent: OnetSzukaj/ User-agent: PicoSearch/ User-agent: PJspider User-agent: DIIbot User-agent: nttdirectory_robot User-agent: super-robot@super.navi.ocn.ne.jp User-agent: griffon User-agent: griffon@super.navi.ocn.ne.jp User-agent: Spider/maxbot.com User-agent: admin@maxbot.com User-agent: various (fakes agent on each access) User-agent: gazz/1.0 User-agent: gazz@nttrd.com User-agent: ??? User-agent: NationalDirectory-SuperSpider User-agent: dloader(NaverRobot)/ User-agent: dumrobo(NaverRobot)/ User-agent: noxtrumbot/ User-agent: Openfind piranha,Shark User-agent: robot-response@openfind.com.tw User-agent: Openbot/ User-agent: www.picsearch.org/bot.html User-agent: CrawlerBoy Pinpoint.com User-agent: user.ip3000.com User-agent: QweeryBot User-agent: http://qweerybot.qweery.com) User-agent: AlkalineBOT User-agent: StackRambler/ User-agent: SeznamBot User-agent: Search-10 User-agent: Fluffy the spider User-agent: info@searchhippo.com) User-agent: Scrubby/ User-agent: asterias User-agent: speedfind ramBot xtreme User-agent: Kototoi/0.1 User-agent: SearchByUsa User-agent: Searchspider/ User-agent: SightQuestBot/ User-agent: http://www.sightquest.com/bot.htm User-agent: Spider_Monkey/ User-agent: Surfnomore Spider v1.1 User-agent: Robot@SuperSnooper.Com User-agent: teoma_agent1 User-agent: teoma_admin@hawkholdings.com User-agent: Teradex_Mapper User-agent: mapper@teradex.com User-agent: ESISmartSpider User-agent: Spider TraficDublu User-agent: Tutorial Crawler User-agent: http://www.tutorgig.com/crawler User-agent: updated/0.1beta User-agent: crawler@updated.com User-agent: UK Searcher Spider User-agent: Vivante Link Checker User-agent: appie User-agent: Nazilla User-agent: www.WebWombat.com.au User-agent: marvin/infoseek User-agent: marvin-team@webseek.de User-agent: MuscatFerret User-agent: WhizBang! Lab User-agent: ZyBorg User-agent: (info@WISEnut.com) User-agent: WIRE WebRefiner: User-agent: webrefiner@wire.co.uk User-agent: WSCbot User-agent: Yandex User-agent: Yellopet-Spider User-agent: Findexa Crawler User-agent: YBSbot search engine indexer User-agent: libwww-perl User-agent: Iron33 Disallow: /files/ Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: / # Downloader User-agent: Alligator User-agent: BatchFTP User-agent: ChinaClaw User-agent: DA User-agent: DLExpert User-agent: Download Demon User-agent: Download Master User-agent: Download Ninja User-agent: Download Wonder User-agent: Ez Auto Downloader User-agent: FreshDownload User-agent: Go!Zilla User-agent: GetRight User-agent: MyGetRight User-agent: GetSmart User-agent: HiDownload User-agent: JetCar (or FlashGet) User-agent: Kapere User-agent: Kontiki Client User-agent: LeechFTP User-agent: LeechGet User-agent: LightningDownload User-agent: Mass Downloader User-agent: MetaProducts Download Express User-agent: NetZip Downloader User-agent: SmartDownload User-agent: NetAnts User-agent: NetButler User-agent: NetPumper User-agent: Net Vampire User-agent: Nitro Downloader User-agent: Octopus User-agent: PuxaRapido User-agent: RealDownload User-agent: SpeedDownload User-agent: WebDownloader for X 1.30 User-agent: WebLeacher User-agent: WebPictures Downloader User-agent: X-Uploader Disallow: /files/ Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: / # robot, Browser, research, validator User-agent: GT::WWW User-agent: HTTPClient User-agent: HTTP::Lite User-agent: IP*Works! User-agent: libghttp User-agent: Macromedia Flash Player User-agent: MFC_Tear_Sample User-agent: PEAR HTTP_Request class User-agent: Python-urllib User-agent: RPT-HTTPClient User-agent: TeamSoft WinInet Component User-agent: wget User-agent: WinScripter iNet Tools User-agent: W3CRobot/ User-agent: W3C-WebCon/ User-agent: wxWidgets User-agent: Zeus Webster Pro User-agent: citenikbot/ User-agent: CLIPS-index User-agent: Computer_and_Automation_Research_Institute_Crawler User-agent: cosmos User-agent: robot@xyleme.com User-agent: D2KWebCrawler User-agent: DiaGem/ User-agent: Digimarc WebReader User-agent: EchO!/2.0 User-agent: FinaleRobot User-agent: robot-master@expressus.com User-agent: Ideare - SignSite User-agent: GentleSpider User-agent: Gulper Web Bot User-agent: sebastien.ailleret@inria.fr User-agent: ghi@lcs.mit.edu User-agent: IRLbot User-agent: KnowItAll User-agent: MJ12bot User-agent: MultiText User-agent: NEC Research Agent User-agent: OntoSpider User-agent: sherlock_spider User-agent: S.T.A.L.K.E.R. User-agent: Steeler User-agent: ru-robot User-agent: 0.1_hseo(at)cs.rutgers.edu User-agent: USyd-NLP-Spider User-agent: WebGather User-agent: xyro User-agent: xcrawler@inria.fr User-agent: Zao/0.2 User-agent: Zao-Crawler User-agent: Googlebot User-agent: AmigaVoyager User-agent: xChaos_Arachne User-agent: IBrowse User-agent: ICab User-agent: JustView User-agent: KMeleon User-agent: Konqueror User-agent: Lynx User-agent: OmniWeb User-agent: Opera User-agent: Plucker User-agent: pwWebSpeak User-agent: QWeb User-agent: retawq User-agent: SlimBrowser User-agent: Sleipnir User-agent: VMS_Mosaic User-agent: WannaBe User-agent: w3m User-agent: abyss.w3.org User-agent: 64.29.16.182 User-agent: selfpromotion.com User-agent: Squid User-agent: SquidClamAV_Redirector Disallow :