360Spider 80legs Abonti Aboundexbot Acunetix AfD-Verbotsverfahren AhrefsBot Aipbot Alexibot AllSubmitter Anarchie Anarchy99 Ankit Apexoo Aspiegel Asterias AwarioSmartBot BBBike BackStreet Barkrowler BatchFTP Bigfoot Black Hole BlackWidow Blackboard Bolt BotALot BuiltWith BuzzSumo CODE87 Calculon CensysInspect CheeseBot Chlooe Citoid Cliqzbot Collector CopyRightCheck Copyscape Cosmos Craftbot CrazyWebCrawler Custo CyotekWebCopy DBLBot Devil Disco Discoverybot Dispatch DnyzBot DomCopBot Dotbot Download Wonder EMail Siphon Ebingbong ExtractorPro FDM Fimap Firefox/7.0 FlashGet Freeuploader FrontPage Fyrebot Gigabot Go!Zilla GoZilla GrapeshotCrawler GridBot HEADMasterSEO HTTP::Lite Harvest Havij Humanlinks IDBTE4M IDBot Iblog IndeedBot InfoTekies InterGET InternetSeer Iskanie JOC Web Spider JamesBOT Jbrofuzz JennyBot JetCar JikeSpider Joomla JustView Keyword Density Kozmosbot LNSpiderguy Lanshanbot LexiBot Lightspeedsystems Likse LinkWalker Linkbot LinkpadBot LinqiaScrapeBot MIIxpc MSFrontPage MSIECrawler Mag-Net Majestic-SEO MarkWatch MauiBot Mb2345Browser MeanPath Bot MegaIndex.ru MicroMessenger Microsoft URL Control Minefield Musobot Nameprotect NearSite Nessus NetAnts NetZIP Netcraft NextGenSearchBot Nibbler NimbleCrawler Nimbostratus Offline Explorer OnCrawl OpenVAS OrangeSpider OutclicksBot PECL::HTTP PHPCrawl POE-Component-Client-HTTP PageAnalyzer PageScorer Pandalytics Pavuk Petalbot Piepmatz ProWebWalker Probethenet Pump PyCurl RankActiveLinkBot RankFlex RankingBot Re-re Reaper Rogerbot SEOkicks-Robot ScanAlert Screaming ScrepyBot Searchestate SearchmetricsBot Seekport SemrushBot SentiBot SeoSiteCheckup Siphon Site Sucker SiteLockSpider Sitebeam Snoopy SocialRankIOBot Sociscraper Sosospider SpaceBison Spammen SpankBot Spbot Spinn3r SputnikBot Sqlmap Sqlworm SuperHTTP Surfbot Swiftbot T0PHackTeam Teleport TeleportPro Titan Tracemyfile Trendictionbot Turingos TurnitinBot URLy Warning URLy.Warning VeriCiteCrawler Virusdie WISENutbot WWW-Collector-E WWW-Mechanize WWW::Mechanize WallpapersHD WeSEE Web Auto Web Collage Web Enhancer WebAuto WebCollage WebFuck WebLeacher WebPix WebWhacker Webalta Webshag Website Quester WebsiteQuester Whacker Whatweb WiseGuys Robot Wotbox Wprecon Xaldon WebSpider Xaldon_WebSpider Xenu YoudaoBot ZmEu ZyBorg archive.org_bot arquivo.pt autoemailspider coccocbot crawler.feedback crawler4j dataforseo.com demandbase-bot evc-batch facebookscraper heritrix ips-agent isitwp.com iubenda-radar linkdexbot lwp-trivial magpie-crawler meanpathbot mediawords polaris version s1z.ru scalaj-http scan.lol sexsearcher tAkeOut trendiction.de ubermetrics-technologies.com voyagerx.com webpros.com x22Mozilla zgrab