360Spider 80legs AIBOT ALittle Client ASPSeek Abonti AfD-Verbotsverfahren AiHitBot Anarchie Anarchy Anarchy99 Ankit Anthill Attach AwarioRssBot BLEXBot BackWeb Bandit Battleztar Bazinga Bigfoot Bitacle Black Hole BlackWidow Blow BlowFish Boardreader Bolt BotALot Brandprotect Buck Buddy Bullseye CATExplorador CCBot CazoodleBot CherryPicker ChinaClaw Chlooe Claritybot Cliqzbot Cloud mapping CopyRightCheck Cosmos Craftbot CrunchBot DBLBot DIIbot DTS Agent DataCha0s Deusu Digincore Discoverybot DittoSpyder DnBCrawler-Analytics DomainCrawler DomainSigmaCrawler Dragonfly Drip EMail Wolf Ecxi Extractor EyeNetIE FDM Firefox/7.0 FlashGet Fuzz GalaxyBot GermCrawler GetRight GetWeb Getintent Gotit GrabNet GrapeshotCrawler HEADMasterSEO HTTP::Lite HaosouSpider Hloader IRLbot Id-search Image Fetch Image Sucker InfoNaviRobot Internet Ninja Iria JamesBOT Jbrofuzz Jetty Joomla JustView Kozmosbot LNSpiderguy Lftp LibWeb LieBaoFast LinkScan LinkWalker Linkbot Linkdexbot LinkextractorPro LinksManager Lipperhey Spider Lmspider Ltx71 MFC_Tear_Sample MIDown tool MIIxpc MJ12bot MSIECrawler Mail.RU_Bot Majestic12 MarkMonitor MarkWatch Mass Downloader Masscan Mata Hari MauiBot Metauri Mojeek Mojolicious Morfeus Fucking Scanner Mozlila NICErsPRO NearSite NetAnts NetLyzer NetZIP Netcraft Netvibes Niki-bot NimbleCrawler Nmap Nutch Octopus Offline Navigator OnCrawl OrangeBot PHPCrawl Page Analyzer PageScorer PageThing.com Pandalytics Panscient Piepmatz PleaseCrawl Pockey ProWebWalker PyCurl Quick-Crawler RSSingBot Rankivabot Re-re ReGet RealDownload RebelMouse Recorder SEOlyticsCrawler SEOprofiler Scanbot ScoutJet Scrapy Screaming ScrepyBot Searchestate SearchmetricsBot SeoSiteCheckup Seomoz Shodan Site Sucker SlySearch Snake Spanner Spbot Spinn3r Sqlworm Sqworm Sucker SuperBot Suzuran T8Abot Telesphorep Thumbor Titan Tracemyfile Trendictionbot True_Robot Turnitin TurnitinBot TwengaBot URLy Warning URLy.Warning UnisterBot VB Project VCI Vacuum VelenPublicWebCrawler VidibleScraper VoidEYE Voltron WASALive-Bot WBSearchBot WISENutbot WWWOFFLE Wallpapers/3.0 WallpapersHD Web Fetch Web Pix Web Sucker WebCopier WebFetch WebPix WebStripper WebZIP Webalta Webshag WebsiteQuester Whacker Widow WinHTTrack Wonderbot Wotbox Zeus ZoominfoBot ZumBot ZyBorg arquivo.pt autoemailspider backlink-check cah.io.community cognitiveseo demandbase-bot evc-batch instabid lwp-request magpie-crawler meanpathbot netEstate NE Crawler page scorer plumanalytics polaris version probe-image-size sitechecker.pro siteripz spyfu sysscan tAkeOut ubermetrics-technologies.com webprosbot x22Mozilla zgrab