80legs
Abonti
AhrefsBot
Aipbot
Alligator
Anarchy
Anarchy99
Anthill
Aspiegel
Asterias
Attach
AwarioRssBot
AwarioSmartBot
BBBike
BDCbot
BackStreet
Badass
BatchFTP
Blackboard
Blow
BlowFish
Bolt
BotALot
Bullseye
CATExplorador
CCBot
CSHttp
Calculon
CazoodleBot
CensysInspect
ChinaClaw
Claritybot
Cloud mapping
Cogentbot
Copier
CopyRightCheck
Cosmos
CrunchBot
DSearch
DatabaseDriverMysqli
Demon
Dirbuster
Discobot
Discoverybot
DittoSpyder
DomainAppender
DomainCrawler
Dragonfly
EMail Siphon
EasyDL
EroCrawler
Exabot
ExtLinksBot
FDM
FHscan
Fimap
Firefox/7.0
FlashGet
Fuzz
GalaxyBot
Genieo
GermCrawler
GetRight
GetWeb
Gigablast
GoZilla
GrabNet
Grabber
GrapeshotCrawler
HTTP::Lite
HaosouSpider
Harvest
Humanlinks
IDBTE4M
Id-search
InfoNaviRobot
InfoTekies
Intelliseek
InterGET
Internet Ninja
Iria
JOC Web Spider
JennyBot
Joomla
Jyxobot
Kenjin Spider
Kinza
Kozmosbot
LNSpiderguy
LWP::Simple
Larbin
LibWeb
LieBaoFast
LinkScan
Linkbot
Linkdexbot
LinksManager
LinqiaRSSBot
Lipperhey
Ltx71
MFC_Tear_Sample
MIDown tool
MIIxpc
Mag-Net
Mail.RU_Bot
Majestic SEO
Majestic-SEO
Majestic12
MarkWatch
Mass Downloader
Masscan
Mata Hari
Meanpathbot
MicroMessenger
Microsoft Data Access
Mister PiX
Mojeek
Mojolicious
Mozlila
Mr.4x3
NPbot
NearSite
Needle
NetLyzer
NetZIP
Nettrack
Netvibes
Niki-bot
Nikto
NimbleCrawler
Nmap
Offline Navigator
OnCrawl
Openvas
OutfoxBot
POE-Component-Client-HTTP
PageGrabber
PageScorer
Pandalytics
Panscient
Pavuk
Petalbot
Pi-Monster
Piepmatz
PleaseCrawl
Pu_iN
Quick-Crawler
RankActive
RankFlex
Rankivabot
RankurBot
RealDownload
RebelMouse
Rogerbot
SEOkicks-Robot
SEOstats
ScanAlert
Screaming
ScreenerBot
SemrushBot
SentiBot
SeoSiteCheckup
Seomoz
Site Sucker
SiteCheckerBotCrawler
SiteLockSpider
Sitevigil
SlySearch
Snoopy
Sosospider
SpaceBison
SpankBot
Spanner
Spinn3r
Sqlmap
SuperHTTP
Surfbot
SurveyBot
T0PHackTeam
Telesoft
Telesphorep
Thumbor
Titan
Toata
Tracemyfile
Trendiction
True_Robot
Turnitin
TurnitinBot
TwengaBot
Twice
Typhoeus
URLy.Warning
Vacuum
VeriCiteCrawler
VidibleScraper
VoidEYE
Voil
Voltron
WASALive-Bot
WEBDAV
WISENutbot
WWW::Mechanize
WeSEE
Web Auto
Web Enhancer
Web Fetch
WebGo IS
WebImageCollector
WebSauger
WebZIP
WebmasterWorldForumBot
Webshag
Website Quester
WebsiteExtractor
WebsiteQuester
Webster
Whacker
Whatweb
Widow
Wonderbot
Woobot
Xaldon WebSpider
Zeus
ZyBorg
arquivo-web-crawler
coccocbot-web
cognitiveseo
crawl.sogou.com
crawler.feedback
crawler4j
evc-batch
gopher
magpie-crawler
meanpathbot
mediawords
page scorer
polaris version
ripz
scan.lol
seobility
serpstatbot
sexsearcher
siteripz
sysscan
tAkeOut
trendiction.de
voyagerx.com
xpymep1.exe
zgrab