404checker
80legs
Aboundexbot
Acunetix
AhrefsBot
AIBOT
AiHitBot
Alligator
AllSubmitter
AlphaBot
Apexoo
arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
BackDoorBot
Badass
BatchFTP
BDCbot
Bitacle
Blackboard
BlackWidow
Blow
BlowFish
Boardreader
BotALot
Brandprotect
Brandwatch
Calculon
CATExplorador
CCBot
Cegbfeieh
CheeseBot
CheTeam
Cloud mapping
coccocbot-web
cognitiveseo
CopyRightCheck
Copyscape
crawler.feedback
CrunchBot
Custo
DatabaseDriverMysqli
DataCha0s
DIIbot
Dirbuster
Discoverybot
DittoSpyder
DnyzBot
DomainAppender
DomainSigmaCrawler
Dragonfly
Drip
Ebingbong
EroCrawler
evc-batch
Exabot
Extreme Picture Finder
Firefox/7.0
FlashGet
Foobot
GalaxyBot
GermCrawler
GetRight
GetWeb
Gigabot
G-i-g-a-b-o-t
Gotit
Grabber
Grafula
GrapeFX
GrapeshotCrawler
GridBot
Harvest
Havij
heritrix
Heritrix
HTTrack
Humanlinks
Iblog
IlseBot
Indy Library
InfoTekies
internetVista monitor
ips-agent
Iskanie
Jbrofuzz
JennyBot
JetCar
Jetty
JikeSpider
JOC Web Spider
Joomla
JustView
Jyxobot
Keyword Density
Kozmosbot
Lanshanbot
LeechGet
LibWeb
Libwhisker
Lightspeedsystems
Likse
LinkextractorPro
LinkScan
LinksManager
LinkWalker
Ltx71
LWP::Simple
lwp-trivial
Magnet
Mag-Net
Mail.RU_Bot
Majestic SEO
Masscan
Mass Downloader
Mata Hari
meanpathbot
MeanPath Bot
MegaIndex.ru
MIIxpc
Mojeek
Mr.4x3
MSFrontPage
MSIECrawler
Msrabot
Musobot
NetAnts
NetMechanic
NetSpider
Nettrack
Net Vampire
Netvibes
NetZIP
NextGenSearchBot
Nibbler
Ninja
Nmap
Nutch
Octopus
Offline Explorer
OnCrawl
Openfind
Openvas
OrangeBot
Page Analyzer
PageGrabber
page scorer
PageScorer
Pandalytics
Pavuk
pcBrowser
PECL::HTTP
PeoplePal
PHPCrawl
PictureFinder
Pimonster
Pi-Monster
PleaseCrawl
plumanalytics
POE-Component-Client-HTTP
PxBroker
QueryN Metasearch
Quick-Crawler
RankActive
Recorder
RedesScrapy
ReGet
Rogerbot
RSSingBot
SBIder
ScanAlert
Scanbot
Scrapy
Screaming
ScreenerBot
Semrush
SemrushBot
SEOlyticsCrawler
SEOprofiler
seoscanners
SEOstats
serpstatbot
Sitebeam
SiteExplorer
Siteimprove
SiteLockSpider
Site Sucker
Sitevigil
SmartDownload
Snake
Snapbot
SocialRankIOBot
Sociscraper
Sogou web spider
Sosospider
Sottopop
Spammen
SpankBot
Spanner
Sqlworm
Sqworm
Sucker
SuperBot
Suzuran
sysscan
T0PHackTeam
T8Abot
tAkeOut
Teleport
Telesphoreo
Thumbor
TightTwatBot
Toata
Toweyabot
Trendiction
Trendictionbot
trendiction.de
True_Robot
Turnitin
Vacuum
VidibleScraper
Voltron
Wallpapers/3.0
WebCopier
WebFuck
WebGo IS
WebImageCollector
WebmasterWorldForumBot
WebReaper
WebSauger
Web Sauger
Webshag
WebsiteExtractor
WebsiteQuester
WebSucker
WeSEE
Whacker
Whatweb
Widow
WISENutbot
Wotbox
WPScan
x22Mozilla
Xenu
xpymep1.exe
YoudaoBot
Zade
Zermelo
Zeus
zgrab