404checker
404enemy
80legs
Abonti
Aboundex
Aboundexbot
Acunetix
ADmantX
AIBOT
Aipbot
Alexibot
autoemailspider
Backlink-Ceck
BackWeb
Badass
Bandit
BatchFTP
BBBike
BetaBot
Bigfoot
Bitacle
BLEXBot
Bolt
BotALot
Brandprotect
Brandwatch
BuiltWith
BuzzSumo
Calculon
CATExplorador
CazoodleBot
CheeseBot
ChinaClaw
Chlooe
Cloud mapping
coccocbot-web
Cogentbot
Copyscape
Cosmos
Craftbot
crawler4j
Curious
Custo
DataCha0s
demandbase-bot
Demon
DigitalPebble
Discobot
Discoverybot
Dispatch
DittoSpyder
DnyzBot
DomainAppender
DomainSigmaCrawler
DomainStatsBot
Dragonfly
Drip
DSearch
DTS Agent
ECCP/1.0
EMail Siphon
EMail Wolf
Evil
facebookscraper
FDM
FemtosearchBot
Fimap
Firefox/7.0
FlashGet
Flunky
Foobot
FyberSpider
Fyrebot
GermCrawler
Gigablast
Gigabot
Go-Ahead-Got-It
GoZilla
GrabNet
GrapeshotCrawler
HaosouSpider
Harvest
Havij
HEADMasterSEO
HMView
HTTP::Lite
Humanlinks
HybridBot
IlseBot
Image Fetch
Image Sucker
Intelliseek
InterGET
IstellaBot
JetCar
Jyxobot
Kenjin Spider
Lanshanbot
Larbin
Libwhisker
LinkpadBot
LinkScan
LinksManager
LinqiaMetadataDownloaderBot
LinqiaRSSBot
LinqiaScrapeBot
Lipperhey Spider
Litemage_walker
Ltx71
lwp-trivial
Majestic12
MarkMonitor
MarkWatch
Mata Hari
meanpathbot
MegaIndex.ru
Microsoft URL Control
MIIxpc
Mister PiX
MJ12bot
Mojeek
MSIECrawler
Name Intelligence
NearSite
Needle
Nessus
netEstate NE Crawler
NetLyzer
Net Vampire
NICErsPRO
Nikto
NimbleCrawler
Nimbostratus
Nmap
NPbot
Nutch
oBot
Offline Explorer
Offline Navigator
OutclicksBot
PageAnalyzer
Page Analyzer
PageGrabber
page scorer
Pandalytics
Pavuk
PECL::HTTP
PeoplePal
PHPCrawl
Picsearch
PictureFinder
Pi-Monster
Pixray
POE-Component-Client-HTTP
ProPowerBot
Pump
Quick-Crawler
RankActiveLinkBot
RankingBot
RankurBot
RealDownload
RepoMonkey
Ripper
Rogerbot
SalesIntelligent
Scrapy
Searchestate
Semrush
SEOstats
serpstatbot
Siphon
Sitebeam
SiteExplorer
Siteimprove
SiteSnagger
Site Sucker
SlySearch
SmartDownload
SMTBot
Snake
SocialRankIOBot
sogouspider
Sogou web spider
Sosospider
Sottopop
Spammen
SpankBot
Spanner
Spbot
Spinn3r
SputnikBot
spyfu
Sqlworm
Sqworm
Sucuri
SuperBot
SuperHTTP
Surfbot
T0PHackTeam
T8Abot
TeleportPro
The Intraformant
TheNomad
Toweyabot
Trendiction
trendiction.com
trendiction.de
Turingos
Turnitin
TurnitinBot
Twice
Typhoeus
Upflow
URLy Warning
Vagabondo
VidibleScraper
Voil
Webalta
WebAuto
WebBandit
WebCollage
WEBDAV
WebFetch
Web Fuck
WebReaper
Website Quester
WebStripper
WebSucker
WebWhacker
WebZIP
Whacker
Who.is Bot
WinHTTrack
WiseGuys Robot
WISENutbot
Wonderbot
Wprecon
WPScan
WWW-Mechanize
x09Mozilla
x22Mozilla
Xaldon WebSpider
Xaldon_WebSpider
YoudaoBot
Zauba
Zermelo
zgrab
ZmEu
ZumBot
ZyBorg