360Spider
404checker
Aboundexbot
Acunetix
AhrefsBot
AIBOT
AiHitBot
Aipbot
Alligator
AlphaBot
Anarchie
autoemailspider
BackDoorBot
backlink-check
BacklinkCrawler
BackWeb
Barkrowler
BBBike
Bigfoot
BlackWidow
Boardreader
Buddy
BuiltBotTough
Calculon
CazoodleBot
CCBot
Cegbfeieh
Cliqzbot
Cloud mapping
Cogentbot
Collector
Copier
Copyscape
Craftbot
crawler4j
crawler.feedback
crawl.sogou.com
CSHttp
DatabaseDriverMysqli
Demon
Deusu
Dirbuster
Disco
Discoverybot
DittoSpyder
DnyzBot
DomainAppender
DomainSigmaCrawler
DomainStatsBot
Download Wonder
Drip
DSearch
DTS Agent
EasyDL
eCatch
EMail Wolf
Evil
ExtLinksBot
Extractor
ExtractorPro
facebookscraper
FDM
FemtosearchBot
Fimap
Firefox/7.0
Flunky
Foobot
FyberSpider
Getintent
GetRight
Gigabot
G-i-g-a-b-o-t
Gotit
Go!Zilla
Grabber
Grafula
GrapeFX
GrapeshotCrawler
HaosouSpider
Harvest
HMView
HTMLparser
HTTP::Lite
Humanlinks
Iblog
IDBot
Image Sucker
IndeedBot
Indy Library
InfoNaviRobot
instabid
Intelliseek
Iria
JamesBOT
Jbrofuzz
JetCar
Jetty
Joomla
Kozmosbot
LexiBot
LibWeb
Linkdexbot
LinkextractorPro
LinkpadBot
LinksManager
LinkWalker
LinqiaMetadataDownloaderBot
LinqiaScrapeBot
Lipperhey
Lipperhey Spider
Lmspider
LNSpiderguy
Ltx71
lwp-request
LWP::Simple
Mag-Net
magpie-crawler
Mail.RU_Bot
Majestic12
Majestic-SEO
Mb2345Browser
Meanpathbot
MFC_Tear_Sample
MicroMessenger
MIDown tool
Mister PiX
Mr.4x3
Msrabot
Musobot
Name Intelligence
Needle
Netcraft
NetLyzer
NetSpider
Nettrack
Net Vampire
Netvibes
NICErsPRO
Nikto
NimbleCrawler
Ninja
Nmap
NPbot
oBot
Octopus
Openfind
OpenLinkProfiler
OrangeBot
OrangeSpider
page scorer
PageScorer
Papa Foto
PECL::HTTP
PHPCrawl
PictureFinder
PleaseCrawl
POE-Component-Client-HTTP
Probethenet
ProPowerBot
Psbot
PxBroker
PyCurl
QueryN Metasearch
RankFlex
RankingBot2
RankurBot
RealDownload
Reaper
ReGet
RocketCrawler
SalesIntelligent
Semrush
SEOkicks
SEOkicks-Robot
SEOlyticsCrawler
Seomoz
SeoSiteCheckup
sexsearcher
Shodan
SiteExplorer
Siteimprove
SiteLockSpider
SiteSnagger
SiteSucker
Site Sucker
SlySearch
SMTBot
Snake
Sogou web spider
Sosospider
Sottopop
SpankBot
Spbot
Spinn3r
SputnikBot
Sqlmap
Sqworm
SuperBot
SuperHTTP
Surfbot
tAkeOut
TeleportPro
Telesoft
Telesphorep
Thumbor
TightTwatBot
Titan
Toweyabot
Trendiction
trendiction.com
trendiction.de
VB Project
VeriCiteCrawler
Virusdie
VoidEYE
Voltron
WallpapersHD
WBSearchBot
WebAuto
Web Auto
WebBandit
WebCopier
WebEnhancer
Web Enhancer
WebFetch
WebGo IS
WebLeacher
WebmasterWorldForumBot
webmeup-crawler
WebPix
Web Pix
WebReaper
WebsiteQuester
Webster
WebSucker
Web Sucker
Whacker
Whatweb
Who.is Bot
WPScan
WWWOFFLE
Xaldon WebSpider
Xenu
YoudaoBot
zauba.io
Zermelo
Zeus
zgrab
ZmEu
ZyBorg