80legs
Aboundex
Aboundexbot
AhrefsBot
AIBOT
Alligator
AlphaBot
Anarchie
ASPSeek
Attach
Backlink-Ceck
backlink-check
Bandit
BDFetch
BetaBot
Bigfoot
Blackboard
Black Hole
BlackWidow
Blow
BlowFish
Boardreader
Buddy
Bullseye
CheeseBot
ChinaClaw
Cliqzbot
coccocbot-web
Cogentbot
CopyRightCheck
Copyscape
Cosmos
crawler4j
CrunchBot
CSHttp
Curious
DBLBot
demandbase-bot
Demon
Digincore
DigitalPebble
DIIbot
Dirbuster
Discobot
DittoSpyder
DnyzBot
DomainAppender
DomainCrawler
DomainSigmaCrawler
DomainStatsBot
Dotbot
Download Wonder
EasyDL
Ebingbong
Ecxi
EirGrabber
evc-batch
Exabot
Express WebPictures
ExtLinksBot
EyeNetIE
FHscan
Foobot
Freeuploader
FrontPage
Fyrebot
GalaxyBot
Gigabot
Go!Zilla
GrabNet
GrapeFX
Haansoft
Harvest
Havij
HEADMasterSEO
Hloader
HTMLparser
Humanlinks
Iblog
Id-search
IlseBot
Image Sucker
IndeedBot
Indy Library
InfoTekies
instabid
Intelliseek
InterGET
InternetSeer
Iria
Iskanie
JamesBOT
Jbrofuzz
Joomla
Keyword Density
Lanshanbot
LeechGet
LexiBot
Lftp
Libwhisker
Lightspeedsystems
LinkpadBot
LinkWalker
LinqiaMetadataDownloaderBot
LinqiaRSSBot
Lipperhey
Lipperhey Spider
Litemage_walker
lwp-trivial
Mag-Net
magpie-crawler
Mail.RU_Bot
Majestic12
MarkMonitor
MarkWatch
Masscan
meanpathbot
Meanpathbot
MeanPath Bot
Mediatoolkitbot
Metauri
Microsoft URL Control
Mister PiX
MSFrontPage
muhstik-scan
netEstate NE Crawler
NetSpider
NetZIP
Nikto
NimbleCrawler
Nimbostratus
NPbot
oBot
Offline Explorer
Offline Navigator
OnCrawl
OpenLinkProfiler
Openvas
OutclicksBot
OutfoxBot
PageAnalyzer
Page Analyzer
PageGrabber
page scorer
PageScorer
Pandalytics
Pavuk
PECL::HTTP
Picsearch
Pimonster
Pixray
PleaseCrawl
Pockey
POE-Component-Client-HTTP
PxBroker
PyCurl
RankFlex
Rankivabot
RealDownload
Reaper
RebelMouse
Recorder
ReGet
Ripper
SalesIntelligent
scan.lol
ScoutJet
SemrushBot
SEOkicks-Robot
SEOprofiler
seoscanners
SeoSiteCheckup
SISTRIX
Sitebeam
SiteExplorer
SiteLockSpider
Site Sucker
SlySearch
SMTBot
Snake
Snoopy
SocialRankIOBot
Sociscraper
Sottopop
SpaceBison
SpankBot
spyfu
Sqlworm
Steeler
Sucuri
SuperHTTP
Surfbot
Suzuran
Szukacz
T0PHackTeam
T8Abot
tAkeOut
Telesoft
Telesphorep
The Intraformant
TheNomad
Thumbor
TightTwatBot
Trendiction
trendiction.com
Turingos
Turnitin
TwengaBot
URLy Warning
URLy.Warning
Vacuum
Vagabondo
Virusdie
WASALive-Bot
Webalta
Web Auto
Web Collage
WebCopier
WEBDAV
Web Fetch
WebFuck
Web Fuck
WebGo IS
WebLeacher
WebmasterWorldForumBot
Web Pix
WebSauger
WebsiteExtractor
WebsiteQuester
WebSucker
WebWhacker
WebZIP
WeSEE
Whacker
Who.is Bot
Widow
WinHTTrack
Wotbox
WWW-Collector-E
WWW::Mechanize
WWWOFFLE
x09Mozilla
Xaldon_WebSpider
Xenu
Zade
zauba.io
zgrab
Zitebot
ZumBot
ZyBorg