404checker
404enemy
Abonti
AhrefsBot
AIBOT
Aipbot
Alexibot
Alligator
AllSubmitter
Anarchie
Apexoo
archive.org_bot
ASPSeek
Asterias
Attach
BackDoorBot
Backlink-Ceck
Bandit
Barkrowler
BatchFTP
BDCbot
Black Hole
BlackWidow
BlowFish
Bolt
Brandwatch
Calculon
CATExplorador
CCBot
CheeseBot
ChinaClaw
Chlooe
Claritybot
Cosmos
Craftbot
crawler.feedback
crawl.sogou.com
Curious
DBLBot
demandbase-bot
Demon
Digincore
DigitalPebble
Dirbuster
Discobot
Discoverybot
DomainAppender
DomainCrawler
DomainSigmaCrawler
Dotbot
Download Wonder
Drip
DTS Agent
EasyDL
Ebingbong
EMail Siphon
Exabot
Extreme Picture Finder
EyeNetIE
Ezooms
facebookscraper
FDM
FHscan
Fimap
Freeuploader
FrontPage
FyberSpider
GalaxyBot
Genieo
Getintent
G-i-g-a-b-o-t
Go!Zilla
GrabNet
GrapeshotCrawler
GridBot
GT::WWW
Haansoft
Havij
HEADMasterSEO
Hloader
HMView
HTMLparser
HTTrack
HybridBot
Iblog
Id-search
Image Fetch
IndeedBot
Indy Library
instabid
InterGET
Internet Ninja
internetVista monitor
Iria
Iskanie
IstellaBot
Jetty
JOC Web Spider
Jorgee
Jyxobot
Kenjin Spider
Keyword Density
Kozmosbot
Lanshanbot
Larbin
LeechFTP
LeechGet
Lightspeedsystems
Likse
Linkdexbot
LinkpadBot
LinqiaMetadataDownloaderBot
LinqiaScrapeBot
Lipperhey Spider
LNSpiderguy
lwp-request
LWP::Simple
Mail.RU_Bot
Majestic SEO
MarkMonitor
Masscan
Mass Downloader
Mata Hari
Meanpathbot
MeanPath Bot
Mediatoolkitbot
mediawords
MFC_Tear_Sample
Microsoft URL Control
MIIxpc
Mojeek
MSIECrawler
Navroad
NearSite
Nettrack
Net Vampire
Netvibes
Nibbler
NPbot
Nutch
oBot
Offline Navigator
Openfind
Openvas
OrangeBot
Page Analyzer
PageScorer
Pandalytics
Papa Foto
pcBrowser
PECL::HTTP
PHPCrawl
Picscout
PictureFinder
Pi-Monster
Pockey
POE-Component-Client-HTTP
ProPowerBot
Pump
Quick-Crawler
RankActiveLinkBot
RankingBot2
Rankivabot
Reaper
RedesScrapy
ReGet
Rogerbot
RSSingBot
SalesIntelligent
ScanAlert
Scrapy
Searchestate
SemrushBot
SEOkicks
SEOkicks-Robot
seoscanners
sexsearcher
Siteimprove
SiteLockSpider
SiteSucker
Sitevigil
SlySearch
SMTBot
Snoopy
Sociscraper
sogouspider
Spammen
spyfu
Sqlmap
Sqlworm
Stripper
Sucuri
SurveyBot
Swiftbot
tAkeOut
Teleport
Telesoft
Telesphoreo
Telesphorep
The Intraformant
Thumbor
TightTwatBot
Titan
Tracemyfile
Trendiction
Trendictionbot
True_Robot
Turingos
Turnitin
TwengaBot
Twice
URLy.Warning
Vacuum
Vagabondo
VB Project
VCI
VidibleScraper
Virusdie
Voil
Wallpapers/3.0
WBSearchBot
Web Auto
WebBandit
WebCopier
WEBDAV
WebFetch
Web Fetch
WebFuck
WebGo IS
webmeup-crawler
WebReaper
WebSauger
Webshag
WebSucker
Web Sucker
Whack
Whatweb
Who.is Bot
WiseGuys Robot
WISENutbot
Wonderbot
Wprecon
WPScan
Xaldon WebSpider
YoudaoBot
Zade
zauba.io
Zermelo
zgrab
ZmEu