nginx-ultimate-bad-bot-blocker/dev-tools/test_units/random-bots-for-whitelist-test.list
2024-07-25 22:04:51 +00:00

100 lines
1 KiB
Text

Anarchie
Anarchy
Backlink-Ceck
BatchFTP
Battleztar Bazinga
BlowFish
Brandprotect
Brandwatch
BuiltWith
CensysInspect
Citoid
Copyscape
Crawling at Home Project
Dispatch
DomCopBot
Drip
Exabot
FemtosearchBot
Foobot
GalaxyBot
GetWeb
Go-Ahead-Got-It
Gotit
GrapeshotCrawler
HTTrack
IDBTE4M
IlseBot
Image Sucker
ImagesiftBot
Intelliseek
Iskanie
Jorgee
Kenjin Spider
Kinza
Leap
LeechFTP
Lftp
MJ12bot
Mail.RU_Bot
Majestic-SEO
Majestic12
Mata Hari
MicroMessenger
NICErsPRO
Nessus
Net Vampire
NetSpider
Netcraft
PageGrabber
Petalbot
PleaseCrawl
Pu_iN
PyCurl
Rankivabot
Re-re
ReGet
RealDownload
RedesScrapy
SISTRIX
Scrapy
Screaming
SemrushBot
SentiBot
Site Sucker
SiteLockSpider
Siteimprove
Snoopy
Spinn3r
SputnikBot
Surfbot
Teleport
Telesoft
Toweyabot
Twice
Typhoeus
V-BOT
VelenPublicWebCrawler
Voil
WallpapersHD
Web Enhancer
Web Pix
Whacker
Who.is Bot
awario.com
cah.io.community
crawler.feedback
crawler4j
facebookscraper
heritrix
instabid
internetVista monitor
lwp-trivial
meanpathbot
page scorer
ripz
seocompany.store
serpstatbot
voyagerx.com
webmeup-crawler
zgrab