mirror of
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git
synced 2025-09-02 18:50:13 +00:00
V3.2017.07.720 [ci skip]
This commit is contained in:
Travis
parent
1224b15c51
commit
54dff4a849
14 changed files with 586 additions and 88 deletions
|
|
@ -1,4 +1,4 @@
|
|||
Last Tested: Fri Jul 21 14:45:45 SAST 2017
|
||||
Last Tested: Fri Jul 21 14:58:12 SAST 2017
|
||||
|
||||
% Total % Received % Xferd Average Speed Time Time Time Current
|
||||
Dload Upload Total Spent Left Speed
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
Last Tested: Fri Jul 21 14:45:45 SAST 2017
|
||||
Last Tested: Fri Jul 21 14:58:12 SAST 2017
|
||||
|
||||
% Total % Received % Xferd Average Speed Time Time Time Current
|
||||
Dload Upload Total Spent Left Speed
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
Last Tested: Fri Jul 21 14:45:45 SAST 2017
|
||||
Last Tested: Fri Jul 21 14:58:12 SAST 2017
|
||||
|
||||
% Total % Received % Xferd Average Speed Time Time Time Current
|
||||
Dload Upload Total Spent Left Speed
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
Last Tested: Fri Jul 21 14:45:45 SAST 2017
|
||||
Last Tested: Fri Jul 21 14:58:12 SAST 2017
|
||||
|
||||
% Total % Received % Xferd Average Speed Time Time Time Current
|
||||
Dload Upload Total Spent Left Speed
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
Last Tested: Fri Jul 21 14:45:45 SAST 2017
|
||||
Last Tested: Fri Jul 21 14:58:12 SAST 2017
|
||||
|
||||
<?php
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
Last Tested: Fri Jul 21 14:45:45 SAST 2017
|
||||
Last Tested: Fri Jul 21 14:58:12 SAST 2017
|
||||
|
||||
<?php
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,498 @@
|
|||
360Spider
|
||||
80legs
|
||||
Abonti
|
||||
Aboundex
|
||||
Acunetix
|
||||
ADmantX
|
||||
AhrefsBot
|
||||
AIBOT
|
||||
AiHitBot
|
||||
Aipbot
|
||||
Alexibot
|
||||
Alligator
|
||||
AllSubmitter
|
||||
Anarchie
|
||||
Apexoo
|
||||
ASPSeek
|
||||
Asterias
|
||||
Attach
|
||||
autoemailspider
|
||||
BackDoorBot
|
||||
Backlink-Ceck
|
||||
backlink-check
|
||||
BacklinkCrawler
|
||||
BackStreet
|
||||
BackWeb
|
||||
Badass
|
||||
Bandit
|
||||
BatchFTP
|
||||
Battleztar Bazinga
|
||||
BBBike
|
||||
BDFetch
|
||||
BetaBot
|
||||
Bigfoot
|
||||
Bitacle
|
||||
Blackboard
|
||||
Black Hole
|
||||
BlackWidow
|
||||
BLEXBot
|
||||
Blow
|
||||
BlowFish
|
||||
Boardreader
|
||||
Bolt
|
||||
BotALot
|
||||
Brandprotect
|
||||
BrandProtect
|
||||
Brandwatch
|
||||
Bubing
|
||||
Buddy
|
||||
BuiltBotTough
|
||||
BuiltWith
|
||||
Bullseye
|
||||
BunnySlippers
|
||||
BuzzSumo
|
||||
Calculon
|
||||
CATExplorador
|
||||
CazoodleBot
|
||||
CCBot
|
||||
Cegbfeieh
|
||||
CheeseBot
|
||||
CherryPicker
|
||||
ChinaClaw
|
||||
Chlooe
|
||||
Claritybot
|
||||
Cliqzbot
|
||||
Cogentbot
|
||||
cognitiveseo
|
||||
Collector
|
||||
Copier
|
||||
CopyRightCheck
|
||||
Copyscape
|
||||
Cosmos
|
||||
Craftbot
|
||||
CrazyWebCrawler
|
||||
Crescent
|
||||
CSHttp
|
||||
Curious
|
||||
Custo
|
||||
DatabaseDriverMysqli
|
||||
DataCha0s
|
||||
DBLBot
|
||||
Demon
|
||||
Deusu
|
||||
Devil
|
||||
DIIbot
|
||||
Dirbuster
|
||||
Disco
|
||||
Discobot
|
||||
Discoverybot
|
||||
DittoSpyder
|
||||
DomainAppender
|
||||
DomainCrawler
|
||||
DomainSigmaCrawler
|
||||
Dotbot
|
||||
Download Demon
|
||||
Download Devil
|
||||
Download Wonder
|
||||
Dragonfly
|
||||
Drip
|
||||
DTS Agent
|
||||
EasyDL
|
||||
Ebingbong
|
||||
eCatch
|
||||
ECCP/1.0
|
||||
Ecxi
|
||||
EirGrabber
|
||||
EMail Collector
|
||||
EMail Extractor
|
||||
EMail Siphon
|
||||
EMail Wolf
|
||||
EroCrawler
|
||||
Evil
|
||||
Exabot
|
||||
Express WebPictures
|
||||
Extractor
|
||||
ExtractorPro
|
||||
Extreme Picture Finder
|
||||
EyeNetIE
|
||||
Ezooms
|
||||
FHscan
|
||||
Fimap
|
||||
Findxbot
|
||||
Firefox/7.0
|
||||
FlashGet
|
||||
Flunky
|
||||
Foobot
|
||||
fq
|
||||
Freeuploader
|
||||
FrontPage
|
||||
Fyrebot
|
||||
GalaxyBot
|
||||
Genieo
|
||||
Getintent
|
||||
GetRight
|
||||
GetWeb
|
||||
Gigablast
|
||||
Gigabot
|
||||
Go-Ahead-Got-It
|
||||
Gotit
|
||||
GoZilla
|
||||
Go!Zilla
|
||||
Grabber
|
||||
GrabNet
|
||||
Grafula
|
||||
GrapeFX
|
||||
GrapeshotCrawler
|
||||
GridBot
|
||||
GT::WWW
|
||||
HaosouSpider
|
||||
Harvest
|
||||
Havij
|
||||
HEADMasterSEO
|
||||
Heritrix
|
||||
Hloader
|
||||
HMView
|
||||
HTMLparser
|
||||
HTTP::Lite
|
||||
HTTrack
|
||||
Humanlinks
|
||||
HybridBot
|
||||
Iblog
|
||||
IDBot
|
||||
Id-search
|
||||
IlseBot
|
||||
Image Fetch
|
||||
Image Stripper
|
||||
Image Sucker
|
||||
Indy Library
|
||||
InfoNaviRobot
|
||||
InfoTekies
|
||||
Intelliseek
|
||||
InterGET
|
||||
Internet Ninja
|
||||
InternetSeer
|
||||
internetVista monitor
|
||||
Iria
|
||||
IRLbot
|
||||
Iskanie
|
||||
JamesBOT
|
||||
Jbrofuzz
|
||||
JennyBot
|
||||
JetCar
|
||||
JikeSpider
|
||||
JOC Web Spider
|
||||
Joomla
|
||||
Jorgee
|
||||
JustView
|
||||
Jyxobot
|
||||
Kenjin Spider
|
||||
Keyword Density
|
||||
Lanshanbot
|
||||
Larbin
|
||||
LeechFTP
|
||||
LeechGet
|
||||
LexiBot
|
||||
Lftp
|
||||
LibWeb
|
||||
Libwhisker
|
||||
Lightspeedsystems
|
||||
Likse
|
||||
Linkdexbot
|
||||
LinkextractorPro
|
||||
LinkpadBot
|
||||
LinkScan
|
||||
LinksManager
|
||||
LinkWalker
|
||||
LinqiaMetadataDownloaderBot
|
||||
LinqiaRSSBot
|
||||
LinqiaScrapeBot
|
||||
Lipperhey
|
||||
Litemage_walker
|
||||
Lmspider
|
||||
LNSpiderguy
|
||||
Ltx71
|
||||
lwp-request
|
||||
LWP::Simple
|
||||
lwp-trivial
|
||||
Magnet
|
||||
Mag-Net
|
||||
magpie-crawler
|
||||
Mail.ru
|
||||
Majestic12
|
||||
MarkMonitor
|
||||
MarkWatch
|
||||
Masscan
|
||||
Mass Downloader
|
||||
Mata Hari
|
||||
Meanpathbot
|
||||
MegaIndex.ru
|
||||
Metauri
|
||||
MFC_Tear_Sample
|
||||
Microsoft Data Access
|
||||
Microsoft URL Control
|
||||
MIDown tool
|
||||
MIIxpc
|
||||
Mister PiX
|
||||
MJ12bot
|
||||
Mojeek
|
||||
Morfeus Fucking Scanner
|
||||
MSFrontPage
|
||||
MSIE 6.0
|
||||
MSIECrawler
|
||||
Msrabot
|
||||
MS Web Services Client Protocol
|
||||
Musobot
|
||||
Name Intelligence
|
||||
Nameprotect
|
||||
Navroad
|
||||
NearSite
|
||||
Needle
|
||||
Nessus
|
||||
NetAnts
|
||||
Netcraft
|
||||
netEstate NE Crawler
|
||||
NetLyzer
|
||||
NetMechanic
|
||||
NetSpider
|
||||
Nettrack
|
||||
Net Vampire
|
||||
NetZIP
|
||||
NextGenSearchBot
|
||||
Nibbler
|
||||
NICErsPRO
|
||||
Niki-bot
|
||||
Nikto
|
||||
NimbleCrawler
|
||||
Ninja
|
||||
Nmap
|
||||
NPbot
|
||||
Nutch
|
||||
Octopus
|
||||
Offline Explorer
|
||||
Offline Navigator
|
||||
Openfind
|
||||
OpenLinkProfiler
|
||||
Openvas
|
||||
OrangeBot
|
||||
OrangeSpider
|
||||
OutfoxBot
|
||||
PageAnalyzer
|
||||
Page Analyzer
|
||||
PageGrabber
|
||||
Page Grabber
|
||||
page scorer
|
||||
PageScorer
|
||||
Panscient
|
||||
Papa Foto
|
||||
Pavuk
|
||||
pcBrowser
|
||||
PECL::HTTP
|
||||
PeoplePal
|
||||
PHPCrawl
|
||||
Picscout
|
||||
Picsearch
|
||||
PictureFinder
|
||||
Pimonster
|
||||
Pi-Monster
|
||||
Pixray
|
||||
PleaseCrawl
|
||||
Pockey
|
||||
POE-Component-Client-HTTP
|
||||
Probethenet
|
||||
ProPowerBot
|
||||
ProWebWalker
|
||||
Proximic
|
||||
Psbot
|
||||
Pump
|
||||
PyCurl
|
||||
QueryN Metasearch
|
||||
Qwantify
|
||||
RankActive
|
||||
RankActiveLinkBot
|
||||
RankFlex
|
||||
RankingBot
|
||||
RankingBot2
|
||||
Rankivabot
|
||||
RankurBot
|
||||
RealDownload
|
||||
Reaper
|
||||
RebelMouse
|
||||
Recorder
|
||||
RedesScrapy
|
||||
ReGet
|
||||
RepoMonkey
|
||||
Ripper
|
||||
RocketCrawler
|
||||
Rogerbot
|
||||
SalesIntelligent
|
||||
SBIder
|
||||
ScanAlert
|
||||
Scanbot
|
||||
Scrapy
|
||||
Screaming
|
||||
Screaming Frog SEO Spider
|
||||
ScreenerBot
|
||||
Searchestate
|
||||
SearchmetricsBot
|
||||
Semrush
|
||||
SemrushBot
|
||||
SEOkicks
|
||||
SEOkicks-Robot
|
||||
SEOlyticsCrawler
|
||||
Seomoz
|
||||
SEOprofiler
|
||||
seoscanners
|
||||
SEOstats
|
||||
Siphon
|
||||
SISTRIX
|
||||
SISTRIX Crawler
|
||||
Sitebeam
|
||||
SiteExplorer
|
||||
Siteimprove
|
||||
SiteLockSpider
|
||||
SiteSnagger
|
||||
SiteSucker
|
||||
Site Sucker
|
||||
Sitevigil
|
||||
Slackbot-LinkExpanding
|
||||
SlySearch
|
||||
SmartDownload
|
||||
Snake
|
||||
Snapbot
|
||||
Snoopy
|
||||
SocialRankIOBot
|
||||
Sogou web spider
|
||||
Sosospider
|
||||
SpaceBison
|
||||
Spammen
|
||||
SpankBot
|
||||
Spanner
|
||||
Spbot
|
||||
Spinn3r
|
||||
SputnikBot
|
||||
Sqlmap
|
||||
Sqlworm
|
||||
Sqworm
|
||||
Steeler
|
||||
Stripper
|
||||
Sucker
|
||||
Sucuri
|
||||
SuperBot
|
||||
SuperHTTP
|
||||
Surfbot
|
||||
SurveyBot
|
||||
Suzuran
|
||||
Swiftbot
|
||||
Szukacz
|
||||
T0PHackTeam
|
||||
T8Abot
|
||||
tAkeOut
|
||||
Teleport
|
||||
TeleportPro
|
||||
Telesoft
|
||||
Telesphoreo
|
||||
Telesphorep
|
||||
The Intraformant
|
||||
TheNomad
|
||||
TightTwatBot
|
||||
Titan
|
||||
Toata
|
||||
Toweyabot
|
||||
Trendictionbot
|
||||
True_Robot
|
||||
Turingos
|
||||
TurnitinBot
|
||||
Turnitin Bot
|
||||
Turnitin Robot
|
||||
TwengaBot
|
||||
Twice
|
||||
Typhoeus
|
||||
UnisterBot
|
||||
URLy Warning
|
||||
URLy.Warning
|
||||
Vacuum
|
||||
Vagabondo
|
||||
VB Project
|
||||
VCI
|
||||
VeriCiteCrawler
|
||||
VidibleScraper
|
||||
Virusdie
|
||||
VoidEYE
|
||||
Voil
|
||||
Voltron
|
||||
Wallpapers/3.0
|
||||
WallpapersHD
|
||||
WASALive-Bot
|
||||
WBSearchBot
|
||||
Webalta
|
||||
WebAuto
|
||||
Web Auto
|
||||
WebBandit
|
||||
Web Bandit
|
||||
WebCollage
|
||||
Web Collage
|
||||
WebCopier
|
||||
Web Copier
|
||||
WEBDAV
|
||||
WEBDAV Client
|
||||
WebEnhancer
|
||||
Web Enhancer
|
||||
WebFetch
|
||||
Web Fetch
|
||||
WebFuck
|
||||
Web Fuck
|
||||
WebGo IS
|
||||
WebImageCollector
|
||||
Web Image Collector
|
||||
WebLeacher
|
||||
WebmasterWorldForumBot
|
||||
webmeup-crawler
|
||||
WebPix
|
||||
Web Pix
|
||||
WebReaper
|
||||
Web Reaper
|
||||
WebSauger
|
||||
Web Sauger
|
||||
Webshag
|
||||
WebsiteExtractor
|
||||
Website Extractor
|
||||
WebsiteQuester
|
||||
Website Quester
|
||||
Webster
|
||||
WebStripper
|
||||
Web Stripper
|
||||
WebSucker
|
||||
Web Sucker
|
||||
WebWhacker
|
||||
Web Whacker
|
||||
WebZIP
|
||||
WeSEE
|
||||
Whack
|
||||
Whacker
|
||||
Whatweb
|
||||
Widow
|
||||
WinHTTrack
|
||||
WiseGuys Robot
|
||||
WISENutbot
|
||||
Wonderbot
|
||||
Woobot
|
||||
Wotbox
|
||||
Wprecon
|
||||
WPScan
|
||||
WWW-Collector-E
|
||||
WWW-Mechanize
|
||||
WWW::Mechanize
|
||||
WWWOFFLE
|
||||
x22Mozilla
|
||||
Xaldon WebSpider
|
||||
Xaldon_WebSpider
|
||||
Xenu
|
||||
YoudaoBot
|
||||
Zade
|
||||
Zermelo
|
||||
Zeus
|
||||
Zgrab
|
||||
Zitebot
|
||||
ZmEu
|
||||
ZumBot
|
||||
ZyBorg
|
||||
|
|
@ -4,8 +4,8 @@
|
|||
|
||||
### VERSION INFORMATION #
|
||||
###################################################
|
||||
### Version: V3.2017.07.719
|
||||
### Updated: Fri Jul 21 14:45:45 SAST 2017
|
||||
### Version: V3.2017.07.720
|
||||
### Updated: Fri Jul 21 14:58:13 SAST 2017
|
||||
### Bad Referrer Count: 5436
|
||||
### Bad Bot Count: 498
|
||||
###################################################
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
|
||||
### Version Information #
|
||||
********************************************
|
||||
#### Version: V3.2017.07.719
|
||||
#### Version: V3.2017.07.720
|
||||
#### Bad Referrer Count: 5436
|
||||
#### Bad Bot Count: 498
|
||||
********************************************
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
|
||||
### Version Information #
|
||||
********************************************
|
||||
#### Version: V3.2017.07.719
|
||||
#### Version: V3.2017.07.720
|
||||
#### Bad Referrer Count: 5436
|
||||
#### Bad Bot Count: 498
|
||||
********************************************
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
|
||||
### Version Information #
|
||||
********************************************
|
||||
#### Version: V3.2017.07.719
|
||||
#### Version: V3.2017.07.720
|
||||
#### Bad Referrer Count: 5436
|
||||
#### Bad Bot Count: 498
|
||||
********************************************
|
||||
|
|
|
|||
|
|
@ -4,8 +4,8 @@
|
|||
|
||||
### VERSION INFORMATION #
|
||||
###################################################
|
||||
### Version: V3.2017.07.719
|
||||
### Updated: Fri Jul 21 14:45:45 SAST 2017
|
||||
### Version: V3.2017.07.720
|
||||
### Updated: Fri Jul 21 14:58:13 SAST 2017
|
||||
### Bad Referrer Count: 5436
|
||||
### Bad Bot Count: 498
|
||||
###################################################
|
||||
|
|
|
|||
|
|
@ -4,8 +4,8 @@
|
|||
|
||||
### VERSION INFORMATION #
|
||||
###################################################
|
||||
### Version: V3.2017.07.719
|
||||
### Updated: Fri Jul 21 14:45:45 SAST 2017
|
||||
### Version: V3.2017.07.720
|
||||
### Updated: Fri Jul 21 14:58:13 SAST 2017
|
||||
### Bad Referrer Count: 5436
|
||||
### Bad Bot Count: 498
|
||||
###################################################
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
### Version Information #
|
||||
###################################################
|
||||
### Version: V3.2017.07.719
|
||||
### Updated: Fri Jul 21 14:45:46 SAST 2017
|
||||
### Version: V3.2017.07.720
|
||||
### Updated: Fri Jul 21 14:58:13 SAST 2017
|
||||
### Bad Bot Count: 498
|
||||
###################################################
|
||||
### Version Information ##
|
||||
|
|
@ -65,7 +65,7 @@ User-agent: Bandit
|
|||
Disallow:/
|
||||
User-agent: BatchFTP
|
||||
Disallow:/
|
||||
User-agent: Battleztar\ Bazinga
|
||||
User-agent: Battleztar Bazinga
|
||||
Disallow:/
|
||||
User-agent: BBBike
|
||||
Disallow:/
|
||||
|
|
@ -79,7 +79,7 @@ User-agent: Bitacle
|
|||
Disallow:/
|
||||
User-agent: Blackboard
|
||||
Disallow:/
|
||||
User-agent: Black\ Hole
|
||||
User-agent: Black Hole
|
||||
Disallow:/
|
||||
User-agent: BlackWidow
|
||||
Disallow:/
|
||||
|
|
@ -195,17 +195,17 @@ User-agent: DomainSigmaCrawler
|
|||
Disallow:/
|
||||
User-agent: Dotbot
|
||||
Disallow:/
|
||||
User-agent: Download\ Demon
|
||||
User-agent: Download Demon
|
||||
Disallow:/
|
||||
User-agent: Download\ Devil
|
||||
User-agent: Download Devil
|
||||
Disallow:/
|
||||
User-agent: Download\ Wonder
|
||||
User-agent: Download Wonder
|
||||
Disallow:/
|
||||
User-agent: Dragonfly
|
||||
Disallow:/
|
||||
User-agent: Drip
|
||||
Disallow:/
|
||||
User-agent: DTS\ Agent
|
||||
User-agent: DTS Agent
|
||||
Disallow:/
|
||||
User-agent: EasyDL
|
||||
Disallow:/
|
||||
|
|
@ -219,13 +219,13 @@ User-agent: Ecxi
|
|||
Disallow:/
|
||||
User-agent: EirGrabber
|
||||
Disallow:/
|
||||
User-agent: EMail\ Collector
|
||||
User-agent: EMail Collector
|
||||
Disallow:/
|
||||
User-agent: EMail\ Extractor
|
||||
User-agent: EMail Extractor
|
||||
Disallow:/
|
||||
User-agent: EMail\ Siphon
|
||||
User-agent: EMail Siphon
|
||||
Disallow:/
|
||||
User-agent: EMail\ Wolf
|
||||
User-agent: EMail Wolf
|
||||
Disallow:/
|
||||
User-agent: EroCrawler
|
||||
Disallow:/
|
||||
|
|
@ -233,13 +233,13 @@ User-agent: Evil
|
|||
Disallow:/
|
||||
User-agent: Exabot
|
||||
Disallow:/
|
||||
User-agent: Express\ WebPictures
|
||||
User-agent: Express WebPictures
|
||||
Disallow:/
|
||||
User-agent: Extractor
|
||||
Disallow:/
|
||||
User-agent: ExtractorPro
|
||||
Disallow:/
|
||||
User-agent: Extreme\ Picture\ Finder
|
||||
User-agent: Extreme Picture Finder
|
||||
Disallow:/
|
||||
User-agent: EyeNetIE
|
||||
Disallow:/
|
||||
|
|
@ -335,13 +335,13 @@ User-agent: Id-search
|
|||
Disallow:/
|
||||
User-agent: IlseBot
|
||||
Disallow:/
|
||||
User-agent: Image\ Fetch
|
||||
User-agent: Image Fetch
|
||||
Disallow:/
|
||||
User-agent: Image\ Stripper
|
||||
User-agent: Image Stripper
|
||||
Disallow:/
|
||||
User-agent: Image\ Sucker
|
||||
User-agent: Image Sucker
|
||||
Disallow:/
|
||||
User-agent: Indy\ Library
|
||||
User-agent: Indy Library
|
||||
Disallow:/
|
||||
User-agent: InfoNaviRobot
|
||||
Disallow:/
|
||||
|
|
@ -351,11 +351,11 @@ User-agent: Intelliseek
|
|||
Disallow:/
|
||||
User-agent: InterGET
|
||||
Disallow:/
|
||||
User-agent: Internet\ Ninja
|
||||
User-agent: Internet Ninja
|
||||
Disallow:/
|
||||
User-agent: InternetSeer
|
||||
Disallow:/
|
||||
User-agent: internetVista\ monitor
|
||||
User-agent: internetVista monitor
|
||||
Disallow:/
|
||||
User-agent: Iria
|
||||
Disallow:/
|
||||
|
|
@ -373,7 +373,7 @@ User-agent: JetCar
|
|||
Disallow:/
|
||||
User-agent: JikeSpider
|
||||
Disallow:/
|
||||
User-agent: JOC\ Web\ Spider
|
||||
User-agent: JOC Web Spider
|
||||
Disallow:/
|
||||
User-agent: Joomla
|
||||
Disallow:/
|
||||
|
|
@ -383,9 +383,9 @@ User-agent: JustView
|
|||
Disallow:/
|
||||
User-agent: Jyxobot
|
||||
Disallow:/
|
||||
User-agent: Kenjin\ Spider
|
||||
User-agent: Kenjin Spider
|
||||
Disallow:/
|
||||
User-agent: Keyword\ Density
|
||||
User-agent: Keyword Density
|
||||
Disallow:/
|
||||
User-agent: Lanshanbot
|
||||
Disallow:/
|
||||
|
|
@ -457,9 +457,9 @@ User-agent: MarkWatch
|
|||
Disallow:/
|
||||
User-agent: Masscan
|
||||
Disallow:/
|
||||
User-agent: Mass\ Downloader
|
||||
User-agent: Mass Downloader
|
||||
Disallow:/
|
||||
User-agent: Mata\ Hari
|
||||
User-agent: Mata Hari
|
||||
Disallow:/
|
||||
User-agent: Meanpathbot
|
||||
Disallow:/
|
||||
|
|
@ -469,35 +469,35 @@ User-agent: Metauri
|
|||
Disallow:/
|
||||
User-agent: MFC_Tear_Sample
|
||||
Disallow:/
|
||||
User-agent: Microsoft\ Data\ Access
|
||||
User-agent: Microsoft Data Access
|
||||
Disallow:/
|
||||
User-agent: Microsoft\ URL\ Control
|
||||
User-agent: Microsoft URL Control
|
||||
Disallow:/
|
||||
User-agent: MIDown\ tool
|
||||
User-agent: MIDown tool
|
||||
Disallow:/
|
||||
User-agent: MIIxpc
|
||||
Disallow:/
|
||||
User-agent: Mister\ PiX
|
||||
User-agent: Mister PiX
|
||||
Disallow:/
|
||||
User-agent: MJ12bot
|
||||
Disallow:/
|
||||
User-agent: Mojeek
|
||||
Disallow:/
|
||||
User-agent: Morfeus\ Fucking\ Scanner
|
||||
User-agent: Morfeus Fucking Scanner
|
||||
Disallow:/
|
||||
User-agent: MSFrontPage
|
||||
Disallow:/
|
||||
User-agent: MSIE\ 6.0
|
||||
User-agent: MSIE 6.0
|
||||
Disallow:/
|
||||
User-agent: MSIECrawler
|
||||
Disallow:/
|
||||
User-agent: Msrabot
|
||||
Disallow:/
|
||||
User-agent: MS\ Web\ Services\ Client\ Protocol
|
||||
User-agent: MS Web Services Client Protocol
|
||||
Disallow:/
|
||||
User-agent: Musobot
|
||||
Disallow:/
|
||||
User-agent: Name\ Intelligence
|
||||
User-agent: Name Intelligence
|
||||
Disallow:/
|
||||
User-agent: Nameprotect
|
||||
Disallow:/
|
||||
|
|
@ -513,7 +513,7 @@ User-agent: NetAnts
|
|||
Disallow:/
|
||||
User-agent: Netcraft
|
||||
Disallow:/
|
||||
User-agent: netEstate\ NE\ Crawler
|
||||
User-agent: netEstate NE Crawler
|
||||
Disallow:/
|
||||
User-agent: NetLyzer
|
||||
Disallow:/
|
||||
|
|
@ -523,7 +523,7 @@ User-agent: NetSpider
|
|||
Disallow:/
|
||||
User-agent: Nettrack
|
||||
Disallow:/
|
||||
User-agent: Net\ Vampire
|
||||
User-agent: Net Vampire
|
||||
Disallow:/
|
||||
User-agent: NetZIP
|
||||
Disallow:/
|
||||
|
|
@ -549,9 +549,9 @@ User-agent: Nutch
|
|||
Disallow:/
|
||||
User-agent: Octopus
|
||||
Disallow:/
|
||||
User-agent: Offline\ Explorer
|
||||
User-agent: Offline Explorer
|
||||
Disallow:/
|
||||
User-agent: Offline\ Navigator
|
||||
User-agent: Offline Navigator
|
||||
Disallow:/
|
||||
User-agent: Openfind
|
||||
Disallow:/
|
||||
|
|
@ -567,19 +567,19 @@ User-agent: OutfoxBot
|
|||
Disallow:/
|
||||
User-agent: PageAnalyzer
|
||||
Disallow:/
|
||||
User-agent: Page\ Analyzer
|
||||
User-agent: Page Analyzer
|
||||
Disallow:/
|
||||
User-agent: PageGrabber
|
||||
Disallow:/
|
||||
User-agent: Page\ Grabber
|
||||
User-agent: Page Grabber
|
||||
Disallow:/
|
||||
User-agent: page\ scorer
|
||||
User-agent: page scorer
|
||||
Disallow:/
|
||||
User-agent: PageScorer
|
||||
Disallow:/
|
||||
User-agent: Panscient
|
||||
Disallow:/
|
||||
User-agent: Papa\ Foto
|
||||
User-agent: Papa Foto
|
||||
Disallow:/
|
||||
User-agent: Pavuk
|
||||
Disallow:/
|
||||
|
|
@ -623,7 +623,7 @@ User-agent: Pump
|
|||
Disallow:/
|
||||
User-agent: PyCurl
|
||||
Disallow:/
|
||||
User-agent: QueryN\ Metasearch
|
||||
User-agent: QueryN Metasearch
|
||||
Disallow:/
|
||||
User-agent: Qwantify
|
||||
Disallow:/
|
||||
|
|
@ -673,7 +673,7 @@ User-agent: Scrapy
|
|||
Disallow:/
|
||||
User-agent: Screaming
|
||||
Disallow:/
|
||||
User-agent: Screaming\ Frog\ SEO\ Spider
|
||||
User-agent: Screaming Frog SEO Spider
|
||||
Disallow:/
|
||||
User-agent: ScreenerBot
|
||||
Disallow:/
|
||||
|
|
@ -703,7 +703,7 @@ User-agent: Siphon
|
|||
Disallow:/
|
||||
User-agent: SISTRIX
|
||||
Disallow:/
|
||||
User-agent: SISTRIX\ Crawler
|
||||
User-agent: SISTRIX Crawler
|
||||
Disallow:/
|
||||
User-agent: Sitebeam
|
||||
Disallow:/
|
||||
|
|
@ -717,7 +717,7 @@ User-agent: SiteSnagger
|
|||
Disallow:/
|
||||
User-agent: SiteSucker
|
||||
Disallow:/
|
||||
User-agent: Site\ Sucker
|
||||
User-agent: Site Sucker
|
||||
Disallow:/
|
||||
User-agent: Sitevigil
|
||||
Disallow:/
|
||||
|
|
@ -735,7 +735,7 @@ User-agent: Snoopy
|
|||
Disallow:/
|
||||
User-agent: SocialRankIOBot
|
||||
Disallow:/
|
||||
User-agent: Sogou\ web\ spider
|
||||
User-agent: Sogou web spider
|
||||
Disallow:/
|
||||
User-agent: Sosospider
|
||||
Disallow:/
|
||||
|
|
@ -797,7 +797,7 @@ User-agent: Telesphoreo
|
|||
Disallow:/
|
||||
User-agent: Telesphorep
|
||||
Disallow:/
|
||||
User-agent: The\ Intraformant
|
||||
User-agent: The Intraformant
|
||||
Disallow:/
|
||||
User-agent: TheNomad
|
||||
Disallow:/
|
||||
|
|
@ -817,9 +817,9 @@ User-agent: Turingos
|
|||
Disallow:/
|
||||
User-agent: TurnitinBot
|
||||
Disallow:/
|
||||
User-agent: Turnitin\ Bot
|
||||
User-agent: Turnitin Bot
|
||||
Disallow:/
|
||||
User-agent: Turnitin\ Robot
|
||||
User-agent: Turnitin Robot
|
||||
Disallow:/
|
||||
User-agent: TwengaBot
|
||||
Disallow:/
|
||||
|
|
@ -829,15 +829,15 @@ User-agent: Typhoeus
|
|||
Disallow:/
|
||||
User-agent: UnisterBot
|
||||
Disallow:/
|
||||
User-agent: URLy.Warning
|
||||
User-agent: URLy Warning
|
||||
Disallow:/
|
||||
User-agent: URLy\ Warning
|
||||
User-agent: URLy.Warning
|
||||
Disallow:/
|
||||
User-agent: Vacuum
|
||||
Disallow:/
|
||||
User-agent: Vagabondo
|
||||
Disallow:/
|
||||
User-agent: VB\ Project
|
||||
User-agent: VB Project
|
||||
Disallow:/
|
||||
User-agent: VCI
|
||||
Disallow:/
|
||||
|
|
@ -865,41 +865,41 @@ User-agent: Webalta
|
|||
Disallow:/
|
||||
User-agent: WebAuto
|
||||
Disallow:/
|
||||
User-agent: Web\ Auto
|
||||
User-agent: Web Auto
|
||||
Disallow:/
|
||||
User-agent: WebBandit
|
||||
Disallow:/
|
||||
User-agent: Web\ Bandit
|
||||
User-agent: Web Bandit
|
||||
Disallow:/
|
||||
User-agent: WebCollage
|
||||
Disallow:/
|
||||
User-agent: Web\ Collage
|
||||
User-agent: Web Collage
|
||||
Disallow:/
|
||||
User-agent: WebCopier
|
||||
Disallow:/
|
||||
User-agent: Web\ Copier
|
||||
User-agent: Web Copier
|
||||
Disallow:/
|
||||
User-agent: WEBDAV
|
||||
Disallow:/
|
||||
User-agent: WEBDAV\ Client
|
||||
User-agent: WEBDAV Client
|
||||
Disallow:/
|
||||
User-agent: WebEnhancer
|
||||
Disallow:/
|
||||
User-agent: Web\ Enhancer
|
||||
User-agent: Web Enhancer
|
||||
Disallow:/
|
||||
User-agent: WebFetch
|
||||
Disallow:/
|
||||
User-agent: Web\ Fetch
|
||||
User-agent: Web Fetch
|
||||
Disallow:/
|
||||
User-agent: WebFuck
|
||||
Disallow:/
|
||||
User-agent: Web\ Fuck
|
||||
User-agent: Web Fuck
|
||||
Disallow:/
|
||||
User-agent: WebGo\ IS
|
||||
User-agent: WebGo IS
|
||||
Disallow:/
|
||||
User-agent: WebImageCollector
|
||||
Disallow:/
|
||||
User-agent: Web\ Image\ Collector
|
||||
User-agent: Web Image Collector
|
||||
Disallow:/
|
||||
User-agent: WebLeacher
|
||||
Disallow:/
|
||||
|
|
@ -909,39 +909,39 @@ User-agent: webmeup-crawler
|
|||
Disallow:/
|
||||
User-agent: WebPix
|
||||
Disallow:/
|
||||
User-agent: Web\ Pix
|
||||
User-agent: Web Pix
|
||||
Disallow:/
|
||||
User-agent: WebReaper
|
||||
Disallow:/
|
||||
User-agent: Web\ Reaper
|
||||
User-agent: Web Reaper
|
||||
Disallow:/
|
||||
User-agent: WebSauger
|
||||
Disallow:/
|
||||
User-agent: Web\ Sauger
|
||||
User-agent: Web Sauger
|
||||
Disallow:/
|
||||
User-agent: Webshag
|
||||
Disallow:/
|
||||
User-agent: WebsiteExtractor
|
||||
Disallow:/
|
||||
User-agent: Website\ Extractor
|
||||
User-agent: Website Extractor
|
||||
Disallow:/
|
||||
User-agent: WebsiteQuester
|
||||
Disallow:/
|
||||
User-agent: Website\ Quester
|
||||
User-agent: Website Quester
|
||||
Disallow:/
|
||||
User-agent: Webster
|
||||
Disallow:/
|
||||
User-agent: WebStripper
|
||||
Disallow:/
|
||||
User-agent: Web\ Stripper
|
||||
User-agent: Web Stripper
|
||||
Disallow:/
|
||||
User-agent: WebSucker
|
||||
Disallow:/
|
||||
User-agent: Web\ Sucker
|
||||
User-agent: Web Sucker
|
||||
Disallow:/
|
||||
User-agent: WebWhacker
|
||||
Disallow:/
|
||||
User-agent: Web\ Whacker
|
||||
User-agent: Web Whacker
|
||||
Disallow:/
|
||||
User-agent: WebZIP
|
||||
Disallow:/
|
||||
|
|
@ -957,7 +957,7 @@ User-agent: Widow
|
|||
Disallow:/
|
||||
User-agent: WinHTTrack
|
||||
Disallow:/
|
||||
User-agent: WiseGuys\ Robot
|
||||
User-agent: WiseGuys Robot
|
||||
Disallow:/
|
||||
User-agent: WISENutbot
|
||||
Disallow:/
|
||||
|
|
@ -981,9 +981,9 @@ User-agent: WWWOFFLE
|
|||
Disallow:/
|
||||
User-agent: x22Mozilla
|
||||
Disallow:/
|
||||
User-agent: Xaldon_WebSpider
|
||||
User-agent: Xaldon WebSpider
|
||||
Disallow:/
|
||||
User-agent: Xaldon\ WebSpider
|
||||
User-agent: Xaldon_WebSpider
|
||||
Disallow:/
|
||||
User-agent: Xenu
|
||||
Disallow:/
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue