vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-09 22:05:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.07.2451 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-07-05 11:05:03 +00:00
parent 606872502d
commit 41967724e3
77 changed files with 123172 additions and 122224 deletions
Download patch file Download diff file Expand all files Collapse all files

368
dev-tools/_robots_input/robots-input.txt
View file

@ -3,19 +3,20 @@
404checker
404enemy
80legs
ADmantX
AIBOT
ASPSeek
Abonti
Aboundex
Aboundexbot
Acunetix
ADmantX
AfD-Verbotsverfahren
AhrefsBot
AIBOT
AiHitBot
Aipbot
Alexibot
Alligator
AllSubmitter
Alligator
AlphaBot
Anarchie
Anarchy
@ -23,37 +24,31 @@ Anarchy99
Ankit
Anthill
Apexoo
archive.org_bot
arquivo.pt
arquivo-web-crawler
Aspiegel
ASPSeek
Asterias
Attach
autoemailspider
AwarioRssBot
AwarioSmartBot
BBBike
BDCbot
BDFetch
BLEXBot
BackDoorBot
Backlink-Ceck
backlink-check
BacklinkCrawler
BackStreet
BackWeb
Backlink-Ceck
BacklinkCrawler
Badass
Bandit
Barkrowler
BatchFTP
Battleztar Bazinga
BBBike
BDCbot
BDFetch
BetaBot
Bigfoot
Bitacle
Blackboard
Black Hole
BlackWidow
BLEXBot
Black Hole
Blackboard
Blow
BlowFish
Boardreader
@ -68,53 +63,47 @@ BuiltWith
Bullseye
BunnySlippers
BuzzSumo
Calculon
CATExplorador
CazoodleBot
CCBot
CODE87
CSHttp
Calculon
CazoodleBot
Cegbfeieh
CensysInspect
check1.exe
CheTeam
CheeseBot
CherryPicker
CheTeam
ChinaClaw
Chlooe
Claritybot
Cliqzbot
Cloud mapping
coccocbot-web
Cocolyzebot
CODE87
Cogentbot
cognitiveseo
Collector
com.plumanalytics
Copier
CopyRightCheck
Copyscape
Cosmos
Craftbot
crawler4j
crawler.feedback
crawl.sogou.com
CrazyWebCrawler
Crescent
CrunchBot
CSHttp
Curious
Custo
CyotekWebCopy
DatabaseDriverMysqli
DataCha0s
DBLBot
demandbase-bot
DIIbot
DSearch
DTS Agent
DataCha0s
DatabaseDriverMysqli
Demon
Deusu
Devil
Digincore
DigitalPebble
DIIbot
Dirbuster
Disco
Discobot
@ -122,29 +111,24 @@ Discoverybot
Dispatch
DittoSpyder
DnyzBot
DomCopBot
DomainAppender
DomainCrawler
DomainSigmaCrawler
Domains Project
domainsproject.org
DomainStatsBot
DomCopBot
Domains Project
Dotbot
Download Wonder
Dragonfly
Drip
DSearch
DTS Agent
EasyDL
Ebingbong
eCatch
ECCP/1.0
Ecxi
EirGrabber
EMail Siphon
EMail Wolf
EasyDL
Ebingbong
Ecxi
EirGrabber
EroCrawler
evc-batch
Evil
Exabot
Express WebPictures
@ -154,10 +138,9 @@ ExtractorPro
Extreme Picture Finder
EyeNetIE
Ezooms
facebookscraper
FDM
FemtosearchBot
FHscan
FemtosearchBot
Fimap
Firefox/7.0
FlashGet
@ -168,44 +151,43 @@ FrontPage
Fuzz
FyberSpider
Fyrebot
G-i-g-a-b-o-t
GT::WWW
GalaxyBot
Genieo
GermCrawler
Getintent
GetRight
GetWeb
Getintent
Gigablast
Gigabot
G-i-g-a-b-o-t
Go-Ahead-Got-It
gopher
Gotit
GoZilla
Go!Zilla
Grabber
Go-Ahead-Got-It
GoZilla
Gotit
GrabNet
Grabber
Grafula
GrapeFX
GrapeshotCrawler
GridBot
GT::WWW
Haansoft
HaosouSpider
Harvest
Havij
HEADMasterSEO
heritrix
Heritrix
Hloader
HMView
HTMLparser
HTTP::Lite
HTTrack
Haansoft
HaosouSpider
Harvest
Havij
Heritrix
Hloader
Humanlinks
HybridBot
Iblog
IDBot
IDBTE4M
IDBot
IRLbot
Iblog
Id-search
IlseBot
Image Fetch
@ -214,25 +196,20 @@ IndeedBot
Indy Library
InfoNaviRobot
InfoTekies
instabid
Intelliseek
InterGET
Internet Ninja
InternetSeer
internetVista monitor
ips-agent
Internet Ninja
Iria
IRLbot
isitwp.com
Iskanie
IstellaBot
JOC Web Spider
JamesBOT
Jbrofuzz
JennyBot
JetCar
Jetty
JikeSpider
JOC Web Spider
Joomla
Jorgee
JustView
@ -241,6 +218,8 @@ Kenjin Spider
Keyword Density
Kinza
Kozmosbot
LNSpiderguy
LWP::Simple
Lanshanbot
Larbin
LeechFTP
@ -252,13 +231,13 @@ Libwhisker
LieBaoFast
Lightspeedsystems
Likse
LinkScan
LinkWalker
Linkbot
Linkdexbot
LinkextractorPro
LinkpadBot
LinkScan
LinksManager
LinkWalker
LinqiaMetadataDownloaderBot
LinqiaRSSBot
LinqiaScrapeBot
@ -266,53 +245,47 @@ Lipperhey
Lipperhey Spider
Litemage_walker
Lmspider
LNSpiderguy
Ltx71
lwp-request
LWP::Simple
lwp-trivial
Magnet
MFC_Tear_Sample
MIDown tool
MIIxpc
MJ12bot
MQQBrowser
MSFrontPage
MSIECrawler
MTRobot
Mag-Net
magpie-crawler
Magnet
Mail.RU_Bot
Majestic12
Majestic-SEO
Majestic12
Majestic SEO
MarkMonitor
MarkWatch
Masscan
Mass Downloader
Masscan
Mata Hari
MauiBot
Mb2345Browser
meanpathbot
Meanpathbot
MeanPath Bot
Meanpathbot
Mediatoolkitbot
mediawords
MegaIndex.ru
Metauri
MFC_Tear_Sample
MicroMessenger
Microsoft Data Access
Microsoft URL Control
MIDown tool
MIIxpc
Mister PiX
MJ12bot
Moblie Safari
Mojeek
Mojolicious
Morfeus Fucking Scanner
Mozlila
MQQBrowser
Mr.4x3
MSFrontPage
MSIECrawler
Msrabot
MTRobot
muhstik-scan
Musobot
NICErsPRO
NPbot
Name Intelligence
Nameprotect
Navroad
@ -320,72 +293,63 @@ NearSite
Needle
Nessus
NetAnts
Netcraft
netEstate NE Crawler
NetLyzer
NetMechanic
NetSpider
Nettrack
Net Vampire
Netvibes
NetZIP
Net Vampire
Netcraft
Nettrack
Netvibes
NextGenSearchBot
Nibbler
NICErsPRO
Niki-bot
Nikto
NimbleCrawler
Nimbostratus
Ninja
Nmap
NPbot
Nuclei
Nutch
oBot
OPPO A33
Octopus
Offline Explorer
Offline Navigator
OnCrawl
Openfind
OpenLinkProfiler
Openvas
OpenVAS
OPPO A33
Openfind
Openvas
OrangeBot
OrangeSpider
OutclicksBot
OutfoxBot
PECL::HTTP
PHPCrawl
POE-Component-Client-HTTP
PageAnalyzer
Page Analyzer
PageGrabber
page scorer
PageScorer
PageThing.com
Page Analyzer
Pandalytics
Panscient
Papa Foto
Pavuk
pcBrowser
PECL::HTTP
PeoplePal
Petalbot
PHPCrawl
Pi-Monster
Picscout
Picsearch
PictureFinder
Piepmatz
Pimonster
Pi-Monster
Pixray
PleaseCrawl
plumanalytics
Pockey
POE-Component-Client-HTTP
polaris version
probe-image-size
Probethenet
ProPowerBot
ProWebWalker
Probethenet
Psbot
Pu_iN
Pump
@ -393,6 +357,7 @@ PxBroker
PyCurl
QueryN Metasearch
Quick-Crawler
RSSingBot
RankActive
RankActiveLinkBot
RankFlex
@ -400,27 +365,28 @@ RankingBot
RankingBot2
Rankivabot
RankurBot
Re-re
ReGet
RealDownload
Reaper
RebelMouse
Recorder
RedesScrapy
ReGet
RepoMonkey
Re-re
Ripper
ripz
RocketCrawler
Rogerbot
RSSingBot
s1z.ru
SalesIntelligent
satoristudio.net
SBIder
scalaj-http
SEOkicks
SEOkicks-Robot
SEOlyticsCrawler
SEOprofiler
SEOstats
SISTRIX
SMTBot
SalesIntelligent
ScanAlert
Scanbot
scan.lol
ScoutJet
Scrapy
Screaming
@ -431,42 +397,27 @@ SearchmetricsBot
Semrush
SemrushBot
SentiBot
seobility
SeobilityBot
SEOkicks
SEOkicks-Robot
SEOlyticsCrawler
Seomoz
SEOprofiler
seoscanners
SeoSiteCheckup
seostar
SEOstats
serpstatbot
sexsearcher
SeobilityBot
Seomoz
Shodan
Siphon
SISTRIX
Sitebeam
SiteCheckerBotCrawler
sitechecker.pro
SiteExplorer
Siteimprove
SiteLockSpider
siteripz
SiteSnagger
SiteSucker
Site Sucker
Sitebeam
Siteimprove
Sitevigil
SlySearch
SmartDownload
SMTBot
Snake
Snapbot
Snoopy
SocialRankIOBot
Sociscraper
sogouspider
Sogou web spider
Sosospider
Sottopop
@ -474,11 +425,9 @@ SpaceBison
Spammen
SpankBot
Spanner
sp_auditbot
Spbot
Spinn3r
SputnikBot
spyfu
Sqlmap
Sqlworm
Sqworm
@ -492,18 +441,16 @@ Surfbot
SurveyBot
Suzuran
Swiftbot
sysscan
Szukacz
T0PHackTeam
T8Abot
tAkeOut
Teleport
TeleportPro
Telesoft
Telesphoreo
Telesphorep
The Intraformant
TheNomad
The Intraformant
Thumbor
TightTwatBot
Titan
@ -512,8 +459,6 @@ Toweyabot
Tracemyfile
Trendiction
Trendictionbot
trendiction.com
trendiction.de
True_Robot
Turingos
Turnitin
@ -521,15 +466,15 @@ TurnitinBot
TwengaBot
Twice
Typhoeus
UnisterBot
Upflow
URLy.Warning
URLy Warning
Vacuum
Vagabondo
UnisterBot
Upflow
V-BOT
VB Project
VCI
Vacuum
Vagabondo
VelenPublicWebCrawler
VeriCiteCrawler
VidibleScraper
@ -537,46 +482,50 @@ Virusdie
VoidEYE
Voil
Voltron
voyagerx.com
Wallpapers/3.0
WallpapersHD
WASALive-Bot
WBSearchBot
Webalta
WEBDAV
WISENutbot
WPScan
WWW-Collector-E
WWW-Mechanize
WWW::Mechanize
WWWOFFLE
Wallpapers/3.0
WallpapersHD
WeSEE
WebAuto
Web Auto
WebBandit
WebCollage
Web Collage
WebCopier
WEBDAV
WebEnhancer
Web Enhancer
WebFetch
Web Fetch
WebFuck
Web Fuck
WebGo IS
WebImageCollector
WebLeacher
WebmasterWorldForumBot
webmeup-crawler
WebPix
Web Pix
WebReaper
WebSauger
WebStripper
WebSucker
WebWhacker
WebZIP
Web Auto
Web Collage
Web Enhancer
Web Fetch
Web Fuck
Web Pix
Web Sauger
Web Sucker
Webalta
WebmasterWorldForumBot
Webshag
WebsiteExtractor
WebsiteQuester
Website Quester
Webster
WebStripper
WebSucker
Web Sucker
WebWhacker
WebZIP
WeSEE
Whack
Whacker
Whatweb
@ -584,32 +533,83 @@ Who.is Bot
Widow
WinHTTrack
WiseGuys Robot
WISENutbot
Wonderbot
Woobot
Wotbox
Wprecon
WPScan
WWW-Collector-E
WWW-Mechanize
WWW::Mechanize
WWWOFFLE
x09Mozilla
x22Mozilla
Xaldon_WebSpider
Xaldon WebSpider
Xaldon_WebSpider
Xenu
xpymep1.exe
YoudaoBot
Zade
Zauba
zauba.io
Zermelo
Zeus
zgrab
Zitebot
ZmEu
ZoomBot
ZoominfoBot
ZumBot
ZyBorg
archive.org_bot
arquivo-web-crawler
arquivo.pt
autoemailspider
backlink-check
check1.exe
coccocbot-web
cognitiveseo
com.plumanalytics
crawl.sogou.com
crawler.feedback
crawler4j
demandbase-bot
domainsproject.org
eCatch
evc-batch
facebookscraper
gopher
heritrix
instabid
internetVista monitor
ips-agent
isitwp.com
lwp-request
lwp-trivial
magpie-crawler
meanpathbot
mediawords
muhstik-scan
netEstate NE Crawler
oBot
page scorer
pcBrowser
plumanalytics
polaris version
probe-image-size
ripz
s1z.ru
satoristudio.net
scalaj-http
scan.lol
seobility
seoscanners
seostar
serpstatbot
sexsearcher
sitechecker.pro
siteripz
sogouspider
sp_auditbot
spyfu
sysscan
tAkeOut
trendiction.com
trendiction.de
voyagerx.com
webmeup-crawler
x09Mozilla
x22Mozilla
xpymep1.exe
zauba.io
zgrab

15472
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/beta_conf_files/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /etc/nginx/bots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
34.66.25.221 1;
10.1.0.59 1;
127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
34.66.25.221 0;
10.1.0.59 0;
127.0.0.1 0;

15472
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/beta_conf_files_ip_whitelist/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /etc/nginx/bots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
34.66.25.221 1;
10.1.0.59 1;
127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
34.66.25.221 0;
10.1.0.59 0;
127.0.0.1 0;

15472
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/beta_conf_files_ratelimiting/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /etc/nginx/bots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
34.66.25.221 1;
10.1.0.59 1;
127.0.0.1 1;

368
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -64,19 +64,20 @@
"~*(?:\b)404checker(?:\b|)" 0;
"~*(?:\b)404enemy(?:\b|)" 0;
"~*(?:\b)80legs(?:\b|)" 0;
"~*(?:\b)ADmantX(?:\b|)" 0;
"~*(?:\b)AIBOT(?:\b|)" 0;
"~*(?:\b)ASPSeek(?:\b|)" 0;
"~*(?:\b)Abonti(?:\b|)" 0;
"~*(?:\b)Aboundex(?:\b|)" 0;
"~*(?:\b)Aboundexbot(?:\b|)" 0;
"~*(?:\b)Acunetix(?:\b|)" 0;
"~*(?:\b)ADmantX(?:\b|)" 0;
"~*(?:\b)AfD-Verbotsverfahren(?:\b|)" 0;
"~*(?:\b)AhrefsBot(?:\b|)" 0;
"~*(?:\b)AIBOT(?:\b|)" 0;
"~*(?:\b)AiHitBot(?:\b|)" 0;
"~*(?:\b)Aipbot(?:\b|)" 0;
"~*(?:\b)Alexibot(?:\b|)" 0;
"~*(?:\b)Alligator(?:\b|)" 0;
"~*(?:\b)AllSubmitter(?:\b|)" 0;
"~*(?:\b)Alligator(?:\b|)" 0;
"~*(?:\b)AlphaBot(?:\b|)" 0;
"~*(?:\b)Anarchie(?:\b|)" 0;
"~*(?:\b)Anarchy(?:\b|)" 0;
@ -84,37 +85,31 @@
"~*(?:\b)Ankit(?:\b|)" 0;
"~*(?:\b)Anthill(?:\b|)" 0;
"~*(?:\b)Apexoo(?:\b|)" 0;
"~*(?:\b)archive.org_bot(?:\b|)" 0;
"~*(?:\b)arquivo.pt(?:\b|)" 0;
"~*(?:\b)arquivo-web-crawler(?:\b|)" 0;
"~*(?:\b)Aspiegel(?:\b|)" 0;
"~*(?:\b)ASPSeek(?:\b|)" 0;
"~*(?:\b)Asterias(?:\b|)" 0;
"~*(?:\b)Attach(?:\b|)" 0;
"~*(?:\b)autoemailspider(?:\b|)" 0;
"~*(?:\b)AwarioRssBot(?:\b|)" 0;
"~*(?:\b)AwarioSmartBot(?:\b|)" 0;
"~*(?:\b)BBBike(?:\b|)" 0;
"~*(?:\b)BDCbot(?:\b|)" 0;
"~*(?:\b)BDFetch(?:\b|)" 0;
"~*(?:\b)BLEXBot(?:\b|)" 0;
"~*(?:\b)BackDoorBot(?:\b|)" 0;
"~*(?:\b)Backlink-Ceck(?:\b|)" 0;
"~*(?:\b)backlink-check(?:\b|)" 0;
"~*(?:\b)BacklinkCrawler(?:\b|)" 0;
"~*(?:\b)BackStreet(?:\b|)" 0;
"~*(?:\b)BackWeb(?:\b|)" 0;
"~*(?:\b)Backlink-Ceck(?:\b|)" 0;
"~*(?:\b)BacklinkCrawler(?:\b|)" 0;
"~*(?:\b)Badass(?:\b|)" 0;
"~*(?:\b)Bandit(?:\b|)" 0;
"~*(?:\b)Barkrowler(?:\b|)" 0;
"~*(?:\b)BatchFTP(?:\b|)" 0;
"~*(?:\b)Battleztar\ Bazinga(?:\b|)" 0;
"~*(?:\b)BBBike(?:\b|)" 0;
"~*(?:\b)BDCbot(?:\b|)" 0;
"~*(?:\b)BDFetch(?:\b|)" 0;
"~*(?:\b)BetaBot(?:\b|)" 0;
"~*(?:\b)Bigfoot(?:\b|)" 0;
"~*(?:\b)Bitacle(?:\b|)" 0;
"~*(?:\b)Blackboard(?:\b|)" 0;
"~*(?:\b)Black\ Hole(?:\b|)" 0;
"~*(?:\b)BlackWidow(?:\b|)" 0;
"~*(?:\b)BLEXBot(?:\b|)" 0;
"~*(?:\b)Black\ Hole(?:\b|)" 0;
"~*(?:\b)Blackboard(?:\b|)" 0;
"~*(?:\b)Blow(?:\b|)" 0;
"~*(?:\b)BlowFish(?:\b|)" 0;
"~*(?:\b)Boardreader(?:\b|)" 0;
@ -129,53 +124,47 @@
"~*(?:\b)Bullseye(?:\b|)" 0;
"~*(?:\b)BunnySlippers(?:\b|)" 0;
"~*(?:\b)BuzzSumo(?:\b|)" 0;
"~*(?:\b)Calculon(?:\b|)" 0;
"~*(?:\b)CATExplorador(?:\b|)" 0;
"~*(?:\b)CazoodleBot(?:\b|)" 0;
"~*(?:\b)CCBot(?:\b|)" 0;
"~*(?:\b)CODE87(?:\b|)" 0;
"~*(?:\b)CSHttp(?:\b|)" 0;
"~*(?:\b)Calculon(?:\b|)" 0;
"~*(?:\b)CazoodleBot(?:\b|)" 0;
"~*(?:\b)Cegbfeieh(?:\b|)" 0;
"~*(?:\b)CensysInspect(?:\b|)" 0;
"~*(?:\b)check1.exe(?:\b|)" 0;
"~*(?:\b)CheTeam(?:\b|)" 0;
"~*(?:\b)CheeseBot(?:\b|)" 0;
"~*(?:\b)CherryPicker(?:\b|)" 0;
"~*(?:\b)CheTeam(?:\b|)" 0;
"~*(?:\b)ChinaClaw(?:\b|)" 0;
"~*(?:\b)Chlooe(?:\b|)" 0;
"~*(?:\b)Claritybot(?:\b|)" 0;
"~*(?:\b)Cliqzbot(?:\b|)" 0;
"~*(?:\b)Cloud\ mapping(?:\b|)" 0;
"~*(?:\b)coccocbot-web(?:\b|)" 0;
"~*(?:\b)Cocolyzebot(?:\b|)" 0;
"~*(?:\b)CODE87(?:\b|)" 0;
"~*(?:\b)Cogentbot(?:\b|)" 0;
"~*(?:\b)cognitiveseo(?:\b|)" 0;
"~*(?:\b)Collector(?:\b|)" 0;
"~*(?:\b)com.plumanalytics(?:\b|)" 0;
"~*(?:\b)Copier(?:\b|)" 0;
"~*(?:\b)CopyRightCheck(?:\b|)" 0;
"~*(?:\b)Copyscape(?:\b|)" 0;
"~*(?:\b)Cosmos(?:\b|)" 0;
"~*(?:\b)Craftbot(?:\b|)" 0;
"~*(?:\b)crawler4j(?:\b|)" 0;
"~*(?:\b)crawler.feedback(?:\b|)" 0;
"~*(?:\b)crawl.sogou.com(?:\b|)" 0;
"~*(?:\b)CrazyWebCrawler(?:\b|)" 0;
"~*(?:\b)Crescent(?:\b|)" 0;
"~*(?:\b)CrunchBot(?:\b|)" 0;
"~*(?:\b)CSHttp(?:\b|)" 0;
"~*(?:\b)Curious(?:\b|)" 0;
"~*(?:\b)Custo(?:\b|)" 0;
"~*(?:\b)CyotekWebCopy(?:\b|)" 0;
"~*(?:\b)DatabaseDriverMysqli(?:\b|)" 0;
"~*(?:\b)DataCha0s(?:\b|)" 0;
"~*(?:\b)DBLBot(?:\b|)" 0;
"~*(?:\b)demandbase-bot(?:\b|)" 0;
"~*(?:\b)DIIbot(?:\b|)" 0;
"~*(?:\b)DSearch(?:\b|)" 0;
"~*(?:\b)DTS\ Agent(?:\b|)" 0;
"~*(?:\b)DataCha0s(?:\b|)" 0;
"~*(?:\b)DatabaseDriverMysqli(?:\b|)" 0;
"~*(?:\b)Demon(?:\b|)" 0;
"~*(?:\b)Deusu(?:\b|)" 0;
"~*(?:\b)Devil(?:\b|)" 0;
"~*(?:\b)Digincore(?:\b|)" 0;
"~*(?:\b)DigitalPebble(?:\b|)" 0;
"~*(?:\b)DIIbot(?:\b|)" 0;
"~*(?:\b)Dirbuster(?:\b|)" 0;
"~*(?:\b)Disco(?:\b|)" 0;
"~*(?:\b)Discobot(?:\b|)" 0;
@ -183,29 +172,24 @@
"~*(?:\b)Dispatch(?:\b|)" 0;
"~*(?:\b)DittoSpyder(?:\b|)" 0;
"~*(?:\b)DnyzBot(?:\b|)" 0;
"~*(?:\b)DomCopBot(?:\b|)" 0;
"~*(?:\b)DomainAppender(?:\b|)" 0;
"~*(?:\b)DomainCrawler(?:\b|)" 0;
"~*(?:\b)DomainSigmaCrawler(?:\b|)" 0;
"~*(?:\b)Domains\ Project(?:\b|)" 0;
"~*(?:\b)domainsproject.org(?:\b|)" 0;
"~*(?:\b)DomainStatsBot(?:\b|)" 0;
"~*(?:\b)DomCopBot(?:\b|)" 0;
"~*(?:\b)Domains\ Project(?:\b|)" 0;
"~*(?:\b)Dotbot(?:\b|)" 0;
"~*(?:\b)Download\ Wonder(?:\b|)" 0;
"~*(?:\b)Dragonfly(?:\b|)" 0;
"~*(?:\b)Drip(?:\b|)" 0;
"~*(?:\b)DSearch(?:\b|)" 0;
"~*(?:\b)DTS\ Agent(?:\b|)" 0;
"~*(?:\b)EasyDL(?:\b|)" 0;
"~*(?:\b)Ebingbong(?:\b|)" 0;
"~*(?:\b)eCatch(?:\b|)" 0;
"~*(?:\b)ECCP/1.0(?:\b|)" 0;
"~*(?:\b)Ecxi(?:\b|)" 0;
"~*(?:\b)EirGrabber(?:\b|)" 0;
"~*(?:\b)EMail\ Siphon(?:\b|)" 0;
"~*(?:\b)EMail\ Wolf(?:\b|)" 0;
"~*(?:\b)EasyDL(?:\b|)" 0;
"~*(?:\b)Ebingbong(?:\b|)" 0;
"~*(?:\b)Ecxi(?:\b|)" 0;
"~*(?:\b)EirGrabber(?:\b|)" 0;
"~*(?:\b)EroCrawler(?:\b|)" 0;
"~*(?:\b)evc-batch(?:\b|)" 0;
"~*(?:\b)Evil(?:\b|)" 0;
"~*(?:\b)Exabot(?:\b|)" 0;
"~*(?:\b)Express\ WebPictures(?:\b|)" 0;
@ -215,10 +199,9 @@
"~*(?:\b)Extreme\ Picture\ Finder(?:\b|)" 0;
"~*(?:\b)EyeNetIE(?:\b|)" 0;
"~*(?:\b)Ezooms(?:\b|)" 0;
"~*(?:\b)facebookscraper(?:\b|)" 0;
"~*(?:\b)FDM(?:\b|)" 0;
"~*(?:\b)FemtosearchBot(?:\b|)" 0;
"~*(?:\b)FHscan(?:\b|)" 0;
"~*(?:\b)FemtosearchBot(?:\b|)" 0;
"~*(?:\b)Fimap(?:\b|)" 0;
"~*(?:\b)Firefox/7.0(?:\b|)" 0;
"~*(?:\b)FlashGet(?:\b|)" 0;
@ -229,44 +212,43 @@
"~*(?:\b)Fuzz(?:\b|)" 0;
"~*(?:\b)FyberSpider(?:\b|)" 0;
"~*(?:\b)Fyrebot(?:\b|)" 0;
"~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0;
"~*(?:\b)GT::WWW(?:\b|)" 0;
"~*(?:\b)GalaxyBot(?:\b|)" 0;
"~*(?:\b)Genieo(?:\b|)" 0;
"~*(?:\b)GermCrawler(?:\b|)" 0;
"~*(?:\b)Getintent(?:\b|)" 0;
"~*(?:\b)GetRight(?:\b|)" 0;
"~*(?:\b)GetWeb(?:\b|)" 0;
"~*(?:\b)Getintent(?:\b|)" 0;
"~*(?:\b)Gigablast(?:\b|)" 0;
"~*(?:\b)Gigabot(?:\b|)" 0;
"~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0;
"~*(?:\b)Go-Ahead-Got-It(?:\b|)" 0;
"~*(?:\b)gopher(?:\b|)" 0;
"~*(?:\b)Gotit(?:\b|)" 0;
"~*(?:\b)GoZilla(?:\b|)" 0;
"~*(?:\b)Go!Zilla(?:\b|)" 0;
"~*(?:\b)Grabber(?:\b|)" 0;
"~*(?:\b)Go-Ahead-Got-It(?:\b|)" 0;
"~*(?:\b)GoZilla(?:\b|)" 0;
"~*(?:\b)Gotit(?:\b|)" 0;
"~*(?:\b)GrabNet(?:\b|)" 0;
"~*(?:\b)Grabber(?:\b|)" 0;
"~*(?:\b)Grafula(?:\b|)" 0;
"~*(?:\b)GrapeFX(?:\b|)" 0;
"~*(?:\b)GrapeshotCrawler(?:\b|)" 0;
"~*(?:\b)GridBot(?:\b|)" 0;
"~*(?:\b)GT::WWW(?:\b|)" 0;
"~*(?:\b)Haansoft(?:\b|)" 0;
"~*(?:\b)HaosouSpider(?:\b|)" 0;
"~*(?:\b)Harvest(?:\b|)" 0;
"~*(?:\b)Havij(?:\b|)" 0;
"~*(?:\b)HEADMasterSEO(?:\b|)" 0;
"~*(?:\b)heritrix(?:\b|)" 0;
"~*(?:\b)Heritrix(?:\b|)" 0;
"~*(?:\b)Hloader(?:\b|)" 0;
"~*(?:\b)HMView(?:\b|)" 0;
"~*(?:\b)HTMLparser(?:\b|)" 0;
"~*(?:\b)HTTP::Lite(?:\b|)" 0;
"~*(?:\b)HTTrack(?:\b|)" 0;
"~*(?:\b)Haansoft(?:\b|)" 0;
"~*(?:\b)HaosouSpider(?:\b|)" 0;
"~*(?:\b)Harvest(?:\b|)" 0;
"~*(?:\b)Havij(?:\b|)" 0;
"~*(?:\b)Heritrix(?:\b|)" 0;
"~*(?:\b)Hloader(?:\b|)" 0;
"~*(?:\b)Humanlinks(?:\b|)" 0;
"~*(?:\b)HybridBot(?:\b|)" 0;
"~*(?:\b)Iblog(?:\b|)" 0;
"~*(?:\b)IDBot(?:\b|)" 0;
"~*(?:\b)IDBTE4M(?:\b|)" 0;
"~*(?:\b)IDBot(?:\b|)" 0;
"~*(?:\b)IRLbot(?:\b|)" 0;
"~*(?:\b)Iblog(?:\b|)" 0;
"~*(?:\b)Id-search(?:\b|)" 0;
"~*(?:\b)IlseBot(?:\b|)" 0;
"~*(?:\b)Image\ Fetch(?:\b|)" 0;
@ -275,25 +257,20 @@
"~*(?:\b)Indy\ Library(?:\b|)" 0;
"~*(?:\b)InfoNaviRobot(?:\b|)" 0;
"~*(?:\b)InfoTekies(?:\b|)" 0;
"~*(?:\b)instabid(?:\b|)" 0;
"~*(?:\b)Intelliseek(?:\b|)" 0;
"~*(?:\b)InterGET(?:\b|)" 0;
"~*(?:\b)Internet\ Ninja(?:\b|)" 0;
"~*(?:\b)InternetSeer(?:\b|)" 0;
"~*(?:\b)internetVista\ monitor(?:\b|)" 0;
"~*(?:\b)ips-agent(?:\b|)" 0;
"~*(?:\b)Internet\ Ninja(?:\b|)" 0;
"~*(?:\b)Iria(?:\b|)" 0;
"~*(?:\b)IRLbot(?:\b|)" 0;
"~*(?:\b)isitwp.com(?:\b|)" 0;
"~*(?:\b)Iskanie(?:\b|)" 0;
"~*(?:\b)IstellaBot(?:\b|)" 0;
"~*(?:\b)JOC\ Web\ Spider(?:\b|)" 0;
"~*(?:\b)JamesBOT(?:\b|)" 0;
"~*(?:\b)Jbrofuzz(?:\b|)" 0;
"~*(?:\b)JennyBot(?:\b|)" 0;
"~*(?:\b)JetCar(?:\b|)" 0;
"~*(?:\b)Jetty(?:\b|)" 0;
"~*(?:\b)JikeSpider(?:\b|)" 0;
"~*(?:\b)JOC\ Web\ Spider(?:\b|)" 0;
"~*(?:\b)Joomla(?:\b|)" 0;
"~*(?:\b)Jorgee(?:\b|)" 0;
"~*(?:\b)JustView(?:\b|)" 0;
@ -302,6 +279,8 @@
"~*(?:\b)Keyword\ Density(?:\b|)" 0;
"~*(?:\b)Kinza(?:\b|)" 0;
"~*(?:\b)Kozmosbot(?:\b|)" 0;
"~*(?:\b)LNSpiderguy(?:\b|)" 0;
"~*(?:\b)LWP::Simple(?:\b|)" 0;
"~*(?:\b)Lanshanbot(?:\b|)" 0;
"~*(?:\b)Larbin(?:\b|)" 0;
"~*(?:\b)LeechFTP(?:\b|)" 0;
@ -313,13 +292,13 @@
"~*(?:\b)LieBaoFast(?:\b|)" 0;
"~*(?:\b)Lightspeedsystems(?:\b|)" 0;
"~*(?:\b)Likse(?:\b|)" 0;
"~*(?:\b)LinkScan(?:\b|)" 0;
"~*(?:\b)LinkWalker(?:\b|)" 0;
"~*(?:\b)Linkbot(?:\b|)" 0;
"~*(?:\b)Linkdexbot(?:\b|)" 0;
"~*(?:\b)LinkextractorPro(?:\b|)" 0;
"~*(?:\b)LinkpadBot(?:\b|)" 0;
"~*(?:\b)LinkScan(?:\b|)" 0;
"~*(?:\b)LinksManager(?:\b|)" 0;
"~*(?:\b)LinkWalker(?:\b|)" 0;
"~*(?:\b)LinqiaMetadataDownloaderBot(?:\b|)" 0;
"~*(?:\b)LinqiaRSSBot(?:\b|)" 0;
"~*(?:\b)LinqiaScrapeBot(?:\b|)" 0;
@ -327,53 +306,47 @@
"~*(?:\b)Lipperhey\ Spider(?:\b|)" 0;
"~*(?:\b)Litemage_walker(?:\b|)" 0;
"~*(?:\b)Lmspider(?:\b|)" 0;
"~*(?:\b)LNSpiderguy(?:\b|)" 0;
"~*(?:\b)Ltx71(?:\b|)" 0;
"~*(?:\b)lwp-request(?:\b|)" 0;
"~*(?:\b)LWP::Simple(?:\b|)" 0;
"~*(?:\b)lwp-trivial(?:\b|)" 0;
"~*(?:\b)Magnet(?:\b|)" 0;
"~*(?:\b)MFC_Tear_Sample(?:\b|)" 0;
"~*(?:\b)MIDown\ tool(?:\b|)" 0;
"~*(?:\b)MIIxpc(?:\b|)" 0;
"~*(?:\b)MJ12bot(?:\b|)" 0;
"~*(?:\b)MQQBrowser(?:\b|)" 0;
"~*(?:\b)MSFrontPage(?:\b|)" 0;
"~*(?:\b)MSIECrawler(?:\b|)" 0;
"~*(?:\b)MTRobot(?:\b|)" 0;
"~*(?:\b)Mag-Net(?:\b|)" 0;
"~*(?:\b)magpie-crawler(?:\b|)" 0;
"~*(?:\b)Magnet(?:\b|)" 0;
"~*(?:\b)Mail.RU_Bot(?:\b|)" 0;
"~*(?:\b)Majestic12(?:\b|)" 0;
"~*(?:\b)Majestic-SEO(?:\b|)" 0;
"~*(?:\b)Majestic12(?:\b|)" 0;
"~*(?:\b)Majestic\ SEO(?:\b|)" 0;
"~*(?:\b)MarkMonitor(?:\b|)" 0;
"~*(?:\b)MarkWatch(?:\b|)" 0;
"~*(?:\b)Masscan(?:\b|)" 0;
"~*(?:\b)Mass\ Downloader(?:\b|)" 0;
"~*(?:\b)Masscan(?:\b|)" 0;
"~*(?:\b)Mata\ Hari(?:\b|)" 0;
"~*(?:\b)MauiBot(?:\b|)" 0;
"~*(?:\b)Mb2345Browser(?:\b|)" 0;
"~*(?:\b)meanpathbot(?:\b|)" 0;
"~*(?:\b)Meanpathbot(?:\b|)" 0;
"~*(?:\b)MeanPath\ Bot(?:\b|)" 0;
"~*(?:\b)Meanpathbot(?:\b|)" 0;
"~*(?:\b)Mediatoolkitbot(?:\b|)" 0;
"~*(?:\b)mediawords(?:\b|)" 0;
"~*(?:\b)MegaIndex.ru(?:\b|)" 0;
"~*(?:\b)Metauri(?:\b|)" 0;
"~*(?:\b)MFC_Tear_Sample(?:\b|)" 0;
"~*(?:\b)MicroMessenger(?:\b|)" 0;
"~*(?:\b)Microsoft\ Data\ Access(?:\b|)" 0;
"~*(?:\b)Microsoft\ URL\ Control(?:\b|)" 0;
"~*(?:\b)MIDown\ tool(?:\b|)" 0;
"~*(?:\b)MIIxpc(?:\b|)" 0;
"~*(?:\b)Mister\ PiX(?:\b|)" 0;
"~*(?:\b)MJ12bot(?:\b|)" 0;
"~*(?:\b)Moblie Safari(?:\b|)" 0;
"~*(?:\b)Mojeek(?:\b|)" 0;
"~*(?:\b)Mojolicious(?:\b|)" 0;
"~*(?:\b)Morfeus\ Fucking\ Scanner(?:\b|)" 0;
"~*(?:\b)Mozlila(?:\b|)" 0;
"~*(?:\b)MQQBrowser(?:\b|)" 0;
"~*(?:\b)Mr.4x3(?:\b|)" 0;
"~*(?:\b)MSFrontPage(?:\b|)" 0;
"~*(?:\b)MSIECrawler(?:\b|)" 0;
"~*(?:\b)Msrabot(?:\b|)" 0;
"~*(?:\b)MTRobot(?:\b|)" 0;
"~*(?:\b)muhstik-scan(?:\b|)" 0;
"~*(?:\b)Musobot(?:\b|)" 0;
"~*(?:\b)NICErsPRO(?:\b|)" 0;
"~*(?:\b)NPbot(?:\b|)" 0;
"~*(?:\b)Name\ Intelligence(?:\b|)" 0;
"~*(?:\b)Nameprotect(?:\b|)" 0;
"~*(?:\b)Navroad(?:\b|)" 0;
@ -381,72 +354,63 @@
"~*(?:\b)Needle(?:\b|)" 0;
"~*(?:\b)Nessus(?:\b|)" 0;
"~*(?:\b)NetAnts(?:\b|)" 0;
"~*(?:\b)Netcraft(?:\b|)" 0;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0;
"~*(?:\b)NetLyzer(?:\b|)" 0;
"~*(?:\b)NetMechanic(?:\b|)" 0;
"~*(?:\b)NetSpider(?:\b|)" 0;
"~*(?:\b)Nettrack(?:\b|)" 0;
"~*(?:\b)Net\ Vampire(?:\b|)" 0;
"~*(?:\b)Netvibes(?:\b|)" 0;
"~*(?:\b)NetZIP(?:\b|)" 0;
"~*(?:\b)Net\ Vampire(?:\b|)" 0;
"~*(?:\b)Netcraft(?:\b|)" 0;
"~*(?:\b)Nettrack(?:\b|)" 0;
"~*(?:\b)Netvibes(?:\b|)" 0;
"~*(?:\b)NextGenSearchBot(?:\b|)" 0;
"~*(?:\b)Nibbler(?:\b|)" 0;
"~*(?:\b)NICErsPRO(?:\b|)" 0;
"~*(?:\b)Niki-bot(?:\b|)" 0;
"~*(?:\b)Nikto(?:\b|)" 0;
"~*(?:\b)NimbleCrawler(?:\b|)" 0;
"~*(?:\b)Nimbostratus(?:\b|)" 0;
"~*(?:\b)Ninja(?:\b|)" 0;
"~*(?:\b)Nmap(?:\b|)" 0;
"~*(?:\b)NPbot(?:\b|)" 0;
"~*(?:\b)Nuclei(?:\b|)" 0;
"~*(?:\b)Nutch(?:\b|)" 0;
"~*(?:\b)oBot(?:\b|)" 0;
"~*(?:\b)OPPO\ A33(?:\b|)" 0;
"~*(?:\b)Octopus(?:\b|)" 0;
"~*(?:\b)Offline\ Explorer(?:\b|)" 0;
"~*(?:\b)Offline\ Navigator(?:\b|)" 0;
"~*(?:\b)OnCrawl(?:\b|)" 0;
"~*(?:\b)Openfind(?:\b|)" 0;
"~*(?:\b)OpenLinkProfiler(?:\b|)" 0;
"~*(?:\b)Openvas(?:\b|)" 0;
"~*(?:\b)OpenVAS(?:\b|)" 0;
"~*(?:\b)OPPO\ A33(?:\b|)" 0;
"~*(?:\b)Openfind(?:\b|)" 0;
"~*(?:\b)Openvas(?:\b|)" 0;
"~*(?:\b)OrangeBot(?:\b|)" 0;
"~*(?:\b)OrangeSpider(?:\b|)" 0;
"~*(?:\b)OutclicksBot(?:\b|)" 0;
"~*(?:\b)OutfoxBot(?:\b|)" 0;
"~*(?:\b)PECL::HTTP(?:\b|)" 0;
"~*(?:\b)PHPCrawl(?:\b|)" 0;
"~*(?:\b)POE-Component-Client-HTTP(?:\b|)" 0;
"~*(?:\b)PageAnalyzer(?:\b|)" 0;
"~*(?:\b)Page\ Analyzer(?:\b|)" 0;
"~*(?:\b)PageGrabber(?:\b|)" 0;
"~*(?:\b)page\ scorer(?:\b|)" 0;
"~*(?:\b)PageScorer(?:\b|)" 0;
"~*(?:\b)PageThing.com(?:\b|)" 0;
"~*(?:\b)Page\ Analyzer(?:\b|)" 0;
"~*(?:\b)Pandalytics(?:\b|)" 0;
"~*(?:\b)Panscient(?:\b|)" 0;
"~*(?:\b)Papa\ Foto(?:\b|)" 0;
"~*(?:\b)Pavuk(?:\b|)" 0;
"~*(?:\b)pcBrowser(?:\b|)" 0;
"~*(?:\b)PECL::HTTP(?:\b|)" 0;
"~*(?:\b)PeoplePal(?:\b|)" 0;
"~*(?:\b)Petalbot(?:\b|)" 0;
"~*(?:\b)PHPCrawl(?:\b|)" 0;
"~*(?:\b)Pi-Monster(?:\b|)" 0;
"~*(?:\b)Picscout(?:\b|)" 0;
"~*(?:\b)Picsearch(?:\b|)" 0;
"~*(?:\b)PictureFinder(?:\b|)" 0;
"~*(?:\b)Piepmatz(?:\b|)" 0;
"~*(?:\b)Pimonster(?:\b|)" 0;
"~*(?:\b)Pi-Monster(?:\b|)" 0;
"~*(?:\b)Pixray(?:\b|)" 0;
"~*(?:\b)PleaseCrawl(?:\b|)" 0;
"~*(?:\b)plumanalytics(?:\b|)" 0;
"~*(?:\b)Pockey(?:\b|)" 0;
"~*(?:\b)POE-Component-Client-HTTP(?:\b|)" 0;
"~*(?:\b)polaris\ version(?:\b|)" 0;
"~*(?:\b)probe-image-size(?:\b|)" 0;
"~*(?:\b)Probethenet(?:\b|)" 0;
"~*(?:\b)ProPowerBot(?:\b|)" 0;
"~*(?:\b)ProWebWalker(?:\b|)" 0;
"~*(?:\b)Probethenet(?:\b|)" 0;
"~*(?:\b)Psbot(?:\b|)" 0;
"~*(?:\b)Pu_iN(?:\b|)" 0;
"~*(?:\b)Pump(?:\b|)" 0;
@ -454,6 +418,7 @@
"~*(?:\b)PyCurl(?:\b|)" 0;
"~*(?:\b)QueryN\ Metasearch(?:\b|)" 0;
"~*(?:\b)Quick-Crawler(?:\b|)" 0;
"~*(?:\b)RSSingBot(?:\b|)" 0;
"~*(?:\b)RankActive(?:\b|)" 0;
"~*(?:\b)RankActiveLinkBot(?:\b|)" 0;
"~*(?:\b)RankFlex(?:\b|)" 0;
@ -461,27 +426,28 @@
"~*(?:\b)RankingBot2(?:\b|)" 0;
"~*(?:\b)Rankivabot(?:\b|)" 0;
"~*(?:\b)RankurBot(?:\b|)" 0;
"~*(?:\b)Re-re(?:\b|)" 0;
"~*(?:\b)ReGet(?:\b|)" 0;
"~*(?:\b)RealDownload(?:\b|)" 0;
"~*(?:\b)Reaper(?:\b|)" 0;
"~*(?:\b)RebelMouse(?:\b|)" 0;
"~*(?:\b)Recorder(?:\b|)" 0;
"~*(?:\b)RedesScrapy(?:\b|)" 0;
"~*(?:\b)ReGet(?:\b|)" 0;
"~*(?:\b)RepoMonkey(?:\b|)" 0;
"~*(?:\b)Re-re(?:\b|)" 0;
"~*(?:\b)Ripper(?:\b|)" 0;
"~*(?:\b)ripz(?:\b|)" 0;
"~*(?:\b)RocketCrawler(?:\b|)" 0;
"~*(?:\b)Rogerbot(?:\b|)" 0;
"~*(?:\b)RSSingBot(?:\b|)" 0;
"~*(?:\b)s1z.ru(?:\b|)" 0;
"~*(?:\b)SalesIntelligent(?:\b|)" 0;
"~*(?:\b)satoristudio.net(?:\b|)" 0;
"~*(?:\b)SBIder(?:\b|)" 0;
"~*(?:\b)scalaj-http(?:\b|)" 0;
"~*(?:\b)SEOkicks(?:\b|)" 0;
"~*(?:\b)SEOkicks-Robot(?:\b|)" 0;
"~*(?:\b)SEOlyticsCrawler(?:\b|)" 0;
"~*(?:\b)SEOprofiler(?:\b|)" 0;
"~*(?:\b)SEOstats(?:\b|)" 0;
"~*(?:\b)SISTRIX(?:\b|)" 0;
"~*(?:\b)SMTBot(?:\b|)" 0;
"~*(?:\b)SalesIntelligent(?:\b|)" 0;
"~*(?:\b)ScanAlert(?:\b|)" 0;
"~*(?:\b)Scanbot(?:\b|)" 0;
"~*(?:\b)scan.lol(?:\b|)" 0;
"~*(?:\b)ScoutJet(?:\b|)" 0;
"~*(?:\b)Scrapy(?:\b|)" 0;
"~*(?:\b)Screaming(?:\b|)" 0;
@ -492,42 +458,27 @@
"~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0;
"~*(?:\b)SentiBot(?:\b|)" 0;
"~*(?:\b)seobility(?:\b|)" 0;
"~*(?:\b)SeobilityBot(?:\b|)" 0;
"~*(?:\b)SEOkicks(?:\b|)" 0;
"~*(?:\b)SEOkicks-Robot(?:\b|)" 0;
"~*(?:\b)SEOlyticsCrawler(?:\b|)" 0;
"~*(?:\b)Seomoz(?:\b|)" 0;
"~*(?:\b)SEOprofiler(?:\b|)" 0;
"~*(?:\b)seoscanners(?:\b|)" 0;
"~*(?:\b)SeoSiteCheckup(?:\b|)" 0;
"~*(?:\b)seostar(?:\b|)" 0;
"~*(?:\b)SEOstats(?:\b|)" 0;
"~*(?:\b)serpstatbot(?:\b|)" 0;
"~*(?:\b)sexsearcher(?:\b|)" 0;
"~*(?:\b)SeobilityBot(?:\b|)" 0;
"~*(?:\b)Seomoz(?:\b|)" 0;
"~*(?:\b)Shodan(?:\b|)" 0;
"~*(?:\b)Siphon(?:\b|)" 0;
"~*(?:\b)SISTRIX(?:\b|)" 0;
"~*(?:\b)Sitebeam(?:\b|)" 0;
"~*(?:\b)SiteCheckerBotCrawler(?:\b|)" 0;
"~*(?:\b)sitechecker.pro(?:\b|)" 0;
"~*(?:\b)SiteExplorer(?:\b|)" 0;
"~*(?:\b)Siteimprove(?:\b|)" 0;
"~*(?:\b)SiteLockSpider(?:\b|)" 0;
"~*(?:\b)siteripz(?:\b|)" 0;
"~*(?:\b)SiteSnagger(?:\b|)" 0;
"~*(?:\b)SiteSucker(?:\b|)" 0;
"~*(?:\b)Site\ Sucker(?:\b|)" 0;
"~*(?:\b)Sitebeam(?:\b|)" 0;
"~*(?:\b)Siteimprove(?:\b|)" 0;
"~*(?:\b)Sitevigil(?:\b|)" 0;
"~*(?:\b)SlySearch(?:\b|)" 0;
"~*(?:\b)SmartDownload(?:\b|)" 0;
"~*(?:\b)SMTBot(?:\b|)" 0;
"~*(?:\b)Snake(?:\b|)" 0;
"~*(?:\b)Snapbot(?:\b|)" 0;
"~*(?:\b)Snoopy(?:\b|)" 0;
"~*(?:\b)SocialRankIOBot(?:\b|)" 0;
"~*(?:\b)Sociscraper(?:\b|)" 0;
"~*(?:\b)sogouspider(?:\b|)" 0;
"~*(?:\b)Sogou\ web\ spider(?:\b|)" 0;
"~*(?:\b)Sosospider(?:\b|)" 0;
"~*(?:\b)Sottopop(?:\b|)" 0;
@ -535,11 +486,9 @@
"~*(?:\b)Spammen(?:\b|)" 0;
"~*(?:\b)SpankBot(?:\b|)" 0;
"~*(?:\b)Spanner(?:\b|)" 0;
"~*(?:\b)sp_auditbot(?:\b|)" 0;
"~*(?:\b)Spbot(?:\b|)" 0;
"~*(?:\b)Spinn3r(?:\b|)" 0;
"~*(?:\b)SputnikBot(?:\b|)" 0;
"~*(?:\b)spyfu(?:\b|)" 0;
"~*(?:\b)Sqlmap(?:\b|)" 0;
"~*(?:\b)Sqlworm(?:\b|)" 0;
"~*(?:\b)Sqworm(?:\b|)" 0;
@ -553,18 +502,16 @@
"~*(?:\b)SurveyBot(?:\b|)" 0;
"~*(?:\b)Suzuran(?:\b|)" 0;
"~*(?:\b)Swiftbot(?:\b|)" 0;
"~*(?:\b)sysscan(?:\b|)" 0;
"~*(?:\b)Szukacz(?:\b|)" 0;
"~*(?:\b)T0PHackTeam(?:\b|)" 0;
"~*(?:\b)T8Abot(?:\b|)" 0;
"~*(?:\b)tAkeOut(?:\b|)" 0;
"~*(?:\b)Teleport(?:\b|)" 0;
"~*(?:\b)TeleportPro(?:\b|)" 0;
"~*(?:\b)Telesoft(?:\b|)" 0;
"~*(?:\b)Telesphoreo(?:\b|)" 0;
"~*(?:\b)Telesphorep(?:\b|)" 0;
"~*(?:\b)The\ Intraformant(?:\b|)" 0;
"~*(?:\b)TheNomad(?:\b|)" 0;
"~*(?:\b)The\ Intraformant(?:\b|)" 0;
"~*(?:\b)Thumbor(?:\b|)" 0;
"~*(?:\b)TightTwatBot(?:\b|)" 0;
"~*(?:\b)Titan(?:\b|)" 0;
@ -573,8 +520,6 @@
"~*(?:\b)Tracemyfile(?:\b|)" 0;
"~*(?:\b)Trendiction(?:\b|)" 0;
"~*(?:\b)Trendictionbot(?:\b|)" 0;
"~*(?:\b)trendiction.com(?:\b|)" 0;
"~*(?:\b)trendiction.de(?:\b|)" 0;
"~*(?:\b)True_Robot(?:\b|)" 0;
"~*(?:\b)Turingos(?:\b|)" 0;
"~*(?:\b)Turnitin(?:\b|)" 0;
@ -582,15 +527,15 @@
"~*(?:\b)TwengaBot(?:\b|)" 0;
"~*(?:\b)Twice(?:\b|)" 0;
"~*(?:\b)Typhoeus(?:\b|)" 0;
"~*(?:\b)UnisterBot(?:\b|)" 0;
"~*(?:\b)Upflow(?:\b|)" 0;
"~*(?:\b)URLy.Warning(?:\b|)" 0;
"~*(?:\b)URLy\ Warning(?:\b|)" 0;
"~*(?:\b)Vacuum(?:\b|)" 0;
"~*(?:\b)Vagabondo(?:\b|)" 0;
"~*(?:\b)UnisterBot(?:\b|)" 0;
"~*(?:\b)Upflow(?:\b|)" 0;
"~*(?:\b)V-BOT(?:\b|)" 0;
"~*(?:\b)VB\ Project(?:\b|)" 0;
"~*(?:\b)VCI(?:\b|)" 0;
"~*(?:\b)Vacuum(?:\b|)" 0;
"~*(?:\b)Vagabondo(?:\b|)" 0;
"~*(?:\b)VelenPublicWebCrawler(?:\b|)" 0;
"~*(?:\b)VeriCiteCrawler(?:\b|)" 0;
"~*(?:\b)VidibleScraper(?:\b|)" 0;
@ -598,46 +543,50 @@
"~*(?:\b)VoidEYE(?:\b|)" 0;
"~*(?:\b)Voil(?:\b|)" 0;
"~*(?:\b)Voltron(?:\b|)" 0;
"~*(?:\b)voyagerx.com(?:\b|)" 0;
"~*(?:\b)Wallpapers/3.0(?:\b|)" 0;
"~*(?:\b)WallpapersHD(?:\b|)" 0;
"~*(?:\b)WASALive-Bot(?:\b|)" 0;
"~*(?:\b)WBSearchBot(?:\b|)" 0;
"~*(?:\b)Webalta(?:\b|)" 0;
"~*(?:\b)WEBDAV(?:\b|)" 0;
"~*(?:\b)WISENutbot(?:\b|)" 0;
"~*(?:\b)WPScan(?:\b|)" 0;
"~*(?:\b)WWW-Collector-E(?:\b|)" 0;
"~*(?:\b)WWW-Mechanize(?:\b|)" 0;
"~*(?:\b)WWW::Mechanize(?:\b|)" 0;
"~*(?:\b)WWWOFFLE(?:\b|)" 0;
"~*(?:\b)Wallpapers/3.0(?:\b|)" 0;
"~*(?:\b)WallpapersHD(?:\b|)" 0;
"~*(?:\b)WeSEE(?:\b|)" 0;
"~*(?:\b)WebAuto(?:\b|)" 0;
"~*(?:\b)Web\ Auto(?:\b|)" 0;
"~*(?:\b)WebBandit(?:\b|)" 0;
"~*(?:\b)WebCollage(?:\b|)" 0;
"~*(?:\b)Web\ Collage(?:\b|)" 0;
"~*(?:\b)WebCopier(?:\b|)" 0;
"~*(?:\b)WEBDAV(?:\b|)" 0;
"~*(?:\b)WebEnhancer(?:\b|)" 0;
"~*(?:\b)Web\ Enhancer(?:\b|)" 0;
"~*(?:\b)WebFetch(?:\b|)" 0;
"~*(?:\b)Web\ Fetch(?:\b|)" 0;
"~*(?:\b)WebFuck(?:\b|)" 0;
"~*(?:\b)Web\ Fuck(?:\b|)" 0;
"~*(?:\b)WebGo\ IS(?:\b|)" 0;
"~*(?:\b)WebImageCollector(?:\b|)" 0;
"~*(?:\b)WebLeacher(?:\b|)" 0;
"~*(?:\b)WebmasterWorldForumBot(?:\b|)" 0;
"~*(?:\b)webmeup-crawler(?:\b|)" 0;
"~*(?:\b)WebPix(?:\b|)" 0;
"~*(?:\b)Web\ Pix(?:\b|)" 0;
"~*(?:\b)WebReaper(?:\b|)" 0;
"~*(?:\b)WebSauger(?:\b|)" 0;
"~*(?:\b)WebStripper(?:\b|)" 0;
"~*(?:\b)WebSucker(?:\b|)" 0;
"~*(?:\b)WebWhacker(?:\b|)" 0;
"~*(?:\b)WebZIP(?:\b|)" 0;
"~*(?:\b)Web\ Auto(?:\b|)" 0;
"~*(?:\b)Web\ Collage(?:\b|)" 0;
"~*(?:\b)Web\ Enhancer(?:\b|)" 0;
"~*(?:\b)Web\ Fetch(?:\b|)" 0;
"~*(?:\b)Web\ Fuck(?:\b|)" 0;
"~*(?:\b)Web\ Pix(?:\b|)" 0;
"~*(?:\b)Web\ Sauger(?:\b|)" 0;
"~*(?:\b)Web\ Sucker(?:\b|)" 0;
"~*(?:\b)Webalta(?:\b|)" 0;
"~*(?:\b)WebmasterWorldForumBot(?:\b|)" 0;
"~*(?:\b)Webshag(?:\b|)" 0;
"~*(?:\b)WebsiteExtractor(?:\b|)" 0;
"~*(?:\b)WebsiteQuester(?:\b|)" 0;
"~*(?:\b)Website\ Quester(?:\b|)" 0;
"~*(?:\b)Webster(?:\b|)" 0;
"~*(?:\b)WebStripper(?:\b|)" 0;
"~*(?:\b)WebSucker(?:\b|)" 0;
"~*(?:\b)Web\ Sucker(?:\b|)" 0;
"~*(?:\b)WebWhacker(?:\b|)" 0;
"~*(?:\b)WebZIP(?:\b|)" 0;
"~*(?:\b)WeSEE(?:\b|)" 0;
"~*(?:\b)Whack(?:\b|)" 0;
"~*(?:\b)Whacker(?:\b|)" 0;
"~*(?:\b)Whatweb(?:\b|)" 0;
@ -645,33 +594,84 @@
"~*(?:\b)Widow(?:\b|)" 0;
"~*(?:\b)WinHTTrack(?:\b|)" 0;
"~*(?:\b)WiseGuys\ Robot(?:\b|)" 0;
"~*(?:\b)WISENutbot(?:\b|)" 0;
"~*(?:\b)Wonderbot(?:\b|)" 0;
"~*(?:\b)Woobot(?:\b|)" 0;
"~*(?:\b)Wotbox(?:\b|)" 0;
"~*(?:\b)Wprecon(?:\b|)" 0;
"~*(?:\b)WPScan(?:\b|)" 0;
"~*(?:\b)WWW-Collector-E(?:\b|)" 0;
"~*(?:\b)WWW-Mechanize(?:\b|)" 0;
"~*(?:\b)WWW::Mechanize(?:\b|)" 0;
"~*(?:\b)WWWOFFLE(?:\b|)" 0;
"~*(?:\b)x09Mozilla(?:\b|)" 0;
"~*(?:\b)x22Mozilla(?:\b|)" 0;
"~*(?:\b)Xaldon_WebSpider(?:\b|)" 0;
"~*(?:\b)Xaldon\ WebSpider(?:\b|)" 0;
"~*(?:\b)Xaldon_WebSpider(?:\b|)" 0;
"~*(?:\b)Xenu(?:\b|)" 0;
"~*(?:\b)xpymep1.exe(?:\b|)" 0;
"~*(?:\b)YoudaoBot(?:\b|)" 0;
"~*(?:\b)Zade(?:\b|)" 0;
"~*(?:\b)Zauba(?:\b|)" 0;
"~*(?:\b)zauba.io(?:\b|)" 0;
"~*(?:\b)Zermelo(?:\b|)" 0;
"~*(?:\b)Zeus(?:\b|)" 0;
"~*(?:\b)zgrab(?:\b|)" 0;
"~*(?:\b)Zitebot(?:\b|)" 0;
"~*(?:\b)ZmEu(?:\b|)" 0;
"~*(?:\b)ZoomBot(?:\b|)" 0;
"~*(?:\b)ZoominfoBot(?:\b|)" 0;
"~*(?:\b)ZumBot(?:\b|)" 0;
"~*(?:\b)ZyBorg(?:\b|)" 0;
"~*(?:\b)archive.org_bot(?:\b|)" 0;
"~*(?:\b)arquivo-web-crawler(?:\b|)" 0;
"~*(?:\b)arquivo.pt(?:\b|)" 0;
"~*(?:\b)autoemailspider(?:\b|)" 0;
"~*(?:\b)backlink-check(?:\b|)" 0;
"~*(?:\b)check1.exe(?:\b|)" 0;
"~*(?:\b)coccocbot-web(?:\b|)" 0;
"~*(?:\b)cognitiveseo(?:\b|)" 0;
"~*(?:\b)com.plumanalytics(?:\b|)" 0;
"~*(?:\b)crawl.sogou.com(?:\b|)" 0;
"~*(?:\b)crawler.feedback(?:\b|)" 0;
"~*(?:\b)crawler4j(?:\b|)" 0;
"~*(?:\b)demandbase-bot(?:\b|)" 0;
"~*(?:\b)domainsproject.org(?:\b|)" 0;
"~*(?:\b)eCatch(?:\b|)" 0;
"~*(?:\b)evc-batch(?:\b|)" 0;
"~*(?:\b)facebookscraper(?:\b|)" 0;
"~*(?:\b)gopher(?:\b|)" 0;
"~*(?:\b)heritrix(?:\b|)" 0;
"~*(?:\b)instabid(?:\b|)" 0;
"~*(?:\b)internetVista\ monitor(?:\b|)" 0;
"~*(?:\b)ips-agent(?:\b|)" 0;
"~*(?:\b)isitwp.com(?:\b|)" 0;
"~*(?:\b)lwp-request(?:\b|)" 0;
"~*(?:\b)lwp-trivial(?:\b|)" 0;
"~*(?:\b)magpie-crawler(?:\b|)" 0;
"~*(?:\b)meanpathbot(?:\b|)" 0;
"~*(?:\b)mediawords(?:\b|)" 0;
"~*(?:\b)muhstik-scan(?:\b|)" 0;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0;
"~*(?:\b)oBot(?:\b|)" 0;
"~*(?:\b)page\ scorer(?:\b|)" 0;
"~*(?:\b)pcBrowser(?:\b|)" 0;
"~*(?:\b)plumanalytics(?:\b|)" 0;
"~*(?:\b)polaris\ version(?:\b|)" 0;
"~*(?:\b)probe-image-size(?:\b|)" 0;
"~*(?:\b)ripz(?:\b|)" 0;
"~*(?:\b)s1z.ru(?:\b|)" 0;
"~*(?:\b)satoristudio.net(?:\b|)" 0;
"~*(?:\b)scalaj-http(?:\b|)" 0;
"~*(?:\b)scan.lol(?:\b|)" 0;
"~*(?:\b)seobility(?:\b|)" 0;
"~*(?:\b)seoscanners(?:\b|)" 0;
"~*(?:\b)seostar(?:\b|)" 0;
"~*(?:\b)serpstatbot(?:\b|)" 0;
"~*(?:\b)sexsearcher(?:\b|)" 0;
"~*(?:\b)sitechecker.pro(?:\b|)" 0;
"~*(?:\b)siteripz(?:\b|)" 0;
"~*(?:\b)sogouspider(?:\b|)" 0;
"~*(?:\b)sp_auditbot(?:\b|)" 0;
"~*(?:\b)spyfu(?:\b|)" 0;
"~*(?:\b)sysscan(?:\b|)" 0;
"~*(?:\b)tAkeOut(?:\b|)" 0;
"~*(?:\b)trendiction.com(?:\b|)" 0;
"~*(?:\b)trendiction.de(?:\b|)" 0;
"~*(?:\b)voyagerx.com(?:\b|)" 0;
"~*(?:\b)webmeup-crawler(?:\b|)" 0;
"~*(?:\b)x09Mozilla(?:\b|)" 0;
"~*(?:\b)x22Mozilla(?:\b|)" 0;
"~*(?:\b)xpymep1.exe(?:\b|)" 0;
"~*(?:\b)zauba.io(?:\b|)" 0;
"~*(?:\b)zgrab(?:\b|)" 0;
# END MAKE BAD BOTS GOOD ### DO NOT EDIT THIS LINE AT ALL ###

4
dev-tools/beta_conf_files_whitelist/bots.d/custom-bad-referrers.conf
View file

@ -498,6 +498,7 @@
"~*(?:\b)anniemation\.com(?:\b)" 0;
"~*(?:\b)anonymizeme\.pro(?:\b)" 0;
"~*(?:\b)anonymous\-redirect\.com(?:\b)" 0;
"~*(?:\b)anonymousfox\.co(?:\b)" 0;
"~*(?:\b)anti\-virus\-removal\.info(?:\b)" 0;
"~*(?:\b)anticrawler\.org(?:\b)" 0;
"~*(?:\b)antons\-transporte\.de(?:\b)" 0;
@ -4637,6 +4638,7 @@
"~*(?:\b)pokemongooo\.ml(?:\b)" 0;
"~*(?:\b)pokerniydom\.ru(?:\b)" 0;
"~*(?:\b)polcin\.de(?:\b)" 0;
"~*(?:\b)poligon\.com(?:\b)" 0;
"~*(?:\b)polimga\.pw(?:\b)" 0;
"~*(?:\b)polska\-poezja\.com(?:\b)" 0;
"~*(?:\b)polybuild\.ru(?:\b)" 0;
@ -4693,6 +4695,7 @@
"~*(?:\b)pornogratisdiario\.com(?:\b)" 0;
"~*(?:\b)pornohd1080\.online(?:\b)" 0;
"~*(?:\b)pornohub\.me(?:\b)" 0;
"~*(?:\b)pornoinn\.com(?:\b)" 0;
"~*(?:\b)pornokajf\.com(?:\b)" 0;
"~*(?:\b)pornoklad\.net(?:\b)" 0;
"~*(?:\b)pornoklad\.ru(?:\b)" 0;
@ -6811,6 +6814,7 @@
"~*(?:\b)xlovecam\.com(?:\b)" 0;
"~*(?:\b)xmladserver\.com(?:\b)" 0;
"~*(?:\b)xmlinde\.com(?:\b)" 0;
"~*(?:\b)xmnb\.net(?:\b)" 0;
"~*(?:\b)xmronta\.com(?:\b)" 0;
"~*(?:\b)xn\-\-\-\-\-\-7cdbapdecfd4ak1bn0amjffj7afu3y\.xn\-\-p1ai(?:\b)" 0;
"~*(?:\b)xn\-\-\-\-\-6kcaabbafhu7cskl7akvongwpo7hvjj\.xn\-\-p1ai(?:\b)" 0;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
34.66.25.221 0;
10.1.0.59 0;
127.0.0.1 0;

15472
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/beta_conf_files_whitelist/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /etc/nginx/bots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2450
2451

16186
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

16164
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

4
dev-tools/referrers-regex-format-whitelist-test.txt
View file

@ -451,6 +451,7 @@
"~*(?:\b)anniemation\.com(?:\b)" 0;
"~*(?:\b)anonymizeme\.pro(?:\b)" 0;
"~*(?:\b)anonymous\-redirect\.com(?:\b)" 0;
"~*(?:\b)anonymousfox\.co(?:\b)" 0;
"~*(?:\b)anti\-virus\-removal\.info(?:\b)" 0;
"~*(?:\b)anticrawler\.org(?:\b)" 0;
"~*(?:\b)antons\-transporte\.de(?:\b)" 0;
@ -4590,6 +4591,7 @@
"~*(?:\b)pokemongooo\.ml(?:\b)" 0;
"~*(?:\b)pokerniydom\.ru(?:\b)" 0;
"~*(?:\b)polcin\.de(?:\b)" 0;
"~*(?:\b)poligon\.com(?:\b)" 0;
"~*(?:\b)polimga\.pw(?:\b)" 0;
"~*(?:\b)polska\-poezja\.com(?:\b)" 0;
"~*(?:\b)polybuild\.ru(?:\b)" 0;
@ -4646,6 +4648,7 @@
"~*(?:\b)pornogratisdiario\.com(?:\b)" 0;
"~*(?:\b)pornohd1080\.online(?:\b)" 0;
"~*(?:\b)pornohub\.me(?:\b)" 0;
"~*(?:\b)pornoinn\.com(?:\b)" 0;
"~*(?:\b)pornokajf\.com(?:\b)" 0;
"~*(?:\b)pornoklad\.net(?:\b)" 0;
"~*(?:\b)pornoklad\.ru(?:\b)" 0;
@ -6764,6 +6767,7 @@
"~*(?:\b)xlovecam\.com(?:\b)" 0;
"~*(?:\b)xmladserver\.com(?:\b)" 0;
"~*(?:\b)xmlinde\.com(?:\b)" 0;
"~*(?:\b)xmnb\.net(?:\b)" 0;
"~*(?:\b)xmronta\.com(?:\b)" 0;
"~*(?:\b)xn\-\-\-\-\-\-7cdbapdecfd4ak1bn0amjffj7afu3y\.xn\-\-p1ai(?:\b)" 0;
"~*(?:\b)xn\-\-\-\-\-6kcaabbafhu7cskl7akvongwpo7hvjj\.xn\-\-p1ai(?:\b)" 0;

1278
dev-tools/referrers-regex-format.txt
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
34.66.200.49 0;
52.170.46.75 0;

16164
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/test1_conf_files/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /etc/nginx/bots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.10.37 1;
10.1.0.59 1;
127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.10.37 0;
10.1.0.59 0;
127.0.0.1 0;

16164
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/test1_conf_files_ip_whitelist/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /etc/nginx/bots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.10.37 1;
10.1.0.59 1;
127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.10.37 0;
10.1.0.59 0;
127.0.0.1 0;

16164
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/test1_conf_files_ratelimiting/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /etc/nginx/bots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.10.37 1;
10.1.0.59 1;
127.0.0.1 1;

368
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -64,19 +64,20 @@
"~*(?:\b)404checker(?:\b|)" 0;
"~*(?:\b)404enemy(?:\b|)" 0;
"~*(?:\b)80legs(?:\b|)" 0;
"~*(?:\b)ADmantX(?:\b|)" 0;
"~*(?:\b)AIBOT(?:\b|)" 0;
"~*(?:\b)ASPSeek(?:\b|)" 0;
"~*(?:\b)Abonti(?:\b|)" 0;
"~*(?:\b)Aboundex(?:\b|)" 0;
"~*(?:\b)Aboundexbot(?:\b|)" 0;
"~*(?:\b)Acunetix(?:\b|)" 0;
"~*(?:\b)ADmantX(?:\b|)" 0;
"~*(?:\b)AfD-Verbotsverfahren(?:\b|)" 0;
"~*(?:\b)AhrefsBot(?:\b|)" 0;
"~*(?:\b)AIBOT(?:\b|)" 0;
"~*(?:\b)AiHitBot(?:\b|)" 0;
"~*(?:\b)Aipbot(?:\b|)" 0;
"~*(?:\b)Alexibot(?:\b|)" 0;
"~*(?:\b)Alligator(?:\b|)" 0;
"~*(?:\b)AllSubmitter(?:\b|)" 0;
"~*(?:\b)Alligator(?:\b|)" 0;
"~*(?:\b)AlphaBot(?:\b|)" 0;
"~*(?:\b)Anarchie(?:\b|)" 0;
"~*(?:\b)Anarchy(?:\b|)" 0;
@ -84,37 +85,31 @@
"~*(?:\b)Ankit(?:\b|)" 0;
"~*(?:\b)Anthill(?:\b|)" 0;
"~*(?:\b)Apexoo(?:\b|)" 0;
"~*(?:\b)archive.org_bot(?:\b|)" 0;
"~*(?:\b)arquivo.pt(?:\b|)" 0;
"~*(?:\b)arquivo-web-crawler(?:\b|)" 0;
"~*(?:\b)Aspiegel(?:\b|)" 0;
"~*(?:\b)ASPSeek(?:\b|)" 0;
"~*(?:\b)Asterias(?:\b|)" 0;
"~*(?:\b)Attach(?:\b|)" 0;
"~*(?:\b)autoemailspider(?:\b|)" 0;
"~*(?:\b)AwarioRssBot(?:\b|)" 0;
"~*(?:\b)AwarioSmartBot(?:\b|)" 0;
"~*(?:\b)BBBike(?:\b|)" 0;
"~*(?:\b)BDCbot(?:\b|)" 0;
"~*(?:\b)BDFetch(?:\b|)" 0;
"~*(?:\b)BLEXBot(?:\b|)" 0;
"~*(?:\b)BackDoorBot(?:\b|)" 0;
"~*(?:\b)Backlink-Ceck(?:\b|)" 0;
"~*(?:\b)backlink-check(?:\b|)" 0;
"~*(?:\b)BacklinkCrawler(?:\b|)" 0;
"~*(?:\b)BackStreet(?:\b|)" 0;
"~*(?:\b)BackWeb(?:\b|)" 0;
"~*(?:\b)Backlink-Ceck(?:\b|)" 0;
"~*(?:\b)BacklinkCrawler(?:\b|)" 0;
"~*(?:\b)Badass(?:\b|)" 0;
"~*(?:\b)Bandit(?:\b|)" 0;
"~*(?:\b)Barkrowler(?:\b|)" 0;
"~*(?:\b)BatchFTP(?:\b|)" 0;
"~*(?:\b)Battleztar\ Bazinga(?:\b|)" 0;
"~*(?:\b)BBBike(?:\b|)" 0;
"~*(?:\b)BDCbot(?:\b|)" 0;
"~*(?:\b)BDFetch(?:\b|)" 0;
"~*(?:\b)BetaBot(?:\b|)" 0;
"~*(?:\b)Bigfoot(?:\b|)" 0;
"~*(?:\b)Bitacle(?:\b|)" 0;
"~*(?:\b)Blackboard(?:\b|)" 0;
"~*(?:\b)Black\ Hole(?:\b|)" 0;
"~*(?:\b)BlackWidow(?:\b|)" 0;
"~*(?:\b)BLEXBot(?:\b|)" 0;
"~*(?:\b)Black\ Hole(?:\b|)" 0;
"~*(?:\b)Blackboard(?:\b|)" 0;
"~*(?:\b)Blow(?:\b|)" 0;
"~*(?:\b)BlowFish(?:\b|)" 0;
"~*(?:\b)Boardreader(?:\b|)" 0;
@ -129,53 +124,47 @@
"~*(?:\b)Bullseye(?:\b|)" 0;
"~*(?:\b)BunnySlippers(?:\b|)" 0;
"~*(?:\b)BuzzSumo(?:\b|)" 0;
"~*(?:\b)Calculon(?:\b|)" 0;
"~*(?:\b)CATExplorador(?:\b|)" 0;
"~*(?:\b)CazoodleBot(?:\b|)" 0;
"~*(?:\b)CCBot(?:\b|)" 0;
"~*(?:\b)CODE87(?:\b|)" 0;
"~*(?:\b)CSHttp(?:\b|)" 0;
"~*(?:\b)Calculon(?:\b|)" 0;
"~*(?:\b)CazoodleBot(?:\b|)" 0;
"~*(?:\b)Cegbfeieh(?:\b|)" 0;
"~*(?:\b)CensysInspect(?:\b|)" 0;
"~*(?:\b)check1.exe(?:\b|)" 0;
"~*(?:\b)CheTeam(?:\b|)" 0;
"~*(?:\b)CheeseBot(?:\b|)" 0;
"~*(?:\b)CherryPicker(?:\b|)" 0;
"~*(?:\b)CheTeam(?:\b|)" 0;
"~*(?:\b)ChinaClaw(?:\b|)" 0;
"~*(?:\b)Chlooe(?:\b|)" 0;
"~*(?:\b)Claritybot(?:\b|)" 0;
"~*(?:\b)Cliqzbot(?:\b|)" 0;
"~*(?:\b)Cloud\ mapping(?:\b|)" 0;
"~*(?:\b)coccocbot-web(?:\b|)" 0;
"~*(?:\b)Cocolyzebot(?:\b|)" 0;
"~*(?:\b)CODE87(?:\b|)" 0;
"~*(?:\b)Cogentbot(?:\b|)" 0;
"~*(?:\b)cognitiveseo(?:\b|)" 0;
"~*(?:\b)Collector(?:\b|)" 0;
"~*(?:\b)com.plumanalytics(?:\b|)" 0;
"~*(?:\b)Copier(?:\b|)" 0;
"~*(?:\b)CopyRightCheck(?:\b|)" 0;
"~*(?:\b)Copyscape(?:\b|)" 0;
"~*(?:\b)Cosmos(?:\b|)" 0;
"~*(?:\b)Craftbot(?:\b|)" 0;
"~*(?:\b)crawler4j(?:\b|)" 0;
"~*(?:\b)crawler.feedback(?:\b|)" 0;
"~*(?:\b)crawl.sogou.com(?:\b|)" 0;
"~*(?:\b)CrazyWebCrawler(?:\b|)" 0;
"~*(?:\b)Crescent(?:\b|)" 0;
"~*(?:\b)CrunchBot(?:\b|)" 0;
"~*(?:\b)CSHttp(?:\b|)" 0;
"~*(?:\b)Curious(?:\b|)" 0;
"~*(?:\b)Custo(?:\b|)" 0;
"~*(?:\b)CyotekWebCopy(?:\b|)" 0;
"~*(?:\b)DatabaseDriverMysqli(?:\b|)" 0;
"~*(?:\b)DataCha0s(?:\b|)" 0;
"~*(?:\b)DBLBot(?:\b|)" 0;
"~*(?:\b)demandbase-bot(?:\b|)" 0;
"~*(?:\b)DIIbot(?:\b|)" 0;
"~*(?:\b)DSearch(?:\b|)" 0;
"~*(?:\b)DTS\ Agent(?:\b|)" 0;
"~*(?:\b)DataCha0s(?:\b|)" 0;
"~*(?:\b)DatabaseDriverMysqli(?:\b|)" 0;
"~*(?:\b)Demon(?:\b|)" 0;
"~*(?:\b)Deusu(?:\b|)" 0;
"~*(?:\b)Devil(?:\b|)" 0;
"~*(?:\b)Digincore(?:\b|)" 0;
"~*(?:\b)DigitalPebble(?:\b|)" 0;
"~*(?:\b)DIIbot(?:\b|)" 0;
"~*(?:\b)Dirbuster(?:\b|)" 0;
"~*(?:\b)Disco(?:\b|)" 0;
"~*(?:\b)Discobot(?:\b|)" 0;
@ -183,29 +172,24 @@
"~*(?:\b)Dispatch(?:\b|)" 0;
"~*(?:\b)DittoSpyder(?:\b|)" 0;
"~*(?:\b)DnyzBot(?:\b|)" 0;
"~*(?:\b)DomCopBot(?:\b|)" 0;
"~*(?:\b)DomainAppender(?:\b|)" 0;
"~*(?:\b)DomainCrawler(?:\b|)" 0;
"~*(?:\b)DomainSigmaCrawler(?:\b|)" 0;
"~*(?:\b)Domains\ Project(?:\b|)" 0;
"~*(?:\b)domainsproject.org(?:\b|)" 0;
"~*(?:\b)DomainStatsBot(?:\b|)" 0;
"~*(?:\b)DomCopBot(?:\b|)" 0;
"~*(?:\b)Domains\ Project(?:\b|)" 0;
"~*(?:\b)Dotbot(?:\b|)" 0;
"~*(?:\b)Download\ Wonder(?:\b|)" 0;
"~*(?:\b)Dragonfly(?:\b|)" 0;
"~*(?:\b)Drip(?:\b|)" 0;
"~*(?:\b)DSearch(?:\b|)" 0;
"~*(?:\b)DTS\ Agent(?:\b|)" 0;
"~*(?:\b)EasyDL(?:\b|)" 0;
"~*(?:\b)Ebingbong(?:\b|)" 0;
"~*(?:\b)eCatch(?:\b|)" 0;
"~*(?:\b)ECCP/1.0(?:\b|)" 0;
"~*(?:\b)Ecxi(?:\b|)" 0;
"~*(?:\b)EirGrabber(?:\b|)" 0;
"~*(?:\b)EMail\ Siphon(?:\b|)" 0;
"~*(?:\b)EMail\ Wolf(?:\b|)" 0;
"~*(?:\b)EasyDL(?:\b|)" 0;
"~*(?:\b)Ebingbong(?:\b|)" 0;
"~*(?:\b)Ecxi(?:\b|)" 0;
"~*(?:\b)EirGrabber(?:\b|)" 0;
"~*(?:\b)EroCrawler(?:\b|)" 0;
"~*(?:\b)evc-batch(?:\b|)" 0;
"~*(?:\b)Evil(?:\b|)" 0;
"~*(?:\b)Exabot(?:\b|)" 0;
"~*(?:\b)Express\ WebPictures(?:\b|)" 0;
@ -215,10 +199,9 @@
"~*(?:\b)Extreme\ Picture\ Finder(?:\b|)" 0;
"~*(?:\b)EyeNetIE(?:\b|)" 0;
"~*(?:\b)Ezooms(?:\b|)" 0;
"~*(?:\b)facebookscraper(?:\b|)" 0;
"~*(?:\b)FDM(?:\b|)" 0;
"~*(?:\b)FemtosearchBot(?:\b|)" 0;
"~*(?:\b)FHscan(?:\b|)" 0;
"~*(?:\b)FemtosearchBot(?:\b|)" 0;
"~*(?:\b)Fimap(?:\b|)" 0;
"~*(?:\b)Firefox/7.0(?:\b|)" 0;
"~*(?:\b)FlashGet(?:\b|)" 0;
@ -229,44 +212,43 @@
"~*(?:\b)Fuzz(?:\b|)" 0;
"~*(?:\b)FyberSpider(?:\b|)" 0;
"~*(?:\b)Fyrebot(?:\b|)" 0;
"~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0;
"~*(?:\b)GT::WWW(?:\b|)" 0;
"~*(?:\b)GalaxyBot(?:\b|)" 0;
"~*(?:\b)Genieo(?:\b|)" 0;
"~*(?:\b)GermCrawler(?:\b|)" 0;
"~*(?:\b)Getintent(?:\b|)" 0;
"~*(?:\b)GetRight(?:\b|)" 0;
"~*(?:\b)GetWeb(?:\b|)" 0;
"~*(?:\b)Getintent(?:\b|)" 0;
"~*(?:\b)Gigablast(?:\b|)" 0;
"~*(?:\b)Gigabot(?:\b|)" 0;
"~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0;
"~*(?:\b)Go-Ahead-Got-It(?:\b|)" 0;
"~*(?:\b)gopher(?:\b|)" 0;
"~*(?:\b)Gotit(?:\b|)" 0;
"~*(?:\b)GoZilla(?:\b|)" 0;
"~*(?:\b)Go!Zilla(?:\b|)" 0;
"~*(?:\b)Grabber(?:\b|)" 0;
"~*(?:\b)Go-Ahead-Got-It(?:\b|)" 0;
"~*(?:\b)GoZilla(?:\b|)" 0;
"~*(?:\b)Gotit(?:\b|)" 0;
"~*(?:\b)GrabNet(?:\b|)" 0;
"~*(?:\b)Grabber(?:\b|)" 0;
"~*(?:\b)Grafula(?:\b|)" 0;
"~*(?:\b)GrapeFX(?:\b|)" 0;
"~*(?:\b)GrapeshotCrawler(?:\b|)" 0;
"~*(?:\b)GridBot(?:\b|)" 0;
"~*(?:\b)GT::WWW(?:\b|)" 0;
"~*(?:\b)Haansoft(?:\b|)" 0;
"~*(?:\b)HaosouSpider(?:\b|)" 0;
"~*(?:\b)Harvest(?:\b|)" 0;
"~*(?:\b)Havij(?:\b|)" 0;
"~*(?:\b)HEADMasterSEO(?:\b|)" 0;
"~*(?:\b)heritrix(?:\b|)" 0;
"~*(?:\b)Heritrix(?:\b|)" 0;
"~*(?:\b)Hloader(?:\b|)" 0;
"~*(?:\b)HMView(?:\b|)" 0;
"~*(?:\b)HTMLparser(?:\b|)" 0;
"~*(?:\b)HTTP::Lite(?:\b|)" 0;
"~*(?:\b)HTTrack(?:\b|)" 0;
"~*(?:\b)Haansoft(?:\b|)" 0;
"~*(?:\b)HaosouSpider(?:\b|)" 0;
"~*(?:\b)Harvest(?:\b|)" 0;
"~*(?:\b)Havij(?:\b|)" 0;
"~*(?:\b)Heritrix(?:\b|)" 0;
"~*(?:\b)Hloader(?:\b|)" 0;
"~*(?:\b)Humanlinks(?:\b|)" 0;
"~*(?:\b)HybridBot(?:\b|)" 0;
"~*(?:\b)Iblog(?:\b|)" 0;
"~*(?:\b)IDBot(?:\b|)" 0;
"~*(?:\b)IDBTE4M(?:\b|)" 0;
"~*(?:\b)IDBot(?:\b|)" 0;
"~*(?:\b)IRLbot(?:\b|)" 0;
"~*(?:\b)Iblog(?:\b|)" 0;
"~*(?:\b)Id-search(?:\b|)" 0;
"~*(?:\b)IlseBot(?:\b|)" 0;
"~*(?:\b)Image\ Fetch(?:\b|)" 0;
@ -275,25 +257,20 @@
"~*(?:\b)Indy\ Library(?:\b|)" 0;
"~*(?:\b)InfoNaviRobot(?:\b|)" 0;
"~*(?:\b)InfoTekies(?:\b|)" 0;
"~*(?:\b)instabid(?:\b|)" 0;
"~*(?:\b)Intelliseek(?:\b|)" 0;
"~*(?:\b)InterGET(?:\b|)" 0;
"~*(?:\b)Internet\ Ninja(?:\b|)" 0;
"~*(?:\b)InternetSeer(?:\b|)" 0;
"~*(?:\b)internetVista\ monitor(?:\b|)" 0;
"~*(?:\b)ips-agent(?:\b|)" 0;
"~*(?:\b)Internet\ Ninja(?:\b|)" 0;
"~*(?:\b)Iria(?:\b|)" 0;
"~*(?:\b)IRLbot(?:\b|)" 0;
"~*(?:\b)isitwp.com(?:\b|)" 0;
"~*(?:\b)Iskanie(?:\b|)" 0;
"~*(?:\b)IstellaBot(?:\b|)" 0;
"~*(?:\b)JOC\ Web\ Spider(?:\b|)" 0;
"~*(?:\b)JamesBOT(?:\b|)" 0;
"~*(?:\b)Jbrofuzz(?:\b|)" 0;
"~*(?:\b)JennyBot(?:\b|)" 0;
"~*(?:\b)JetCar(?:\b|)" 0;
"~*(?:\b)Jetty(?:\b|)" 0;
"~*(?:\b)JikeSpider(?:\b|)" 0;
"~*(?:\b)JOC\ Web\ Spider(?:\b|)" 0;
"~*(?:\b)Joomla(?:\b|)" 0;
"~*(?:\b)Jorgee(?:\b|)" 0;
"~*(?:\b)JustView(?:\b|)" 0;
@ -302,6 +279,8 @@
"~*(?:\b)Keyword\ Density(?:\b|)" 0;
"~*(?:\b)Kinza(?:\b|)" 0;
"~*(?:\b)Kozmosbot(?:\b|)" 0;
"~*(?:\b)LNSpiderguy(?:\b|)" 0;
"~*(?:\b)LWP::Simple(?:\b|)" 0;
"~*(?:\b)Lanshanbot(?:\b|)" 0;
"~*(?:\b)Larbin(?:\b|)" 0;
"~*(?:\b)LeechFTP(?:\b|)" 0;
@ -313,13 +292,13 @@
"~*(?:\b)LieBaoFast(?:\b|)" 0;
"~*(?:\b)Lightspeedsystems(?:\b|)" 0;
"~*(?:\b)Likse(?:\b|)" 0;
"~*(?:\b)LinkScan(?:\b|)" 0;
"~*(?:\b)LinkWalker(?:\b|)" 0;
"~*(?:\b)Linkbot(?:\b|)" 0;
"~*(?:\b)Linkdexbot(?:\b|)" 0;
"~*(?:\b)LinkextractorPro(?:\b|)" 0;
"~*(?:\b)LinkpadBot(?:\b|)" 0;
"~*(?:\b)LinkScan(?:\b|)" 0;
"~*(?:\b)LinksManager(?:\b|)" 0;
"~*(?:\b)LinkWalker(?:\b|)" 0;
"~*(?:\b)LinqiaMetadataDownloaderBot(?:\b|)" 0;
"~*(?:\b)LinqiaRSSBot(?:\b|)" 0;
"~*(?:\b)LinqiaScrapeBot(?:\b|)" 0;
@ -327,53 +306,47 @@
"~*(?:\b)Lipperhey\ Spider(?:\b|)" 0;
"~*(?:\b)Litemage_walker(?:\b|)" 0;
"~*(?:\b)Lmspider(?:\b|)" 0;
"~*(?:\b)LNSpiderguy(?:\b|)" 0;
"~*(?:\b)Ltx71(?:\b|)" 0;
"~*(?:\b)lwp-request(?:\b|)" 0;
"~*(?:\b)LWP::Simple(?:\b|)" 0;
"~*(?:\b)lwp-trivial(?:\b|)" 0;
"~*(?:\b)Magnet(?:\b|)" 0;
"~*(?:\b)MFC_Tear_Sample(?:\b|)" 0;
"~*(?:\b)MIDown\ tool(?:\b|)" 0;
"~*(?:\b)MIIxpc(?:\b|)" 0;
"~*(?:\b)MJ12bot(?:\b|)" 0;
"~*(?:\b)MQQBrowser(?:\b|)" 0;
"~*(?:\b)MSFrontPage(?:\b|)" 0;
"~*(?:\b)MSIECrawler(?:\b|)" 0;
"~*(?:\b)MTRobot(?:\b|)" 0;
"~*(?:\b)Mag-Net(?:\b|)" 0;
"~*(?:\b)magpie-crawler(?:\b|)" 0;
"~*(?:\b)Magnet(?:\b|)" 0;
"~*(?:\b)Mail.RU_Bot(?:\b|)" 0;
"~*(?:\b)Majestic12(?:\b|)" 0;
"~*(?:\b)Majestic-SEO(?:\b|)" 0;
"~*(?:\b)Majestic12(?:\b|)" 0;
"~*(?:\b)Majestic\ SEO(?:\b|)" 0;
"~*(?:\b)MarkMonitor(?:\b|)" 0;
"~*(?:\b)MarkWatch(?:\b|)" 0;
"~*(?:\b)Masscan(?:\b|)" 0;
"~*(?:\b)Mass\ Downloader(?:\b|)" 0;
"~*(?:\b)Masscan(?:\b|)" 0;
"~*(?:\b)Mata\ Hari(?:\b|)" 0;
"~*(?:\b)MauiBot(?:\b|)" 0;
"~*(?:\b)Mb2345Browser(?:\b|)" 0;
"~*(?:\b)meanpathbot(?:\b|)" 0;
"~*(?:\b)Meanpathbot(?:\b|)" 0;
"~*(?:\b)MeanPath\ Bot(?:\b|)" 0;
"~*(?:\b)Meanpathbot(?:\b|)" 0;
"~*(?:\b)Mediatoolkitbot(?:\b|)" 0;
"~*(?:\b)mediawords(?:\b|)" 0;
"~*(?:\b)MegaIndex.ru(?:\b|)" 0;
"~*(?:\b)Metauri(?:\b|)" 0;
"~*(?:\b)MFC_Tear_Sample(?:\b|)" 0;
"~*(?:\b)MicroMessenger(?:\b|)" 0;
"~*(?:\b)Microsoft\ Data\ Access(?:\b|)" 0;
"~*(?:\b)Microsoft\ URL\ Control(?:\b|)" 0;
"~*(?:\b)MIDown\ tool(?:\b|)" 0;
"~*(?:\b)MIIxpc(?:\b|)" 0;
"~*(?:\b)Mister\ PiX(?:\b|)" 0;
"~*(?:\b)MJ12bot(?:\b|)" 0;
"~*(?:\b)Moblie Safari(?:\b|)" 0;
"~*(?:\b)Mojeek(?:\b|)" 0;
"~*(?:\b)Mojolicious(?:\b|)" 0;
"~*(?:\b)Morfeus\ Fucking\ Scanner(?:\b|)" 0;
"~*(?:\b)Mozlila(?:\b|)" 0;
"~*(?:\b)MQQBrowser(?:\b|)" 0;
"~*(?:\b)Mr.4x3(?:\b|)" 0;
"~*(?:\b)MSFrontPage(?:\b|)" 0;
"~*(?:\b)MSIECrawler(?:\b|)" 0;
"~*(?:\b)Msrabot(?:\b|)" 0;
"~*(?:\b)MTRobot(?:\b|)" 0;
"~*(?:\b)muhstik-scan(?:\b|)" 0;
"~*(?:\b)Musobot(?:\b|)" 0;
"~*(?:\b)NICErsPRO(?:\b|)" 0;
"~*(?:\b)NPbot(?:\b|)" 0;
"~*(?:\b)Name\ Intelligence(?:\b|)" 0;
"~*(?:\b)Nameprotect(?:\b|)" 0;
"~*(?:\b)Navroad(?:\b|)" 0;
@ -381,72 +354,63 @@
"~*(?:\b)Needle(?:\b|)" 0;
"~*(?:\b)Nessus(?:\b|)" 0;
"~*(?:\b)NetAnts(?:\b|)" 0;
"~*(?:\b)Netcraft(?:\b|)" 0;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0;
"~*(?:\b)NetLyzer(?:\b|)" 0;
"~*(?:\b)NetMechanic(?:\b|)" 0;
"~*(?:\b)NetSpider(?:\b|)" 0;
"~*(?:\b)Nettrack(?:\b|)" 0;
"~*(?:\b)Net\ Vampire(?:\b|)" 0;
"~*(?:\b)Netvibes(?:\b|)" 0;
"~*(?:\b)NetZIP(?:\b|)" 0;
"~*(?:\b)Net\ Vampire(?:\b|)" 0;
"~*(?:\b)Netcraft(?:\b|)" 0;
"~*(?:\b)Nettrack(?:\b|)" 0;
"~*(?:\b)Netvibes(?:\b|)" 0;
"~*(?:\b)NextGenSearchBot(?:\b|)" 0;
"~*(?:\b)Nibbler(?:\b|)" 0;
"~*(?:\b)NICErsPRO(?:\b|)" 0;
"~*(?:\b)Niki-bot(?:\b|)" 0;
"~*(?:\b)Nikto(?:\b|)" 0;
"~*(?:\b)NimbleCrawler(?:\b|)" 0;
"~*(?:\b)Nimbostratus(?:\b|)" 0;
"~*(?:\b)Ninja(?:\b|)" 0;
"~*(?:\b)Nmap(?:\b|)" 0;
"~*(?:\b)NPbot(?:\b|)" 0;
"~*(?:\b)Nuclei(?:\b|)" 0;
"~*(?:\b)Nutch(?:\b|)" 0;
"~*(?:\b)oBot(?:\b|)" 0;
"~*(?:\b)OPPO\ A33(?:\b|)" 0;
"~*(?:\b)Octopus(?:\b|)" 0;
"~*(?:\b)Offline\ Explorer(?:\b|)" 0;
"~*(?:\b)Offline\ Navigator(?:\b|)" 0;
"~*(?:\b)OnCrawl(?:\b|)" 0;
"~*(?:\b)Openfind(?:\b|)" 0;
"~*(?:\b)OpenLinkProfiler(?:\b|)" 0;
"~*(?:\b)Openvas(?:\b|)" 0;
"~*(?:\b)OpenVAS(?:\b|)" 0;
"~*(?:\b)OPPO\ A33(?:\b|)" 0;
"~*(?:\b)Openfind(?:\b|)" 0;
"~*(?:\b)Openvas(?:\b|)" 0;
"~*(?:\b)OrangeBot(?:\b|)" 0;
"~*(?:\b)OrangeSpider(?:\b|)" 0;
"~*(?:\b)OutclicksBot(?:\b|)" 0;
"~*(?:\b)OutfoxBot(?:\b|)" 0;
"~*(?:\b)PECL::HTTP(?:\b|)" 0;
"~*(?:\b)PHPCrawl(?:\b|)" 0;
"~*(?:\b)POE-Component-Client-HTTP(?:\b|)" 0;
"~*(?:\b)PageAnalyzer(?:\b|)" 0;
"~*(?:\b)Page\ Analyzer(?:\b|)" 0;
"~*(?:\b)PageGrabber(?:\b|)" 0;
"~*(?:\b)page\ scorer(?:\b|)" 0;
"~*(?:\b)PageScorer(?:\b|)" 0;
"~*(?:\b)PageThing.com(?:\b|)" 0;
"~*(?:\b)Page\ Analyzer(?:\b|)" 0;
"~*(?:\b)Pandalytics(?:\b|)" 0;
"~*(?:\b)Panscient(?:\b|)" 0;
"~*(?:\b)Papa\ Foto(?:\b|)" 0;
"~*(?:\b)Pavuk(?:\b|)" 0;
"~*(?:\b)pcBrowser(?:\b|)" 0;
"~*(?:\b)PECL::HTTP(?:\b|)" 0;
"~*(?:\b)PeoplePal(?:\b|)" 0;
"~*(?:\b)Petalbot(?:\b|)" 0;
"~*(?:\b)PHPCrawl(?:\b|)" 0;
"~*(?:\b)Pi-Monster(?:\b|)" 0;
"~*(?:\b)Picscout(?:\b|)" 0;
"~*(?:\b)Picsearch(?:\b|)" 0;
"~*(?:\b)PictureFinder(?:\b|)" 0;
"~*(?:\b)Piepmatz(?:\b|)" 0;
"~*(?:\b)Pimonster(?:\b|)" 0;
"~*(?:\b)Pi-Monster(?:\b|)" 0;
"~*(?:\b)Pixray(?:\b|)" 0;
"~*(?:\b)PleaseCrawl(?:\b|)" 0;
"~*(?:\b)plumanalytics(?:\b|)" 0;
"~*(?:\b)Pockey(?:\b|)" 0;
"~*(?:\b)POE-Component-Client-HTTP(?:\b|)" 0;
"~*(?:\b)polaris\ version(?:\b|)" 0;
"~*(?:\b)probe-image-size(?:\b|)" 0;
"~*(?:\b)Probethenet(?:\b|)" 0;
"~*(?:\b)ProPowerBot(?:\b|)" 0;
"~*(?:\b)ProWebWalker(?:\b|)" 0;
"~*(?:\b)Probethenet(?:\b|)" 0;
"~*(?:\b)Psbot(?:\b|)" 0;
"~*(?:\b)Pu_iN(?:\b|)" 0;
"~*(?:\b)Pump(?:\b|)" 0;
@ -454,6 +418,7 @@
"~*(?:\b)PyCurl(?:\b|)" 0;
"~*(?:\b)QueryN\ Metasearch(?:\b|)" 0;
"~*(?:\b)Quick-Crawler(?:\b|)" 0;
"~*(?:\b)RSSingBot(?:\b|)" 0;
"~*(?:\b)RankActive(?:\b|)" 0;
"~*(?:\b)RankActiveLinkBot(?:\b|)" 0;
"~*(?:\b)RankFlex(?:\b|)" 0;
@ -461,27 +426,28 @@
"~*(?:\b)RankingBot2(?:\b|)" 0;
"~*(?:\b)Rankivabot(?:\b|)" 0;
"~*(?:\b)RankurBot(?:\b|)" 0;
"~*(?:\b)Re-re(?:\b|)" 0;
"~*(?:\b)ReGet(?:\b|)" 0;
"~*(?:\b)RealDownload(?:\b|)" 0;
"~*(?:\b)Reaper(?:\b|)" 0;
"~*(?:\b)RebelMouse(?:\b|)" 0;
"~*(?:\b)Recorder(?:\b|)" 0;
"~*(?:\b)RedesScrapy(?:\b|)" 0;
"~*(?:\b)ReGet(?:\b|)" 0;
"~*(?:\b)RepoMonkey(?:\b|)" 0;
"~*(?:\b)Re-re(?:\b|)" 0;
"~*(?:\b)Ripper(?:\b|)" 0;
"~*(?:\b)ripz(?:\b|)" 0;
"~*(?:\b)RocketCrawler(?:\b|)" 0;
"~*(?:\b)Rogerbot(?:\b|)" 0;
"~*(?:\b)RSSingBot(?:\b|)" 0;
"~*(?:\b)s1z.ru(?:\b|)" 0;
"~*(?:\b)SalesIntelligent(?:\b|)" 0;
"~*(?:\b)satoristudio.net(?:\b|)" 0;
"~*(?:\b)SBIder(?:\b|)" 0;
"~*(?:\b)scalaj-http(?:\b|)" 0;
"~*(?:\b)SEOkicks(?:\b|)" 0;
"~*(?:\b)SEOkicks-Robot(?:\b|)" 0;
"~*(?:\b)SEOlyticsCrawler(?:\b|)" 0;
"~*(?:\b)SEOprofiler(?:\b|)" 0;
"~*(?:\b)SEOstats(?:\b|)" 0;
"~*(?:\b)SISTRIX(?:\b|)" 0;
"~*(?:\b)SMTBot(?:\b|)" 0;
"~*(?:\b)SalesIntelligent(?:\b|)" 0;
"~*(?:\b)ScanAlert(?:\b|)" 0;
"~*(?:\b)Scanbot(?:\b|)" 0;
"~*(?:\b)scan.lol(?:\b|)" 0;
"~*(?:\b)ScoutJet(?:\b|)" 0;
"~*(?:\b)Scrapy(?:\b|)" 0;
"~*(?:\b)Screaming(?:\b|)" 0;
@ -492,42 +458,27 @@
"~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0;
"~*(?:\b)SentiBot(?:\b|)" 0;
"~*(?:\b)seobility(?:\b|)" 0;
"~*(?:\b)SeobilityBot(?:\b|)" 0;
"~*(?:\b)SEOkicks(?:\b|)" 0;
"~*(?:\b)SEOkicks-Robot(?:\b|)" 0;
"~*(?:\b)SEOlyticsCrawler(?:\b|)" 0;
"~*(?:\b)Seomoz(?:\b|)" 0;
"~*(?:\b)SEOprofiler(?:\b|)" 0;
"~*(?:\b)seoscanners(?:\b|)" 0;
"~*(?:\b)SeoSiteCheckup(?:\b|)" 0;
"~*(?:\b)seostar(?:\b|)" 0;
"~*(?:\b)SEOstats(?:\b|)" 0;
"~*(?:\b)serpstatbot(?:\b|)" 0;
"~*(?:\b)sexsearcher(?:\b|)" 0;
"~*(?:\b)SeobilityBot(?:\b|)" 0;
"~*(?:\b)Seomoz(?:\b|)" 0;
"~*(?:\b)Shodan(?:\b|)" 0;
"~*(?:\b)Siphon(?:\b|)" 0;
"~*(?:\b)SISTRIX(?:\b|)" 0;
"~*(?:\b)Sitebeam(?:\b|)" 0;
"~*(?:\b)SiteCheckerBotCrawler(?:\b|)" 0;
"~*(?:\b)sitechecker.pro(?:\b|)" 0;
"~*(?:\b)SiteExplorer(?:\b|)" 0;
"~*(?:\b)Siteimprove(?:\b|)" 0;
"~*(?:\b)SiteLockSpider(?:\b|)" 0;
"~*(?:\b)siteripz(?:\b|)" 0;
"~*(?:\b)SiteSnagger(?:\b|)" 0;
"~*(?:\b)SiteSucker(?:\b|)" 0;
"~*(?:\b)Site\ Sucker(?:\b|)" 0;
"~*(?:\b)Sitebeam(?:\b|)" 0;
"~*(?:\b)Siteimprove(?:\b|)" 0;
"~*(?:\b)Sitevigil(?:\b|)" 0;
"~*(?:\b)SlySearch(?:\b|)" 0;
"~*(?:\b)SmartDownload(?:\b|)" 0;
"~*(?:\b)SMTBot(?:\b|)" 0;
"~*(?:\b)Snake(?:\b|)" 0;
"~*(?:\b)Snapbot(?:\b|)" 0;
"~*(?:\b)Snoopy(?:\b|)" 0;
"~*(?:\b)SocialRankIOBot(?:\b|)" 0;
"~*(?:\b)Sociscraper(?:\b|)" 0;
"~*(?:\b)sogouspider(?:\b|)" 0;
"~*(?:\b)Sogou\ web\ spider(?:\b|)" 0;
"~*(?:\b)Sosospider(?:\b|)" 0;
"~*(?:\b)Sottopop(?:\b|)" 0;
@ -535,11 +486,9 @@
"~*(?:\b)Spammen(?:\b|)" 0;
"~*(?:\b)SpankBot(?:\b|)" 0;
"~*(?:\b)Spanner(?:\b|)" 0;
"~*(?:\b)sp_auditbot(?:\b|)" 0;
"~*(?:\b)Spbot(?:\b|)" 0;
"~*(?:\b)Spinn3r(?:\b|)" 0;
"~*(?:\b)SputnikBot(?:\b|)" 0;
"~*(?:\b)spyfu(?:\b|)" 0;
"~*(?:\b)Sqlmap(?:\b|)" 0;
"~*(?:\b)Sqlworm(?:\b|)" 0;
"~*(?:\b)Sqworm(?:\b|)" 0;
@ -553,18 +502,16 @@
"~*(?:\b)SurveyBot(?:\b|)" 0;
"~*(?:\b)Suzuran(?:\b|)" 0;
"~*(?:\b)Swiftbot(?:\b|)" 0;
"~*(?:\b)sysscan(?:\b|)" 0;
"~*(?:\b)Szukacz(?:\b|)" 0;
"~*(?:\b)T0PHackTeam(?:\b|)" 0;
"~*(?:\b)T8Abot(?:\b|)" 0;
"~*(?:\b)tAkeOut(?:\b|)" 0;
"~*(?:\b)Teleport(?:\b|)" 0;
"~*(?:\b)TeleportPro(?:\b|)" 0;
"~*(?:\b)Telesoft(?:\b|)" 0;
"~*(?:\b)Telesphoreo(?:\b|)" 0;
"~*(?:\b)Telesphorep(?:\b|)" 0;
"~*(?:\b)The\ Intraformant(?:\b|)" 0;
"~*(?:\b)TheNomad(?:\b|)" 0;
"~*(?:\b)The\ Intraformant(?:\b|)" 0;
"~*(?:\b)Thumbor(?:\b|)" 0;
"~*(?:\b)TightTwatBot(?:\b|)" 0;
"~*(?:\b)Titan(?:\b|)" 0;
@ -573,8 +520,6 @@
"~*(?:\b)Tracemyfile(?:\b|)" 0;
"~*(?:\b)Trendiction(?:\b|)" 0;
"~*(?:\b)Trendictionbot(?:\b|)" 0;
"~*(?:\b)trendiction.com(?:\b|)" 0;
"~*(?:\b)trendiction.de(?:\b|)" 0;
"~*(?:\b)True_Robot(?:\b|)" 0;
"~*(?:\b)Turingos(?:\b|)" 0;
"~*(?:\b)Turnitin(?:\b|)" 0;
@ -582,15 +527,15 @@
"~*(?:\b)TwengaBot(?:\b|)" 0;
"~*(?:\b)Twice(?:\b|)" 0;
"~*(?:\b)Typhoeus(?:\b|)" 0;
"~*(?:\b)UnisterBot(?:\b|)" 0;
"~*(?:\b)Upflow(?:\b|)" 0;
"~*(?:\b)URLy.Warning(?:\b|)" 0;
"~*(?:\b)URLy\ Warning(?:\b|)" 0;
"~*(?:\b)Vacuum(?:\b|)" 0;
"~*(?:\b)Vagabondo(?:\b|)" 0;
"~*(?:\b)UnisterBot(?:\b|)" 0;
"~*(?:\b)Upflow(?:\b|)" 0;
"~*(?:\b)V-BOT(?:\b|)" 0;
"~*(?:\b)VB\ Project(?:\b|)" 0;
"~*(?:\b)VCI(?:\b|)" 0;
"~*(?:\b)Vacuum(?:\b|)" 0;
"~*(?:\b)Vagabondo(?:\b|)" 0;
"~*(?:\b)VelenPublicWebCrawler(?:\b|)" 0;
"~*(?:\b)VeriCiteCrawler(?:\b|)" 0;
"~*(?:\b)VidibleScraper(?:\b|)" 0;
@ -598,46 +543,50 @@
"~*(?:\b)VoidEYE(?:\b|)" 0;
"~*(?:\b)Voil(?:\b|)" 0;
"~*(?:\b)Voltron(?:\b|)" 0;
"~*(?:\b)voyagerx.com(?:\b|)" 0;
"~*(?:\b)Wallpapers/3.0(?:\b|)" 0;
"~*(?:\b)WallpapersHD(?:\b|)" 0;
"~*(?:\b)WASALive-Bot(?:\b|)" 0;
"~*(?:\b)WBSearchBot(?:\b|)" 0;
"~*(?:\b)Webalta(?:\b|)" 0;
"~*(?:\b)WEBDAV(?:\b|)" 0;
"~*(?:\b)WISENutbot(?:\b|)" 0;
"~*(?:\b)WPScan(?:\b|)" 0;
"~*(?:\b)WWW-Collector-E(?:\b|)" 0;
"~*(?:\b)WWW-Mechanize(?:\b|)" 0;
"~*(?:\b)WWW::Mechanize(?:\b|)" 0;
"~*(?:\b)WWWOFFLE(?:\b|)" 0;
"~*(?:\b)Wallpapers/3.0(?:\b|)" 0;
"~*(?:\b)WallpapersHD(?:\b|)" 0;
"~*(?:\b)WeSEE(?:\b|)" 0;
"~*(?:\b)WebAuto(?:\b|)" 0;
"~*(?:\b)Web\ Auto(?:\b|)" 0;
"~*(?:\b)WebBandit(?:\b|)" 0;
"~*(?:\b)WebCollage(?:\b|)" 0;
"~*(?:\b)Web\ Collage(?:\b|)" 0;
"~*(?:\b)WebCopier(?:\b|)" 0;
"~*(?:\b)WEBDAV(?:\b|)" 0;
"~*(?:\b)WebEnhancer(?:\b|)" 0;
"~*(?:\b)Web\ Enhancer(?:\b|)" 0;
"~*(?:\b)WebFetch(?:\b|)" 0;
"~*(?:\b)Web\ Fetch(?:\b|)" 0;
"~*(?:\b)WebFuck(?:\b|)" 0;
"~*(?:\b)Web\ Fuck(?:\b|)" 0;
"~*(?:\b)WebGo\ IS(?:\b|)" 0;
"~*(?:\b)WebImageCollector(?:\b|)" 0;
"~*(?:\b)WebLeacher(?:\b|)" 0;
"~*(?:\b)WebmasterWorldForumBot(?:\b|)" 0;
"~*(?:\b)webmeup-crawler(?:\b|)" 0;
"~*(?:\b)WebPix(?:\b|)" 0;
"~*(?:\b)Web\ Pix(?:\b|)" 0;
"~*(?:\b)WebReaper(?:\b|)" 0;
"~*(?:\b)WebSauger(?:\b|)" 0;
"~*(?:\b)WebStripper(?:\b|)" 0;
"~*(?:\b)WebSucker(?:\b|)" 0;
"~*(?:\b)WebWhacker(?:\b|)" 0;
"~*(?:\b)WebZIP(?:\b|)" 0;
"~*(?:\b)Web\ Auto(?:\b|)" 0;
"~*(?:\b)Web\ Collage(?:\b|)" 0;
"~*(?:\b)Web\ Enhancer(?:\b|)" 0;
"~*(?:\b)Web\ Fetch(?:\b|)" 0;
"~*(?:\b)Web\ Fuck(?:\b|)" 0;
"~*(?:\b)Web\ Pix(?:\b|)" 0;
"~*(?:\b)Web\ Sauger(?:\b|)" 0;
"~*(?:\b)Web\ Sucker(?:\b|)" 0;
"~*(?:\b)Webalta(?:\b|)" 0;
"~*(?:\b)WebmasterWorldForumBot(?:\b|)" 0;
"~*(?:\b)Webshag(?:\b|)" 0;
"~*(?:\b)WebsiteExtractor(?:\b|)" 0;
"~*(?:\b)WebsiteQuester(?:\b|)" 0;
"~*(?:\b)Website\ Quester(?:\b|)" 0;
"~*(?:\b)Webster(?:\b|)" 0;
"~*(?:\b)WebStripper(?:\b|)" 0;
"~*(?:\b)WebSucker(?:\b|)" 0;
"~*(?:\b)Web\ Sucker(?:\b|)" 0;
"~*(?:\b)WebWhacker(?:\b|)" 0;
"~*(?:\b)WebZIP(?:\b|)" 0;
"~*(?:\b)WeSEE(?:\b|)" 0;
"~*(?:\b)Whack(?:\b|)" 0;
"~*(?:\b)Whacker(?:\b|)" 0;
"~*(?:\b)Whatweb(?:\b|)" 0;
@ -645,33 +594,84 @@
"~*(?:\b)Widow(?:\b|)" 0;
"~*(?:\b)WinHTTrack(?:\b|)" 0;
"~*(?:\b)WiseGuys\ Robot(?:\b|)" 0;
"~*(?:\b)WISENutbot(?:\b|)" 0;
"~*(?:\b)Wonderbot(?:\b|)" 0;
"~*(?:\b)Woobot(?:\b|)" 0;
"~*(?:\b)Wotbox(?:\b|)" 0;
"~*(?:\b)Wprecon(?:\b|)" 0;
"~*(?:\b)WPScan(?:\b|)" 0;
"~*(?:\b)WWW-Collector-E(?:\b|)" 0;
"~*(?:\b)WWW-Mechanize(?:\b|)" 0;
"~*(?:\b)WWW::Mechanize(?:\b|)" 0;
"~*(?:\b)WWWOFFLE(?:\b|)" 0;
"~*(?:\b)x09Mozilla(?:\b|)" 0;
"~*(?:\b)x22Mozilla(?:\b|)" 0;
"~*(?:\b)Xaldon_WebSpider(?:\b|)" 0;
"~*(?:\b)Xaldon\ WebSpider(?:\b|)" 0;
"~*(?:\b)Xaldon_WebSpider(?:\b|)" 0;
"~*(?:\b)Xenu(?:\b|)" 0;
"~*(?:\b)xpymep1.exe(?:\b|)" 0;
"~*(?:\b)YoudaoBot(?:\b|)" 0;
"~*(?:\b)Zade(?:\b|)" 0;
"~*(?:\b)Zauba(?:\b|)" 0;
"~*(?:\b)zauba.io(?:\b|)" 0;
"~*(?:\b)Zermelo(?:\b|)" 0;
"~*(?:\b)Zeus(?:\b|)" 0;
"~*(?:\b)zgrab(?:\b|)" 0;
"~*(?:\b)Zitebot(?:\b|)" 0;
"~*(?:\b)ZmEu(?:\b|)" 0;
"~*(?:\b)ZoomBot(?:\b|)" 0;
"~*(?:\b)ZoominfoBot(?:\b|)" 0;
"~*(?:\b)ZumBot(?:\b|)" 0;
"~*(?:\b)ZyBorg(?:\b|)" 0;
"~*(?:\b)archive.org_bot(?:\b|)" 0;
"~*(?:\b)arquivo-web-crawler(?:\b|)" 0;
"~*(?:\b)arquivo.pt(?:\b|)" 0;
"~*(?:\b)autoemailspider(?:\b|)" 0;
"~*(?:\b)backlink-check(?:\b|)" 0;
"~*(?:\b)check1.exe(?:\b|)" 0;
"~*(?:\b)coccocbot-web(?:\b|)" 0;
"~*(?:\b)cognitiveseo(?:\b|)" 0;
"~*(?:\b)com.plumanalytics(?:\b|)" 0;
"~*(?:\b)crawl.sogou.com(?:\b|)" 0;
"~*(?:\b)crawler.feedback(?:\b|)" 0;
"~*(?:\b)crawler4j(?:\b|)" 0;
"~*(?:\b)demandbase-bot(?:\b|)" 0;
"~*(?:\b)domainsproject.org(?:\b|)" 0;
"~*(?:\b)eCatch(?:\b|)" 0;
"~*(?:\b)evc-batch(?:\b|)" 0;
"~*(?:\b)facebookscraper(?:\b|)" 0;
"~*(?:\b)gopher(?:\b|)" 0;
"~*(?:\b)heritrix(?:\b|)" 0;
"~*(?:\b)instabid(?:\b|)" 0;
"~*(?:\b)internetVista\ monitor(?:\b|)" 0;
"~*(?:\b)ips-agent(?:\b|)" 0;
"~*(?:\b)isitwp.com(?:\b|)" 0;
"~*(?:\b)lwp-request(?:\b|)" 0;
"~*(?:\b)lwp-trivial(?:\b|)" 0;
"~*(?:\b)magpie-crawler(?:\b|)" 0;
"~*(?:\b)meanpathbot(?:\b|)" 0;
"~*(?:\b)mediawords(?:\b|)" 0;
"~*(?:\b)muhstik-scan(?:\b|)" 0;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0;
"~*(?:\b)oBot(?:\b|)" 0;
"~*(?:\b)page\ scorer(?:\b|)" 0;
"~*(?:\b)pcBrowser(?:\b|)" 0;
"~*(?:\b)plumanalytics(?:\b|)" 0;
"~*(?:\b)polaris\ version(?:\b|)" 0;
"~*(?:\b)probe-image-size(?:\b|)" 0;
"~*(?:\b)ripz(?:\b|)" 0;
"~*(?:\b)s1z.ru(?:\b|)" 0;
"~*(?:\b)satoristudio.net(?:\b|)" 0;
"~*(?:\b)scalaj-http(?:\b|)" 0;
"~*(?:\b)scan.lol(?:\b|)" 0;
"~*(?:\b)seobility(?:\b|)" 0;
"~*(?:\b)seoscanners(?:\b|)" 0;
"~*(?:\b)seostar(?:\b|)" 0;
"~*(?:\b)serpstatbot(?:\b|)" 0;
"~*(?:\b)sexsearcher(?:\b|)" 0;
"~*(?:\b)sitechecker.pro(?:\b|)" 0;
"~*(?:\b)siteripz(?:\b|)" 0;
"~*(?:\b)sogouspider(?:\b|)" 0;
"~*(?:\b)sp_auditbot(?:\b|)" 0;
"~*(?:\b)spyfu(?:\b|)" 0;
"~*(?:\b)sysscan(?:\b|)" 0;
"~*(?:\b)tAkeOut(?:\b|)" 0;
"~*(?:\b)trendiction.com(?:\b|)" 0;
"~*(?:\b)trendiction.de(?:\b|)" 0;
"~*(?:\b)voyagerx.com(?:\b|)" 0;
"~*(?:\b)webmeup-crawler(?:\b|)" 0;
"~*(?:\b)x09Mozilla(?:\b|)" 0;
"~*(?:\b)x22Mozilla(?:\b|)" 0;
"~*(?:\b)xpymep1.exe(?:\b|)" 0;
"~*(?:\b)zauba.io(?:\b|)" 0;
"~*(?:\b)zgrab(?:\b|)" 0;
# END MAKE BAD BOTS GOOD ### DO NOT EDIT THIS LINE AT ALL ###

4
dev-tools/test1_conf_files_whitelist/bots.d/custom-bad-referrers.conf
View file

@ -498,6 +498,7 @@
"~*(?:\b)anniemation\.com(?:\b)" 0;
"~*(?:\b)anonymizeme\.pro(?:\b)" 0;
"~*(?:\b)anonymous\-redirect\.com(?:\b)" 0;
"~*(?:\b)anonymousfox\.co(?:\b)" 0;
"~*(?:\b)anti\-virus\-removal\.info(?:\b)" 0;
"~*(?:\b)anticrawler\.org(?:\b)" 0;
"~*(?:\b)antons\-transporte\.de(?:\b)" 0;
@ -4637,6 +4638,7 @@
"~*(?:\b)pokemongooo\.ml(?:\b)" 0;
"~*(?:\b)pokerniydom\.ru(?:\b)" 0;
"~*(?:\b)polcin\.de(?:\b)" 0;
"~*(?:\b)poligon\.com(?:\b)" 0;
"~*(?:\b)polimga\.pw(?:\b)" 0;
"~*(?:\b)polska\-poezja\.com(?:\b)" 0;
"~*(?:\b)polybuild\.ru(?:\b)" 0;
@ -4693,6 +4695,7 @@
"~*(?:\b)pornogratisdiario\.com(?:\b)" 0;
"~*(?:\b)pornohd1080\.online(?:\b)" 0;
"~*(?:\b)pornohub\.me(?:\b)" 0;
"~*(?:\b)pornoinn\.com(?:\b)" 0;
"~*(?:\b)pornokajf\.com(?:\b)" 0;
"~*(?:\b)pornoklad\.net(?:\b)" 0;
"~*(?:\b)pornoklad\.ru(?:\b)" 0;
@ -6811,6 +6814,7 @@
"~*(?:\b)xlovecam\.com(?:\b)" 0;
"~*(?:\b)xmladserver\.com(?:\b)" 0;
"~*(?:\b)xmlinde\.com(?:\b)" 0;
"~*(?:\b)xmnb\.net(?:\b)" 0;
"~*(?:\b)xmronta\.com(?:\b)" 0;
"~*(?:\b)xn\-\-\-\-\-\-7cdbapdecfd4ak1bn0amjffj7afu3y\.xn\-\-p1ai(?:\b)" 0;
"~*(?:\b)xn\-\-\-\-\-6kcaabbafhu7cskl7akvongwpo7hvjj\.xn\-\-p1ai(?:\b)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.10.37 0;
10.1.0.59 0;
127.0.0.1 0;

16164
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/test1_conf_files_whitelist/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /etc/nginx/bots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

3
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,5 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
35.202.245.105 0;
35.192.91.101 0;
52.170.46.75 0;

16164
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/test2_conf_files/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /usr/local/nginx/bots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

3
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,5 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
35.192.91.101 0;
35.202.245.105 0;
52.170.46.75 0;

16164
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9
dev-tools/test3_conf_files/default.vhost
View file

@ -1,7 +1,6 @@
server {
listen *:9000;
root /var/www/html;
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
charset UTF-8;
@ -14,6 +13,10 @@ server {
include /etc/nginx/mybots.d/blockbots.conf;
index index.html;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /var/www/html/;

6
dev-tools/test_units/allowed-bots-for-test.list
View file

@ -1,6 +1,6 @@
jetmon
libwww-perl
Lynx
munin
Presto
Wget/1.15
jetmon
libwww-perl
munin

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
34.66.25.221 1;
10.1.0.59 1;
127.0.0.1 1;

368
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -64,19 +64,20 @@
"~*(?:\b)404checker(?:\b|)" 0;
"~*(?:\b)404enemy(?:\b|)" 0;
"~*(?:\b)80legs(?:\b|)" 0;
"~*(?:\b)ADmantX(?:\b|)" 0;
"~*(?:\b)AIBOT(?:\b|)" 0;
"~*(?:\b)ASPSeek(?:\b|)" 0;
"~*(?:\b)Abonti(?:\b|)" 0;
"~*(?:\b)Aboundex(?:\b|)" 0;
"~*(?:\b)Aboundexbot(?:\b|)" 0;
"~*(?:\b)Acunetix(?:\b|)" 0;
"~*(?:\b)ADmantX(?:\b|)" 0;
"~*(?:\b)AfD-Verbotsverfahren(?:\b|)" 0;
"~*(?:\b)AhrefsBot(?:\b|)" 0;
"~*(?:\b)AIBOT(?:\b|)" 0;
"~*(?:\b)AiHitBot(?:\b|)" 0;
"~*(?:\b)Aipbot(?:\b|)" 0;
"~*(?:\b)Alexibot(?:\b|)" 0;
"~*(?:\b)Alligator(?:\b|)" 0;
"~*(?:\b)AllSubmitter(?:\b|)" 0;
"~*(?:\b)Alligator(?:\b|)" 0;
"~*(?:\b)AlphaBot(?:\b|)" 0;
"~*(?:\b)Anarchie(?:\b|)" 0;
"~*(?:\b)Anarchy(?:\b|)" 0;
@ -84,37 +85,31 @@
"~*(?:\b)Ankit(?:\b|)" 0;
"~*(?:\b)Anthill(?:\b|)" 0;
"~*(?:\b)Apexoo(?:\b|)" 0;
"~*(?:\b)archive.org_bot(?:\b|)" 0;
"~*(?:\b)arquivo.pt(?:\b|)" 0;
"~*(?:\b)arquivo-web-crawler(?:\b|)" 0;
"~*(?:\b)Aspiegel(?:\b|)" 0;
"~*(?:\b)ASPSeek(?:\b|)" 0;
"~*(?:\b)Asterias(?:\b|)" 0;
"~*(?:\b)Attach(?:\b|)" 0;
"~*(?:\b)autoemailspider(?:\b|)" 0;
"~*(?:\b)AwarioRssBot(?:\b|)" 0;
"~*(?:\b)AwarioSmartBot(?:\b|)" 0;
"~*(?:\b)BBBike(?:\b|)" 0;
"~*(?:\b)BDCbot(?:\b|)" 0;
"~*(?:\b)BDFetch(?:\b|)" 0;
"~*(?:\b)BLEXBot(?:\b|)" 0;
"~*(?:\b)BackDoorBot(?:\b|)" 0;
"~*(?:\b)Backlink-Ceck(?:\b|)" 0;
"~*(?:\b)backlink-check(?:\b|)" 0;
"~*(?:\b)BacklinkCrawler(?:\b|)" 0;
"~*(?:\b)BackStreet(?:\b|)" 0;
"~*(?:\b)BackWeb(?:\b|)" 0;
"~*(?:\b)Backlink-Ceck(?:\b|)" 0;
"~*(?:\b)BacklinkCrawler(?:\b|)" 0;
"~*(?:\b)Badass(?:\b|)" 0;
"~*(?:\b)Bandit(?:\b|)" 0;
"~*(?:\b)Barkrowler(?:\b|)" 0;
"~*(?:\b)BatchFTP(?:\b|)" 0;
"~*(?:\b)Battleztar\ Bazinga(?:\b|)" 0;
"~*(?:\b)BBBike(?:\b|)" 0;
"~*(?:\b)BDCbot(?:\b|)" 0;
"~*(?:\b)BDFetch(?:\b|)" 0;
"~*(?:\b)BetaBot(?:\b|)" 0;
"~*(?:\b)Bigfoot(?:\b|)" 0;
"~*(?:\b)Bitacle(?:\b|)" 0;
"~*(?:\b)Blackboard(?:\b|)" 0;
"~*(?:\b)Black\ Hole(?:\b|)" 0;
"~*(?:\b)BlackWidow(?:\b|)" 0;
"~*(?:\b)BLEXBot(?:\b|)" 0;
"~*(?:\b)Black\ Hole(?:\b|)" 0;
"~*(?:\b)Blackboard(?:\b|)" 0;
"~*(?:\b)Blow(?:\b|)" 0;
"~*(?:\b)BlowFish(?:\b|)" 0;
"~*(?:\b)Boardreader(?:\b|)" 0;
@ -129,53 +124,47 @@
"~*(?:\b)Bullseye(?:\b|)" 0;
"~*(?:\b)BunnySlippers(?:\b|)" 0;
"~*(?:\b)BuzzSumo(?:\b|)" 0;
"~*(?:\b)Calculon(?:\b|)" 0;
"~*(?:\b)CATExplorador(?:\b|)" 0;
"~*(?:\b)CazoodleBot(?:\b|)" 0;
"~*(?:\b)CCBot(?:\b|)" 0;
"~*(?:\b)CODE87(?:\b|)" 0;
"~*(?:\b)CSHttp(?:\b|)" 0;
"~*(?:\b)Calculon(?:\b|)" 0;
"~*(?:\b)CazoodleBot(?:\b|)" 0;
"~*(?:\b)Cegbfeieh(?:\b|)" 0;
"~*(?:\b)CensysInspect(?:\b|)" 0;
"~*(?:\b)check1.exe(?:\b|)" 0;
"~*(?:\b)CheTeam(?:\b|)" 0;
"~*(?:\b)CheeseBot(?:\b|)" 0;
"~*(?:\b)CherryPicker(?:\b|)" 0;
"~*(?:\b)CheTeam(?:\b|)" 0;
"~*(?:\b)ChinaClaw(?:\b|)" 0;
"~*(?:\b)Chlooe(?:\b|)" 0;
"~*(?:\b)Claritybot(?:\b|)" 0;
"~*(?:\b)Cliqzbot(?:\b|)" 0;
"~*(?:\b)Cloud\ mapping(?:\b|)" 0;
"~*(?:\b)coccocbot-web(?:\b|)" 0;
"~*(?:\b)Cocolyzebot(?:\b|)" 0;
"~*(?:\b)CODE87(?:\b|)" 0;
"~*(?:\b)Cogentbot(?:\b|)" 0;
"~*(?:\b)cognitiveseo(?:\b|)" 0;
"~*(?:\b)Collector(?:\b|)" 0;
"~*(?:\b)com.plumanalytics(?:\b|)" 0;
"~*(?:\b)Copier(?:\b|)" 0;
"~*(?:\b)CopyRightCheck(?:\b|)" 0;
"~*(?:\b)Copyscape(?:\b|)" 0;
"~*(?:\b)Cosmos(?:\b|)" 0;
"~*(?:\b)Craftbot(?:\b|)" 0;
"~*(?:\b)crawler4j(?:\b|)" 0;
"~*(?:\b)crawler.feedback(?:\b|)" 0;
"~*(?:\b)crawl.sogou.com(?:\b|)" 0;
"~*(?:\b)CrazyWebCrawler(?:\b|)" 0;
"~*(?:\b)Crescent(?:\b|)" 0;
"~*(?:\b)CrunchBot(?:\b|)" 0;
"~*(?:\b)CSHttp(?:\b|)" 0;
"~*(?:\b)Curious(?:\b|)" 0;
"~*(?:\b)Custo(?:\b|)" 0;
"~*(?:\b)CyotekWebCopy(?:\b|)" 0;
"~*(?:\b)DatabaseDriverMysqli(?:\b|)" 0;
"~*(?:\b)DataCha0s(?:\b|)" 0;
"~*(?:\b)DBLBot(?:\b|)" 0;
"~*(?:\b)demandbase-bot(?:\b|)" 0;
"~*(?:\b)DIIbot(?:\b|)" 0;
"~*(?:\b)DSearch(?:\b|)" 0;
"~*(?:\b)DTS\ Agent(?:\b|)" 0;
"~*(?:\b)DataCha0s(?:\b|)" 0;
"~*(?:\b)DatabaseDriverMysqli(?:\b|)" 0;
"~*(?:\b)Demon(?:\b|)" 0;
"~*(?:\b)Deusu(?:\b|)" 0;
"~*(?:\b)Devil(?:\b|)" 0;
"~*(?:\b)Digincore(?:\b|)" 0;
"~*(?:\b)DigitalPebble(?:\b|)" 0;
"~*(?:\b)DIIbot(?:\b|)" 0;
"~*(?:\b)Dirbuster(?:\b|)" 0;
"~*(?:\b)Disco(?:\b|)" 0;
"~*(?:\b)Discobot(?:\b|)" 0;
@ -183,29 +172,24 @@
"~*(?:\b)Dispatch(?:\b|)" 0;
"~*(?:\b)DittoSpyder(?:\b|)" 0;
"~*(?:\b)DnyzBot(?:\b|)" 0;
"~*(?:\b)DomCopBot(?:\b|)" 0;
"~*(?:\b)DomainAppender(?:\b|)" 0;
"~*(?:\b)DomainCrawler(?:\b|)" 0;
"~*(?:\b)DomainSigmaCrawler(?:\b|)" 0;
"~*(?:\b)Domains\ Project(?:\b|)" 0;
"~*(?:\b)domainsproject.org(?:\b|)" 0;
"~*(?:\b)DomainStatsBot(?:\b|)" 0;
"~*(?:\b)DomCopBot(?:\b|)" 0;
"~*(?:\b)Domains\ Project(?:\b|)" 0;
"~*(?:\b)Dotbot(?:\b|)" 0;
"~*(?:\b)Download\ Wonder(?:\b|)" 0;
"~*(?:\b)Dragonfly(?:\b|)" 0;
"~*(?:\b)Drip(?:\b|)" 0;
"~*(?:\b)DSearch(?:\b|)" 0;
"~*(?:\b)DTS\ Agent(?:\b|)" 0;
"~*(?:\b)EasyDL(?:\b|)" 0;
"~*(?:\b)Ebingbong(?:\b|)" 0;
"~*(?:\b)eCatch(?:\b|)" 0;
"~*(?:\b)ECCP/1.0(?:\b|)" 0;
"~*(?:\b)Ecxi(?:\b|)" 0;
"~*(?:\b)EirGrabber(?:\b|)" 0;
"~*(?:\b)EMail\ Siphon(?:\b|)" 0;
"~*(?:\b)EMail\ Wolf(?:\b|)" 0;
"~*(?:\b)EasyDL(?:\b|)" 0;
"~*(?:\b)Ebingbong(?:\b|)" 0;
"~*(?:\b)Ecxi(?:\b|)" 0;
"~*(?:\b)EirGrabber(?:\b|)" 0;
"~*(?:\b)EroCrawler(?:\b|)" 0;
"~*(?:\b)evc-batch(?:\b|)" 0;
"~*(?:\b)Evil(?:\b|)" 0;
"~*(?:\b)Exabot(?:\b|)" 0;
"~*(?:\b)Express\ WebPictures(?:\b|)" 0;
@ -215,10 +199,9 @@
"~*(?:\b)Extreme\ Picture\ Finder(?:\b|)" 0;
"~*(?:\b)EyeNetIE(?:\b|)" 0;
"~*(?:\b)Ezooms(?:\b|)" 0;
"~*(?:\b)facebookscraper(?:\b|)" 0;
"~*(?:\b)FDM(?:\b|)" 0;
"~*(?:\b)FemtosearchBot(?:\b|)" 0;
"~*(?:\b)FHscan(?:\b|)" 0;
"~*(?:\b)FemtosearchBot(?:\b|)" 0;
"~*(?:\b)Fimap(?:\b|)" 0;
"~*(?:\b)Firefox/7.0(?:\b|)" 0;
"~*(?:\b)FlashGet(?:\b|)" 0;
@ -229,44 +212,43 @@
"~*(?:\b)Fuzz(?:\b|)" 0;
"~*(?:\b)FyberSpider(?:\b|)" 0;
"~*(?:\b)Fyrebot(?:\b|)" 0;
"~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0;
"~*(?:\b)GT::WWW(?:\b|)" 0;
"~*(?:\b)GalaxyBot(?:\b|)" 0;
"~*(?:\b)Genieo(?:\b|)" 0;
"~*(?:\b)GermCrawler(?:\b|)" 0;
"~*(?:\b)Getintent(?:\b|)" 0;
"~*(?:\b)GetRight(?:\b|)" 0;
"~*(?:\b)GetWeb(?:\b|)" 0;
"~*(?:\b)Getintent(?:\b|)" 0;
"~*(?:\b)Gigablast(?:\b|)" 0;
"~*(?:\b)Gigabot(?:\b|)" 0;
"~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0;
"~*(?:\b)Go-Ahead-Got-It(?:\b|)" 0;
"~*(?:\b)gopher(?:\b|)" 0;
"~*(?:\b)Gotit(?:\b|)" 0;
"~*(?:\b)GoZilla(?:\b|)" 0;
"~*(?:\b)Go!Zilla(?:\b|)" 0;
"~*(?:\b)Grabber(?:\b|)" 0;
"~*(?:\b)Go-Ahead-Got-It(?:\b|)" 0;
"~*(?:\b)GoZilla(?:\b|)" 0;
"~*(?:\b)Gotit(?:\b|)" 0;
"~*(?:\b)GrabNet(?:\b|)" 0;
"~*(?:\b)Grabber(?:\b|)" 0;
"~*(?:\b)Grafula(?:\b|)" 0;
"~*(?:\b)GrapeFX(?:\b|)" 0;
"~*(?:\b)GrapeshotCrawler(?:\b|)" 0;
"~*(?:\b)GridBot(?:\b|)" 0;
"~*(?:\b)GT::WWW(?:\b|)" 0;
"~*(?:\b)Haansoft(?:\b|)" 0;
"~*(?:\b)HaosouSpider(?:\b|)" 0;
"~*(?:\b)Harvest(?:\b|)" 0;
"~*(?:\b)Havij(?:\b|)" 0;
"~*(?:\b)HEADMasterSEO(?:\b|)" 0;
"~*(?:\b)heritrix(?:\b|)" 0;
"~*(?:\b)Heritrix(?:\b|)" 0;
"~*(?:\b)Hloader(?:\b|)" 0;
"~*(?:\b)HMView(?:\b|)" 0;
"~*(?:\b)HTMLparser(?:\b|)" 0;
"~*(?:\b)HTTP::Lite(?:\b|)" 0;
"~*(?:\b)HTTrack(?:\b|)" 0;
"~*(?:\b)Haansoft(?:\b|)" 0;
"~*(?:\b)HaosouSpider(?:\b|)" 0;
"~*(?:\b)Harvest(?:\b|)" 0;
"~*(?:\b)Havij(?:\b|)" 0;
"~*(?:\b)Heritrix(?:\b|)" 0;
"~*(?:\b)Hloader(?:\b|)" 0;
"~*(?:\b)Humanlinks(?:\b|)" 0;
"~*(?:\b)HybridBot(?:\b|)" 0;
"~*(?:\b)Iblog(?:\b|)" 0;
"~*(?:\b)IDBot(?:\b|)" 0;
"~*(?:\b)IDBTE4M(?:\b|)" 0;
"~*(?:\b)IDBot(?:\b|)" 0;
"~*(?:\b)IRLbot(?:\b|)" 0;
"~*(?:\b)Iblog(?:\b|)" 0;
"~*(?:\b)Id-search(?:\b|)" 0;
"~*(?:\b)IlseBot(?:\b|)" 0;
"~*(?:\b)Image\ Fetch(?:\b|)" 0;
@ -275,25 +257,20 @@
"~*(?:\b)Indy\ Library(?:\b|)" 0;
"~*(?:\b)InfoNaviRobot(?:\b|)" 0;
"~*(?:\b)InfoTekies(?:\b|)" 0;
"~*(?:\b)instabid(?:\b|)" 0;
"~*(?:\b)Intelliseek(?:\b|)" 0;
"~*(?:\b)InterGET(?:\b|)" 0;
"~*(?:\b)Internet\ Ninja(?:\b|)" 0;
"~*(?:\b)InternetSeer(?:\b|)" 0;
"~*(?:\b)internetVista\ monitor(?:\b|)" 0;
"~*(?:\b)ips-agent(?:\b|)" 0;
"~*(?:\b)Internet\ Ninja(?:\b|)" 0;
"~*(?:\b)Iria(?:\b|)" 0;
"~*(?:\b)IRLbot(?:\b|)" 0;
"~*(?:\b)isitwp.com(?:\b|)" 0;
"~*(?:\b)Iskanie(?:\b|)" 0;
"~*(?:\b)IstellaBot(?:\b|)" 0;
"~*(?:\b)JOC\ Web\ Spider(?:\b|)" 0;
"~*(?:\b)JamesBOT(?:\b|)" 0;
"~*(?:\b)Jbrofuzz(?:\b|)" 0;
"~*(?:\b)JennyBot(?:\b|)" 0;
"~*(?:\b)JetCar(?:\b|)" 0;
"~*(?:\b)Jetty(?:\b|)" 0;
"~*(?:\b)JikeSpider(?:\b|)" 0;
"~*(?:\b)JOC\ Web\ Spider(?:\b|)" 0;
"~*(?:\b)Joomla(?:\b|)" 0;
"~*(?:\b)Jorgee(?:\b|)" 0;
"~*(?:\b)JustView(?:\b|)" 0;
@ -302,6 +279,8 @@
"~*(?:\b)Keyword\ Density(?:\b|)" 0;
"~*(?:\b)Kinza(?:\b|)" 0;
"~*(?:\b)Kozmosbot(?:\b|)" 0;
"~*(?:\b)LNSpiderguy(?:\b|)" 0;
"~*(?:\b)LWP::Simple(?:\b|)" 0;
"~*(?:\b)Lanshanbot(?:\b|)" 0;
"~*(?:\b)Larbin(?:\b|)" 0;
"~*(?:\b)LeechFTP(?:\b|)" 0;
@ -313,13 +292,13 @@
"~*(?:\b)LieBaoFast(?:\b|)" 0;
"~*(?:\b)Lightspeedsystems(?:\b|)" 0;
"~*(?:\b)Likse(?:\b|)" 0;
"~*(?:\b)LinkScan(?:\b|)" 0;
"~*(?:\b)LinkWalker(?:\b|)" 0;
"~*(?:\b)Linkbot(?:\b|)" 0;
"~*(?:\b)Linkdexbot(?:\b|)" 0;
"~*(?:\b)LinkextractorPro(?:\b|)" 0;
"~*(?:\b)LinkpadBot(?:\b|)" 0;
"~*(?:\b)LinkScan(?:\b|)" 0;
"~*(?:\b)LinksManager(?:\b|)" 0;
"~*(?:\b)LinkWalker(?:\b|)" 0;
"~*(?:\b)LinqiaMetadataDownloaderBot(?:\b|)" 0;
"~*(?:\b)LinqiaRSSBot(?:\b|)" 0;
"~*(?:\b)LinqiaScrapeBot(?:\b|)" 0;
@ -327,53 +306,47 @@
"~*(?:\b)Lipperhey\ Spider(?:\b|)" 0;
"~*(?:\b)Litemage_walker(?:\b|)" 0;
"~*(?:\b)Lmspider(?:\b|)" 0;
"~*(?:\b)LNSpiderguy(?:\b|)" 0;
"~*(?:\b)Ltx71(?:\b|)" 0;
"~*(?:\b)lwp-request(?:\b|)" 0;
"~*(?:\b)LWP::Simple(?:\b|)" 0;
"~*(?:\b)lwp-trivial(?:\b|)" 0;
"~*(?:\b)Magnet(?:\b|)" 0;
"~*(?:\b)MFC_Tear_Sample(?:\b|)" 0;
"~*(?:\b)MIDown\ tool(?:\b|)" 0;
"~*(?:\b)MIIxpc(?:\b|)" 0;
"~*(?:\b)MJ12bot(?:\b|)" 0;
"~*(?:\b)MQQBrowser(?:\b|)" 0;
"~*(?:\b)MSFrontPage(?:\b|)" 0;
"~*(?:\b)MSIECrawler(?:\b|)" 0;
"~*(?:\b)MTRobot(?:\b|)" 0;
"~*(?:\b)Mag-Net(?:\b|)" 0;
"~*(?:\b)magpie-crawler(?:\b|)" 0;
"~*(?:\b)Magnet(?:\b|)" 0;
"~*(?:\b)Mail.RU_Bot(?:\b|)" 0;
"~*(?:\b)Majestic12(?:\b|)" 0;
"~*(?:\b)Majestic-SEO(?:\b|)" 0;
"~*(?:\b)Majestic12(?:\b|)" 0;
"~*(?:\b)Majestic\ SEO(?:\b|)" 0;
"~*(?:\b)MarkMonitor(?:\b|)" 0;
"~*(?:\b)MarkWatch(?:\b|)" 0;
"~*(?:\b)Masscan(?:\b|)" 0;
"~*(?:\b)Mass\ Downloader(?:\b|)" 0;
"~*(?:\b)Masscan(?:\b|)" 0;
"~*(?:\b)Mata\ Hari(?:\b|)" 0;
"~*(?:\b)MauiBot(?:\b|)" 0;
"~*(?:\b)Mb2345Browser(?:\b|)" 0;
"~*(?:\b)meanpathbot(?:\b|)" 0;
"~*(?:\b)Meanpathbot(?:\b|)" 0;
"~*(?:\b)MeanPath\ Bot(?:\b|)" 0;
"~*(?:\b)Meanpathbot(?:\b|)" 0;
"~*(?:\b)Mediatoolkitbot(?:\b|)" 0;
"~*(?:\b)mediawords(?:\b|)" 0;
"~*(?:\b)MegaIndex.ru(?:\b|)" 0;
"~*(?:\b)Metauri(?:\b|)" 0;
"~*(?:\b)MFC_Tear_Sample(?:\b|)" 0;
"~*(?:\b)MicroMessenger(?:\b|)" 0;
"~*(?:\b)Microsoft\ Data\ Access(?:\b|)" 0;
"~*(?:\b)Microsoft\ URL\ Control(?:\b|)" 0;
"~*(?:\b)MIDown\ tool(?:\b|)" 0;
"~*(?:\b)MIIxpc(?:\b|)" 0;
"~*(?:\b)Mister\ PiX(?:\b|)" 0;
"~*(?:\b)MJ12bot(?:\b|)" 0;
"~*(?:\b)Moblie Safari(?:\b|)" 0;
"~*(?:\b)Mojeek(?:\b|)" 0;
"~*(?:\b)Mojolicious(?:\b|)" 0;
"~*(?:\b)Morfeus\ Fucking\ Scanner(?:\b|)" 0;
"~*(?:\b)Mozlila(?:\b|)" 0;
"~*(?:\b)MQQBrowser(?:\b|)" 0;
"~*(?:\b)Mr.4x3(?:\b|)" 0;
"~*(?:\b)MSFrontPage(?:\b|)" 0;
"~*(?:\b)MSIECrawler(?:\b|)" 0;
"~*(?:\b)Msrabot(?:\b|)" 0;
"~*(?:\b)MTRobot(?:\b|)" 0;
"~*(?:\b)muhstik-scan(?:\b|)" 0;
"~*(?:\b)Musobot(?:\b|)" 0;
"~*(?:\b)NICErsPRO(?:\b|)" 0;
"~*(?:\b)NPbot(?:\b|)" 0;
"~*(?:\b)Name\ Intelligence(?:\b|)" 0;
"~*(?:\b)Nameprotect(?:\b|)" 0;
"~*(?:\b)Navroad(?:\b|)" 0;
@ -381,72 +354,63 @@
"~*(?:\b)Needle(?:\b|)" 0;
"~*(?:\b)Nessus(?:\b|)" 0;
"~*(?:\b)NetAnts(?:\b|)" 0;
"~*(?:\b)Netcraft(?:\b|)" 0;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0;
"~*(?:\b)NetLyzer(?:\b|)" 0;
"~*(?:\b)NetMechanic(?:\b|)" 0;
"~*(?:\b)NetSpider(?:\b|)" 0;
"~*(?:\b)Nettrack(?:\b|)" 0;
"~*(?:\b)Net\ Vampire(?:\b|)" 0;
"~*(?:\b)Netvibes(?:\b|)" 0;
"~*(?:\b)NetZIP(?:\b|)" 0;
"~*(?:\b)Net\ Vampire(?:\b|)" 0;
"~*(?:\b)Netcraft(?:\b|)" 0;
"~*(?:\b)Nettrack(?:\b|)" 0;
"~*(?:\b)Netvibes(?:\b|)" 0;
"~*(?:\b)NextGenSearchBot(?:\b|)" 0;
"~*(?:\b)Nibbler(?:\b|)" 0;
"~*(?:\b)NICErsPRO(?:\b|)" 0;
"~*(?:\b)Niki-bot(?:\b|)" 0;
"~*(?:\b)Nikto(?:\b|)" 0;
"~*(?:\b)NimbleCrawler(?:\b|)" 0;
"~*(?:\b)Nimbostratus(?:\b|)" 0;
"~*(?:\b)Ninja(?:\b|)" 0;
"~*(?:\b)Nmap(?:\b|)" 0;
"~*(?:\b)NPbot(?:\b|)" 0;
"~*(?:\b)Nuclei(?:\b|)" 0;
"~*(?:\b)Nutch(?:\b|)" 0;
"~*(?:\b)oBot(?:\b|)" 0;
"~*(?:\b)OPPO\ A33(?:\b|)" 0;
"~*(?:\b)Octopus(?:\b|)" 0;
"~*(?:\b)Offline\ Explorer(?:\b|)" 0;
"~*(?:\b)Offline\ Navigator(?:\b|)" 0;
"~*(?:\b)OnCrawl(?:\b|)" 0;
"~*(?:\b)Openfind(?:\b|)" 0;
"~*(?:\b)OpenLinkProfiler(?:\b|)" 0;
"~*(?:\b)Openvas(?:\b|)" 0;
"~*(?:\b)OpenVAS(?:\b|)" 0;
"~*(?:\b)OPPO\ A33(?:\b|)" 0;
"~*(?:\b)Openfind(?:\b|)" 0;
"~*(?:\b)Openvas(?:\b|)" 0;
"~*(?:\b)OrangeBot(?:\b|)" 0;
"~*(?:\b)OrangeSpider(?:\b|)" 0;
"~*(?:\b)OutclicksBot(?:\b|)" 0;
"~*(?:\b)OutfoxBot(?:\b|)" 0;
"~*(?:\b)PECL::HTTP(?:\b|)" 0;
"~*(?:\b)PHPCrawl(?:\b|)" 0;
"~*(?:\b)POE-Component-Client-HTTP(?:\b|)" 0;
"~*(?:\b)PageAnalyzer(?:\b|)" 0;
"~*(?:\b)Page\ Analyzer(?:\b|)" 0;
"~*(?:\b)PageGrabber(?:\b|)" 0;
"~*(?:\b)page\ scorer(?:\b|)" 0;
"~*(?:\b)PageScorer(?:\b|)" 0;
"~*(?:\b)PageThing.com(?:\b|)" 0;
"~*(?:\b)Page\ Analyzer(?:\b|)" 0;
"~*(?:\b)Pandalytics(?:\b|)" 0;
"~*(?:\b)Panscient(?:\b|)" 0;
"~*(?:\b)Papa\ Foto(?:\b|)" 0;
"~*(?:\b)Pavuk(?:\b|)" 0;
"~*(?:\b)pcBrowser(?:\b|)" 0;
"~*(?:\b)PECL::HTTP(?:\b|)" 0;
"~*(?:\b)PeoplePal(?:\b|)" 0;
"~*(?:\b)Petalbot(?:\b|)" 0;
"~*(?:\b)PHPCrawl(?:\b|)" 0;
"~*(?:\b)Pi-Monster(?:\b|)" 0;
"~*(?:\b)Picscout(?:\b|)" 0;
"~*(?:\b)Picsearch(?:\b|)" 0;
"~*(?:\b)PictureFinder(?:\b|)" 0;
"~*(?:\b)Piepmatz(?:\b|)" 0;
"~*(?:\b)Pimonster(?:\b|)" 0;
"~*(?:\b)Pi-Monster(?:\b|)" 0;
"~*(?:\b)Pixray(?:\b|)" 0;
"~*(?:\b)PleaseCrawl(?:\b|)" 0;
"~*(?:\b)plumanalytics(?:\b|)" 0;
"~*(?:\b)Pockey(?:\b|)" 0;
"~*(?:\b)POE-Component-Client-HTTP(?:\b|)" 0;
"~*(?:\b)polaris\ version(?:\b|)" 0;
"~*(?:\b)probe-image-size(?:\b|)" 0;
"~*(?:\b)Probethenet(?:\b|)" 0;
"~*(?:\b)ProPowerBot(?:\b|)" 0;
"~*(?:\b)ProWebWalker(?:\b|)" 0;
"~*(?:\b)Probethenet(?:\b|)" 0;
"~*(?:\b)Psbot(?:\b|)" 0;
"~*(?:\b)Pu_iN(?:\b|)" 0;
"~*(?:\b)Pump(?:\b|)" 0;
@ -454,6 +418,7 @@
"~*(?:\b)PyCurl(?:\b|)" 0;
"~*(?:\b)QueryN\ Metasearch(?:\b|)" 0;
"~*(?:\b)Quick-Crawler(?:\b|)" 0;
"~*(?:\b)RSSingBot(?:\b|)" 0;
"~*(?:\b)RankActive(?:\b|)" 0;
"~*(?:\b)RankActiveLinkBot(?:\b|)" 0;
"~*(?:\b)RankFlex(?:\b|)" 0;
@ -461,27 +426,28 @@
"~*(?:\b)RankingBot2(?:\b|)" 0;
"~*(?:\b)Rankivabot(?:\b|)" 0;
"~*(?:\b)RankurBot(?:\b|)" 0;
"~*(?:\b)Re-re(?:\b|)" 0;
"~*(?:\b)ReGet(?:\b|)" 0;
"~*(?:\b)RealDownload(?:\b|)" 0;
"~*(?:\b)Reaper(?:\b|)" 0;
"~*(?:\b)RebelMouse(?:\b|)" 0;
"~*(?:\b)Recorder(?:\b|)" 0;
"~*(?:\b)RedesScrapy(?:\b|)" 0;
"~*(?:\b)ReGet(?:\b|)" 0;
"~*(?:\b)RepoMonkey(?:\b|)" 0;
"~*(?:\b)Re-re(?:\b|)" 0;
"~*(?:\b)Ripper(?:\b|)" 0;
"~*(?:\b)ripz(?:\b|)" 0;
"~*(?:\b)RocketCrawler(?:\b|)" 0;
"~*(?:\b)Rogerbot(?:\b|)" 0;
"~*(?:\b)RSSingBot(?:\b|)" 0;
"~*(?:\b)s1z.ru(?:\b|)" 0;
"~*(?:\b)SalesIntelligent(?:\b|)" 0;
"~*(?:\b)satoristudio.net(?:\b|)" 0;
"~*(?:\b)SBIder(?:\b|)" 0;
"~*(?:\b)scalaj-http(?:\b|)" 0;
"~*(?:\b)SEOkicks(?:\b|)" 0;
"~*(?:\b)SEOkicks-Robot(?:\b|)" 0;
"~*(?:\b)SEOlyticsCrawler(?:\b|)" 0;
"~*(?:\b)SEOprofiler(?:\b|)" 0;
"~*(?:\b)SEOstats(?:\b|)" 0;
"~*(?:\b)SISTRIX(?:\b|)" 0;
"~*(?:\b)SMTBot(?:\b|)" 0;
"~*(?:\b)SalesIntelligent(?:\b|)" 0;
"~*(?:\b)ScanAlert(?:\b|)" 0;
"~*(?:\b)Scanbot(?:\b|)" 0;
"~*(?:\b)scan.lol(?:\b|)" 0;
"~*(?:\b)ScoutJet(?:\b|)" 0;
"~*(?:\b)Scrapy(?:\b|)" 0;
"~*(?:\b)Screaming(?:\b|)" 0;
@ -492,42 +458,27 @@
"~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0;
"~*(?:\b)SentiBot(?:\b|)" 0;
"~*(?:\b)seobility(?:\b|)" 0;
"~*(?:\b)SeobilityBot(?:\b|)" 0;
"~*(?:\b)SEOkicks(?:\b|)" 0;
"~*(?:\b)SEOkicks-Robot(?:\b|)" 0;
"~*(?:\b)SEOlyticsCrawler(?:\b|)" 0;
"~*(?:\b)Seomoz(?:\b|)" 0;
"~*(?:\b)SEOprofiler(?:\b|)" 0;
"~*(?:\b)seoscanners(?:\b|)" 0;
"~*(?:\b)SeoSiteCheckup(?:\b|)" 0;
"~*(?:\b)seostar(?:\b|)" 0;
"~*(?:\b)SEOstats(?:\b|)" 0;
"~*(?:\b)serpstatbot(?:\b|)" 0;
"~*(?:\b)sexsearcher(?:\b|)" 0;
"~*(?:\b)SeobilityBot(?:\b|)" 0;
"~*(?:\b)Seomoz(?:\b|)" 0;
"~*(?:\b)Shodan(?:\b|)" 0;
"~*(?:\b)Siphon(?:\b|)" 0;
"~*(?:\b)SISTRIX(?:\b|)" 0;
"~*(?:\b)Sitebeam(?:\b|)" 0;
"~*(?:\b)SiteCheckerBotCrawler(?:\b|)" 0;
"~*(?:\b)sitechecker.pro(?:\b|)" 0;
"~*(?:\b)SiteExplorer(?:\b|)" 0;
"~*(?:\b)Siteimprove(?:\b|)" 0;
"~*(?:\b)SiteLockSpider(?:\b|)" 0;
"~*(?:\b)siteripz(?:\b|)" 0;
"~*(?:\b)SiteSnagger(?:\b|)" 0;
"~*(?:\b)SiteSucker(?:\b|)" 0;
"~*(?:\b)Site\ Sucker(?:\b|)" 0;
"~*(?:\b)Sitebeam(?:\b|)" 0;
"~*(?:\b)Siteimprove(?:\b|)" 0;
"~*(?:\b)Sitevigil(?:\b|)" 0;
"~*(?:\b)SlySearch(?:\b|)" 0;
"~*(?:\b)SmartDownload(?:\b|)" 0;
"~*(?:\b)SMTBot(?:\b|)" 0;
"~*(?:\b)Snake(?:\b|)" 0;
"~*(?:\b)Snapbot(?:\b|)" 0;
"~*(?:\b)Snoopy(?:\b|)" 0;
"~*(?:\b)SocialRankIOBot(?:\b|)" 0;
"~*(?:\b)Sociscraper(?:\b|)" 0;
"~*(?:\b)sogouspider(?:\b|)" 0;
"~*(?:\b)Sogou\ web\ spider(?:\b|)" 0;
"~*(?:\b)Sosospider(?:\b|)" 0;
"~*(?:\b)Sottopop(?:\b|)" 0;
@ -535,11 +486,9 @@
"~*(?:\b)Spammen(?:\b|)" 0;
"~*(?:\b)SpankBot(?:\b|)" 0;
"~*(?:\b)Spanner(?:\b|)" 0;
"~*(?:\b)sp_auditbot(?:\b|)" 0;
"~*(?:\b)Spbot(?:\b|)" 0;
"~*(?:\b)Spinn3r(?:\b|)" 0;
"~*(?:\b)SputnikBot(?:\b|)" 0;
"~*(?:\b)spyfu(?:\b|)" 0;
"~*(?:\b)Sqlmap(?:\b|)" 0;
"~*(?:\b)Sqlworm(?:\b|)" 0;
"~*(?:\b)Sqworm(?:\b|)" 0;
@ -553,18 +502,16 @@
"~*(?:\b)SurveyBot(?:\b|)" 0;
"~*(?:\b)Suzuran(?:\b|)" 0;
"~*(?:\b)Swiftbot(?:\b|)" 0;
"~*(?:\b)sysscan(?:\b|)" 0;
"~*(?:\b)Szukacz(?:\b|)" 0;
"~*(?:\b)T0PHackTeam(?:\b|)" 0;
"~*(?:\b)T8Abot(?:\b|)" 0;
"~*(?:\b)tAkeOut(?:\b|)" 0;
"~*(?:\b)Teleport(?:\b|)" 0;
"~*(?:\b)TeleportPro(?:\b|)" 0;
"~*(?:\b)Telesoft(?:\b|)" 0;
"~*(?:\b)Telesphoreo(?:\b|)" 0;
"~*(?:\b)Telesphorep(?:\b|)" 0;
"~*(?:\b)The\ Intraformant(?:\b|)" 0;
"~*(?:\b)TheNomad(?:\b|)" 0;
"~*(?:\b)The\ Intraformant(?:\b|)" 0;
"~*(?:\b)Thumbor(?:\b|)" 0;
"~*(?:\b)TightTwatBot(?:\b|)" 0;
"~*(?:\b)Titan(?:\b|)" 0;
@ -573,8 +520,6 @@
"~*(?:\b)Tracemyfile(?:\b|)" 0;
"~*(?:\b)Trendiction(?:\b|)" 0;
"~*(?:\b)Trendictionbot(?:\b|)" 0;
"~*(?:\b)trendiction.com(?:\b|)" 0;
"~*(?:\b)trendiction.de(?:\b|)" 0;
"~*(?:\b)True_Robot(?:\b|)" 0;
"~*(?:\b)Turingos(?:\b|)" 0;
"~*(?:\b)Turnitin(?:\b|)" 0;
@ -582,15 +527,15 @@
"~*(?:\b)TwengaBot(?:\b|)" 0;
"~*(?:\b)Twice(?:\b|)" 0;
"~*(?:\b)Typhoeus(?:\b|)" 0;
"~*(?:\b)UnisterBot(?:\b|)" 0;
"~*(?:\b)Upflow(?:\b|)" 0;
"~*(?:\b)URLy.Warning(?:\b|)" 0;
"~*(?:\b)URLy\ Warning(?:\b|)" 0;
"~*(?:\b)Vacuum(?:\b|)" 0;
"~*(?:\b)Vagabondo(?:\b|)" 0;
"~*(?:\b)UnisterBot(?:\b|)" 0;
"~*(?:\b)Upflow(?:\b|)" 0;
"~*(?:\b)V-BOT(?:\b|)" 0;
"~*(?:\b)VB\ Project(?:\b|)" 0;
"~*(?:\b)VCI(?:\b|)" 0;
"~*(?:\b)Vacuum(?:\b|)" 0;
"~*(?:\b)Vagabondo(?:\b|)" 0;
"~*(?:\b)VelenPublicWebCrawler(?:\b|)" 0;
"~*(?:\b)VeriCiteCrawler(?:\b|)" 0;
"~*(?:\b)VidibleScraper(?:\b|)" 0;
@ -598,46 +543,50 @@
"~*(?:\b)VoidEYE(?:\b|)" 0;
"~*(?:\b)Voil(?:\b|)" 0;
"~*(?:\b)Voltron(?:\b|)" 0;
"~*(?:\b)voyagerx.com(?:\b|)" 0;
"~*(?:\b)Wallpapers/3.0(?:\b|)" 0;
"~*(?:\b)WallpapersHD(?:\b|)" 0;
"~*(?:\b)WASALive-Bot(?:\b|)" 0;
"~*(?:\b)WBSearchBot(?:\b|)" 0;
"~*(?:\b)Webalta(?:\b|)" 0;
"~*(?:\b)WEBDAV(?:\b|)" 0;
"~*(?:\b)WISENutbot(?:\b|)" 0;
"~*(?:\b)WPScan(?:\b|)" 0;
"~*(?:\b)WWW-Collector-E(?:\b|)" 0;
"~*(?:\b)WWW-Mechanize(?:\b|)" 0;
"~*(?:\b)WWW::Mechanize(?:\b|)" 0;
"~*(?:\b)WWWOFFLE(?:\b|)" 0;
"~*(?:\b)Wallpapers/3.0(?:\b|)" 0;
"~*(?:\b)WallpapersHD(?:\b|)" 0;
"~*(?:\b)WeSEE(?:\b|)" 0;
"~*(?:\b)WebAuto(?:\b|)" 0;
"~*(?:\b)Web\ Auto(?:\b|)" 0;
"~*(?:\b)WebBandit(?:\b|)" 0;
"~*(?:\b)WebCollage(?:\b|)" 0;
"~*(?:\b)Web\ Collage(?:\b|)" 0;
"~*(?:\b)WebCopier(?:\b|)" 0;
"~*(?:\b)WEBDAV(?:\b|)" 0;
"~*(?:\b)WebEnhancer(?:\b|)" 0;
"~*(?:\b)Web\ Enhancer(?:\b|)" 0;
"~*(?:\b)WebFetch(?:\b|)" 0;
"~*(?:\b)Web\ Fetch(?:\b|)" 0;
"~*(?:\b)WebFuck(?:\b|)" 0;
"~*(?:\b)Web\ Fuck(?:\b|)" 0;
"~*(?:\b)WebGo\ IS(?:\b|)" 0;
"~*(?:\b)WebImageCollector(?:\b|)" 0;
"~*(?:\b)WebLeacher(?:\b|)" 0;
"~*(?:\b)WebmasterWorldForumBot(?:\b|)" 0;
"~*(?:\b)webmeup-crawler(?:\b|)" 0;
"~*(?:\b)WebPix(?:\b|)" 0;
"~*(?:\b)Web\ Pix(?:\b|)" 0;
"~*(?:\b)WebReaper(?:\b|)" 0;
"~*(?:\b)WebSauger(?:\b|)" 0;
"~*(?:\b)WebStripper(?:\b|)" 0;
"~*(?:\b)WebSucker(?:\b|)" 0;
"~*(?:\b)WebWhacker(?:\b|)" 0;
"~*(?:\b)WebZIP(?:\b|)" 0;
"~*(?:\b)Web\ Auto(?:\b|)" 0;
"~*(?:\b)Web\ Collage(?:\b|)" 0;
"~*(?:\b)Web\ Enhancer(?:\b|)" 0;
"~*(?:\b)Web\ Fetch(?:\b|)" 0;
"~*(?:\b)Web\ Fuck(?:\b|)" 0;
"~*(?:\b)Web\ Pix(?:\b|)" 0;
"~*(?:\b)Web\ Sauger(?:\b|)" 0;
"~*(?:\b)Web\ Sucker(?:\b|)" 0;
"~*(?:\b)Webalta(?:\b|)" 0;
"~*(?:\b)WebmasterWorldForumBot(?:\b|)" 0;
"~*(?:\b)Webshag(?:\b|)" 0;
"~*(?:\b)WebsiteExtractor(?:\b|)" 0;
"~*(?:\b)WebsiteQuester(?:\b|)" 0;
"~*(?:\b)Website\ Quester(?:\b|)" 0;
"~*(?:\b)Webster(?:\b|)" 0;
"~*(?:\b)WebStripper(?:\b|)" 0;
"~*(?:\b)WebSucker(?:\b|)" 0;
"~*(?:\b)Web\ Sucker(?:\b|)" 0;
"~*(?:\b)WebWhacker(?:\b|)" 0;
"~*(?:\b)WebZIP(?:\b|)" 0;
"~*(?:\b)WeSEE(?:\b|)" 0;
"~*(?:\b)Whack(?:\b|)" 0;
"~*(?:\b)Whacker(?:\b|)" 0;
"~*(?:\b)Whatweb(?:\b|)" 0;
@ -645,33 +594,84 @@
"~*(?:\b)Widow(?:\b|)" 0;
"~*(?:\b)WinHTTrack(?:\b|)" 0;
"~*(?:\b)WiseGuys\ Robot(?:\b|)" 0;
"~*(?:\b)WISENutbot(?:\b|)" 0;
"~*(?:\b)Wonderbot(?:\b|)" 0;
"~*(?:\b)Woobot(?:\b|)" 0;
"~*(?:\b)Wotbox(?:\b|)" 0;
"~*(?:\b)Wprecon(?:\b|)" 0;
"~*(?:\b)WPScan(?:\b|)" 0;
"~*(?:\b)WWW-Collector-E(?:\b|)" 0;
"~*(?:\b)WWW-Mechanize(?:\b|)" 0;
"~*(?:\b)WWW::Mechanize(?:\b|)" 0;
"~*(?:\b)WWWOFFLE(?:\b|)" 0;
"~*(?:\b)x09Mozilla(?:\b|)" 0;
"~*(?:\b)x22Mozilla(?:\b|)" 0;
"~*(?:\b)Xaldon_WebSpider(?:\b|)" 0;
"~*(?:\b)Xaldon\ WebSpider(?:\b|)" 0;
"~*(?:\b)Xaldon_WebSpider(?:\b|)" 0;
"~*(?:\b)Xenu(?:\b|)" 0;
"~*(?:\b)xpymep1.exe(?:\b|)" 0;
"~*(?:\b)YoudaoBot(?:\b|)" 0;
"~*(?:\b)Zade(?:\b|)" 0;
"~*(?:\b)Zauba(?:\b|)" 0;
"~*(?:\b)zauba.io(?:\b|)" 0;
"~*(?:\b)Zermelo(?:\b|)" 0;
"~*(?:\b)Zeus(?:\b|)" 0;
"~*(?:\b)zgrab(?:\b|)" 0;
"~*(?:\b)Zitebot(?:\b|)" 0;
"~*(?:\b)ZmEu(?:\b|)" 0;
"~*(?:\b)ZoomBot(?:\b|)" 0;
"~*(?:\b)ZoominfoBot(?:\b|)" 0;
"~*(?:\b)ZumBot(?:\b|)" 0;
"~*(?:\b)ZyBorg(?:\b|)" 0;
"~*(?:\b)archive.org_bot(?:\b|)" 0;
"~*(?:\b)arquivo-web-crawler(?:\b|)" 0;
"~*(?:\b)arquivo.pt(?:\b|)" 0;
"~*(?:\b)autoemailspider(?:\b|)" 0;
"~*(?:\b)backlink-check(?:\b|)" 0;
"~*(?:\b)check1.exe(?:\b|)" 0;
"~*(?:\b)coccocbot-web(?:\b|)" 0;
"~*(?:\b)cognitiveseo(?:\b|)" 0;
"~*(?:\b)com.plumanalytics(?:\b|)" 0;
"~*(?:\b)crawl.sogou.com(?:\b|)" 0;
"~*(?:\b)crawler.feedback(?:\b|)" 0;
"~*(?:\b)crawler4j(?:\b|)" 0;
"~*(?:\b)demandbase-bot(?:\b|)" 0;
"~*(?:\b)domainsproject.org(?:\b|)" 0;
"~*(?:\b)eCatch(?:\b|)" 0;
"~*(?:\b)evc-batch(?:\b|)" 0;
"~*(?:\b)facebookscraper(?:\b|)" 0;
"~*(?:\b)gopher(?:\b|)" 0;
"~*(?:\b)heritrix(?:\b|)" 0;
"~*(?:\b)instabid(?:\b|)" 0;
"~*(?:\b)internetVista\ monitor(?:\b|)" 0;
"~*(?:\b)ips-agent(?:\b|)" 0;
"~*(?:\b)isitwp.com(?:\b|)" 0;
"~*(?:\b)lwp-request(?:\b|)" 0;
"~*(?:\b)lwp-trivial(?:\b|)" 0;
"~*(?:\b)magpie-crawler(?:\b|)" 0;
"~*(?:\b)meanpathbot(?:\b|)" 0;
"~*(?:\b)mediawords(?:\b|)" 0;
"~*(?:\b)muhstik-scan(?:\b|)" 0;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0;
"~*(?:\b)oBot(?:\b|)" 0;
"~*(?:\b)page\ scorer(?:\b|)" 0;
"~*(?:\b)pcBrowser(?:\b|)" 0;
"~*(?:\b)plumanalytics(?:\b|)" 0;
"~*(?:\b)polaris\ version(?:\b|)" 0;
"~*(?:\b)probe-image-size(?:\b|)" 0;
"~*(?:\b)ripz(?:\b|)" 0;
"~*(?:\b)s1z.ru(?:\b|)" 0;
"~*(?:\b)satoristudio.net(?:\b|)" 0;
"~*(?:\b)scalaj-http(?:\b|)" 0;
"~*(?:\b)scan.lol(?:\b|)" 0;
"~*(?:\b)seobility(?:\b|)" 0;
"~*(?:\b)seoscanners(?:\b|)" 0;
"~*(?:\b)seostar(?:\b|)" 0;
"~*(?:\b)serpstatbot(?:\b|)" 0;
"~*(?:\b)sexsearcher(?:\b|)" 0;
"~*(?:\b)sitechecker.pro(?:\b|)" 0;
"~*(?:\b)siteripz(?:\b|)" 0;
"~*(?:\b)sogouspider(?:\b|)" 0;
"~*(?:\b)sp_auditbot(?:\b|)" 0;
"~*(?:\b)spyfu(?:\b|)" 0;
"~*(?:\b)sysscan(?:\b|)" 0;
"~*(?:\b)tAkeOut(?:\b|)" 0;
"~*(?:\b)trendiction.com(?:\b|)" 0;
"~*(?:\b)trendiction.de(?:\b|)" 0;
"~*(?:\b)voyagerx.com(?:\b|)" 0;
"~*(?:\b)webmeup-crawler(?:\b|)" 0;
"~*(?:\b)x09Mozilla(?:\b|)" 0;
"~*(?:\b)x22Mozilla(?:\b|)" 0;
"~*(?:\b)xpymep1.exe(?:\b|)" 0;
"~*(?:\b)zauba.io(?:\b|)" 0;
"~*(?:\b)zgrab(?:\b|)" 0;
# END MAKE BAD BOTS GOOD ### DO NOT EDIT THIS LINE AT ALL ###

4
dev-tools/test_units/custom-bad-referrers.conf
View file

@ -498,6 +498,7 @@
"~*(?:\b)anniemation\.com(?:\b)" 0;
"~*(?:\b)anonymizeme\.pro(?:\b)" 0;
"~*(?:\b)anonymous\-redirect\.com(?:\b)" 0;
"~*(?:\b)anonymousfox\.co(?:\b)" 0;
"~*(?:\b)anti\-virus\-removal\.info(?:\b)" 0;
"~*(?:\b)anticrawler\.org(?:\b)" 0;
"~*(?:\b)antons\-transporte\.de(?:\b)" 0;
@ -4637,6 +4638,7 @@
"~*(?:\b)pokemongooo\.ml(?:\b)" 0;
"~*(?:\b)pokerniydom\.ru(?:\b)" 0;
"~*(?:\b)polcin\.de(?:\b)" 0;
"~*(?:\b)poligon\.com(?:\b)" 0;
"~*(?:\b)polimga\.pw(?:\b)" 0;
"~*(?:\b)polska\-poezja\.com(?:\b)" 0;
"~*(?:\b)polybuild\.ru(?:\b)" 0;
@ -4693,6 +4695,7 @@
"~*(?:\b)pornogratisdiario\.com(?:\b)" 0;
"~*(?:\b)pornohd1080\.online(?:\b)" 0;
"~*(?:\b)pornohub\.me(?:\b)" 0;
"~*(?:\b)pornoinn\.com(?:\b)" 0;
"~*(?:\b)pornokajf\.com(?:\b)" 0;
"~*(?:\b)pornoklad\.net(?:\b)" 0;
"~*(?:\b)pornoklad\.ru(?:\b)" 0;
@ -6811,6 +6814,7 @@
"~*(?:\b)xlovecam\.com(?:\b)" 0;
"~*(?:\b)xmladserver\.com(?:\b)" 0;
"~*(?:\b)xmlinde\.com(?:\b)" 0;
"~*(?:\b)xmnb\.net(?:\b)" 0;
"~*(?:\b)xmronta\.com(?:\b)" 0;
"~*(?:\b)xn\-\-\-\-\-\-7cdbapdecfd4ak1bn0amjffj7afu3y\.xn\-\-p1ai(?:\b)" 0;
"~*(?:\b)xn\-\-\-\-\-6kcaabbafhu7cskl7akvongwpo7hvjj\.xn\-\-p1ai(?:\b)" 0;

30
dev-tools/test_units/good-bots-for-test.list
View file

@ -1,34 +1,34 @@
adidxbot
AdsBot-Google
aolbuild
bingbot
bingpreview
developers.facebook.com
DoCoMo
duckduckgo
facebookexternalhit
facebookplatform
Feedfetcher-Google
Google-HTTP-Java-Client
Googlebot
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot/Test
Googlebot-Video
Google-HTTP-Java-Client
Googlebot/Test
Gravityscan
gsa-crawler
Jakarta Commons
Kraken/0.1
LinkedInBot
Mediapartners-Google
msnbot
msnbot-media
SAMSUNG
Slackbot
Slackbot-LinkExpanding
slurp
teoma
TwitterBot
Wordpress
adidxbot
aolbuild
bingbot
bingpreview
developers.facebook.com
duckduckgo
facebookexternalhit
facebookplatform
gsa-crawler
msnbot
msnbot-media
slurp
teoma
yahoo

8
dev-tools/test_units/limited-bots-for-test.list
View file

@ -1,18 +1,18 @@
Alexa
archive.org
ArchiveTeam
Baidu
BUbiNG
Baidu
FlipboardProxy
ia_archiver
MSIE 7.0
Proximic
R6_CommentReader
R6_FeedFetcher
RED/1
RPT-HTTPClient
sfFeedReader/0.9
Spaidu
UptimeRobot/2.0
YandexBot
YandexImages
archive.org
ia_archiver
sfFeedReader/0.9

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Anarchie
Bigfoot
Drip
Linkbot
LWP::Simple
Openvas
Ripper
sogouspider
WebFuck
Web Sauger
HMView
Jyxobot
Linkdexbot
Lmspider
Mag-Net
Microsoft Data Access
SpaceBison
Toata
Voil
sp_auditbot

356
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com
404checker
Aboundex
AIBOT
Abonti
Aboundexbot
Acunetix
ADmantX
AhrefsBot
Anarchy
AfD-Verbotsverfahren
Alexibot
AllSubmitter
Alligator
AlphaBot
Anarchie
Anarchy99
Ankit
Anthill
Apexoo
archive.org_bot
arquivo-web-crawler
Aspiegel
Asterias
AwarioRssBot
BBBike
BDCbot
BDFetch
BLEXBot
BackDoorBot
BackStreet
Bandit
Bigfoot
BacklinkCrawler
BetaBot
Bitacle
Black Hole
BlackWidow
Blow
BlowFish
Boardreader
Brandwatch
BuiltWith
Bolt
BotALot
Buck
BuiltBotTough
Bullseye
BunnySlippers
CheeseBot
CATExplorador
CODE87
CazoodleBot
CensysInspect
CheTeam
CheeseBot
ChinaClaw
Chlooe
com.plumanalytics
Cocolyzebot
Collector
Copier
Craftbot
crawler.feedback
crawl.sogou.com
Crescent
CSHttp
Custo
CyotekWebCopy
DatabaseDriverMysqli
DataCha0s
CrunchBot
DBLBot
Deusu
Digincore
DigitalPebble
DIIbot
DSearch
DataCha0s
Demon
Dirbuster
Disco
Discobot
Dispatch
Discoverybot
DittoSpyder
DomainAppender
DnyzBot
DomainSigmaCrawler
DomCopBot
DomainStatsBot
Dotbot
Dragonfly
DTS Agent
Ebingbong
eCatch
Drip
ECCP/1.0
EirGrabber
evc-batch
Evil
EMail Wolf
Ebingbong
Ecxi
EroCrawler
Exabot
Extreme Picture Finder
FemtosearchBot
Firefox/7.0
Flunky
Fuzz
Extractor
ExtractorPro
FDM
FHscan
FrontPage
Fyrebot
GT::WWW
GalaxyBot
Getintent
GermCrawler
GetRight
Gigablast
gopher
GoZilla
GridBot
GrabNet
Grafula
GrapeshotCrawler
HEADMasterSEO
HTMLparser
Humanlinks
HybridBot
IDBot
HTTP::Lite
Haansoft
Hloader
IDBTE4M
Id-search
IlseBot
Image Fetch
Indy Library
InfoNaviRobot
ips-agent
Jbrofuzz
InfoTekies
Intelliseek
Internet Ninja
InternetSeer
Iria
Iskanie
JikeSpider
JOC Web Spider
Joomla
Kenjin Spider
Kozmosbot
Lanshanbot
Larbin
LeechGet
LexiBot
Libwhisker
LieBaoFast
LinksManager
LinqiaScrapeBot
Lipperhey Spider
Lftp
LinkScan
LinkWalker
LinkextractorPro
LinkpadBot
LinqiaRSSBot
Lmspider
Ltx71
lwp-request
Mail.RU_Bot
Majestic12
Majestic SEO
MarkWatch
MauiBot
Meanpathbot
MeanPath Bot
mediawords
MegaIndex.ru
MFC_Tear_Sample
Microsoft Data Access
MJ12bot
Mojeek
Mojolicious
Morfeus Fucking Scanner
MIDown tool
MQQBrowser
Mail.RU_Bot
MarkMonitor
Meanpathbot
MegaIndex.ru
Microsoft URL Control
Mozlila
Msrabot
muhstik-scan
Musobot
Name Intelligence
NetAnts
Netcraft
netEstate NE Crawler
NetLyzer
NetMechanic
NetZIP
NICErsPRO
NPbot
Nameprotect
NearSite
Needle
Nessus
Net Vampire
Nettrack
NextGenSearchBot
Nuclei
Nibbler
Niki-bot
NimbleCrawler
Ninja
Nmap
OPPO A33
Octopus
Offline Explorer
OnCrawl
OpenLinkProfiler
OpenVAS
Openfind
Openvas
OrangeSpider
OutclicksBot
PageAnalyzer
PageScorer
Pavuk
PECL::HTTP
Petalbot
PHPCrawl
Page Analyzer
PageGrabber
Pandalytics
Papa Foto
PeoplePal
Picscout
Picsearch
Pi-Monster
PictureFinder
Pimonster
Pixray
polaris version
Psbot
Pu_iN
Pump
Pockey
PyCurl
QueryN Metasearch
Quick-Crawler
RankActiveLinkBot
RSSingBot
RankFlex
RankingBot
Rankivabot
RankurBot
RealDownload
RebelMouse
ReGet
Reaper
Recorder
RepoMonkey
Re-re
Ripper
RocketCrawler
RSSingBot
satoristudio.net
Rogerbot
SEOkicks
SEOlyticsCrawler
SEOprofiler
SMTBot
SalesIntelligent
Scanbot
scan.lol
Scrapy
Screaming
ScrepyBot
ScoutJet
SearchmetricsBot
Semrush
SemrushBot
seobility
SeobilityBot
Seomoz
seoscanners
SEOstats
serpstatbot
Siphon
SISTRIX
Siteimprove
SiteExplorer
SiteLockSpider
Site Sucker
SiteSnagger
Sitevigil
SlySearch
Snoopy
SocialRankIOBot
sogouspider
Sociscraper
Sosospider
SpaceBison
Spammen
SpankBot
Spanner
Spbot
Spinn3r
SputnikBot
Sqlmap
Sqlworm
Sqworm
Sucuri
SuperBot
SuperHTTP
Surfbot
SurveyBot
sysscan
Szukacz
T0PHackTeam
T8Abot
tAkeOut
Telesphorep
Suzuran
Telesphoreo
TheNomad
TightTwatBot
Titan
Toata
Toweyabot
Tracemyfile
Trendiction
Turingos
Turnitin
TwengaBot
Typhoeus
URLy.Warning
UnisterBot
Upflow
V-BOT
Vacuum
Vagabondo
VB Project
VCI
VelenPublicWebCrawler
VeriCiteCrawler
VidibleScraper
Virusdie
VoidEYE
Voil
Wallpapers/3.0
WallpapersHD
WASALive-Bot
WBSearchBot
Webalta
Web Auto
WebCollage
Web Fuck
WebLeacher
WebPix
Web Pix
WebSauger
WebsiteQuester
Website Quester
Webster
WebSucker
WeSEE
Whacker
Whatweb
Who.is Bot
WiseGuys Robot
WEBDAV
WISENutbot
Wotbox
WWW-Collector-E
WWW-Mechanize
x22Mozilla
WPScan
WWWOFFLE
Wallpapers/3.0
Web Fetch
Web Pix
Web Sucker
WebFetch
WebImageCollector
WebReaper
WebSauger
WebStripper
Webalta
Whack
Who.is Bot
Wprecon
Xaldon_WebSpider
YoudaoBot
Zade
zgrab
Zitebot
ZumBot
ZmEu
ZoominfoBot
arquivo-web-crawler
coccocbot-web
com.plumanalytics
internetVista monitor
ips-agent
isitwp.com
magpie-crawler
netEstate NE Crawler
page scorer
pcBrowser
plumanalytics
polaris version
probe-image-size
scan.lol
seoscanners
seostar
siteripz
tAkeOut
trendiction.com
trendiction.de
voyagerx.com

336
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com
404checker
404enemy
80legs
Aboundex
Acunetix
ADmantX
AIBOT
AiHitBot
Alexibot
AllSubmitter
AlphaBot
ASPSeek
Aipbot
Anarchie
Anarchy99
Ankit
Anthill
archive.org_bot
arquivo.pt
Aspiegel
AwarioSmartBot
BBBike
BDCbot
BackDoorBot
BackStreet
Backlink-Ceck
Barkrowler
BatchFTP
BetaBot
Bitacle
Black Hole
BlackWidow
Blow
BlowFish
BotALot
Brandprotect
Buck
BuiltBotTough
BuiltWith
BunnySlippers
Calculon
CATExplorador
CensysInspect
CheeseBot
CheTeam
ChinaClaw
Chlooe
CCBot
CODE87
Cogentbot
Copier
CSHttp
CazoodleBot
CheTeam
Claritybot
Cosmos
Craftbot
Custo
Crescent
Curious
CyotekWebCopy
DatabaseDriverMysqli
DIIbot
DTS Agent
DataCha0s
DBLBot
Demon
Deusu
DigitalPebble
Dirbuster
Disco
Discobot
DittoSpyder
DomainAppender
DomainSigmaCrawler
DomainStatsBot
Devil
Digincore
Discoverybot
Dispatch
DnyzBot
DomCopBot
DomainAppender
DomainCrawler
DomainSigmaCrawler
Dotbot
Dragonfly
DSearch
EasyDL
Ebingbong
EirGrabber
EMail Wolf
Exabot
Express WebPictures
Extreme Picture Finder
Drip
EMail Siphon
Ecxi
ExtLinksBot
Extractor
EyeNetIE
FDM
FemtosearchBot
FlashGet
Ezooms
Flunky
Foobot
Fuzz
Gigablast
Go-Ahead-Got-It
gopher
GoZilla
G-i-g-a-b-o-t
GT::WWW
Havij
HEADMasterSEO
heritrix
Hloader
HMView
GermCrawler
Gigabot
Go!Zilla
GrabNet
Grabber
Grafula
HTMLparser
HTTP::Lite
HTTrack
Iblog
Id-search
Hloader
IRLbot
Image Sucker
IndeedBot
InfoTekies
Intelliseek
InterGET
internetVista monitor
Iria
IRLbot
IstellaBot
JamesBOT
InternetSeer
Jbrofuzz
Jetty
JikeSpider
JOC Web Spider
Joomla
JustView
Kenjin Spider
Jyxobot
Kinza
LNSpiderguy
LWP::Simple
Larbin
Libwhisker
LeechGet
LexiBot
Lftp
Lightspeedsystems
Linkdexbot
Likse
LinkScan
LinqiaMetadataDownloaderBot
LinkWalker
LinkpadBot
LinqiaRSSBot
LinqiaScrapeBot
Lipperhey
Litemage_walker
lwp-trivial
Magnet
magpie-crawler
Ltx71
MJ12bot
Mail.RU_Bot
Majestic SEO
Majestic-SEO
MarkMonitor
Mata Hari
MarkWatch
Mass Downloader
Masscan
MauiBot
meanpathbot
Meanpathbot
MegaIndex.ru
MFC_Tear_Sample
MQQBrowser
Mb2345Browser
MicroMessenger
Microsoft URL Control
Moblie Safari
Mozlila
Mr.4x3
Msrabot
Name Intelligence
Navroad
NearSite
Netcraft
netEstate NE Crawler
NetLyzer
NetSpider
Netvibes
NextGenSearchBot
Ninja
Needle
NetAnts
NetMechanic
NetZIP
NimbleCrawler
Nmap
NPbot
Nutch
oBot
Octopus
Offline Explorer
Offline Navigator
OnCrawl
Openfind
OpenLinkProfiler
OrangeBot
Openvas
OrangeSpider
OutclicksBot
Page Analyzer
PageGrabber
page scorer
Papa Foto
PECL::HTTP
Picsearch
Piepmatz
Pixray
POE-Component-Client-HTTP
probe-image-size
Probethenet
Pump
PageThing.com
Panscient
Pavuk
Petalbot
Picscout
Pimonster
PleaseCrawl
Pockey
ProPowerBot
Psbot
PyCurl
QueryN Metasearch
Quick-Crawler
RSSingBot
RankActiveLinkBot
RankFlex
RankingBot2
Rankivabot
Reaper
Re-re
ReGet
RealDownload
RebelMouse
Recorder
RedesScrapy
RepoMonkey
Re-re
Ripper
ripz
RocketCrawler
Rogerbot
RSSingBot
s1z.ru
satoristudio.net
SBIder
scalaj-http
Scrapy
Screaming
Searchestate
SentiBot
seobility
SEOkicks
SEOkicks-Robot
SEOlyticsCrawler
SEOprofiler
Searchestate
Semrush
SeoSiteCheckup
serpstatbot
sexsearcher
SeobilityBot
Seomoz
Shodan
Siphon
SiteExplorer
siteripz
SiteLockSpider
SiteSnagger
Sitevigil
SMTBot
Snapbot
SiteSucker
Sociscraper
Sogou web spider
Sosospider
Sottopop
SpaceBison
SpankBot
Spanner
Spbot
Spinn3r
SputnikBot
Sqlmap
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
tAkeOut
SurveyBot
Swiftbot
Teleport
Telesphoreo
Telesphorep
Toweyabot
TightTwatBot
Titan
Toata
Trendiction
Trendictionbot
trendiction.com
trendiction.de
Turnitin
TwengaBot
True_Robot
TurnitinBot
Twice
Typhoeus
URLy Warning
URLy.Warning
UnisterBot
Upflow
V-BOT
VB Project
Voil
Voltron
voyagerx.com
Wallpapers/3.0
WASALive-Bot
WBSearchBot
WWW-Mechanize
WWW::Mechanize
WWWOFFLE
WallpapersHD
WeSEE
Web Enhancer
WebAuto
Web Auto
WebCollage
WebEnhancer
WebFetch
WebFuck
WebImageCollector
WebLeacher
WebPix
WebSauger
Webshag
WebsiteExtractor
Website Quester
Webster
WebStripper
WebSucker
Web Sucker
WeSEE
WiseGuys Robot
WebZIP
Webalta
Webshag
Website Quester
WebsiteExtractor
WebsiteQuester
Webster
Who.is Bot
Widow
Wonderbot
WWWOFFLE
x22Mozilla
xpymep1.exe
zauba.io
Zermelo
Woobot
Wotbox
Xaldon WebSpider
Xaldon_WebSpider
Xenu
Zitebot
ZmEu
ZoominfoBot
ZumBot
ZyBorg
arquivo-web-crawler
autoemailspider
backlink-check
eCatch
facebookscraper
gopher
instabid
internetVista monitor
lwp-request
muhstik-scan
netEstate NE Crawler
oBot
scan.lol
seobility
seostar
sitechecker.pro
tAkeOut
x09Mozilla
x22Mozilla
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
bonux.nextview.ru
brus-vsem.ru
freshwallpapers.info
nuup.info
pacificair.com
petedrummond.com
rock-cafe.info
sinel.info
telsis.com
torrentred.games
celebsopics.com
indiasourcemart.in
jus0wil.pp.ua
nfljerseyscheapbiz.us
oda.as
onlinetvseries.me
ororodnik.goodbb.ru
scanmarine.info
titangel-vietnam.com
urldelivery.com

592
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,100 @@
00webcams.com
2020iscoming.info
24x7-server-support.site
273-fz.ru
2nt.ru
2x2fan.ru
5u.com
66cpwgln.space
70casino.online
8558l.jobs.net
8jn.dating-app.ru
a96527gi.beget.tech
aa8b68101d388c446389283820863176e7.com
aadroid.net
aarbur.com
addray.pro
adimmix.com
ads-cool.pro
adzpower.com
aerogo.com
akuhni.by
alienwheel.es
allnews24.in
aludecor.info
ampower.me
angigreene.com
anticrawler.org
aosheng-tech.com
apiappenableinfo-a.akamaihd.net
apifasterlightin-a.akamaihd.net
appartement-stumm.at
apple.com-webbrowsing-security.science
ap.senai.br
arabgirls.adultgalls.com
arendovalka.xyz
artpicso.com
asacopaco.tk
ascat.porn
asrv-a.akamaihd.net
asseenontv.ru
avkzarabotok.com
azartniy-bonus.com
azinoofficial777.ru
babieca.com
backgroundpictures.net
balitouroffice.com
bangla-chat-uk-paralud.ga
bankiem.pl
bannertesting.com
beauby.ru
becuo.com
begalka.xyz
belstaffstore.online
bestserials.com
bezsporno.ru
bigcities.org
bingoporno.com
bio.trade-jp.net
bitporno.sx
biznesrealnost.ru
blagovest-med.ru
blobar.org
1pamm.ru
2000k.ru
4ur.click
alert-hgd.xyz
alienwheels.de
alphahoverboards.com
amehdaily.com
amoi.tn
aproposde.com
atmovs.com
ayodhya.co
bayareaaeroclub.org
bdsmgalls.net
bestcalovebracelet.cn
big-cash.net
bizcheapjerseyswholesalechina.com
blog.f00kclan.de
blog.koorg.ru
blogqpot.com
bmwhighperformers.com
boanonihaca.tk
bonanza-fortune.men
bonkers.name
brainboosting.club
brainxs.us
brandehk.dk
brandov.ru
bscodecs.com
buscarfoto.com
business-made-fun.com
bus-offer.com
buyessaynow.biz
buypillsorderonline.com
buzzsumo.com
cablecar.us
callcafe.info
calvet.altervista.org
camdolls.com
canyougethighofftizanidine.blogspot.com
carson.getenjoyment.net
cartierreplica.xyz
cashkitten-a.akamaihd.net
caulderwoodkennels.com
caveavins.fr
cejewelry.xyz
c-english.ru
censys.io
cfsrating.sonicwall.com
c-gainsbourg.com
chatroulette.online
chinacheapelitejerseys.com
cialisndbrx.com
classiquebijoux.ru
claytransformations.info
clayvasedesigns.tk
cleanallspyware.com
comeondog.info
comissionka.net
companies-catalog.com
compliance-george.xyz
compliance-ivan.top
compliance-nelson.top
compucelunlock.net
conciergegroup.org
connectionstrenth.com
cosmeticswomens-womensports.rhcloud.com
cottageofgrace.com
crackzplanet.com
credit-online.ws
crynet.cc
dalavia.ru
dayibiao.com
derevesendeco.com
deshevo-nedorogo.ru
detskie-konstruktory.ru
digitalfaq.com
directivepub.com
domainroam.win
domashneeruporno.com
doska-vsem.ru
drhomes.biz
dvd-famille.com
eandsgallery.com
editors.choice6912650.hulfingtonpost.com
educatemetv.com
eldiariodeguadalajara.com
electro-prom.com
elektronischezigarettekaufen2.tumblr.com
elektrozigaretten2.yn.lt
elite-sex-finders.com
energydiet-info.ru
enginebay.ru
englate.com
enskedesquashclub.se
enternet.ee
este-line.com.ua
eu-cookie-law.blogspot.com
euronis-free.com
evogarage.com
eyelike.com.ua
eyessurgery.ru
ezigarettekaufen2.yolasite.com
f0815.de
f5mtrack.com
facialporntube.com
faptitans.com
fatfasts-4tmz.com
fefo.gdn
ferieboligkbh.dk
filesvine.com
filmidivx.com
findanysex.com
findsexguide.com
finejewelryshop.ru
fishtauto.ru
fitnesspiks.com
flashahead.info
flooringinstallation-edmonton.com
focalink.com
forex21.ru
forum.poker4life.ru
foto-sisek.porngalleries.top
foxinsocks.ru
foxjuegos.com
free-fb-traffic.com
freelotto.com
free-traffic.xyz
freeuploader.com
freevpn.space
fullgirl.ru
g7m.pl
games.kolossale.ru
garciniaxt.us
globetrotting-culture.ru
go2jump.org
goblacked.com
golden-praga.ru
cbox.ws
cdncash.org
cheapsergiorossi.online
cindymatches.com
clean-start.net
demon-tweeks.com
dosug-lux.ru
drivotracker.com
elektir.ru
feeriaclub.ru
fm-upgrade.ru
forum20.smailik.org
foto-telok.net
funnymama.com
getoutofdebtfree.org
goldpanningtools.com
goodwinmetals.co
googlepositions.com
groupmoney.ru
growshop.es
guruofcasino.com
handsandlegs.ru
harmonyglen.us
hawaiielectriclight.com
hd720kino.ru
healmytrauma.info
hentai-manga.porn
histock.info
hitmuzik.ru
hjaoopoa.top
hobbyhuren-datenbank.com
hobild.net
homeandhealth.ru
hoverboard360.es
hptwaakw.blog.fc2.com
huimin764128.com
humbmosquina.tk
hvd-store.com
icaseclub.ru
ideibiznesa2015.ru
iframe-toloka.com
igithab.com
igtools.club
ilikevitaly.com
imabase.com
imfamous.info
imgata.com
imicrovision.com
importchinacoach-teach.com
indo-export.ru
insurple.com
interesnie-faktu.ru
internetproviderstucson.com
iomoio.net
iqoption.pro
iqupdatetmz.win
isistaylorporn.info
itag.pw
ivoiretechnocom.ci
iwanttodeliver.com
izismile.com
jeremyeaton.co
jerseywholesalebizchina.com
jongose.ninja
jpcycles.com
jrpmakati.com
jurajskie.info
jyvopys.com
kamorel.com
karaganda.xkaz.org
keywordhouse.com
kihi.gdn
kino2018.cc
kleine-titten.biz
klejonka.info
klosetkitten.com
kochanelli.com
konteiner24.com
kovrenok.ru
kozhniebolezni.com
krafte.ru
krasula.pp.ua
ks1234.com
kumuk.info
ladsup.com
landscapebackgrounds.blogspot.com
lawyers.cafe
ledpolice.ru
lerporn.info
lider-zhaluzi.kiev.ua
liffa.ru
lineavabit.it
littleberry.ru
livefixer.com
liver-chintai.org
lizace.com
localflirtbuddies.com
lyrster.com
m1media.net
mabdoola.blogspot.com
macresource.co.uk
magda-gadalka.ru
makis.nu
managerpak204.weebly.com
map028.com
marblestyle.ru
marinetraffic.com
marketingtechniques.info
marktforschung-stuttgart.com
masserect.com
master-muznachas.ru
mathgym.com.au
mecash.ru
michaelkorsoutletstore.net
michaelkorssaleoutletonline.net
middlerush-a.akamaihd.net
minneapoliscopiers.com
mirobuvi.com.ua
mnogabukaff.net
mobilemedia.md
mockupui.com
moesonce.com
monclercheap.xyz
moncleroutletonline.win
mp3downloadhq.com
mp3ritm.top
myanyone.net
mydeathspace.com
myfreetutorials.com
myhitmp3.club
myliveblog.ru
myshopmatemac.com
naughtyconnect.com
nero-us.com
net-profits.xyz
nextconseil.com
nfljerseys.online
notasprensa.info
nudepatch.net
obiavo.net
onefilms.net
onescreen.cc
oneshotdate.com
onlinadverts.com
online-podarki.com
onlinewritingjobs17.blogspot.ru
optitrade24.com
oralsexfilme.net
ossmalta.com
outrageousdeal-a.akamaihd.net
ownshop.cf
pandarastore.top
parlament.biz
partner-pop.men
partner-print.men
pc-test.net
perso.wanadoo.es
piccdata.com
picquery.com
pictures-and-images.com
picturesboss.com
picturesmania.com
pinapchik.com
pinsdaddy.com
popup.matchmaker.com
pornodojd.ru
pornohd1080.online
pornokorol.com
pornolook.net
pornosmola.info
pornozhara.com
porn-w.org
greatzip.com
hmmm.cz
hotel-mkad.ru
ivanstroi.ru
ix20.ru
jackpotchances.com
laptoper.net
larutti.ru
linkpulse.com
lion.snapmap.us
lkbennettstore.online
loginduepunti.it
makemoneyonline.com
martlinker.com
metascephe.com
moivestiy.biz
moskva.nodup.ru
newsperuse.com
o-dachnik.ru
o.light.d0t.ru
offside2.5v.pl
onclkads.com
onlainbesplatno.ru
ourtherapy.ru
pills24h.com
pornoted.com
portadd.men
potoideas.us
pozdrawleniya.ru
pricheskaonline.ru
princevc.com
privacyassistant.net
producm.ru
pro-okis.ru
prostitutki-novgoroda.org
ptr.ruvds.com
pts163.ru
purchasepillsnorx.com
pyrodesigns.com.au
qor360.com
queerspace.com
questionmarque.ch
quickloanbank.com
rankexperience.com
rapidsites.pro
rebrand.ly
redirectme.net
refererx.com
remmling.de
remontgruzovik.ru
rewardit.com
rfd-split.hr
riralmolamsaca.tk
robot-forex.biz
rockma.se
rp9.ru
ruclicks.com
runstocks.com
russkoe-zdorovie.ru
rusvideos.su
s8-nowy-wygraj.comli.com
sady-urala.ru
salmonfishingsacramentoriver.com
sanjosestartups.com
sax-sex.com
scanner-george.top
scanner-margo.top
schoolfiles.net
scoopquest.com
scopich.com
seofied.com
seo-platform.com
seorankinglinks.xyz
sexfreepornoxxx.com
sexy.babes.frontend-stack.top
sheerseo.com
shohanb.com
sigmund-freud.co.uk
siha.de
silvermature.net
site5.com
sitevaluation.org
sloopyjoes.com
small-games.biz
sms2x2.ru
snip.tw
sobecjvuwa.com.ru
soc-econom-problems.ru
socialtrade.biz
sotechco.co
souvenir.cc
spasswelt.net
spidtest.space
spl63.fr
sportbetfair.com
srecorder.com
stop-zavisimost.com
strangeduckfilms.com
streamin.to
strfls.com
strv.se
supermama.top
superstats.com
tdsing.ru
technika-remont.ru
telefonsexi.com
templates.radiodigital.co
tetracsaudi.com
thebluffs.com
thecoolimages.net
thefarmergame.com
thefds.net
thepornsex.org
thetoiletpaper.com
thin.me.pn
tizanidine4mgtablets.blogspot.com
tizanidinehcl4mgtablet.blogspot.com
tnctrx.com
token-lab.org
toloka.hurtom.com
tootoo.to
top10-online-games.com
top10-way.com
torontoplumbinggroup.com
torrentgamer.net
traffic100.com
traffmonster.info
trk-4.net
tuberkulezanet.ru
tula.howotorg.ru
tvory.predmety.in.ua
ublaze.ru
u-cheats.ru
udayavani.com
uggsale.online
ukrtextbook.com
ukrup.com
ultimatesetnewfreeallsoftupgradesystems.pw
um-razum.ru
underthesite.com
unrealcommander.biz
uptime-alpha.net
urlopener.blogspot.com.au
vacuumscleaner.com
vclicks.net
vektorpress.ru
velpanex.ru
vertaform.com
via-gra.webstarts.com
video-hollywood.ru
videosbox.ru
vip.51.la
virus-schutzmasken.de
visa-china.ru
vod.com.ua
vpnhowto.info
vuclip.com
vzlomtw.com
vzubah.com
watchmygf.net
webix.me
weburok.com
wejdz-tu.pl
westvilletowingservices.co.za
wikes.20fr.com
witclub.info
wjgony.com
woodyguthrie.se
wsgames.ru
privhosting.com
rabotaetvse.ru
rangoman.date
realmonte.net
rewardpoll.com
ricorsogiustizia.org
sashagreyblog.ga
saveindex.xyz
scanner-fred.top
secret.xn--oogle-wmc.com
sexreliz.com
sexy-screen-savers.com
shivafurnishings.com
soda.media
sports-supplements.us
stroilka.info
textads.men
the-usa-games.blogspot.com
thegolfclub.info
tilido.com
tizanidinehcl2mg.blogspot.com
topsiteminecraft.com
torrent-newgames.com
uhod-za-sobakoj.ru
ukkelberg.no
utiblog.fr
vepad.com
versaut.xxx-cam.webcam
vip2ch.com
whatsupinfoley.com
wordpress-crew.net
ww2awards.info
xchat26.myfreecams.com
xep.info
xn--80aebbcbcdemfkhba4byaehoejh8dza3v.xn--p1ai
xn----btbdvdh4aafrfciljm6k.xn--p1ai
xpresscare.ru
x-rates.ru
yellowads.men
yoopsie.com
youbrainboost.asia
zakon-ob-obrazovanii.ru
zeikopay.com
znakomstva-piter78.ru
znaniyapolza.ru
zophim.me
zx6.ru
zzlgxh.com
xcc24.pl
xn----9sbebi2bvzr7h.xn--p1ai
xn--80aafb2a.xn--p1ai
xn--80ak6aa92e.com
youjizz.com
zarenica.net
zigarettenonl.canalblog.com
zlatnajesen.com

476
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
12bet.com
1688.com
0912701309f8ce.com
123locker.com
1kinobig.ru
34.gs
3dracergames.com
7makemoneyonline.com
acgs.tk
adtech.us
allboard.xobor.de
alldubai.biz
am15.net
amazon-adsystem.com
amazon-seo-service.com
analytics-ads.xyz
andpolice.com
animal-drawings.com
apartamentwroclaw.eu
apioasisspacenet-a.akamaihd.net
apioutoboxnet-a.akamaihd.net
apple.com-cleaner.systems
as5000.com
ascat.porn
azazu.ru
40cg.com
4k-player.pl
7a2rnuey1tw9ar.ru
aad1f4acb0a373420d9b0c4202d38d94fa.com
aasoldes.fr
absugars.com
acheterviagrafr24.com
adelly.bg
alienwheel.es
alive-ua.com
all-streaming-media.com
allesohnegirls.net
allnews24.in
alveris.ru
androids-store.com
anmysite.com
apiholdingmypage-a.akamaihd.net
arabsexxxtube.com
atmagroup.ru
auto-zapchasti.org
avramstroy.ru
avtovolop.ru
ayodhya.co
badavit.com.ua
balans.shahterworld.org
bannerads.de
beauby.ru
bensbargains.net
bestadbid.com
bezsporno.ru
bitporno.sx
bluerobot.info
boazpower.com
bongiornos.info
bookmaker-bet.com
braip.com.br
bus-offer.com
capsderfudd.tk
carmuffler.net
cartierlovebraceletreplica.xyz
cartierlovestore.xyz
casablancamanor.co.za
cete.ru
cheapjerseysfootballshop.com
balitouroffice.com
bannerpower.com
beachtoday.ru
beauty-b0x.pl
beclean-nn.ru
best-way.men
bestserials.com
beyan.host.sk
bigpenisguide.com
bitcoinpile.com
bitcoinremote.com
bmw.afora.ru
boole.onlinewebshop.net
bravegirlsclub.com
bugshoot.cn
bupropion-sr-150-mg.us
buron.pw
buttons-for-your-website.com
cardiosport.com.ua
cascadelink.org
cctva.tv
charmstroy.info
cheap-pills-norx.com
chocolateslim-u-srbiji.com
cinemaflix.website
claytransformations.info
co.lumb.co
coaching-netz.info
compliance-jessica.top
compliance-julia.top
conciergegroup.org
concordexoticrentals.com
converse.ddsoldes.fr
cookie-law-enforcement-aa.xyz
copro.pw
covetnica.com
cupday.com
dailyrank.net
dear-diary.ru
delayreferat.ru
dev.citetick.com
digital-sale.su
discountbarbour.online
dstroy.su
electrik-avenue.com
eleimgo.pw
elektrozigaretten2.yn.lt
engines-usa.com
envaseslotusama.com
erotik-kostenlos.net
erotikstories.ru
errorfixing.space
creditnation.ru
cymbaltaandtizanidine.blogspot.com
daisye.top
darodar.com
data.vtc.pw
davebestdeals.com
ddospanel.com
design-lands.ru
dinkolove.ya.ru
discountliv.com
dneprsvet.com.ua
doeco.ru
domaincheck.io
doxysexy.com
dumuelave.xyz
dvr.biz.ua
e-buyeasy.com
e-commerce-seo1.com
ekto.ee
elmacho.xyz
epngo.bz
ertelecom.ru
es5.com
essaypro.com
facebook-mobile.xyz
fanoboi.com
farmingworm.com
filesvine.com
filmidivx.com
fingerprintjs.com
firstdrugmall.ru
foxjuegos.com
free-fbook-traffic.com
freelotto.com
funnel.co.za
funnypica.com
gambarkatabaru.com
getlaid-xxxhookupdirect.com
gk170.ru
globetrotting-culture.ru
gopro-online.info
gox.com.ua
grupografico-pilar.com.ar
gsasearchenginerankerdiscount.com
hao123.com
hawaiielectriclight.com
healbio.ru
homemature.net
hotblog.top
housemilan.ru
houseofgaga.ru
https-legalrc.biz
husky-shop.cz
hyhj.info
imitex-plus.ru
inboxdollars.com
esopini.com
europages.com.ru
extrabot.com
ezofest.sk
factorynightclub.com
familienzahnaerzte.com
fickluder69.com
folowsite.com
foto-basa.com
foto-sisek.porngalleries.top
foxtechfpv.com
franch.info
freshberry.com.ua
fyl.com.ru
gamingspark.com
gaz-voshod.ru
gdcentre.ru
gembird.com
gerhardhealer.com
greendream.com.ua
happy.new.yeartwit.com
heartofbeijing.blogspot.com
helpmymacfaster.trade
histock.info
hotcore.info
hoztorg-opt.ru
ido3.com
ilte.info
imgarcade.com
imperia31.ru
increasewwwtraffic.info
indo-export.ru
infodocsportal.com
internet-apteka.ru
irkutsk.zrus.org
je7.us
justdating.online
kalb.ru
kazrent.com
keywordhut.com
keywordspay.com
kinofak.net
klumba55.ru
komputernaya-pomosh-moscow.ru
labplus.ru
lacave.ntic.fr
lampokrat.ws
landreferat.ru
instasexyblog.com
iplusbit.blogspot.co.za
istanbulit.com
ix20.ru
izoll.ru
jacago.com
jav-idol.com
jerseyschinabizwholesale.us
jongose.ninja
junglenet-a.akamaihd.net
jurajskie.info
justprofit.xyz
kakadu-interior.com.ua
kchaxton.com
kino-key.info
kleine-titten.biz
klosetkitten.com
kochanelli.com
kountrylife.com
kraljeva-sutjeska.com
krasnodar.zrus.org
ladov.ru
larchik.net
lflash.ru
ledpolice.ru
lexixxx.com
liran-locks.com
littlesunraiser.com
live-sexcam.tk
mafa-free.com
mainlinehobby.net
maladot.com
masthopehomes.com
matsdale.com
mearns-tractors.co.uk
mega-bony-2017.pl
locationdesaison.com
lovelycraftyhome.com
macdamaged.space
magnetpress.sk
massage-info.nl
medizinreisen.de
metaxalonevstizanidine.blogspot.com
mirobuvi.com.ua
mmofreegames.online
mojpregled.com
money-every-day.com
moneyzzz.ru
moscow.online-podarki.com
moskva.nodup.ru
movies-in-theaters.net
musezone.ru
na15.ru
namecrumilchlet.tk
napalm51.nut.cc
natali-forex.com
noscrapleftbehind.co
nrjmobile.fr
obiavo.su
onescreen.cc
openmediasoft.com
orsonet.ru
outrageousdeal-a.akamaihd.net
parajumpersstore.online
pathwhelp.org
pc-virus-d0l92j2.pw
mexicotravelnet.com
minneapoliscopiers.com
mir-limuzinov.ru
misandesign.se
mobstarr.com
msk-diplomat.com
myseoconsultant.com
narkologiya-penza.ru
neodownload.webcam
nfvsz.com
nonameread45.live
november-lax.com
panchro.xyz
photochki.com
photokitchendesign.com
pivka.xyz
plusnetwork.com
poquoson.org
pornofeuer.com
pornophoto.xyz
postclass.com
pokerniydom.ru
popup-fdm.xyz
priceg.com
pricheski-video.com
privacylocationforloc.com
privatov-zapisi.ru
privetsochi.ru
produkto.net
prostitutki-novgoroda.org
putevka24.ru
qexyfu.bugs3.com
quick-offer.com
radiogambling.com
razvratnoe.org
research.ifmo.ru
rialp.getenjoyment.net
rimedia.org
rniaeba.ga
robertefuller.com
romhacking.ru
royalcar-ufa.ru
rubbed.us
savememoney.co.za
scenarii-1-sentyabrya.uroki.org.ua
scottbywater.com
scripted.com
securesmrt-dt.com
see-your-website-here.com
semalt.com
seminarykansas904.ml
seorankinglinks.us
seo-smm.kz
servicecenter.co.ua
serviporno.com
sexfreepornoxxx.com
sextracker.com
sex-znakomstva.online
sharebutton.org
shisha-swag.de
shop.xz618.com
sibdevice.ru
silvermature.net
sitevaluation.org
professionaldieselcare.com
profitfx.online
projectforte.ru
prpops.com
rangjued.com
rapevideosmovies.com
raschtextil.com.ua
razleton.com
rcb101.ru
referencemoi.com
reklama1.ru
remontvsamare.su
rvzr-a.akamaihd.net
sakhboard.ru
satoristudio.net
sbornik-zakonov.ru
scanner-walt.top
sdelai-prosto.ru
searchengineranker.email
searchimage.co
seccioncontrabajo.com
seorank.info
shmetall.com.ua
sierraapps.com
sims-sims.ru
skuteczna-dieta.co.pl
skylta.com
smart-balancewheel.com
smart-scripts.com
snegozaderzhatel.ru
snsdeainavi.info
socialmediasuggest.com
spidtest.org
spin2016.cf
startufa.ru
svarbit.com
sweepstakes.rewardit.com
swinon.site
tagil.zrus.org
thebitcoincode.com
thebluenoodle.com
thegameriders.com
thetattoohut.com
tizanidineandgabapentin.blogspot.com
tizanidineorflexeril.blogspot.com
tizanidinezanaflex.blogspot.com
top1-seo-service.com
socialbuttons.xyz
socialsignals24.com
soft-program.com
spy-app.info
start.myplaycity.com
strangeduckfilms.com
stromerrealty.com
studiofaca.com
studiofmp.com
sugvant.ru
tattooreligion.ru
techart24.com
tgtclick.com
tizanidine4mgtablets.blogspot.com
tizanidine6mg.blogspot.com
tizanidinepain.blogspot.com
tnctrx.com
tonerbox.kz
topsiteminecraft.com
torrentgamer.net
totu.us
tour-line.net
tracklead.net
trafficmania.com
tramadolandtizanidine.blogspot.com
tripper.de
trumpetedextremes.com
tuckpointingmasonrysystems.com
unblocksit.es
unionmarkt.de
uptime.com
vertaform.com
video-chat.cn
video-chat.in
vita.com.hr
vsdshnik.com
watchmyfb.pl
weboptimizes.com
wholesalejerseys-cheapest.com
wieseversa.no
williamrobsonproperty.com
wsgames.ru
wtsindia.in
www.888.com
www.labves.ru
xaylapdiendanang.com
xn--b1adccaf1bzj.xn--p1ai
xxx-treker.ru
yellocloud.be
yourserverisdown.com
yuarra.pluto.ro
zarenica.net
zvetki.ru
track-rankings.online
tracksurf.daooda.com
trahic.ru
turist-strani.ru
twodollarshows.com
uginekologa.com
underthesite.com
unrealcommander.biz
url-extractor.xyz
utiblog.fr
via-energy-order.com
videochat.cafe
vinsit.ru
vltai.com
vote-up.ru
vykup-avto-krasnodar.ru
webjam.com
wetgames.ru
whos.amung.us
winwotgold.pl
wnhjavlhezp.gq
wzgyyq.com
xblog.in
xmladserver.com
xnxxandxvideos.com
yeartwit.com
youdao.com
zarabotok--doma.ru
zatjmuzu.info
zero1.it
zhuravlev.info
zojirushi-products.ru
zytpirwai.net

10
dev-tools/test_units/ratelimittest-beta.txt
View file

@ -581,4 +581,14 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html>

14
dev-tools/test_units/ratelimittest-master.txt
View file

@ -1,11 +1,21 @@
<html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
ml><html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
34.66.25.221 0;
10.1.0.59 0;
127.0.0.1 0;

1248
dev-tools/tmp/robots.txt
View file

File diff suppressed because it is too large Load diff