vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-03 11:10:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.11.2698 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-11-06 22:07:10 +00:00
parent 25fb172ddd
commit dafc20668c
47 changed files with 65507 additions and 65500 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.11.2697 #### Version: V4.2021.11.2698
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.11.2697 #### Version: V4.2021.11.2698
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -11,7 +11,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2021.11.2697 #### Version: V4.2021.11.2698
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

8050
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2021.11.2697 #### Version: V4.2021.11.2698
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

8054
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

8054
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

8054
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

8054
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 1; 10.1.0.238 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 0; 10.1.0.238 0;
127.0.0.1 0; 127.0.0.1 0;

8054
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 1; 10.1.0.238 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 0; 10.1.0.238 0;
127.0.0.1 0; 127.0.0.1 0;

8054
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 1; 10.1.0.238 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 0; 10.1.0.238 0;
127.0.0.1 0; 127.0.0.1 0;

8054
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2697 2698

8054
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

8054
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.121.211.120 0; 20.119.55.253 0;

8054
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 1; 10.1.0.238 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 0; 10.1.0.238 0;
127.0.0.1 0; 127.0.0.1 0;

8054
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 1; 10.1.0.238 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 0; 10.1.0.238 0;
127.0.0.1 0; 127.0.0.1 0;

8054
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 1; 10.1.0.238 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 0; 10.1.0.238 0;
127.0.0.1 0; 127.0.0.1 0;

8054
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.121.211.120 0; 20.119.55.253 0;

8054
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.121.211.120 0; 20.119.55.253 0;

8054
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 1; 10.1.0.238 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Buck DTS Agent
Discobot FemtosearchBot
EasyDL Page Analyzer
Jbrofuzz Picscout
OutfoxBot Recorder
SpankBot Ripper
WebCopier heritrix
ZoominfoBot scan.lol
archive.org_bot serpstatbot
spyfu trendiction.de

324
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 404checker
360Spider Aboundexbot
404enemy Aipbot
80legs AllSubmitter
ADmantX Anarchie
ALittle Client Anarchy99
ASPSeek Ankit
AfD-Verbotsverfahren
Alexibot
Alligator
Anarchy
Anthill Anthill
Apexoo Asterias
Attach
AwarioSmartBot AwarioSmartBot
BLEXBot BBBike
BackWeb BDFetch
Badass BackDoorBot
Bandit Backlink-Ceck
BetaBot BacklinkCrawler
Bitacle Black Hole
BlackWidow BlackWidow
Blackboard Blow
Boardreader Bolt
Brandwatch BuiltWith
Bullseye BunnySlippers
BuzzSumo BuzzSumo
CCBot CATExplorador
CSHttp Calculon
CazoodleBot CheTeam
Cegbfeieh
CheeseBot
ChinaClaw ChinaClaw
Claritybot Cliqzbot
Cocolyzebot Crawling at Home Project
Cogentbot CrunchBot
Craftbot DIIbot
CrazyWebCrawler
Curious
CyotekWebCopy
DSearch
DTS Agent
DatabaseDriverMysqli
Demon Demon
Deusu DigitalPebble
Devil DittoSpyder
Digincore
Discobot
Discoverybot
DnBCrawler-Analytics DnBCrawler-Analytics
DnyzBot
DomCopBot
DomainAppender DomainAppender
Dotbot DomainSigmaCrawler
Domains Project
Download Wonder Download Wonder
Dragonfly Dragonfly
Drip Drip
ECCP/1.0 ECCP/1.0
EMail Siphon
EMail Wolf EMail Wolf
EasyDL
Ebingbong Ebingbong
ExtractorPro EirGrabber
EyeNetIE EroCrawler
Ezooms Exabot
FHscan ExtLinksBot
Fimap FemtosearchBot
Flunky
Foobot Foobot
Fyrebot Fuzz
G-i-g-a-b-o-t
GalaxyBot
GermCrawler GermCrawler
GetRight GetRight
GetWeb
Go!Zilla Go!Zilla
Go-Ahead-Got-It Gotit
GoZilla
GrabNet GrabNet
HEADMasterSEO Grabber
Harvest GrapeFX
Havij GrapeshotCrawler
HMView
HTMLparser
Haansoft
HaosouSpider
Heritrix
Humanlinks Humanlinks
IDBTE4M HybridBot
IRLbot IRLbot
Iblog Id-search
IlseBot InfoTekies
Image Sucker Intelliseek
InterGET
Internet Ninja Internet Ninja
InternetSeer
Iskanie Iskanie
JOC Web Spider
JamesBOT JamesBOT
JetCar Jetty
JikeSpider
Jorgee Jorgee
JustView LNSpiderguy
Jyxobot
LWP::Simple
Larbin Larbin
Leap
LeechFTP LeechFTP
Libwhisker LeechGet
LieBaoFast LexiBot
Likse Lftp
LinkWalker
Linkdexbot Linkdexbot
LinqiaScrapeBot LinkpadBot
Lmspider LinqiaRSSBot
Lipperhey
Litemage_walker
Ltx71 Ltx71
MJ12bot Majestic SEO
MSFrontPage MarkWatch
MTRobot Mass Downloader
Majestic-SEO Mata Hari
Majestic12
MauiBot MauiBot
MeanPath Bot MeanPath Bot
Meanpathbot
Mediatoolkitbot Mediatoolkitbot
MegaIndex.ru MegaIndex.ru
MicroMessenger Metauri
Microsoft Data Access Microsoft URL Control
Moblie Safari Mister PiX
Mojeek
Mojolicious
Morfeus Fucking Scanner Morfeus Fucking Scanner
Musobot Msrabot
NPbot Nameprotect
Navroad NearSite
NetAnts Nessus
NetMechanic Net Vampire
NetSpider NetZIP
Netcraft Netvibes
Nibbler Nibbler
Nikto NimbleCrawler
Nimbostratus
Nmap Nmap
Nutch
Offline Explorer
OnCrawl OnCrawl
OpenVAS
Openfind Openfind
OrangeSpider
OutclicksBot OutclicksBot
OutfoxBot OutfoxBot
PECL::HTTP PHPCrawl
POE-Component-Client-HTTP POE-Component-Client-HTTP
Papa Foto Page Analyzer
Pavuk PageAnalyzer
Picsearch PageGrabber
PageScorer
PeoplePal
Pi-Monster
Picscout
PictureFinder PictureFinder
Pimonster Pimonster
PleaseCrawl Pixray
Pockey
ProWebWalker ProWebWalker
Probethenet Probethenet
Psbot
Quick-Crawler Quick-Crawler
RSSingBot RankActive
RankActiveLinkBot
RankFlex RankFlex
RankingBot Rankivabot
ReGet ReGet
RebelMouse Reaper
Recorder RepoMonkey
Ripper RocketCrawler
Rogerbot
SBIder SBIder
SEOkicks-Robot
SEOprofiler
SEOstats SEOstats
SalesIntelligent SalesIntelligent
ScanAlert
Scanbot Scanbot
Scrapy ScreenerBot
Screaming
ScrepyBot ScrepyBot
SearchmetricsBot
Semrush
SemrushBot
SeobilityBot SeobilityBot
Seomoz Seomoz
SiteExplorer
SiteLockSpider SiteLockSpider
Sitebeam SiteSnagger
SlySearch Siteimprove
Snapbot
Snoopy
Sociscraper Sociscraper
Sosospider Sosospider
SpaceBison Sottopop
Spanner
Spbot
Spinn3r Spinn3r
Sqlworm SputnikBot
Stripper Sqworm
Sucker Sucker
Surfbot SuperHTTP
SurveyBot SurveyBot
Swiftbot
T0PHackTeam T0PHackTeam
T8Abot Teleport
Telesoft
Telesphoreo Telesphoreo
TightTwatBot Telesphorep
The Intraformant
Titan Titan
Toweyabot
Tracemyfile
Trendictionbot Trendictionbot
True_Robot True_Robot
TurnitinBot Turingos
TwengaBot URLy Warning
UnisterBot URLy.Warning
Upflow Upflow
V-BOT
VB Project VB Project
Vagabondo VeriCiteCrawler
Virusdie Voil
VoidEYE WISENutbot
WASALive-Bot
WWW-Collector-E WWW-Collector-E
WWWOFFLE
Web Auto Web Auto
Web Fetch Web Fetch
Web Sauger Web Pix
Web Sucker WebLeacher
WebAuto WebSauger
WebBandit
WebFetch
WebFuck
WebStripper WebStripper
WebZIP WebSucker
WebmasterWorldForumBot Website Quester
Whatweb WebsiteExtractor
Who.is Bot WebsiteQuester
Whack
Widow
WinHTTrack WinHTTrack
Wonderbot
Woobot Woobot
Wotbox YoudaoBot
Xaldon WebSpider Zauba
Xaldon_WebSpider Zeus
Xenu Zitebot
Zermelo ZoominfoBot
ZoomBot
ZyBorg ZyBorg
arquivo-web-crawler autoemailspider
cah.io.community backlink-check
check1.exe
coccocbot-web coccocbot-web
crawler4j cognitiveseo
domainsproject.org crawl.sogou.com
crawler.feedback
eCatch
evc-batch evc-batch
facebookscraper ips-agent
internetVista monitor isitwp.com
lwp-request lwp-trivial
meanpathbot satoristudio.net
muhstik-scan
netEstate NE Crawler
oBot
page scorer
pcBrowser
plumanalytics
scalaj-http scalaj-http
scan.lol seobility
seoscanners
seostar seostar
serpstatbot
sexsearcher sexsearcher
sitechecker.pro sitechecker.pro
siteripz
sogouspider sogouspider
spyfu spyfu
sysscan sysscan
tAkeOut tAkeOut
trendiction.com trendiction.com
trendiction.de
ubermetrics-technologies.com
voyagerx.com voyagerx.com
webmeup-crawler
webpros.com
x09Mozilla
x22Mozilla
zauba.io zauba.io
zgrab

284
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,248 +1,248 @@
404enemy 01h4x.com
404checker
80legs 80legs
Aboundexbot ADmantX
Alexibot AIBOT
AllSubmitter Abonti
AfD-Verbotsverfahren
Anarchie Anarchie
Ankit
Anthill Anthill
BDCbot Aspiegel
Attach
AwarioSmartBot
BackDoorBot BackDoorBot
BackStreet BackStreet
BackWeb BackWeb
Backlink-Ceck
BacklinkCrawler BacklinkCrawler
Badass
Bandit
Barkrowler Barkrowler
Battleztar Bazinga Battleztar Bazinga
BetaBot BetaBot
Bitacle Bitacle
Blackboard Black Hole
Blow BlackWidow
BotALot BlowFish
Brandprotect Buck
BuiltWith
Bullseye Bullseye
CCBot CCBot
Calculon CODE87
CazoodleBot
Cegbfeieh Cegbfeieh
CheTeam
CheeseBot
CherryPicker
ChinaClaw ChinaClaw
Chlooe
Cloud mapping Cloud mapping
Cocolyzebot
Collector
Copier Copier
CopyRightCheck
Copyscape
Cosmos Cosmos
Craftbot Craftbot
Crescent Crescent
CrunchBot CrunchBot
Custo CyotekWebCopy
DTS Agent DIIbot
DSearch
DatabaseDriverMysqli
Demon Demon
Deusu Deusu
Discobot Discobot
Dispatch
DittoSpyder DittoSpyder
DnyzBot DnBCrawler-Analytics
Dotbot DomCopBot
Download Wonder DomainCrawler
Drip DomainSigmaCrawler
EMail Siphon
EMail Wolf EMail Wolf
EasyDL
Ebingbong Ebingbong
EirGrabber EroCrawler
Exabot Express WebPictures
Extractor
EyeNetIE
Fimap
Firefox/7.0 Firefox/7.0
Foobot G-i-g-a-b-o-t
FrontPage
Fuzz
GT::WWW GT::WWW
GalaxyBot GalaxyBot
GetRight Genieo
GetWeb Getintent
Gigabot Gigabot
Go!Zilla
Go-Ahead-Got-It Go-Ahead-Got-It
GoZilla GrapeshotCrawler
GrabNet
Grabber
Grafula
GridBot GridBot
HaosouSpider HEADMasterSEO
Heritrix Heritrix
Humanlinks Humanlinks
HybridBot IDBTE4M
IRLbot IRLbot
IlseBot
Indy Library Indy Library
InfoNaviRobot InterGET
Intelliseek
Internet Ninja
InternetSeer
Iria
Iskanie Iskanie
IstellaBot IstellaBot
Jbrofuzz JOC Web Spider
JamesBOT
JennyBot JennyBot
JetCar Jetty
JustView JikeSpider
Kenjin Spider Jorgee
Keyword Density Kinza
Kozmosbot Larbin
LNSpiderguy LeechGet
LWP::Simple Libwhisker
Lanshanbot
LeechFTP
LexiBot
LibWeb
LieBaoFast
Lightspeedsystems Lightspeedsystems
Likse
Linkbot Linkbot
LinkextractorPro LinkextractorPro
LinqiaScrapeBot LinkpadBot
Lipperhey LinksManager
Lipperhey Spider LinqiaRSSBot
Lmspider Lmspider
Ltx71 Ltx71
MFC_Tear_Sample
MIIxpc
MSFrontPage MSFrontPage
MTRobot MTRobot
Mag-Net Mag-Net
Majestic SEO MarkMonitor
Majestic12
MarkWatch
Mass Downloader Mass Downloader
Mata Hari MauiBot
MeanPath Bot MegaIndex.ru
Meanpathbot MicroMessenger
Metauri
Microsoft URL Control Microsoft URL Control
Moblie Safari
Morfeus Fucking Scanner Morfeus Fucking Scanner
Msrabot Mozlila
NICErsPRO NICErsPRO
Name Intelligence Name Intelligence
Navroad NearSite
Needle
NetAnts NetAnts
NetLyzer NetLyzer
NetMechanic NetZIP
Netcraft Nettrack
Netvibes Netvibes
NextGenSearchBot
Nibbler Nibbler
NimbleCrawler NimbleCrawler
Nuclei Nimbostratus
Ninja
Nutch Nutch
Octopus Octopus
Offline Explorer OpenLinkProfiler
OpenVAS Openfind
OrangeBot
OrangeSpider OrangeSpider
OutclicksBot OutclicksBot
Page Analyzer POE-Component-Client-HTTP
PageGrabber PageGrabber
PageScorer PageScorer
Pavuk PageThing.com
Picsearch Panscient
PleaseCrawl Picscout
ProWebWalker ProWebWalker
Probethenet Probethenet
Psbot Pu_iN
Pump Pump
PxBroker Quick-Crawler
PyCurl
RankActive
RankFlex RankFlex
RankingBot
RankingBot2
Rankivabot Rankivabot
RealDownload Re-re
Reaper ReGet
RebelMouse RebelMouse
RedesScrapy RepoMonkey
SEOkicks Ripper
SEOkicks-Robot SMTBot
SISTRIX
ScanAlert ScanAlert
ScoutJet Screaming
Searchestate Searchestate
SearchmetricsBot SentiBot
Semrush
SeoSiteCheckup
SeobilityBot SeobilityBot
Seomoz
Site Sucker
SiteCheckerBotCrawler SiteCheckerBotCrawler
SiteLockSpider
SiteSucker SiteSucker
Sitebeam Siteimprove
Snapbot Sitevigil
SocialRankIOBot Snake
Snoopy
Sociscraper
Sogou web spider Sogou web spider
Sosospider
Sottopop Sottopop
SpaceBison SpankBot
SputnikBot Spanner
Sqlworm Sqlmap
Steeler
Sucuri Sucuri
SuperBot SurveyBot
Surfbot
Suzuran Suzuran
Swiftbot Swiftbot
Szukacz
T0PHackTeam T0PHackTeam
Telesphoreo Teleport
The Intraformant TeleportPro
Telesoft
TheNomad TheNomad
Titan Thumbor
Tracemyfile TightTwatBot
Trendiction
True_Robot
Turnitin
TurnitinBot TurnitinBot
URLy Warning TwengaBot
URLy.Warning VCI
VB Project
VelenPublicWebCrawler VelenPublicWebCrawler
VeriCiteCrawler
VoidEYE VoidEYE
WEBDAV Voil
WWW-Collector-E
WWW-Mechanize
WWW::Mechanize WWW::Mechanize
WWWOFFLE Wallpapers
Wallpapers/3.0 WeSEE
Web Auto Web Fuck
Web Collage Web Sucker
Web Enhancer WebCopier
Web Sauger WebEnhancer
WebBandit
WebFetch
WebFuck
WebGo IS
WebImageCollector WebImageCollector
WebZIP WebSauger
WebStripper
WebSucker
WebWhacker
WebmasterWorldForumBot WebmasterWorldForumBot
Website Quester Webster
WebsiteExtractor Whack
Whacker
Whatweb Whatweb
Who.is Bot WinHTTrack
Widow
WiseGuys Robot
Wonderbot Wonderbot
Woobot Woobot
Xaldon_WebSpider Xaldon_WebSpider
YoudaoBot Xenu
Zade
Zauba Zauba
Zermelo ZyBorg
Zeus backlink-check
Zitebot
arquivo-web-crawler
check1.exe
crawler.feedback
crawler4j crawler4j
dataforseo.com
demandbase-bot
domainsproject.org domainsproject.org
gopher facebookscraper
heritrix heritrix
internetVista monitor
ips-agent ips-agent
mediawords lwp-trivial
magpie-crawler
pcBrowser pcBrowser
plumanalytics plumanalytics
s1z.ru
scalaj-http scalaj-http
sexsearcher seoscanners
sitechecker.pro
siteripz siteripz
sogouspider
sp_auditbot sp_auditbot
sysscan sysscan
voyagerx.com trendiction.de
ubermetrics-technologies.com
webmeup-crawler
webprosbot webprosbot
x09Mozilla x09Mozilla
xpymep1.exe xpymep1.exe

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
bezlimitko.xyz fishingwholesale.us
buypillsorderonline.com fuck-paid-share-buttons.xyz
galaxy-family.ru gk-atlant.info
paphoselectricianandplumber.com intellektmedia.at
predominant-invent.tk mobilierland.com
sbornik-zakonov.ru roll123.com
simply.net sispe.com.br
sptslmtrafms.com ublaze.ru
thecoral.com.br uhod-za-sobakoj.ru
vrnelectro.ru xxxkaz.org

924
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
1001desert.com 00webcams.com
2ads.co.uk 101billion.com
300richmond.co.nz 123cha.com
3weekdiet.com 24x7-server-support.site
4kplayer.pl 2728fb936f0.com
6hopping.com 2pxg8bcf.top
aa81bf391151884adfa3dd677e41f94be1.com 34.gs
aa8780bb28a1de4eb5bff33c28a218a930.com 3w24.com
aac52006ec82a24e08b665f4db2b5013f7.com 5000-cotydzien.com
abbanreddy.com 5ws.dating-app.ru
abcguru.xyz 66cpwgln.space
acc.eu.org 7minuteworkout.com
addbin.men 883zy.com
admanaerofoil.com 8kisses.eu
adult-shop.com.ua aa625d84f1587749c1ab011d6f269f7d64.com
adultnet.in aad1f4acb0a373420d9b0c4202d38d94fa.com
agahinameh.com abovetherivernc.com
agysya.ru adf.ly
aihelen.net adhome.biz
aimayangzhi.com adloads.net
akama.com admatic.com.tr
alarmobninsk.ru adtiger.tk
alert-hgd.xyz agreda.pluto.ro
alienwheel.es ahmedabadwebs.com
all-streaming-media.com aktivator-windows10.blogspot.com
allproblog.com alvtank.se
alta-realestate.com amazingninja.com
anal-acrobats.hol.es ameblo.jp
android-systems.ru animalrank.com
android-vsem.org anniemation.com
angel1777.kz anonymous-redirect.com
anjalika.co.in antons-transporte.de
apartmentbay.ru apiappenableinfo-a.akamaihd.net
apimegabrowsebiz-a.akamaihd.net apisecretsaucebi-a.akamaihd.net
appearance-cool.com appiq.mobi
appleid-verification.com aproposde.com
architecturebest.com architecturebest.com
arendakvartir.kz arius.tech
arewater.com
artavenuegardenstudios.com artavenuegardenstudios.com
articlesdirectoryme.info
asdfz.pro
audiobangout.com
audiofree.ru
ausmepa.org.au ausmepa.org.au
aviav.org autogrep.ru
aviav.ru aviav.eu
avitocars.ru b2b-lounge.com
avkzarabotok.info babyfactory.fr
avon-novosib.ru babyguns.ru
azbuka-mo.ru banki76.ru
b00kmarks.com
babespcs.com
bad-stars.net
baersaratov.ru
balkanfarma.ru
bavsac.com bavsac.com
bazaronline24.ru bdsmgalls.net
beclean-nn.ru beatifullandscape.co
beauty-clinic.ru
best-way.men
bestbrainprod.win
bestbuy.ca
besthomemadepornsites.com
bestkfiledxd.cf
bestmarriages.com bestmarriages.com
binomo.kz bestmobilityscooterstoday.com
bioinnovate.co bestpriceninja.com
bitcoins-live.ru bestwaystogetpaid.us
bluerobot.info bet365.com
bocoarchives.org betonka.pro
bolezniorganov.ru betterhdporn.com
bongiornos.info betteroffers.review
book-bets.com beztuberkuleza.ru
bosefux.esy.es bget.ru
bouda.kvalitne.cz billiard-classic.com.ua
breastaugmentation.co.za bingo8888.com
brewdom.ru bio-optomarket.ru
brianhenry.co birzha-truda.eu
bloggedporn.com
boo-arts.com
brainsandeggs.blogspot.com
budpost.com.ua budpost.com.ua
builtwith.com bumskontakte.org
businesn.men buydissertation.net
buy-forum.ru bytimedance.ru
candlehandmade.com candiceloves.us
cannibalcheerleader.com cannibalcheerleader.com
canvas.gloverid.site canyougethighofftizanidine.blogspot.com
cartierlove.xyz carsdined.org
cartierreplica.top carson.getenjoyment.net
cartiertrinity.xyz carstrends2015.com
cascadelink.org cayado.snn.gr
castingbank.ru cdncash.com
cbcseward.com cdncash.net
chatroulette.world cejewelry.xyz
cheapsergiorossi.online centrumcoachingu.com
chastnoeporno.com
chatroulette.video
cheapestjerseysonwholesale.com
cheapjerseysap.com
checkpagerank.net checkpagerank.net
chiblackhawks-jerseys.com chimiver.info
chlooe.com chinajerseyswholesalecoupons.com
christianlouboutinshoes.xyz chocolateslim-original.com
circlesl.com chocolateslim-u-srbiji.com
clash-clans.ru cialis-samples.com
cleansearch.net cicaki.net
clicksor.com civilwartheater.com
clmforexeu.com cl.s7.exct.net
club-samodelkin.ru clubfashionista.com
coinsspb.com colehaanoutlet.store
community.allhiphop.com compliance-barak.top
compliance-alexa.top compliance-elena.top
contentsexpress.com compliance-john.top
copypaste.traderzplanet.in crazyboost.pro
covadhosting.biz culad.com
cpajunkies.com culmimg.pw
crackzplanet.com cvety24.by
cryptoswap.biz d-black.bz
customsua.com.ua d2jsp.org
cymbaltaandtizanidine.blogspot.com damasarenai.info
dailyfinancefix.com daymusam.com
dayibiao.com ddospanel.com
deanmoore.ie deplim.com
dekoration.us dev.citetick.com
demian.kz diesel-parts28.ru
den-noch24.ru dieswaene.com
dikx.gdn discountbarbour.online
disruptingdinnerparties.com discountliv.com
divku.ru div.as
doctormakarova.ru djonwatch.ru
domcran.net djstools.com
domik-derevne.ru domain-tracker.com
dominateforex.ml domaineaneblanc.com
domznaniy.ru domznaniy.ru
dorratex.tn dostavimvdom.ru
doska-vsem.ru downloaddy.net
downtuptv.gq doyouknowtheword-flummox.ml
dvervmoskvu.ru
dvrlists.com dvrlists.com
e-c.al dyshagi.ru
e-collantes.com e-buyeasy.com
e-commerce-seo1.com e-commerce-seo1.com
ednorxmed.com easync.io
ee77ee.com elitedollars.com
ekaterinburg.xrus.org embedle.com
el-nation.com enternet.ee
eldiariodeguadalajara.com enthuse.computernetworksonline.com
elektrozigaretten2.yn.lt erotik0049.com
elidelcream.weebly.com essaypro.com
eropho.com european-torches.ru
erotiktreff24.info exonline.info
event-tracking.com
everflora.ru
exchanges-bet.com
exhibitionplus.eu
eyelike.com.ua eyelike.com.ua
eywords-monitoring-your-success.com
ezigarettenkaufen2.dreamwidth.org
ezigarettenshop2.myblog.de ezigarettenshop2.myblog.de
ezigaretteshop2.mywapblog.com facialporntube.com
falcoware.com familienzahnaerzte.com
fapgon.com farmprofi.net
faracontrol.ir fickenbumsen.net
fashion-stickers.ru filmi-onlain.info
faz99.com findsexguide.com
fickkontaktehobbyhuren.com fitfloponline.store
fidelityfunding.com flashlarevista.com
fikasound.tk floating-share-buttons.com
finstroy.net floridamobilebillboards.com
floridahuntingfishingadventures.com forms-mtm.ru
flowersbazar.com formulaantiuban.com
footballfarrago.com frankofficial.ru
forensicpsychiatry.ru freelinkbuilding.website.tk
frameimage.org frivgame250.com
free-laptop-reward.com fsalas.com
free-share-buttons.com gamebackyard.com
free411games.com gamerscorps.com
freshberry.com.ua geilehausfrauen.net
freshwallpapers.info getlaid-xxxhookupdirect.com
front.ru glavtral.ru
fsakhalin.ru globatur.ru
fuel-gas.com
fugarif.ga
fun2cell.net
gallery.rennlist.com
gambarxkata.co
gamewrath.com
gaypornmovie.net
gayxperience.com
gemgrab-a.akamaihd.net
generic-pills-online.com
gfaq.ru
ghernnqr.skyrock.com
gilsonchiro.xyz
girlgamerdaily.com
girlsatgames.ru
gktt.ru
go2mike.ru go2mike.ru
goatse.ru goldandcard.ru
goblacked.com goldpanningtools.com
gobongo.info gratuitxblcodes.com
golden-catalog.pro green-tea.tv
goodhousekeeping.com gsmtlf.ru
goodwriterssales.com gsou.cf
googleseo.com.tr handicapvansarea.com
greatgrace.ru haveinc.xyz
growmyfunds.ca healthcarestore.info
guardlink.org hermesreplica.pw
guruofcasino.com histock.info
handsandlegs.ru hit-kino.ru
hanink.biz.ly homelygarden.com
hccoder.info host-protection.com
hd-filmy.net hostingclub.lk
hd720kino.ru hotblog.top
healgastro.com howtowhitenteethfast.xyz
healmytrauma.info hully.altervista.org
hermesbracelets.xyz hyipmanager.in
hitmuzik.ru ibmdatamanagement.co
hobild.net iflycapetown.co.za
honyaku.yahoofs.jp ilbe.club
hotchatdate.com ilmen.net
hothor.se imagerydatabase.com
hotsocialz.com img.wallpaperstock.net
houseofrose.com incomekey.net
hspline.com internetartfair.com
hulfingtonpost.com investingclub.ru
humanorightswatch.org
hundejo.com
hunterboots.online
i-hobot.ru
ilmexico.com
imabase.com
images.gyffu.com
imfamous.info
imgarcade.com
imitex-plus.ru
imperiafilm.ru
inboxdollars.com
infazavr.ru
invest-pamm.ru
investyb.com investyb.com
invitefashion.com invivo.hu
iptool.xyz iqs.biz.ua
istizanidineanopiate.blogspot.com isotoner.com
itag.pw
iwanttodeliver.com
janavibekken.no
jav-fetish.com
javatex.co.id javatex.co.id
javidol.site javspace.net
javitas.info
job.icivil.ir job.icivil.ir
kabbalah-red-bracelets.com jobgirl24.ru
kambasoft.com joessmogtestonly.com
kashubadesign.ru juliadiets.com
katadhin.co jwss.cc
kenaba.su kamen-e.ru
keywordsuggest.org karachev-city.ru
kinky-fetishes.com kazinogames.lv
kino-filmi.com ketoanhanoi.info
kino-key.info kino2018.cc
komputers-best.ru kit-opt.ru
konpax.com kollesa.ru
kovrenok.ru kongruan.com
kretpicf.pw
kurwa.win kurwa.win
kvartir-remont.biz kvartira-sutochno.com
kwzf.net lakomka.com.ua
landreferat.ru lampokrat.ws
landmania.ru
languagecode.com languagecode.com
laparfumotec.com laptop-4-less.com
larose.jb2c.me leftofcentrist.com
larutti.ru lightinghomes.net
ldrtrack.com likesuccess.com
lecbter-relationships.ga lineshops.biz
lemon-ade.site locustdesign.co
letslowbefast.today lunchrock.co
lineavabit.it
listiseltemournlan.gq
littlesexdolls.com
liver-chintai.org
llastbuy.ru
loginduepunti.it
lost-alpha.ru
ltvperf.com
macrotek.ru macrotek.ru
mahnwachen-helfen.info mactechinfo.info
mail.allnews24.in mafa-free.com
maju.bluesalt.co malls.com
makemoneyonline.com mamasuper.prom.ua
makis.nu manifestyourmillion.com
malwareremovalcenter.com manualterap.roleforum.ru
mcar.in.ua marketland.ml
meccadumps.net martlinker.com
medicinacom.ru mcnamaratech.com
medicine-4u.org mebeldekor.com.ua
megakino.net medmajor.ru
mesto-x.com metallosajding.ru
metabar.ru mezaruk.info
michaelkorssaleoutletonline.net
mil-stak.com
millionare.com millionare.com
mir-betting.ru minyetki.ru
misswell.net missis.top
mister-shop.com
mmofreegames.online mmofreegames.online
monarhs.info mmog-play.ru
moneymaster.ru mobstarr.com
morepoweronmac.trade moivestiy.biz
mossmesi.com monclercheap.xyz
mozello.ru moncleroutletonline.xyz
mp3films.ru mosrif.ru
mrmoneymustache.com mtmtv.info
musicas.baixar-musicas-gratis.com mug-na-chas-moscow.ru
music7s.me
mxgetcode.com mxgetcode.com
mybackgroundlandscape.blogspot.com myfreecams.com
mydownlodablefiles.com nakozhe.com
nacap.ru narkologiya-belgorod.ru
naturalbreakthroughsresearch.com narkologiya-penza.ru
navalwiki.info netfacet.net
ningessaybe.me netoil.no
noclegonline.info networkcheck.xyz
novgorod.xrus.org news-readers.ru
newstraveller.ru
nfljerseysforsalewholesaler.com
nofreezingmac.click
nofreezingmac.work
nudo.ca
nuit-artisanale.com nuit-artisanale.com
nullrefer.com nuup.info
obiavo.uz
obnal.org
obsessionphrases.com
odywpjtw.bloger.index.hr
oecnhs.info oecnhs.info
of-ireland.info of-ireland.info
omgtnc.com oksrv.com
onclkads.com okuos.com
openfrost.net omsk.xrus.org
oren-cats.ru online-hit.info
ownshop.cf onlineserialy.ru
ownshop.win optom-deshevo.ru
ovirus.ru
oxotl.com
oynat.info
p-business.ru p-business.ru
page2rss.com paintingplanet.ru
paleohub.info papasdelivery.ru
parajumpersjakkesalgnorge.info parajumpersjakkesalgnorge.info
partners-ship.pro paydayonlinecom.com
pay2me.pl petrovka-online.com
photochki.com photochki.com
picurams.pw picscout.com
pinapchik.com picturesfrom.com
pinstake.com piluli.info
pinwallpaper.xyz pizdeishn.com
pio.polytopesexempt.com pkr1hand.com
piter.xrus.org platesauto.com
plastjulgranar.se plusnetwork.com
platezhka.net polybuild.ru
play.leadzupc.com popserve.adscpm.net
plohaya-kreditnaya-istoriya.ru porn-w.org
pochtovyi-index.ru porndairy.in
podshipniki-nsk.ru porno-home365.com
poquoson.org porno-play.net
porno.simple-image.com.ua pornodojd.ru
pornzone.tv pornoelita.info
portadd.men pornoslive.net
potolokelekor.ru pornotubexxx.name
pozdravleniya-c.ru pornpost.in
predmety.in.ua pozdrawleniya.ru
priceg.com pp-budpostach.com.ua
profitfx.online pr-ten.de
promalp-universal.ru prefersurvey.net
promoheads.com printingpeach.com
push-ad.com privatamateure.com
pushdata.sendpulse.com privatov-zapisi.ru
pussyspace.net produkto.net
quality-traffic.com professionalwritingservices15.blogspot.ru
prombudpostach.com.ua
pssucai.info
ptr.ruvds.com
pwwysydh.com
quick-seeker.com
quickchange.cc quickchange.cc
raschtextil.com.ua qwarckoine.com
rasteniya-vs-zombi.ru rating-casino2021.ru
remybutler.fr rcb101.ru
rixpix.ru realizmobi.com
robot-forex.biz redhotfreebies.co.uk
rrutw.com remontbiz.ru
running-line.ru rethinkwasteni.info
ruscopybook.com rewardpoll.com
sabizonline.com rezeptiblud.ru
safe-app.net ribieiendom.no
ric.info
rniaeba.ga
rostov.xrus.org
ruf777.com
saletool.ru
sampleletters.net
sardinie.us sardinie.us
sbornik-zakonov.ru
sbricur.com sbricur.com
sbt-aqua.ru sbt-aqua.ru
scanner-margo.top scanner-fred.top
scanner-marwin.top scanner-jess.top
scmor.ilxc.cc scansafe.net
searchimage.co scanspyware.net
securesmrt-dt.com scat.porn
seeingmeerkat.com scenarii-1-sentyabrya.uroki.org.ua
seeresultshub-a.akamaihd.net scenicmissouri.us
seoanalyses.com schalke04fc.info
seozoom.it school-diplomat.ru
service.adtech.fr scottbywater.com
sdi-pme.com
sdsjweb.com
search.1and1.com
searchencrypt.com
semalt.com
serialsway.ucoz.ru
servicecenter.co.ua
sevendays.com.ua sevendays.com.ua
sexcamamateurchat.com sex-znakomstva.online
sexkontakte-seite.com sexblog.pw
sexstream.pl sexpornotales.net
shinikiev.com.ua sexspornotub.com
shoesonlinebuy.xyz sexuria.net
shop.acim.org shopcheermakeup.info
sientalyric.co sibdevice.ru
sitechecker.pro sibtest.ru
siteexpress.co.il sideeffectsoftizanidine.blogspot.com
siteonomy.com sildenafilcitratemed.com
sitopreferito.it simplepooltips.com
skanninge.se site-analyzer.com
slonechka.ru sitevaluation.org
sobecjvuwa.com.ru skutecznetabletkinaporostwlosow.pl
soc-econom-problems.ru smartbalanceworld.com
social-buttons.com snsdeainavi.info
sockshare.net social-fun.ru
sofit-dmd.ru soft-terminal.ru
softwaretrend.net softonicads.com
solnplast.ru sortthemesitesby.com
soserfis.com spb.ru
sovinsteel.ru speechfoodie.com
speeddream.xyz sport7777.net
splendorsearch-a.akamaihd.net sps-shop.com
sports-supplements.us stariy-baku.com
spymac.net statistici.ro
stard.shop stillmiracle.com
stardevine.com stjamesschool.info
stoki.ru storehouse.ua
stpolice.com stpolice.com
streamin.to streetfooduncovered.com
supers.com.ua stroy-portal22.ru
surintech.ac.th stuff-about-money.com
svolze.com suralink.com
szucs.ru susanholtphotography.com
taihouse.ru swagbucks.com
tam-gde-more.ru swinger-mobil.net
taraz.xkaz.org swingerseiten.com
td-33.ru tanieaukcje.com.pl
telefonsexsofort.tk technopellet.gr
template-kid.com teenporn18.net
tfxiq.com telefonsexkostenlos.tk
themestotal.com tengohydar.tk
thepokertimer.com terrafootwear.us
thexart.club thebluffs.com
tizanidineinfo.blogspot.com tiens2010.ru
tizanidinepain.blogspot.com tizanidine4mg.blogspot.com
tizanidineshowupondrugtest.blogspot.com tizanidineforopiatewithdrawal.blogspot.com
tizanidinezanaflex.blogspot.com tizanidinehcl4mgisitanarcotic.blogspot.com
toon-families.com tizanidinesideeffects.blogspot.com
toondinsey.com tomck.com
topvidos.ru topquality.cf
trackzapper.com torrents-tracker.com
trafaret74.ru track112.site
traffic100.com trafficgenius.xyz
trafficnetzwerk.de truck-rece.life
trainoffend.ml tube8.com
trichizobswiv.agddns.net twiclub.in
trustedhealthtips.com twinderbella.com
tubo360.com ukrup.com
tzritel.tk unisexjewelry.org
uhodzalijami.ru upupa.net
underthesite.com urccvfmc.bloger.index.hr
uptime-delta.net urengoy.pro
urzedowski.eu vandrie-ict.nl
us-america.ru vchulkah.net
usswrite.com vektorpress.ru
utrolive.ru velobikestock.com
uytmaster.ru venta-prom.ru
valkiria-tk.ru viandpet.com
vbtracker.net videos-for-your-business.com
vesnatehno.ru vip.51.la
vigrx-original.ru vip2ch.com
virus-schutzmasken.de viven.host.sk
viteonlusarezzo.it viveresaniesnelli.it
vizitki.net voloslove.ru
vladhistory.com volume-pills.biz
vsexkontakte.net voluumtracker1.com
vzubah.com voluumtrk.com
wallpapers-best.com voronezh.xrus.org
warezkeeper.com vostoktrade.info
vrotike.ru
vseuznaem.com
wait3sec.org
walkme.com
wapsite.me
warning.or.kr
waterefficiency.co waterefficiency.co
web-analytics.date we-are-gamers.com
webalan.ru webix.biz
webnode.me webshoppermac.com
website-analyzer.info website-audit.com.ua
websiteaccountant.de website-stealer.nufaq.com
westsextube.com weightatraining.com
wetgames.ru wesharepics.info
wholesalejerseychinashop.com whereverdesperate.gq
winterclassichockeyjerseys.com whosonmyserver.com
woman-h.ru workle.website
womensterritory.ru worldis.me
world-mmo.com
worldinternetauthority.com
wowcasinoonline.ooo wowcasinoonline.ooo
wsgames.ru wstroika.ru
ww1943.ru ww2awards.info
x-true.info www.pinnacle-bets.com
xbaboon.com x-diesel.com
xfluro.com x-diesel.org
xn----8sbdbjgb1ap7a9c4czbh.xn--p1acf xaylapdiendanang.com
xn----8sblgmbj1a1bk8l.xn----161-4vemb6cjl7anbaea3afninj.xn--p1ai xcc24.pl
xn----9sbebi2bvzr7h.xn--p1ai xev.ru
xn--80abgj3a5acid6ghs.top xmlinde.com
xn--80aodinpgi.xn--p1ai xn----7sbho2agebbhlivy.xn--p1ai
xn--e1afanlbnfckd7c3d.xn--p1ai xn----8sbarihbihxpxqgaf0g1e.xn--80adxhks
xn--hxazdsfy.blogspot.com xn----btbdvdh4aafrfciljm6k.xn--p1ai
xn--q1a.xn--b1aube0e.xn--c1acygb.xn--p1ai xn--d1aifoe0a9a.top
xxxhdvideo.site xnxx-n.com
yachts-cruise.info xsion.net
yadro.ru yellocloud.be
yoopsie.com yoluxuryevents.com
yougotanewdomain.com youporn-ru.com
yoursearch.me yourtemplatefinder.com
yur-p.ru youtube-downloader.savetubevideo.com
z-master.ru youtuhe.com
zajm-pod-zalog-nedvizhimosti.ru zdesformula.ru
zakazfutbolki.com zero1.it
zeroredirect8.com zeroredirect2.com
zetmaster.ru zigarettenonlinekaufen2.bloog.pl
zigarettenonlinekaufen2.drupalgardens.com zirondelli.it
zona-aqua.ru zone-kev717.info
zoomovies.org zoosexart.com
zootravel.com
zryydi.com zryydi.com
zvezdagedon.ru

482
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
00it.com 08800.top
123movies.love 10bet.com
4inn.ru 214jaluwobapef.cf
5kstore.com 28n2gl3wfyb0.ru
a.frcls.fr 4kpics.rocks
accessoires-mode.in 7zap.com
adanih.com 8gold.com
adssafeprotected.com adloads.net
ahrefs.com admanaerofoil.com
akusherok.ru ahmedabadwebs.com
alert-fjg.xyz aksonural.ru
all-streaming-media.com amateurlivechat.org
allboard.xobor.de anjalika.co.in
allornamenti.com apiappenableinfo-a.akamaihd.net
animalrank.com atmovs.com
anime.dougasouko.com autobudpostach.club
anlimebel.kiev.ua autolombard-krasnodar.ru
anonymous-redirect.com autonew.biz
anonymousfox.co avek.ru
apccargo.com avto-oligarh.ru
apibatbrowsecom-a.akamaihd.net avtocredit-legko.ru
apiwoodensealcom-a.akamaihd.net bad-stars.net
arate.ru bag77.ru
arcteryxsale.online bget.ru
arpelsreplica.xyz bibys.com
arraty.altervista.org bio-optomarket.ru
artel116.ru bird1.ru
as5000.com bizzliving.com
astana.xxxkaz.org blackwitchcraft.ru
auto4style.ru blogsfunky672.weebly.com
avtolombard-krasnodar.ru btc4u.ru
azte.ch bumascloset.com
bambi.ck.ua camel-beach.com
bannerconnect.net cartierlove2u.com
bausparen.bz.it cete.ru
best-seo-offer.com cheatcode-lita12.rhcloud.com
bestcoin.cc chee-by.biz
bharatdefencekavach.com chlooe.com
bizcheapjerseyswholesalechina.com cleanallspyware.com
bizru.info coinsspb.com
bonux.nextview.ru
bosman.pluto.ro
brainboostingsupplements.org
brandedleadgeneration.com
brimstonehillfortress.org
castingbank.ru
cctva.tv
celebsopics.com
cheapbarbour.online
choosecuisine.com
christianlouboutinreplica.pw
clknsee.com
club-lukojl.ru
comissionka.net
compliance-alex.top
compliance-george.xyz
compliance-walter.top compliance-walter.top
converse.ddsoldes.fr cornerstone-countertops.com
coolyarddecorations.com
cordstrap.cc
courtshipgift.com
crafthubs.com
creditnation.ru creditnation.ru
dieswaene.com dailyrank.net
digital-sale.su daisye.top
dimfour.com dellalimov.com
dividendo.ru derevesendeco.com
dogclothing.store diffbot.com
doktoronline.no digest-project.ru
domainsatcost.com dotmass.top
dyshagi.ru download-wallpaper.net
ebonyporn.site duplicashapp.com
ed-shop01.ru dvervmoskvu.ru
editors.choice6912650.hulfingtonpost.com eduserver.net
ekspresihati.info elektrozigarette1.dreamwidth.org
elektrischezigarette2.onsugar.com elitesportsadvisor.com
eorogo.top eloxal.ru
errorfixing.space energydiet24.ru
escort-russian.com erotikchat-24.com
essaypro.com evidencecleanergold.com
eu-cookie-law.blogspot.com exchanges-bet.com
expertnaya-ocenka.ru falcoware.com
fable.in.ua family1st.ca
factorynightclub.com fashion-mk.net
fanrto.com femmesdenudees.com
fetroshok.ru filkhbr.com
filmania-x.ru forum69.info
finstroy.net free-social-buttons.xyz
fitness-video.net free-today.com
flex4launch.ru free-traffic.xyz
forex21.ru freegamesplay.online
forums.d2jsp.org freejabs.com
foto-weinberger.at fungirlsgames.net
fr-bearings.ru fz139.ttk.ru
franklinfire.co galeria-zdjec.com
freetangodownload.com galleryawesome.com
frighteningremain.cf gallictures.com
fsakhalin.ru gay-site.store
funtoonez.com gigixo.com
get-your-social-buttons.info gobongo.info
ghernnqr.skyrock.com goforexvps.com
gok-kasten.net goldpanningtools.com
greatgrace.ru gotomontenegro.net
hdpixent.com guarrasdelporno.xxx
hostcritique.com gwebtools.com.br
hoverboardforsaledirect.com hanink.biz.ly
iboss.com hidefiles.org
ideibiznesa2015.ru hopeonthestreet.co.uk
iklysha.ml imitex-plus.ru
irkutsk.online-podarki.com infobabki.ru
istock-mebel.ru internet-apteka.ru
itag.pw ip-guide.com
itronics.ca javidol.site
iwanttodeliver.com jo24news.com
javrip.net joy-penguin.com
joingames.org karaganda.xkaz.org
katadhin.co kartiniresto.com
kinohit1.ru kellyonline.xyz
klejonka.info klikbonus.com
knowyournextmove.com konica.kz
kolotiloff.ru kustanay.kz
komputers-best.ru landoftracking.com
kostenlos-sexvideos.com life-instyle.com
ladov.ru lollypopgaming.com
landscapebackgrounds.blogspot.com lyrics.home-task.com
lashstudia.ru macdamaged.tech
likesuccess.com mactechinfo.info
linerdrilling.com marktforschung-stuttgart.com
lion.playtap.us marsgatan.com
lirunet.ru medicineseasybuy.com
livefixer.com metabar.ru
llastbuy.ru metaxalonevstizanidine.blogspot.com
lost-alpha.ru misandesign.se
mafa-free.com mitrasound.ru
magda-gadalka.ru mockupui.com
makis.nu moneymaster.ru
marketland.ml moscow.online-podarki.com
matchpal-a.akamaihd.net moviemail-online.co.uk
med-zdorovie.com.ua mp3s.club
member-quiz.com msfsaar.de
michaelkorsoutletstore.net
mir-limuzinov.ru
mp3ringtone.info
mp3ritm.top
mulberryoutletonlineeu.com mulberryoutletonlineeu.com
music.utrolive.ru mundoaberrante.com
myliveblog.ru mydownlodablefiles.com
nastydollars.com na15.ru
negociosdasha.com nalogovyykodeks.ru
njkmznnb.ru nastroyke.net
nodding-passion.tk naturalpharm.com.ua
nyfinance.ml nevansk.ru
oivcvx.website ngps1.ru
online-x.ru nomuos.it
openmultipleurl.com notebook-pro.ru
nottyu.xyz
nucia.biz.ly
o-o-8-o-o.com
oakleyglassesonline.us
obiavo.su
oda.as
offf.info
openfrost.net
ostrovtaxi.ru ostrovtaxi.ru
paighambot.com pacificair.com
pandroid.co partner-stop.men
partner-pop.men partnerads.men
pcgroup.com.uy parvezmia.xyz
photorepair.ru pearlisland.ru
picquery.com photochki.com
picturesfrom.com pic-re.blogspot.com
picsearch.com
pizdeishn.com
pk-services.ru pk-services.ru
placid-rounded-coast.glitch.me pkr1hand.com
plusnetwork.com pornoreino.com
pochemychka.net pp-budpostach.com.ua
podshipniki-ntn.ru privatefx-in.ru
poptool.net pro-tec.kz
porno.simple-image.com.ua projectforte.ru
pornofiljmi.com prostitutki-kharkova.org
portside.cc pts163.ru
profitkode.com pukaporn.com
pskcijdc.bloger.index.hr puteshestvennik.com
quelle.ru qexyfu.bugs3.com
realmonte.net qld10000.net
richinvestmonitor.com qualitymarketzone.com
roma-kukareku.livejournal.com r-control.ru
rukino.org rankscanner.com
saltspray.ru razvratnoe.org
samoiedo.it realitykings.com
rednise.com
regionshop.biz
remote-dba.de
responsive-test.net
ricorsogiustizia.org
robotixix.com
rock-cafe.info
rosbalt.com.ua
rossmark.ru
rusvideos.su
rybalka-opt.ru
sammlungfotos.online
sanidumps.com
sanyuprojects.com
sarafangel.ru
sardinie.us sardinie.us
scanner-donald.top saugatuck.com
scansafe.net sbetodiodnye-lampy.ru
seansonline24.pl seansonline24.pl
seeresultshub-a.akamaihd.net seeresultshub-a.akamaihd.net
seo-smm.kz seo-traffic-ranking.info
seo18.su
seotoolsagency.com seotoolsagency.com
servicecenter.co.ua seozoom.it
servisural.ru sethrollins.net
sientalyric.co sexkrasivo.net
simply.net sharebutton.net
sledstvie-veli.net shivafurnishings.com
slujbauborki.ru shodanhq.com
smallseotools.com shtaketniki.ru
soc-proof.su signal03.ru
social-s-iii.xyz site3.free-share-buttons.com
socialbuttons.xyz sitevaluation.org
statistici.ro skutecznetabletkinaporostwlosow.pl
stats-collector.org skypasss.com
streetfooduncovered.com slowmac.tech
studiokamyk.com.pl social-s-ggg.xyz
suchenindeutschland.com sockshares.tv
taaaak.com sugarlyflex.pw
teastory.co summerlinhomes411.info
teesdaleflyballclub.co.uk surcentro.com
texbaza.by surflinksmedical.com
thaisamkok.com surintech.ac.th
thenetinfo.com svarbit.com
time-japan.ru swimpool.ca
tizanidine4mgprice.blogspot.com syndicate.fun
top10-online-games.com techart24.com
toxicwap.com telesvoboda.ru
traffic2cash.org thebestweightlosspills.ovh
trkur.com thecoolimages.net
trustl.life theporndude.com
tubeoffline.com tizanidinedosage.blogspot.com
tvorozhnaja-zapekanka-recept.ru tizanidinerecreationaluse.blogspot.com
twojebook.pl tonivedu.it
uchetunet.su top10registrycleaners.com
unacittaconte.org topkarkas.com
unitygame3d.com torrentgamer.net
valoresito.com turizm.bz
vgoloveboli.net twittruth.com
viagraneggrx.com twodollarshows.com
videosbox.ru u555u.info
vip.51.la uhod-za-sobakoj.ru
voluumtracker1.com uni.me
web-betting.ru uptime-as.net
webcamdevochka.com validdomain.xyz
website-datenbank.de voloomoney.com
websitevaluebot.com volume-pills.biz
whereiskentoday.com vseigru.one
wholesalenfljerseys.us.com vzubah.com
womens-journal.net wellcome2slovenia.ru
ww2awards.info worldinternetauthority.com
www.get-free-traffic-now.com worldlovers.ru
www.timer4web.com wygraj-teraz.com
xjlottery.com xn--lifehacer-1rb.com
xn--80ajjbdhgmudixfjc8c5a9df8b.xn--p1ai yoursite.com
xn--90acenikpebbdd4f6d.xn--p1ai zakon-ob-obrazovanii.ru
xn--oogle-wmc.com zarabotok--doma.ru
xsion.net zeroredirect10.com
xxxguitars.com zeroredirect11.com
yellowstonevisitortours.com znakomstva-piter78.ru
yes-com.com znakomstvaonlain.ru
yottos.com zoominfo.com
yur-p.ru zygophyceous.womanstars.site
zakonobosago.ru
zigarettenonlinekaufen2.drupalgardens.com

41
dev-tools/test_units/ratelimittest-master.txt
View file

@ -8,23 +8,6 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html>ml>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><html> </html><html>
<head><title>503 Service Temporarily Unavailable</title></head> <head><title>503 Service Temporarily Unavailable</title></head>
<body> <body>
@ -494,6 +477,20 @@
<hr><center>nginx/1.18.0 (Ubuntu)</center> <hr><center>nginx/1.18.0 (Ubuntu)</center>
</body> </body>
</html> </html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<!DOCTYPE html> <!DOCTYPE html>
<html> <html>
<head> <head>
@ -584,4 +581,14 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html> </html>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.210 0; 10.1.0.238 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.11.2697 ### Version: V4.2021.11.2698
### Updated: Sat Nov 6 10:03:35 UTC 2021 ### Updated: Sat Nov 6 22:01:17 UTC 2021
### Bad Bot Count: 624 ### Bad Bot Count: 624
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.11.2697 ### Version: V4.2021.11.2698
### Updated: Sat Nov 6 10:03:35 UTC 2021 ### Updated: Sat Nov 6 22:01:17 UTC 2021
### Bad Bot Count: 624 ### Bad Bot Count: 624
################################################### ###################################################
### Version Information ## ### Version Information ##