vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-05 12:09:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.10.2632 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-10-05 10:09:15 +00:00
parent 66df4fd433
commit edb81df949
47 changed files with 82902 additions and 82892 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.10.2631 #### Version: V4.2021.10.2632
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.10.2631 #### Version: V4.2021.10.2632
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -11,7 +11,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2021.10.2631 #### Version: V4.2021.10.2632
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

10226
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2021.10.2631 #### Version: V4.2021.10.2632
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

10230
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

10230
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

10230
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

10230
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 1; 10.1.1.69 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 0; 10.1.1.69 0;
127.0.0.1 0; 127.0.0.1 0;

10230
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 1; 10.1.1.69 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 0; 10.1.1.69 0;
127.0.0.1 0; 127.0.0.1 0;

10230
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 1; 10.1.1.69 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 0; 10.1.1.69 0;
127.0.0.1 0; 127.0.0.1 0;

10230
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2631 2632

10230
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

10230
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
104.211.47.82 0; 13.82.87.85 0;

10230
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 1; 10.1.1.69 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 0; 10.1.1.69 0;
127.0.0.1 0; 127.0.0.1 0;

10230
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 1; 10.1.1.69 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 0; 10.1.1.69 0;
127.0.0.1 0; 127.0.0.1 0;

10230
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 1; 10.1.1.69 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 0; 10.1.1.69 0;
127.0.0.1 0; 127.0.0.1 0;

10230
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
104.211.47.82 0; 13.82.87.85 0;

10230
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
104.211.47.82 0; 13.82.87.85 0;

10230
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 1; 10.1.1.69 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Aboundex 01h4x.com
Anarchie Battleztar Bazinga
Claritybot G-i-g-a-b-o-t
Domains Project JOC Web Spider
GrapeFX Lanshanbot
MSIECrawler MJ12bot
Probethenet Masscan
RankurBot Mojeek
RebelMouse SiteCheckerBotCrawler
Snoopy Trendiction

296
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com
360Spider 360Spider
404checker
80legs 80legs
ADmantX ADmantX
AIBOT AIBOT
Abonti ASPSeek
Aboundex Aboundex
Acunetix
AiHitBot AiHitBot
Aipbot Aipbot
Alexibot
AllSubmitter
AlphaBot
Anarchie Anarchie
Asterias Attach
AwarioSmartBot AwarioSmartBot
BDFetch
BackStreet
BackWeb BackWeb
BacklinkCrawler
Badass
Bandit Bandit
Battleztar Bazinga
BetaBot
Bigfoot Bigfoot
Black Hole Blow
Blackboard Brandprotect
Boardreader Brandwatch
Bolt
Buck Buck
Buddy
Bullseye
BuzzSumo BuzzSumo
CATExplorador CCBot
CODE87 CODE87
CSHttp CensysInspect
CazoodleBot CheTeam
Cegbfeieh CherryPicker
ChinaClaw Chlooe
Cloud mapping Claritybot
Cliqzbot
Copier Copier
Craftbot CopyRightCheck
Copyscape
Crawling at Home Project
CrunchBot CrunchBot
Custo DBLBot
Demon Demon
Digincore Devil
DigitalPebble Discoverybot
Dispatch DomainCrawler
DittoSpyder DomainSigmaCrawler
DomainStatsBot DomainStatsBot
Download Wonder
Drip
ECCP/1.0 ECCP/1.0
EMail Siphon
EMail Wolf EMail Wolf
Ecxi EasyDL
Ebingbong
EirGrabber EirGrabber
Exabot FDM
Express WebPictures FHscan
EyeNetIE Firefox/7.0
Ezooms
Fimap
Flunky Flunky
Freeuploader
FrontPage FrontPage
Fyrebot Fyrebot
G-i-g-a-b-o-t GermCrawler
Genieo GetWeb
Getintent Getintent
GrabNet GoZilla
Grafula Grafula
GrapeshotCrawler HEADMasterSEO
HMView
HTMLparser HTMLparser
Haansoft Haansoft
Harvest Heritrix
Havij Hloader
Humanlinks Humanlinks
Iblog HybridBot
IlseBot IDBot
Image Fetch
Image Sucker Image Sucker
Indy Library InfoNaviRobot
InfoTekies
InterGET
Internet Ninja
Iria
Iskanie Iskanie
JOC Web Spider
JamesBOT JamesBOT
JennyBot
JetCar
Jetty Jetty
JikeSpider Joomla
Jorgee Jorgee
JustView
Jyxobot
Kenjin Spider Kenjin Spider
Kinza Keyword Density
LWP::Simple Kozmosbot
LNSpiderguy
Lanshanbot Lanshanbot
Larbin Larbin
Leap LeechGet
LeechFTP
LibWeb LibWeb
LieBaoFast
Lightspeedsystems Lightspeedsystems
Likse
LinkScan LinkScan
Linkbot LinkWalker
Linkdexbot LinksManager
LinkextractorPro
LinkpadBot
LinqiaRSSBot
Lipperhey Spider Lipperhey Spider
Litemage_walker MIDown tool
Lmspider MIIxpc
Ltx71
MJ12bot MJ12bot
MQQBrowser MQQBrowser
MSFrontPage MSFrontPage
Majestic-SEO Mag-Net
Mb2345Browser Magnet
MarkMonitor
Mass Downloader
MeanPath Bot MeanPath Bot
Meanpathbot Meanpathbot
Mediatoolkitbot Metauri
MicroMessenger MicroMessenger
Mojeek Mister PiX
Morfeus Fucking Scanner Mojolicious
Mr.4x3
NICErsPRO NICErsPRO
NPbot NPbot
Name Intelligence Name Intelligence
Nameprotect
Navroad Navroad
NearSite Needle
Nessus
NetAnts
NetMechanic NetMechanic
Netcraft Netcraft
NimbleCrawler Nettrack
Nimbostratus
Ninja Ninja
Nuclei Nmap
Nutch
OnCrawl OnCrawl
Openfind OpenLinkProfiler
OrangeBot OrangeBot
OrangeSpider
OutclicksBot OutclicksBot
PECL::HTTP PECL::HTTP
POE-Component-Client-HTTP PageGrabber
PageAnalyzer
PageScorer PageScorer
PageThing.com
Panscient
Pavuk
PeoplePal PeoplePal
Piepmatz Piepmatz
Pimonster
PleaseCrawl PleaseCrawl
ProPowerBot ProPowerBot
ProWebWalker ProWebWalker
Psbot PxBroker
PyCurl RankActive
QueryN Metasearch
Quick-Crawler
RankFlex
RankingBot RankingBot
RankurBot Re-re
RepoMonkey ReGet
SBIder RebelMouse
SEOkicks Recorder
Ripper
RocketCrawler
Rogerbot
SEOkicks-Robot SEOkicks-Robot
SMTBot SEOlyticsCrawler
ScanAlert SEOstats
Scrapy SalesIntelligent
Scanbot
ScoutJet
Screaming Screaming
Searchestate SentiBot
SearchmetricsBot SeoSiteCheckup
SemrushBot Seomoz
SeobilityBot Siphon
SiteExplorer
Sitebeam
Siteimprove Siteimprove
SlySearch SlySearch
SmartDownload
Snake Snake
Snoopy Snoopy
Sosospider Sosospider
SpankBot Sottopop
Spammen
Spanner Spanner
Spinn3r Sqlworm
SurveyBot Sqworm
Sucuri
SuperBot
Surfbot
Swiftbot Swiftbot
T8Abot T8Abot
Teleport Teleport
TeleportPro
Telesoft
Telesphoreo Telesphoreo
The Intraformant Thumbor
TheNomad TightTwatBot
Titan Titan
True_Robot Toweyabot
Tracemyfile
Turingos Turingos
Turnitin Turnitin
TurnitinBot TurnitinBot
TwengaBot UnisterBot
Twice VB Project
Typhoeus VCI
Upflow Vagabondo
V-BOT Voil
VelenPublicWebCrawler Voltron
VidibleScraper
Virusdie
WASALive-Bot WASALive-Bot
WBSearchBot
WPScan WPScan
WWW-Collector-E WWW-Mechanize
WWW::Mechanize Wallpapers
WWWOFFLE Wallpapers/3.0
WallpapersHD
WeSEE
Web Enhancer
Web Pix
Web Sauger Web Sauger
WebAuto
WebBandit WebBandit
WebCollage WebCopier
WebEnhancer
WebFetch WebFetch
WebFuck WebGo IS
WebImageCollector
WebLeacher
WebPix
WebReaper WebReaper
WebStripper WebSauger
WebZIP WebSucker
Webshag Webshag
WebsiteExtractor WebsiteExtractor
WebsiteQuester
Webster Webster
Whacker Whacker
Who.is Bot Widow
Woobot WiseGuys Robot
Wotbox Xenu
Wprecon
Xaldon WebSpider
Xaldon_WebSpider
YoudaoBot
Zade Zade
ZmEu Zermelo
ZoomBot ZoomBot
ZoominfoBot archive.org_bot
backlink-check arquivo-web-crawler
coccocbot-web cah.io.community
com.plumanalytics
crawl.sogou.com
crawler4j crawler4j
eCatch domainsproject.org
gopher gopher
heritrix
instabid
internetVista monitor internetVista monitor
magpie-crawler ips-agent
meanpathbot mediawords
netEstate NE Crawler netEstate NE Crawler
oBot plumanalytics
polaris version
ripz ripz
s1z.ru satoristudio.net
scan.lol scalaj-http
seobility seobility
seoscanners seoscanners
sexsearcher seostar
sogouspider siteripz
spyfu
sysscan sysscan
ubermetrics-technologies.com trendiction.com
webmeup-crawler trendiction.de
webprosbot zauba.io
x22Mozilla

306
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
360Spider 404enemy
80legs Abonti
Aboundexbot Aboundex
Acunetix
AiHitBot
AllSubmitter AllSubmitter
AlphaBot AlphaBot
Anarchie Anarchy
Anarchy99 Asterias
Ankit Attach
Anthill
Apexoo
Aspiegel
AwarioRssBot
AwarioSmartBot AwarioSmartBot
BDCbot BBBike
BDFetch
BLEXBot
BackDoorBot
BackStreet BackStreet
BackWeb
Backlink-Ceck Backlink-Ceck
BacklinkCrawler
Bandit Bandit
Bigfoot Bigfoot
Bitacle Bitacle
Black Hole BlackWidow
BlowFish BotALot
Bolt
Brandprotect Brandprotect
Brandwatch
Buck Buck
BuiltBotTough
Bullseye Bullseye
CATExplorador CCBot
CODE87
Calculon
CazoodleBot
CheTeam CheTeam
ChinaClaw CheeseBot
Cliqzbot Claritybot
Cloud mapping
Cocolyzebot
Cogentbot Cogentbot
Copyscape Copier
Craftbot Cosmos
Crawling at Home Project Crescent
CrazyWebCrawler
CrunchBot
Custo
CyotekWebCopy CyotekWebCopy
DataCha0s DSearch
DomCopBot DatabaseDriverMysqli
Digincore
Discoverybot
Dispatch
DnyzBot
DomainAppender DomainAppender
Domains Project DomainCrawler
Dotbot Dotbot
Dragonfly Drip
ECCP/1.0
EMail Siphon EMail Siphon
Ebingbong
EirGrabber EirGrabber
EroCrawler
Evil Evil
Extractor Exabot
Express WebPictures
Extreme Picture Finder
EyeNetIE
Ezooms
FDM
FHscan FHscan
FemtosearchBot Flunky
Fimap
Firefox/7.0
Foobot Foobot
Freeuploader
FyberSpider
Fyrebot
G-i-g-a-b-o-t
GT::WWW GT::WWW
Genieo GalaxyBot
Getintent
Go!Zilla Go!Zilla
Go-Ahead-Got-It GrabNet
Grabber Grabber
Grafula Grafula
GrapeFX GrapeshotCrawler
HMView HTTP::Lite
Haansoft
Harvest
Id-search Id-search
IlseBot IlseBot
IndeedBot Indy Library
InfoTekies InfoNaviRobot
Internet Ninja
InternetSeer InternetSeer
Iskanie Iria
JamesBOT
Jbrofuzz
JennyBot
JikeSpider JikeSpider
Jorgee Jorgee
Jyxobot Jyxobot
Keyword Density Keyword Density
Lanshanbot Kozmosbot
Larbin LNSpiderguy
LeechFTP LeechGet
Libwhisker LibWeb
Likse LieBaoFast
Linkdexbot Lightspeedsystems
LinkextractorPro Linkbot
LinkpadBot
LinksManager LinksManager
LinqiaMetadataDownloaderBot LinqiaMetadataDownloaderBot
LinqiaRSSBot MSFrontPage
Lipperhey MSIECrawler
Lipperhey Spider MTRobot
MFC_Tear_Sample Majestic12
MIDown tool
MIIxpc
Majestic-SEO
MarkWatch MarkWatch
Mass Downloader Mass Downloader
Masscan
Mata Hari Mata Hari
Mb2345Browser
MeanPath Bot
Mediatoolkitbot
MegaIndex.ru MegaIndex.ru
MicroMessenger Microsoft Data Access
Mojeek Mister PiX
Moblie Safari
Morfeus Fucking Scanner
Mr.4x3
Msrabot Msrabot
Musobot
NPbot NPbot
Needle Navroad
Net Vampire NearSite
Nessus
NetAnts NetAnts
NetLyzer NetMechanic
NetZIP
Nettrack
Netvibes Netvibes
Nibbler Niki-bot
Nikto
Nimbostratus
Ninja Ninja
Nmap
Nutch Nutch
Octopus Octopus
OpenLinkProfiler OpenLinkProfiler
OpenVAS Openvas
OutfoxBot OutfoxBot
PHPCrawl
POE-Component-Client-HTTP POE-Component-Client-HTTP
Page Analyzer Page Analyzer
PageAnalyzer
PageScorer
PageThing.com
Pandalytics
Panscient Panscient
Pavuk Papa Foto
PeoplePal PictureFinder
Picscout
Piepmatz Piepmatz
Pimonster Probethenet
Pockey PxBroker
Pu_iN
PyCurl PyCurl
RSSingBot Quick-Crawler
RankFlex RankActiveLinkBot
RankingBot
RankingBot2 RankingBot2
Rankivabot
RankurBot
RealDownload RealDownload
Reaper
Recorder Recorder
RedesScrapy
RepoMonkey RepoMonkey
RocketCrawler Ripper
Rogerbot
SEOkicks-Robot SEOkicks-Robot
SEOlyticsCrawler SEOprofiler
SEOstats
SMTBot SMTBot
SalesIntelligent SalesIntelligent
Scanbot ScoutJet
Screaming
ScreenerBot ScreenerBot
ScrepyBot
Semrush Semrush
SiteCheckerBotCrawler SemrushBot
SiteExplorer SentiBot
SiteLockSpider SeoSiteCheckup
Siphon
Site Sucker Site Sucker
SiteCheckerBotCrawler
SiteLockSpider
SiteSnagger
Sitevigil Sitevigil
SmartDownload SmartDownload
Snoopy Snapbot
SocialRankIOBot SocialRankIOBot
Sogou web spider
Sosospider
Sottopop
SpaceBison
SpankBot SpankBot
Sqlmap Spanner
Sqworm Spbot
Steeler Steeler
Stripper
Sucuri Sucuri
SuperHTTP SuperBot
Surfbot Swiftbot
SurveyBot T0PHackTeam
Teleport T8Abot
TeleportPro The Intraformant
Telesoft
Telesphorep
TheNomad TheNomad
TightTwatBot Toata
Titan Tracemyfile
Toweyabot Trendictionbot
Twice TurnitinBot
Upflow TwengaBot
VeriCiteCrawler URLy Warning
UnisterBot
VB Project
Vacuum
Vagabondo
VelenPublicWebCrawler
Virusdie Virusdie
Voltron Voltron
WASALive-Bot
WEBDAV WEBDAV
WISENutbot Wallpapers
WWW::Mechanize Wallpapers/3.0
Web Pix Web Auto
WebCopier
Web Enhancer Web Enhancer
WebImageCollector Web Fetch
WebAuto
WebBandit
WebFetch
WebGo IS
WebLeacher WebLeacher
WebReaper WebPix
WebSauger WebStripper
WebWhacker WebZIP
Webalta
Webshag Webshag
Webster Who.is Bot
Whatweb
Widow Widow
WinHTTrack
WiseGuys Robot WiseGuys Robot
Wotbox Wonderbot
Wprecon Woobot
Xaldon WebSpider
Xaldon_WebSpider
YoudaoBot YoudaoBot
Zauba Zeus
Zermelo
Zitebot Zitebot
ZoomBot ZoomBot
ZumBot
ZyBorg ZyBorg
arquivo-web-crawler
arquivo.pt
backlink-check backlink-check
check1.exe
coccocbot-web coccocbot-web
cognitiveseo cognitiveseo
com.plumanalytics com.plumanalytics
crawl.sogou.com
dataforseo.com dataforseo.com
demandbase-bot
domainsproject.org domainsproject.org
gopher eCatch
evc-batch
facebookscraper
instabid
internetVista monitor
isitwp.com isitwp.com
meanpathbot lwp-request
mediawords mediawords
netEstate NE Crawler netEstate NE Crawler
page scorer page scorer
pcBrowser
plumanalytics
polaris version
probe-image-size probe-image-size
ripz
scalaj-http scalaj-http
scan.lol
seobility seobility
siteripz serpstatbot
spyfu sexsearcher
sogouspider
sp_auditbot
sysscan
tAkeOut tAkeOut
trendiction.de trendiction.com
ubermetrics-technologies.com voyagerx.com
webmeup-crawler webmeup-crawler
webpros.com webpros.com
webprosbot

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
acunetix-referrer.com 11235813.webzdarma.cz
bannerconnect.net adtech.de
credit-online.ws hotgirlhdwallpaper.com
diamond-necklace.info jasonpartington.com
istizanidineanopiate.blogspot.com liumimgx.pw
kartiniresto.com louboutinbooties.xyz
mhi-systems.ru macotool.com
monetizer.com-01.site semp.net
reyel1985.webnode.fr versaut.xxx-cam.webcam
topkarkas.com website-speed-check.site

924
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
01apple.com 000free.us
0n-line.tv 1001watch.com.ua
1000n1.ru 1314dh.com
2345.com 1hwy.com
2pxg8bcf.top 2015god.org
4senses.co 4kplayer.pl
5elementov.ru 4rent.ru
6128786.com 4ureyesonly.com
7a2rnuey1tw9ar.ru 777-club.ru
8gold.com 8lufu.com
a.pr-cy.ru a.frcls.fr
a1security.com.ua aa8b68101d388c446389283820863176e7.com
aa08daf7e13b6345e09e92f771507fa5f4.com aadroid.net
aa14ab57a3339c4064bd9ae6fad7495b5f.com addbin.men
aaa38852e886ac4af1a3cff9b47cab6272.com ads-cool.pro
aac500b7a15b2646968f6bd8c6305869d7.com adultmeetonline.info
aad1f4acb0a373420d9b0c4202d38d94fa.com adventureparkcostarica.com
abclauncher.com agreda.pluto.ro
acunetix-referrer.com agroeconom.kz
ads-seo.men akuhni.by
advocatemsk.ru alert-jdh.xyz
adzpower.com alevinefesleri.com
ahrntal.verymes.xyz alfapro.ru
alekseevec.ru all4invest.info
alert-fdm.xyz
alice.it
alkoravto.ru
all4invest.ru all4invest.ru
alldubai.biz alldubai.biz
aloofly.com allpdfmags.net
amateurmatch.com allsilver925.co.il
amoi.tn amung.us
androids-store.com amyfoxfitness.com
animalrank.com anti-virus-removal.info
apartment.ru apelsinnik.website
apiallgeniusinfo-a.akamaihd.net apiwoodensealcom-a.akamaihd.net
apibatbrowsecom-a.akamaihd.net appartement-stumm.at
architecturebest.com appiq.mobi
arquapetrarca.info applyneedy.xyz
asscenihotosidea.blogspot.co.za autolombard-krasnodar.ru
atovh.local-finders.com autowebmarket.com.ua
ausergrubhof.info aviav.org
autoua.top avon-novosib.ru
azartmix.com avto-oligarh.ru
babespcs.com awency.com
babs.com.ua axbocz.net
baikaleminer.com azartclub.org
balans.shahterworld.org azbukadiets.ru
balkanfarma.ru babieca.com
barbourjackets.online backiee.com
batanga.net baksman.com
bazaronline24.ru baladur.ru
bdsmgalls.net bannerspace.com
bedandbreakfast.com bayareaaeroclub.org
belinvestdom.by belstaffstore.online
bestadbid.com best-seo-solution.com
bestdooz.com bestimagecoollibrary.com
bestdraws.com bget.ru
bestmobilityscooterstoday.com bingo8888.com
bestplacetobuyeriacta.jw.lt bio.trade-jp.net
bestpriceninja.com bnt-team.com
bestsexyblog.com bobinoz.com
bildsuche.ru bpro1.top
blackle.com brainzpod.win
blagovest-med.ru brandbucket.com
blavia.00author.com brianhenry.co
blockety.co bronzeaid-a.akamaihd.net
blog.koorg.ru browsepulse-a.akamaihd.net
bmwhighperformers.com bscodecs.com
bongiornos.info bukleteg.xyz
bookhome.info buy-cheap-online.info
boostmyppc.com buypanicdisorderpill.com
brainboostingsupplements.org buypuppies.ca
braip.com.br buytizanidineonlinenoprescription.blogspot.com
briomotor.co buzz-porno.info
buehne-fuer-menschenrechte.de c1.onedmp.com
c-english.ru calcularpagerank.com
camsex.xxx-cam.webcam camdolls.com
cardiosport.com.ua cannibalcheerleader.com
carson.getenjoyment.net
cartechnic.ru cartechnic.ru
cartierjusteunclou.xyz cartierlove2u.com
cartierring.xyz cementaresearch.se
cbozhe.com
ceotrk.com ceotrk.com
cheapwebsitehoster.com certifywebsite.win
chinaelitecheapjerseys.com chastnoeporno.com
chinajerseyswholesalecoupons.com checktext.org
christianlouboutinreplica.pw chee-by.biz
citysecurity.nu chinacheapelitejerseys.com
chiptuninger.com
choosecuisine.com
cialgenisrx.com
cityadspix.com
claytransformations.info
clayvasedesigns.tk
clickintext.net
clickprotects.com clickprotects.com
clksite.com clicksor.com
clubfashionista.com clippingphotoindia.com
comeondog.info codysbbq.com
compliance-viktor.top colobit.biz
compliance-andrew.top
compliance-barak.xyz
compliance-ivan.xyz
compliance-josh.top
compliance-julianna.top
compliance-willy.top compliance-willy.top
coolgramgoods.com confib.ifmo.ru
copro.pw cookingmeat.ru
coprofam.org coolgamechannel.com
cornerstone-countertops.com copenergo.ru
countbertwistdisp26.soup.io cornomase.win
counter.bloke.com courtshipgift.com
cracksplanet.com covetnica.com
crafthubs.com covid-schutzmasken.de
craftinsta.ru cowblog.fr
cpabegins.ru
cranly.net cranly.net
creditwell.ru customboxes4less.com
crynet.cc dairyindia.in
customergrowthsystems.com daisye.top
cutt.us date-withme.com
czat.wp.pl dev-seo.blog
dailyfinancefix.com diamond-necklace.info
day-news.info dickssportinggoods.com
delitime.info dikx.gdn
depositfiles-porn.ga discovertreasure-a.akamaihd.net
digital-sale.su disruptingdinnerparties.com
digitalfaq.com dnsrsearch.com
dogbreedspicture.net dogoimage.com
dokumentalkino.net domain.webkeyit.com
domain-submit.info dostavka-v-ukrainu.ru
domaincdn.xyz drugs-no-rx.info
dosug-lux.ru dustyorate.com
download-of-the-warez.blogspot.com e-biznes.info
downloadeer.net e-poker-2005.com
downvids.net ed-shop01.ru
dvrlists.com editors.choice6912650.hulfingtonpost.com
dynainbox.com ekaterinburg.xrus.org
dyt.net elektroniksigaraankara.info
ec-file.info elkacentr.ru
efkt.jp elstal.com.pl
ekosmetyki.net.pl eredijovon.com
elexies.info erot.co
elitepcgames.com erotische-geschichten-xxl.com
empirepoker.com everflora.ru
englishdictionaryfree.com excitacion.info
enthuse.computernetworksonline.com extlabs.io
ertelecom.ru extremez.net
esopini.com eyelike.com.ua
etur.ru ezofest.sk
eugenevaultstorage.com fanrto.com
evidencecleanergold.com faracontrol.ir
executehosting.com farmprofi.net
extractorandburner.com fast-torrent.ru
ezigarettenkaufen1.deviantart.com felizporno.com
facebook-mobile.xyz femdom.twiclub.in
falcon-images.blogspot.com feorina.ru
fickenbumsen.net ff30236ddef1465f88547e760973d70a.com
filesvine.com fickkontakte.org
filmgo.ru fickluder69.com
filmci.pro
filmi-onlain.info filmi-onlain.info
financepoints.eu financepoints.eu
finansov.info
findpics.pw findpics.pw
findpik.com findpik.com
finteks.ru findsexguide.com
firstsiteguide.com firma-legion.ru
flashbannernow.com foodcrafts.website
flex4launch.ru forminecrafters.ru
flipper.top free-social-buttons.xyz
floridamhca.org
forensicpsychiatry.ru
foto-telok.net
fototravel.eu
free-social-buttons.com
free-stock-illustration.com
free-traffic.xyz
free411games.com free411games.com
freelotto.com front.to
freemags.cc fsalas.com
freetangodownload.com fugarif.ga
funnel.co.za fyl.com.ru
game-mmorpg.net fz139.ttk.ru
gamedayassist.com galaxy-family.ru
ge0ip.net galeria-zdjec.com
gemara.com gambarkataku.co
gg-arena.ru gaygalls.net
gigapeta.com glcomputers.ru
gk170.ru glicol.kz
goforexvps.com gogalleryawesome.com
good-mummy.ru goldenggames.com
goodprotein.ru goldpanningtools.com
goodwinmetals.co golmau.host.sk
google-liar.ru goroda-vsego-mira.ru
gopro-online.info guruofcasino.com
gox.com.ua gyffu.com
gstatey.net hamacapty.com
gwebtools.com hdxnxxtube.mobi
hackers-crackers.tk headpharmacy.com
hahashka.ru
hcate.com
hdfreeporno.net
headpress.ru
helicalpile.us
helpmymacfaster.trade helpmymacfaster.trade
hentai-manga.porn hentai-manga.porn
heroz.fr hexpilot.com
highstairs-a.akamaihd.net homeandhealth.ru
hikesearch.net homegardenlova.com
hkladys.com homemade.gq
hobbyhuren24.net hotblog.top
host-protection.com
hotchatdate.com hotchatdate.com
howtoclean.club hotgirlhdwallpaper.com
huimin764128.com hothor.se
hully.altervista.org hothot.ru
hoverboardforsaledirect.com
howopen.ru
hut1.ru hut1.ru
idc.com.ua hvd-store.com
ilmexico.com i-midias.net.br
imagefinder.site ideawheel.com
incest-ru.com ideoworld.org
incolors.club igithab.com
indiakino.net imfamous.info
innoslicon.com inet-traffic.com
interferencer.ru instabid.tech
investpamm.ru invest-pamm.ru
iphantom.com
iplogger.org
ipostroika.ru ipostroika.ru
istizanidinelikexanax.blogspot.com itbc.kiev.ua
itronics.ca iwantedmoney.com
itservicesthatworkforyou.com
izhevsk.xrus.org izhevsk.xrus.org
janettabridal.com jagg.info
japfm.com janerikholst.se
jerseychinabizwholesale.com jjbabskoe.ru
jewelryandfiligree.com journeydownthescale.info
justprofit.xyz jus0wil.pp.ua
jwcialislrt.com jwss.cc
kalb.ru kamen-e.ru
kaliningrad.zrus.org karadene.com
karting196.ru karaganda.xkaz.org
kashubadesign.ru
kasino-money.pw kasino-money.pw
katadhin.co ketoanhanoi.info
keywordspay.com keywordteam.net
kicknights.gq kinofak.net
kinotorka.ru
knogg.net
koleso24.com.ua
kollesa.ru kollesa.ru
kosmetyki.tm.pl konturkrasoty.ru
kovrenok.ru kosynka-games.ru
krasnodar-avtolombards.ru
krasnodar.zrus.org krasnodar.zrus.org
ladov.ru lacloop.info
lanadelreyfans.us landscapebackgrounds.blogspot.com
lancheck.net larger.io
landscaping.center laserpen.club
law-check-two.xyz law-check-eight.xyz
le-clos-des-alouettes.com leadwayau.com
lego4x4.ru lenvred.org
lezbiyanki.net
li-er.ru
lignofix.ua lignofix.ua
littleberry.ru lion.misba.us
liver-chintai.org liran-locks.com
llastbuy.ru littlecity.ch
localflirtbuddies.com liveinternet.ru
lombia.co lizace.com
louisvuittonoutletstore.net lombardfinder.ru
machicon-akihabara.info lonfon.xyz
madot.onlinewebshop.net luchshie-topcasino.ru
magda-gadalka.ru luckyshop.net.ua
map028.com mackeeper-center.club
marblestyle.ru magicdiet.gq
marsgatan.com
marwer.info
masserect.com
me-ke.com
mebelcomplekt.ru
mebeldekor.com.ua
meccadumps.net meccadumps.net
medanestesia.ru mediawhirl.net
megaindex.ru meget.co.za
member-quiz.com mexicosleevegastrectomy.com
misandesign.se microstatic.pl
missclub.info middlerush-a.akamaihd.net
mmoguider.ru mil-stak.com
mmostrike.ru miniature.io
mnimmigrantrights.net mojaocena.com
mole.pluto.ro moncleroutletonline.pw
monclercoats.xyz
monclerjacketsoutlet.win
money-for-placing-articles.com
moneyteam24.com moneyteam24.com
morlat.altervista.org
moroccosurfadventures.com moroccosurfadventures.com
moscow.online-podarki.com mrpornogratis.xxx
mountainstream.ms musezone.ru
mozello.ru mvpicton.co.uk
mriyadh.com my-bc.ru
mttwtrack.com mycaf.it
multgo.ru mylovelibrabry.com
music.utrolive.ru mypornfree.ru
musicspire.online myxdate.info
myblogregistercm.tk nacap.ru
myplaycity.com namenectar.com
nakrutka.cc newfilmsonline.ru
napalm51.nut.cc newstudio.tv
naturalpharm.com.ua nextrent-crimea.ru
nbsproject.ru nfljerseysforsalewholesaler.com
nelc.edu.eg ngps1.ru
nodup.ru nhl17coins.exblog.jp
nofreezingmac.click nisuturnetdgu.tk
nordstar.pro no-rx.info
novosibirsk.xrus.org notaria-desalas.com
oakridgemo.com nuit-artisanale.com
oballergiya.ru nyfinance.ml
odoratus.net nzfilecloud.weebly.com
of-ireland.info obsessionphrases.com
onlinadverts.com ofanda.com
online-hit.info offergroup.info
ons-add.men office-windows.ru
ooo-olni.ru offside2.5v.pl
ostrovtaxi.ru oksrv.com
outrageousdeal-a.akamaihd.net onthemarch.co
p-business.ru ops.picscout.com
paighambot.com oralsexfilme.net
pammik.ru oranga.host.sk
parajumpersoutlet.online panamaforbeginners.com
park.above.com papasdelivery.ru
pc-virus-d0l92j2.pw pcboa.se
pekori.to penisvergrotendepillennl.ovh
perosan.com
pharmacyincity.com pharmacyincity.com
piter.xrus.org phelissota.xyz
placid-rounded-coast.glitch.me phobia.us
phrcialiled.com
pics-group.com
picswe.com
picturesfrom.com
pinup-casino1.ru
pirateday.ru
playmsn.com
podshipniki-ntn.ru podshipniki-ntn.ru
popup-fdm.xyz polska-poezja.com
poquoson.org porno-play.net
pornofiljmi.com pornodojd.ru
pornogig.com pornoelita.info
pornosee.info pornosexrolik.com
pornoxxx.com.mx positive2b.ru
potoideas.us powenlite24.ru
pr-ten.de pp-budpostach.com.ua
predmety.in.ua pr0fit-b0x.com
preparevideosafesystem4unow.site privatefx-in.ru
priceg.com proekt-mos.ru
primedice.com pron.pro
prizeestates.cricket prosperent.com
profolan.pl prostitutki-kiev.org
prointer.net.ua pssucai.info
promalp-universal.ru pussysaga.com
prostitutki-odessa.org pyrodesigns.com.au
psa48.ru qexyfu.bugs3.com
pukaporn.com qld10000.net
pulse33.ru qpypcx.com
puzzleweb.ru queerspace.com
qor360.com randalljhoward.com
qwesa.ru rangjued.com
rankingchart.de ranksignals.com
rating-bestcasino.com ranksonic.com
razyboard.com
rcb101.ru rcb101.ru
realitykings.com recipedays.ru
realizmobi.com registratciya-v-moskve.ru
referencemoi.com reklamuss.ru
relax.ru relax.ru
reversing.cc remontbiz.ru
revistaindustria.com ric.info
rfd-split.hr rockma.se
rusexy.xyz rossanasaavedra.net
rustic-quiver.win rrutw.com
rusvideos.su ruclicks.com
rvzr-a.akamaihd.net ruf777.com
sabaapress.com running-line.ru
sanatorrii.ru runstocks.com
sanyuprojects.com ryetaw.com
sarafangel.ru salut-camp.ru
scanner-ivan.top sardinie.us
scanner-nelson.top sashagreyblog.ga
schlampen-treffen.com sax-sex.com
scrapy.org sbwealthsolutions.ca
se-welding.ru scanmyphones.com
seansonline24.pl scanner-brian.top
search-error.com scanner-walter.top
search-goo.com screentoolkit.com
searchkut.com se.bnt-team.com
seeresultshub-a.akamaihd.net search.xtconnect.com
semaltmedia.com searchimage.co
seo-platform.com sendearnings.com
seofirmreviewsus.info seolab.top
seorankinglinks.xyz serving.adbetclickin.pink
seotoolsagency.com serviporno.com
serw.clicksor.com
seryeznie-znakomstva.ru
sexkrasivo.net sexkrasivo.net
sexsaoy.com sexobzor.info
sextracker.be sexpartygirls.net
shopperifymac.com sexsearch.com
silverage.ru sexuria.net
site5.com sexy-screen-savers.com
siteaero.com shtaketniki.kz
sitiz.club sientalyric.co
slavic-magic.ru simplesite.com
smart-balancewheel.com sitechecker.pro
sixcooler.de
skyway24.ru
slkrm.ru
slonechka.ru
slowmacfaster.trade
smartpet.ru
smsactivator.ru
sneakyboy.com sneakyboy.com
snip.to snworks.com
socas.pluto.ro snyatie-lomki-v-stacionare.ru
social-buttons.com socialbutton.xyz
softomix.com sockshare.net
solinf.co srgwebmail.nl
spabali.org stroicol.net
space-worry.ml stroimajor.ru
speeddream.xyz styro.ru
spidtest.space sugvant.ru
sportbetfair.com superlist.biz
squidoo.com
stillmiracle.com
stoki.ru
strana-solnca.ru
strigkaomsk.ru
sucsesofinspiration.com
sukirgenk.dvrlists.com
sundrugstore.com
supernew.org
surgut.zrus.org surgut.zrus.org
swagbucks.com symbaloo.com
talant-factory.ru syndicate.fun
tattooha.com tabletkinaodchudzanie.com.pl
td-33.ru tam-gde-more.ru
template-kid.com teenforporn.com
testbotprocessor44.com teletype.in
thegamerznetwork.com tengohydar.tk
themestotal.com terraclicks.com
theporndude.com thaisamkok.com
theprofitsmaker.net thebestphotos.eu
timdreby.com thebluffs.com
tizanidineandgabapentin.blogspot.com thedownloadfreeonlinegames.blogspot.com
tizanidinedruginteractions.blogspot.com theendivechronicles.com
tizanidinestreetprice.blogspot.com thejournal.ru
tmearegion26.com thetattoohut.com
token-lab.org thruport.com
top-karkas.ru ticketsys.inetwd.com
topdownloads.ru timecrimea.ru
torontoplumbinggroup.com titan-ads.life
titangel-vietnam.com
tizanidinehydrochloride2mg.blogspot.com
tizanidinevssoma.blogspot.com
tnaionline.org
toon-families.com
top10registrycleaners.com
torrents.life torrents.life
tour-line.net tourismvictoria.com
trafficnetzwerk.de toys.erolove.in
tripper.de tracksz.co
trubywriting.com tradedeals.biz
truck-addzilla.life trafficjunky.com
tuckermktg.com trichizobswiv.agddns.net
tupper-posuda.ru trudogolik.net
twittrading.com turbo-suslik.org
ugguk.online tv-spoty.info
ultimateclassicrock.com tvorozhnaja-zapekanka-recept.ru
unrealcommander.com uchil.net
urlopener.com ufa.xrus.org
uzungil.com univerfiles.com
vabasa.inwtrade.com uptime.com
valsalud.com urengoy.pro
video-chat.cn utiblog.fr
videochat.cafe vedomstvo.net
vegan-foods.us
veselokloun.ru
vesnatehno.ru
via-energy-acquistare.com
via-energy-order.com
viagra.pp.ua
videochat.ph
videogamesecrets.com
videos-for-your-business.com
vids18.site
villacoloniale.com villacoloniale.com
vip.51.la villakohlanta.nu
vipsiterip.org vinylvault.co.uk
vmnmvzsmn.over-blog.com vmnmvzsmn.over-blog.com
vodaodessa.com vod.com.ua
voltrknc1.com vpnmouse.com
vpnhowto.info vrotike.ru
vzlomfb.com vzlom-na-zakaz.com
vzlomsn.org waterpurifier.club
w-journal.ru weblo.com
w3data.co
wanker.us
wdfdocando.com
webcamdevochka.com
weblibrary.win
webmasterhome.cn webmasterhome.cn
webmonetizer.net websites-reviews.com
webnode.me wedding0venues.tk
weboptimizes.com westen-z.life
webpromotion.ae whatsupinfoley.com
webs.com whiteelephantwellington.com
whipme.yopoint.in wholesalejerseys-cheapest.com
wholesalejerseyscheapjerseys.us.com wholesalejerseyscheapjerseys.us.com
wjgony.com williamrobsonproperty.com
worldinternetauthority.com wingsofrefuge.net
worldlovers.ru winwotgold.pl
wowas31.ucoz.ru wleuaprpxuvr.ga
wufak.com wpthemedetector.co.uk
www.888.com xedserver.com
www.ehscloud.cn xn----7sbbagbq7bd5aheftfllo4m.xn--p1ai
xchangetrak.com xn----7sbbahaq9bb5afgiqfliv4m.xn--p1ai
xn----7sbaphztdjeboffeiof6c.xn--p1ai xn----9sbubg3ambdfl1j.xn--p1ai
xn----7sbho2agebbhlivy.xn--p1ai xn--80aaafbn2bc2ahdfrfkln6l.xn--p1ai
xn----itbkqkfiq.xn--p1ai xn--80aagddcgkbcqbad7amllnejg6dya.xn--p1ai
xn--80aafb2a.xn--p1ai
xn--80aeahghtf8ac5i.xn--p1ai
xn--80ak6aa92e.com xn--80ak6aa92e.com
xn--b1addnj3cah.xn--p1ai xn--c1acygb.xn--p1ai
xvideospornoru.com xn--hxazdsfy.blogspot.com
xxlargepop.com xn--oogle-wmc.com
xxxmania.top yopoint.in
xz618.com yougetsignal.com
yellowproxy.net yugk.net
your-bearings.com zaapplesales.blogspot.com
zaimite.ru zakon-ob-obrazovanii.ru
zambini.ru zaloadi.ru
zooggames.com zarabotat-na-sajte.ru
zoominfo.com zarabotok--doma.ru
zootoplist.com zeroredirect12.com
zrizvtrnpale.tk zeroredirect2.com
zionstar.net
zscalerone.net
zzlgxh.com

490
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
000free.us 0912701309f8ce.com
1-99seo.com 123any.com
1pamm.ru 18ps.ru
2ads.co.uk 273-fz.ru
2pxg8bcf.top 4kepics.com
above.com 4rent.ru
acc.eu.org 6hopping.com
acgs.tk a.frcls.fr
adobereader-free.ru aanapa.ru
alphahoverboards.com abbanreddy.com
analnoeporno.tv academiacsmendoza.org
ananumous.ru admanaerofoil.com
apiwoodensealcom-a.akamaihd.net advokateg.ru
arpelsreplica.xyz akusherok.ru
asiengirls.net alarmobninsk.ru
asseenontvonline.ru albamargroup.com
atmagroup.ru alert-jdh.xyz
atmovs.com alienwheel.es
avto-oligarh.ru alpinism.ru
ayakino.net anaksma.info
babyguns.ru anatomiy.com
baksman.com anticrawler.org
banki76.ru apartmentratings.com
bavsac.com autoua.top
bestbrainprod.win avkzarabotok.info
bestmobilityscooterstoday.com azbukafree.com
beta.hotkeys.com bad-stars.net
bodybuilding-shop.biz balitouroffice.com
bongiornos.info bamo.xsl.pt
brainboostingsupplements.org battle.net
brimstonehillfortress.org bestcalovebracelet.cn
cand.jp bestofpicture.com
cartierlove2u.com biznesschat.net
cbozhe.com blockety.co
cfacarrosserie74.com bloglag.com
boostmyppc.com
brianhenry.co
brk-rti.ru
bumble.cheapwebsitehoster.com
bvlgariwallet.xyz
c-english.ru
canyougethighofftizanidine.blogspot.com
capacitacionyaprendizaje.com
cb.iphantom.com
celebsopics.com
chatroulette.online chatroulette.online
cheapestjerseys-wholesale.com chiptuninger.com
cheapmoncler.xyz clickintext.net
click2pawn.com concealthyself.com
clickaider.com covadhosting.biz
clickcash.com cpabegins.ru
clicksor.net custodieva.ru
clickzzs.nl cvety24.by
clmforexeu.com daptravel.com
coderstate.com davebestdeals.com
coolwallpapers-hd.com dealitright.click
corta.co detalizaciya-tut.biz
daisye.top dividendo.ru
delayreferat.ru dkvorota.ru
deniven.1bb.ru domashneeruporno.com
dfwu1019.info doxyporno.com
dojki365.online drillsaw.ru
dotnetdotcom.org drugstoreforyou.com
dtm-spain.com dyshagi.ru
dvervmoskvu.ru e-biznes.info
e-buyeasy.com
e-commerce-seo1.com
education-cz.ru
edudocs.net edudocs.net
eduserver.net elektrozigarettekaufen1.devhub.com
eldiariodeguadalajara.com elitesportsadvisor.com
energydiet-info.ru emailaccountlogin.co
epngo.bz evening-dating-club.info
ereko.ru everytalk.tv
eupornstar.info exhibitionplus.eu
extractorandburner.com expresstoplivo.ru
ezigarettekaufen2.mpbloggar.se falcon-images.blogspot.com
ezigarettenkaufen2.dreamwidth.org
facialporntube.com
fast-wordpress-start.com fast-wordpress-start.com
fetishinside.com faz99.com
floridamobilebillboards.com fickblock18.com
forex-procto.ru fil.ru
fotopop.club fitness-video.net
foxtechfpv.com food.dtu.dk
fr-bearings.ru foxydeal.com
franecki.net free-floating-buttons.com
free-gluten.ru free-laptop-reward.com
free-share-buttons.blogspot.com freeseedsonline.com
freshberry.com.ua freshwallpapers.info
game-top.su fx-brokers-review.com
gay.adultgalls.com fyxabomiw.ru
gemara.com gallerylisting.com
glopages.ru ge0ip.net
golmau.host.sk geile-lelly.eu
googlemare.com gheus.altervista.org
gopro-online.info glastecfilms.com.my
h2monline.com global-ics.co.za
hamilton.ca golden-praga.ru
hao123.com googst2.ru
hdseriale.pl green-tea.tv
herokuapp.com gribokstop.com
hit-men.men helvetia.com.ua
hledejvshopech.cz homeandhealth.ru
hop.clickbank.net house.sieraddns.com
hotshoppymac.com howmuchdoestizanidinecost.blogspot.com
houseofrose.com hptwaakw.blog.fc2.com
images-free.net hydropump.su
immobiliaremassaro.com iboss.com
imperiafilm.ru ie.57883.net
in-tandem.co ikritikimou.gr
infobanks.ru insomniagamingfestival.com
infospot.pt intellego.info
instabid.tech interior-stickers.ru
ipchicken.com istizanidineacontrolledsubstance.blogspot.com
irkutsk.online-podarki.com itsdp3.com
ispaniya-costa-blanca.ru ivanstroi.ru
istripper.com james13prix.info
itrevolution.cf janettabridal.com
j-times.ru jerseywholesalechinabiz.com
jav-fetish.site jerseywholesaleelitestore.com
javstock.com jovencitas.gratis
jeremyeaton.co kamen-e.ru
job.icivil.ir karachev-city.ru
joingames.org kollesa.ru
kamalsinha.com kovrenok.ru
katadhin.co kraljeva-sutjeska.com
kinogonew.ru krynica.info
kit-opt.ru ks1234.com
kliksaya.com ledis.top
knowsitall.info likesdesign.com
koleso24.com.ua linkbuddies.com
larutti.ru lion.playtap.us
lomza.info lionking-1994.blogspot.com
macotool.com live-cam6.info
maju.bluesalt.co liveinternet.ro
manyresultshub-a.akamaihd.net lovi-moment.com.ua
marcowebonyodziezowe.pl lunamedia.co
marmitaco.cat luxup.ru
masturbate.co.uk macfix.life
maxthon.com maladot.com
megakino.net marcogrup.com
menetie.ru meddesk.ru
metarip.ru medictube.ru
mhtr.be midst.eu
mnimmigrantrights.net moroccosurfadventures.com
mockupui.com musicspire.online
mojpregled.com muz-baza.net
moncleroutletonline.pw myblogregistercm.tk
moneyteam24.com mydirtyhobby.com
moskva.nodup.ru myplaycity.com
mountainstream.ms na-telefon.biz
mrbitsandbytes.com narkologiya-peterburg.ru
mturkcontent.com netallergy.ru
muschisexbilder.com newtechspb.ru
net-profits.xyz oivcvx.website
networkad.net
newsperuse.com
nibbler.silktide.com
nmrk.ru
nomuos.it
novosti-hi-tech.ru
nudo.ca
nufaq.com
o00.in
obiavo.kz
offer.wpsecurity.website
online-zaymy.ru
online.ktc45.ru online.ktc45.ru
onlyporno.ru online247.ml
ops.picscout.com onlinebay.ru
pavlodar.xkaz.org onlineku.com
pawli.eu orgasmatrix.com
pdn-4.com outrageousdeal-a.akamaihd.net
pensplan4u.com oynat.info
perederni.net panchro.co.uk
pills24h.com petsblogroll.com
platezhka.net picture-group.com
pochtovyi-index.ru play.leadzupc.com
poem-paying.gq poffet.net
polytopesexempt.com pooleroadmedicalcentre.co.uk
porn555.com porno-dojki.net
pornpost.in pornotubexxx.name
powenlite24.ru potoideas.us
pozvonim.com progress-upakovka.ru
preventheadacheguide.info proposal-engine.com
projectforte.ru psa48.ru
pxhdwsm.com pukaporn.com
qexyfu.bugs3.com r-e-f-e-r-e-r.com
radiogambling.com rainbowice.ru
razleton.com realizmobi.com
realitykings.com recipedays.com
registry-cleaner.net regionshop.biz
renhacklids.tk remont-comp-pomosh.ru
rentehno.ru rentaremotecomputer.com
restaurantlescampi.com rff-cfal.info
rock-cafe.info royalvegascasino.com
rosbalt.com.ua salutmontreal.com
rozalli.com sammlungfotos.online
saletool.ru saneitconsulting.com
sbprabooks.com sbetodiodnye-lampy.ru
scopich.com scanner-andrew.top
scrnet.biz.ua scoopquest.com
search.alot.com scrapinghub.com
sel-hoz.com sell-fb-group-here.com
seo-2-0.com seonetwizard.com
seo-prof1.xyz serpstat.com
seoprofiler.com servethis.com
sexfreepornoxxx.com sex-tracker.de
sexpornotales.com sexkontakteao.info
sexuria.net shopcheermakeup.info
shareyards.com silvercash.com
shemalegalls.blogporn.in sim-service.net
shoppingmiracles.co.uk simple-share-buttons.com
sibecoprom.ru siteaero.com
sildenafilcitratemed.com
simplepooltips.com
sinestesia.host.sk
site-analyzer.com
site-speed-checker.site
smart-balancewheel.com smart-balancewheel.com
snts.shell-pmr.ru snworks.com
sodexo.com star61.de
stmassage.ru sticken.co
stroilka.info stickers-market.ru
stroyhelp-dv.ru superstarfloraluk.com
sudexpert66.ru svnuppsalaorebro.se
summerlinhomes411.info svtrd.com
taaaak.com telefonsexsofort.tk
tabletkinaodchudzanie.com.pl tfxiq.com
takethatad.com thebitcoincode.com
tam-gde-more.ru themestotal.com
tattooreligion.ru tizanidineingredients.blogspot.com
tiandeural.ru tizanidinerecreationaluse.blogspot.com
tizanidinehcl2mgsideeffects.blogspot.com tizanidinevsvalium.blogspot.com
tizanidineinfo.blogspot.com tksn.ru
tizanidineusedfor.blogspot.com tour-line.net
top-study.work track112.site
topclickguru.com traffic-club.info
torrent-newgames.com trafficstars.com
twodollarshows.com tube8.com
umekana.ru turbodsp.com
underthesite.com turizm.bz
veles.shop uchetunet.su
vod.com.ua uggbootsoutletsale.us
vseigry.fun unacittaconte.org
wemedinc.com urlopener.com
weprik.ru videospornogratisx.net
wesharepics.site vip-dom.in
wetgames.ru vmnmvzsmn.over-blog.com
whatsappdownload10.com vzube.com
wleuaprpxuvr.ga warezaccess.com
womens-journal.net wedding0venues.tk
xev.ru wellcome2slovenia.ru
xn----8sbhefaln6acifdaon5c6f4axh.xn--p1ai wingsoffury2.com
xn--hxazdsfy.blogspot.com woman-tampon.ru
xxxdatinglocal.us wormix-cheats.ru
xxxtubesafari.com x-diesel.info
yebocasino.com xn--80aeahghtf8ac5i.xn--p1ai
youtubologia.it xn--80ahdheogk5l.xn--p1ai
zatjmuzu.info yaderenergy.ru
zigarettenonlinekaufen1.bloog.pl yellowads.men
zigarettenonlinekaufen1.blox.pl yopoint.in
youdesigner.kz
yugo-star.ru
zebradudka.com
zeroredirect.com
zoominfo.com
zoomovies.org

10
dev-tools/test_units/ratelimittest-beta.txt
View file

@ -581,4 +581,14 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html> </html>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.64 0; 10.1.1.69 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.10.2631 ### Version: V4.2021.10.2632
### Updated: Mon Oct 4 22:01:15 UTC 2021 ### Updated: Tue Oct 5 10:03:25 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.10.2631 ### Version: V4.2021.10.2632
### Updated: Mon Oct 4 22:01:15 UTC 2021 ### Updated: Tue Oct 5 10:03:25 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##