vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-10 15:34:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.10.3385 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-10-02 10:09:22 +00:00
parent ae3b04cf5c
commit 5f24e5cb63
47 changed files with 93453 additions and 93463 deletions
Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.10.3384 #### Version: V4.2022.10.3385
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.10.3384 #### Version: V4.2022.10.3385
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.10.3384 #### Version: V4.2022.10.3385
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

11546
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.10.3384 #### Version: V4.2022.10.3385
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

11550
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11550
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

11550
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11550
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 1; 10.1.0.29 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 0; 10.1.0.29 0;
127.0.0.1 0; 127.0.0.1 0;

11550
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 1; 10.1.0.29 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 0; 10.1.0.29 0;
127.0.0.1 0; 127.0.0.1 0;

11550
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 1; 10.1.0.29 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 0; 10.1.0.29 0;
127.0.0.1 0; 127.0.0.1 0;

11550
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3384 3385

11550
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

11550
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.165.219.199 0; 168.61.4.191 0;

11550
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 1; 10.1.0.29 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 0; 10.1.0.29 0;
127.0.0.1 0; 127.0.0.1 0;

11550
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 1; 10.1.0.29 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 0; 10.1.0.29 0;
127.0.0.1 0; 127.0.0.1 0;

11550
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 1; 10.1.0.29 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 0; 10.1.0.29 0;
127.0.0.1 0; 127.0.0.1 0;

11550
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.165.219.199 0; 168.61.4.191 0;

11550
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.165.219.199 0; 168.61.4.191 0;

11550
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 1; 10.1.0.29 1;
127.0.0.1 1; 127.0.0.1 1;

12
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
360Spider AiHitBot
BBBike BBBike
BDCbot
BetaBot
Grafula Grafula
Lmspider InternetSeer
LinkWalker
Mr.4x3
Rankivabot Rankivabot
RocketCrawler
Upflow Upflow
internetVista monitor WWW-Collector-E
zgrab

314
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
404enemy
80legs
ALittle Client
ASPSeek ASPSeek
Aboundexbot Aboundexbot
AiHitBot Acunetix
Alexibot AfD-Verbotsverfahren
Alligator AllSubmitter
AlphaBot
Anarchie Anarchie
Anarchy99 AwarioRssBot
Ankit BDCbot
Anthill BLEXBot
Asterias
AwarioSmartBot
BackDoorBot BackDoorBot
Backlink-Ceck
BacklinkCrawler BacklinkCrawler
BetaBot Brandwatch
Black Hole Bullseye
Blow
Bolt
BuiltWith
BunnySlippers BunnySlippers
BuzzSumo
CCBot CCBot
CSHttp Calculon
Cegbfeieh CazoodleBot
CheeseBot ChinaClaw
CherryPicker Chlooe
Citoid Cloud mapping
Cliqzbot Copier
Cocolyzebot Copyscape
Craftbot
Crescent Crescent
CrunchBot CrunchBot
Curious
Custo Custo
CyotekWebCopy DIIbot
DBLBot DataCha0s
DSearch
DTS Agent
DatabaseDriverMysqli DatabaseDriverMysqli
Demon Demon
Devil Deusu
Dirbuster Digincore
Disco DigitalPebble
Discoverybot DomainAppender
Dispatch
DnyzBot
DomCopBot
DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
DomainStatsBot DomainStatsBot
Domains Project
Dotbot
Download Wonder
Dragonfly Dragonfly
EMail Siphon Drip
EMail Wolf
EasyDL
Evil Evil
Exabot Express WebPictures
Extractor Extractor
Firefox/7.0 ExtractorPro
FrontPage EyeNetIE
FDM
FlashGet
Flunky
Fuzz Fuzz
Genieo Fyrebot
GermCrawler G-i-g-a-b-o-t
GT::WWW
GetRight
GetWeb GetWeb
Getintent Getintent
Gigabot Gotit
GoZilla GrabNet
Grabber Grabber
Grafula HTTP::Lite
HEADMasterSEO HTTrack
HTMLparser HaosouSpider
Haansoft
Humanlinks Humanlinks
HybridBot
IDBTE4M IDBTE4M
Id-search Indy Library
Intelliseek InterGET
InternetSeer Iria
Iskanie
IstellaBot IstellaBot
JamesBOT JOC Web Spider
Jbrofuzz JikeSpider
JennyBot Joomla
Jorgee JustView
Jyxobot Jyxobot
Keybot Translation-Search-Machine Keybot Translation-Search-Machine
Keyword Density Keyword Density
Kinza Kinza
Kozmosbot
Lanshanbot Lanshanbot
Larbin Lftp
Libwhisker Likse
LinkScan
LinkextractorPro LinkextractorPro
LinqiaRSSBot LinkpadBot
LinqiaScrapeBot LinqiaMetadataDownloaderBot
Lipperhey Spider Litemage_walker
MJ12bot Lmspider
MQQBrowser Ltx71
MSFrontPage Majestic12
Magnet
Mail.RU_Bot
MarkMonitor
Mass Downloader
Masscan Masscan
MauiBot MauiBot
Meanpathbot Meanpathbot
Microsoft Data Access Mediatoolkitbot
Microsoft URL Control Microsoft URL Control
Moblie Safari Moblie Safari
Mojolicious Mr.4x3
MolokaiBot
Morfeus Fucking Scanner
Musobot
NICErsPRO
NPbot NPbot
Name Intelligence Name Intelligence
Nameprotect Nameprotect
Navroad
NearSite NearSite
Net Vampire NetAnts
NetLyzer NetLyzer
Netcraft Netcraft
Nettrack
Netvibes
Niki-bot Niki-bot
NimbleCrawler Nikto
Nmap Ninja
Not Nuclei
Offline Explorer Offline Explorer
Offline Navigator OrangeSpider
OpenLinkProfiler
OrangeBot
OutclicksBot OutclicksBot
OutfoxBot
PHPCrawl PHPCrawl
Page Analyzer Page Analyzer
PageScorer PageAnalyzer
Pandalytics PageGrabber
Panscient
Papa Foto Papa Foto
PeoplePal Pavuk
Petalbot Petalbot
Picsearch
PictureFinder PictureFinder
Piepmatz
Pimonster
Pixray
PleaseCrawl
ProWebWalker ProWebWalker
Pu_iN
Pump Pump
Quick-Crawler PxBroker
RSSingBot
RankActive RankActive
RankFlex RankActiveLinkBot
RankurBot
Re-re Re-re
ReGet RealDownload
Reaper
Recorder
RedesScrapy
RepoMonkey RepoMonkey
Ripper Ripper
RocketCrawler Rogerbot
SBIder SBIder
SEOprofiler SEOkicks
SEOkicks-Robot
SEOlyticsCrawler
SEOstats
SISTRIX
SMTBot SMTBot
Scanbot ScoutJet
Scrapy
Searchestate Searchestate
Semrush SearchmetricsBot
SemrushBot Seekport
SentiBot
SeoSiteCheckup SeoSiteCheckup
SeobilityBot SeobilityBot
Site Sucker Seomoz
SocialRankIOBot Shodan
Sottopop Siphon
SiteLockSpider
SiteSnagger
SiteSucker
Sitebeam
Siteimprove
Snapbot
Sociscraper
Sosospider
SpaceBison
Spammen
Sqlworm Sqlworm
SurveyBot Sqworm
Stripper
Sucuri
SuperBot
T0PHackTeam T0PHackTeam
T8Abot TeleportPro
Teleport Telesphoreo
Telesphorep
The Intraformant
TheNomad
TightTwatBot TightTwatBot
Titan Toata
Trendictionbot Tracemyfile
Turingos Trendiction
Turnitin
TurnitinBot TurnitinBot
TwengaBot TwengaBot
UnisterBot URLy.Warning
Upflow VCI
V-BOT
Vagabondo
VelenPublicWebCrawler
VeriCiteCrawler
VidibleScraper VidibleScraper
Virusdie
VoidEYE VoidEYE
WEBDAV Voltron
WBSearchBot
WPScan WPScan
WWW-Collector-E
WWW::Mechanize
WWWOFFLE
Wallpapers Wallpapers
Wallpapers/3.0
WallpapersHD
WeSEE
Web Enhancer Web Enhancer
Web Fuck Web Sauger
Web Pix
Web Sucker
WebAuto WebAuto
WebBandit
WebCollage WebCollage
WebCopier
WebEnhancer WebEnhancer
WebFetch WebFetch
WebFuck
WebGo IS
WebLeacher WebLeacher
WebPix WebReaper
WebSauger WebSauger
WebStripper WebWhacker
WebZIP Webshag
WebmasterWorldForumBot
Website Quester Website Quester
Whack WebsiteQuester
Webster
Whacker Whacker
Whatweb Whatweb
Who.is Bot Who.is Bot
Woobot Widow
Wonderbot
Wotbox Wotbox
Xaldon WebSpider Wprecon
YoudaoBot Xaldon_WebSpider
Zitebot Xenu
ZyBorg Zade
ZoomBot
ZumBot
adscanner
archive.org_bot archive.org_bot
arquivo-web-crawler arquivo.pt
autoemailspider
cah.io.community
coccocbot coccocbot
crawl.sogou.com crawler4j
crawler.feedback
domainsproject.org domainsproject.org
eCatch
evc-batch evc-batch
facebookscraper facebookscraper
gopher heritrix
ips-agent ips-agent
isitwp.com
iubenda-radar
linkdexbot linkdexbot
muhstik-scan magpie-crawler
probe-image-size meanpathbot
oBot
polaris version
ripz ripz
s1z.ru
scalaj-http
scan.lol
seobility
seocompany.store
seoscanners
serpstatbot serpstatbot
trendiction.com sogouspider
trendiction.de sp_auditbot
sysscan
ubermetrics-technologies.com ubermetrics-technologies.com
voyagerx.com
webgains-bot
webmeup-crawler webmeup-crawler
webprosbot webprosbot
x22Mozilla xpymep1.exe

308
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
404checker 360Spider
80legs 404enemy
ASPSeek ALittle Client
Aboundexbot Alexibot
AfD-Verbotsverfahren AllSubmitter
AhrefsBot
Anarchie
Anarchy Anarchy
Aspiegel Aspiegel
Atomseobot Asterias
Attach
AwarioSmartBot AwarioSmartBot
BDCbot BDCbot
BDFetch BacklinkCrawler
Backlink-Ceck Badass
Bandit
BatchFTP BatchFTP
Bigfoot Battleztar Bazinga
Black Hole Bitacle
Blackboard
BotALot
Brandwatch Brandwatch
Buck Buck
Buddy
BuiltBotTough
BuiltWith BuiltWith
Bullseye
BuzzSumo
CCBot CCBot
CODE87
CSHttp CSHttp
Cegbfeieh
CensysInspect CensysInspect
CheTeam CherryPicker
ChinaClaw ChinaClaw
Cliqzbot Chlooe
Citoid
Cloud mapping Cloud mapping
Cogentbot Copyscape
Cosmos
Craftbot Craftbot
CrazyWebCrawler
Crescent Crescent
DBLBot CrunchBot
Demon Curious
DTS Agent
Deusu Deusu
Devil Devil
DigitalPebble Discoverybot
Dirbuster Dispatch
DomainCrawler DnyzBot
DomainSigmaCrawler DomCopBot
Domains Project Download Wonder
Dragonfly
ECCP/1.0
EMail Wolf
EasyDL EasyDL
EirGrabber Ebingbong
Evil Ecxi
Exabot ExtLinksBot
Express WebPictures Extractor
EyeNetIE
FDM
Fimap Fimap
Firefox/7.0 FlashGet
Flunky
Freeuploader Freeuploader
FrontPage
FyberSpider FyberSpider
G-i-g-a-b-o-t Getintent
GT::WWW
Genieo
Gigabot Gigabot
Grabber Go-Ahead-Got-It
Grafula GoZilla
GrapeshotCrawler Gotit
HEADMasterSEO HaosouSpider
HTMLparser
Haansoft
Havij Havij
HonoluluBot Heritrix
IRLbot Humanlinks
HybridBot
Iblog Iblog
Image Fetch Image Fetch
Image Sucker
IndeedBot IndeedBot
Indy Library Intelliseek
Iria Internet Ninja
Iskanie
IstellaBot IstellaBot
JOC Web Spider
JamesBOT JamesBOT
Jbrofuzz
JennyBot JennyBot
Jetty Keybot Translation-Search-Machine
JustView Kinza
Kenjin Spider
Keyword Density
Kozmosbot Kozmosbot
LNSpiderguy
LWP::Simple
Lanshanbot Lanshanbot
LeechFTP LibWeb
LexiBot Libwhisker
LieBaoFast Lightspeedsystems
LinkScan
LinkWalker LinkWalker
LinkextractorPro Linkbot
LinqiaMetadataDownloaderBot
LinqiaScrapeBot LinqiaScrapeBot
Lipperhey Spider
Litemage_walker
Lmspider
Ltx71 Ltx71
MIIxpc MIDown tool
MSIECrawler MJ12bot
MTRobot MQQBrowser
Magnet Mail.RU_Bot
Majestic SEO Majestic12
Majestic-SEO MarkWatch
MarkMonitor MeanPath Bot
Microsoft URL Control Meanpathbot
Minefield Minefield
Mister PiX Mister PiX
Mojeek Mojolicious
Mr.4x3 Mozlila
Msrabot NICErsPRO
Musobot
NPbot NPbot
Nameprotect Navroad
NearSite Nessus
Needle
Net Vampire
NetAnts
NetLyzer
NetMechanic NetMechanic
NetZIP Netcraft
NextGenSearchBot Netvibes
Niki-bot
Nikto Nikto
Not Nimbostratus
Nuclei
Nutch Nutch
Offline Explorer Offline Explorer
OpenLinkProfiler
Openfind
OrangeBot OrangeBot
OutfoxBot OutfoxBot
PECL::HTTP
PHPCrawl PHPCrawl
POE-Component-Client-HTTP
Page Analyzer Page Analyzer
PageGrabber PageScorer
PageThing.com PageThing.com
Pandalytics Pavuk
PeoplePal
Petalbot Petalbot
Pi-Monster
Picscout Picscout
Pimonster Piepmatz
Pixray
PleaseCrawl
ProPowerBot ProPowerBot
ProWebWalker ProWebWalker
Pu_iN Psbot
Pump Pump
PxBroker
QueryN Metasearch QueryN Metasearch
RankFlex Quick-Crawler
RankActiveLinkBot
ReGet ReGet
RealDownload RealDownload
RebelMouse Reaper
RepoMonkey Recorder
Ripper SEOprofiler
SEOkicks SEOstats
SEOlyticsCrawler SMTBot
SalesIntelligent ScreenerBot
ScoutJet
Scrapy
ScrepyBot
SearchmetricsBot SearchmetricsBot
SemrushBot SemanticJuice
Semrush
SeoSiteCheckup SeoSiteCheckup
Shodan
Siphon Siphon
SiteExplorer
SiteLockSpider SiteLockSpider
SiteSucker Sitebeam
Siteimprove Siteimprove
Sitevigil SlySearch
Snoopy
SocialRankIOBot
Sogou web spider
Sosospider Sosospider
Sottopop
Spammen Spammen
Spanner SpankBot
SputnikBot Sqlmap
Sqlworm
Sqworm
Stripper
Sucker Sucker
Surfbot Sucuri
SuperBot
SuperHTTP
SurveyBot
Suzuran Suzuran
Teleport Szukacz
T0PHackTeam
T8Abot
TeleportPro
Telesoft
Telesphorep
The Intraformant
TheNomad TheNomad
Thumbor
TightTwatBot TightTwatBot
Toata
Trendiction Trendiction
Trendictionbot
True_Robot True_Robot
Turingos Turingos
Turnitin TurnitinBot
VB Project TwengaBot
URLy Warning
UnisterBot
V-BOT
Vacuum Vacuum
VeriCiteCrawler VelenPublicWebCrawler
VidibleScraper VidibleScraper
Virusdie Virusdie
Voil Voil
WASALive-Bot Voltron
WPScan
WWW::Mechanize WWW::Mechanize
Wallpapers/3.0
Web Auto
Web Fetch Web Fetch
WebBandit Web Sucker
WebFetch WebEnhancer
WebGo IS
WebImageCollector WebImageCollector
WebReaper WebLeacher
WebSauger WebPix
WebStripper WebStripper
WebSucker
WebWhacker
WebZIP WebZIP
Webalta Webalta
Webshag
Website Quester Website Quester
WebsiteQuester
Whacker
Widow Widow
Wonderbot WiseGuys Robot
Woobot Woobot
Wprecon Wprecon
Zade Xaldon_WebSpider
Zitebot Xenu
ZumBot Zauba
Zermelo
ZoominfoBot
ZyBorg ZyBorg
arquivo.pt arquivo.pt
autoemailspider
backlink-check backlink-check
check1.exe cah.io.community
clark-crawler clark-crawler
coccocbot
com.plumanalytics
crawler.feedback crawler.feedback
crawler4j
dataforseo.com dataforseo.com
domainsproject.org demandbase-bot
eCatch evc-batch
facebookscraper lwp-request
heritrix lwp-trivial
internetVista monitor magpie-crawler
isitwp.com muhstik-scan
iubenda-radar oBot
mediawords plumanalytics
netEstate NE Crawler
page scorer
polaris version polaris version
probe-image-size
ripz ripz
s1z.ru
scalaj-http scalaj-http
scan.lol
seoscanners
serpstatbot
sitechecker.pro
sogouspider
sp_auditbot sp_auditbot
sysscan spyfu
trendiction.com tAkeOut
ubermetrics-technologies.com
voyagerx.com voyagerx.com
webpros.com
x09Mozilla x09Mozilla
x22Mozilla
zauba.io zauba.io
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
artavenuegardenstudios.com abbanreddy.com
cartechnic.ru gotcher.us
eropho.com keywesthideaways.co
flytourisme.org kiwi237au.tk
iscblog.info rogervivierforsale.com
mncrftpcs.com social-buttons.com
orgasmus-virtual.com tako3.com
rentalcarnavi.info themestotal.com
taqplayer.info turist-strani.ru
worldinternetauthority.com unmaroll.ya.ru

920
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
0lovespells0.blogspot.com
1-99seo.com
1-free-share-buttons.com 1-free-share-buttons.com
108shot.com 123any.com
1adult.com 2girls1cup-free.com
1bet.com 4kepics.com
1pamm.ru 51.la
2drittel.de aa8780bb28a1de4eb5bff33c28a218a930.com
3th.co.in acc.eu.org
3weekdiet.com ad-words.ru
404.mn adloads.com
45en.ru admatic.com.tr
4inn.ru adobereader-free.ru
4pp13.com
5kstore.com
8lufu.com
DomainStatsBot
abbanreddy.com
abcdefh.xyz
adelly.bg
admanaerofoil.com
adrenalin-stk.ru adrenalin-stk.ru
adsfresh.men adssafeprotected.com
ahrntal.verymes.xyz advancedmassagebysara.com
aibolita.com advertex.info
akuhni.by affordablewebsitesandmobileapps.com
ahrefs.com
alborzan.com
albumsuper.info
alert-fdm.xyz
alf-img.com alf-img.com
allwidewallpapers.com all-streaming-media.com
alphahoverboards.com alt-servis.ru
am-se.com altamayoztourism.com
amigobulls.com amanda-porn.ga
angel1777.kz amospalla.es
angigreene.com anabolics.shop
anmysite.com android-style.com
anticrawler.org animal-drawings.com
antons-transporte.de anniemation.com
apiitsthirteende-a.akamaihd.net apiqualitinknet-a.akamaihd.net
apispringsmartne-a.akamaihd.net artparquet.ru
appleid-verification.com asophoto.com
approved.su
ardimobileinfo.ml
arendovalka.xyz
astana.xxxkaz.org astana.xxxkaz.org
astrochicks.com atley.eu.pn
ausergrubhof.info atofilms.com
audiobangout.com
australia-opening-times.com australia-opening-times.com
autobudpostach.club autogrep.ru
autonew.biz autoseo-traffic.com
autorn.ru aviav.ru
aviav.org avramstroy.ru
avon-ukraine.com azazu.ru
babosas.com azbuka-mo.ru
babyguns.ru babieca.com
balans.shahterworld.org backiee.com
bang-hotties.com bamo.xsl.pt
bangla-chat-uk-paralud.ga bazaronline24.ru
bannertesting.com bestmarriages.com
bard-real.com.ua bestmobilityscooterstoday.com
bedcapdealers.com betune.onlinewebshop.net
berdasovivan.ru bigcareer.info
berrymall.ru bin-brokers.com
bestwrinklecreamnow.com bio.trade-jp.net
betteroffers.review biographiya.com
biocn.dx.am bioinnovate.co
biteg.xyz
blackbabesporn.com
blockworld.ru
blogos.kz blogos.kz
blogporn.in
bmsco.co bmsco.co
boanonihaca.tk board.f00d.de
bolezniorganov.ru brains2.biz
bookmark4you.com.biz brakehawk.com
breget74.ru brandehk.dk
burger-imperia.com bristolhotel.com
buyessayonline19.blogspot.ru brokergid.ru
bycontext.com burger-tycoon.com
camsex.xxx-cam.webcam businesxxl.com
canacopegdl.com buypillsorderonline.com
candelluxsklep.pl bytimedance.ru
candy-glam-hp.com c-english.ru
capture-room.com carapics.com
cartierlove.xyz cardul.ru
cartierlove2u.com carrauterie.be
cartierreplica.pw carsdined.org
centre-indigo.org.ua carsplate.com
ceotrk.com casinos4dummies.co.uk
chatroulette.si cbox.ws
cheapjerseysa.com cdncash.net
check-host.net celejihad.info
checkhit.com cfsrating.sonicwall.com
christianlouboutinsaleoutletonline.us children-learningreading.info
cinemaenergy-hd.ru choice-credit.ru
cl.s7.exct.net citysecurity.nu
clarithromycin500mg.com civilwartheater.com
clash-clans.ru cleanallvirus.com
clickaider.com click2pawn.com
clippingphotoindia.com clicksor.net
cntravelre.com cobaltpro.ru
cookie-law-enforcement-hh.xyz compliance-don.xyz
counterbot.com conboy.us
crafthubs.com conocer-sanabria.com
credit-online.ws construmac.com.mx
cutalltheshit.com cool-wedding.net
dalavia.ru coolwallpapers-hd.com
data-mining.tk copenergo.ru
davebestdeals.com cottageofgrace.com
dbmkfhqk.bloger.index.hr counter.yadro.ru
dear-diary.ru creams.makeforum.eu
demon-tweeks.com culad.com
den-noch24.ru culmimg.pw
dengi-pod-zalog-nedvizhimosti.ru cyse.tk
dev.citetick.com damasarenai.info
diminishedvalueoforegon.com damedingel.ya.ru
dolohen.com data-ox.com
doublepimp.com data1.scopich.com
downloadeer.net datract.com
downloadkakaotalk.com datsun-do.com
dptaughtme.com dcj-nn.ru
druzhbany.ru ddospanel.com
druzhininevgeniy63.blogspot.com deadlinkchecker.com
dtm-spain.com design-lands.ru
dumpsmania24.com diamond-necklace.info
dvrlists.com dichvuvesinhhanoi.com
dikqlyremy.info
discountliv.com
diy-handmade-ideas.com
doctissimo.fr
doctormakarova.ru
dokfilms.net
domainroam.win
donvito.unas.cz
dotnetdotcom.org
downtuptv.gq
dprtb.com
e-buyeasy.com e-buyeasy.com
e-collantes.com
eager-nash.188-93-233-196.plesk.page
eas-seo.com eas-seo.com
ebonyporn.site editors.choice6912650.hulfingtonpost.com
elektir.ru elektrischezigarette1.blog.pl
elektrozigaretten1.postbit.com elektroniksigaraankara.info
elektronischezigarettekaufen1.beeplog.com
elektrozigaretten2.yn.lt
elitedollars.com elitedollars.com
eloconcream.blogspot.com eloconcream.blogspot.com
elvel.com.ua elvel.com.ua
engines-usa.com energy-ua.com
envaseslotusama.com eonpal.com
erotag.com eredijovon.com
escort-russian.com erotik0049.com
euroskat.ru eshop.md
evehemming.blogspot.com.au essay-edu.biz
eyelike.com.ua etenininrade.ga
ezigarettekaufen1.hpage.com etur.ru
f-online.de europages.com.ru
fdzone.org european-torches.ru
fetroshok.ru excaliburfilms.com
floridamobilebillboards.com extstat.com
forensicpsychiatry.ru eyessurgery.ru
forsex.info ezofest.sk
forums.toucharcade.com f5mtrack.com
facialporntube.com
family1st.ca
fantasticpixcool.com
fashion-stickers.ru
fatfasts-4tmz.com
favoritemoney.ru
fetlifeblog.com
fickenprivat.info
filerockstar298.weebly.com
filkhbr.com
filmidivx.com
firstsiteguide.com
fiuxy.com
flash4fun.com
flashahead.info
flooringinstallation-edmonton.com
fok.nl
forminecrafters.ru
foxweber.com foxweber.com
free-traffic.xyz free-today.com
free-video-tool.com freewlan.info
fuckbuddybestgilf.info fuck-paid-share-buttons.xyz
futbolkisales.ru fullfileaccess.com
galaxy-family.ru fungamelands.com
fxgallery.com
g33.org
galeria-zdjec.com
gallerylisting.com gallerylisting.com
gamerextra.com gazobeton-p.com.ua
gdebestkupit.ru genetworx.com
generousdeal-a.akamaihd.net getaclueamerica.com
gentamicineyedrops.blogspot.com
germetiki.com.ua
girlsatgames.ru girlsatgames.ru
glasof.es goforexvps.com
gobongo.info goldpanningtools.com
gonextmedia.com goodwriterssales.com
gopro-online.info googlefeud.com
gotorussia.com googlemare.com
gox.com.ua
gpms.org.my gpms.org.my
greenidesign.co greenidesign.co
gribokstop.com growshop.es
headpharmacy.com gsou.cf
headpress.ru guardlink.com
healbio.ru handicapbathtubarea.com
highspeed5.net hasshe.com
houseofrose.com hayate.biz
hccoder.info
healmytrauma.info
hobild.net
hol.es
homedecoguide.info
hop.clickbank.net
hostsshop.ru
hotblognetwork.com
housekuba.org
housemilan.ru
hoverboard360.de hoverboard360.de
hoverboard360.es hplaserjetpdriver8y.pen.io
howopen.ru hptwaakw.blog.fc2.com
humanelydrew.com hully.altervista.org
hunthillfarmtrust.org hyipmanager.in
hyiphunter.org ideibiznesa2015.ru
ictizanidinehcl4mg.blogspot.com ido3.com
idealtits.net igrovyeavtomaty777.ru
idegenvezeto.eu imagerydatabase.com
ideoworld.org inboundlinks.win
ilmexico.com inmate-locator.us
imk.com.ua
incep.imagine-ex.co
indiasourcemart.in
indo-export.ru
intermesh.net intermesh.net
investzalog.ru ipostroika.ru
inzn.ru iqbazar.ru
iplogger.org iskalko.ru
ipornox.xxx islamtoday.co.za
ixora.pro istizanidinelikexanax.blogspot.com
jamiembrown.com itrevolution.cf
jbl-charge.info janerikholst.se
jerseychinabizwholesale.com javidol.site
jobmarket.com.ua jillepille.com
kalandranis.gr jongose.ninja
kalb.ru jovencitas.gratis
kambasoft.com jrcigars.com
kanimage.com junketjuice.blogspot.com
jurajskie.info
kadashihotel.com
kamalsinha.com
karting196.ru
kazan.zrus.org
kerei.ru
kerwinandcariza.com
keywesthideaways.co
keywords-monitoring-success.com keywords-monitoring-success.com
keywordsking.com kicknights.gq
khadastoafarde.tk kino-key.info
kino-filmi.com kino-rf.ru
kndxbkdx.bloger.index.hr kletkimehan.ru
koleso24.com.ua knogg.net
kollesa.ru kollekcioner.ru
koptims.tiu.ru koptims.tiu.ru
kuzinsp.ru koral.se
la-fa.ru kosova.de
kostenlos-sexvideos.com
krasivoe-hd.com
kretpicf.pw
kvartiry-remont.ucoz.ru
lacapilla.info lacapilla.info
lafourmiliaire.com lafourmiliaire.com
lawyersinfo.org lalalove.ru
lazy-z.com larutti.ru
lazymanyoga.com leboard.ru
lesbian.xyz lechenie-gemorroya.com
letto.by leftofcentrist.com
lexiangwan.com letmacworkfaster.world
linkbux.ru lincolntheatre.com
linkdebrideur.xyz linkarena.com
lionking-1994.blogspot.com
littlesunraiser.com littlesunraiser.com
llastbuy.ru live-sexchat.ru
lob.com.ru ljusihus.se
lombardfinder.ru lombia.com
long-beach-air-conditioning.com lonfon.xyz
lost-alpha.ru lostaruban.com
louboutinreplica.pw louboutinreplica.pw
ltvperf.com
luckyshop.net.ua
lutherstable.org lutherstable.org
m3gadownload.pl max-eclat.men
machicon-akihabara.info mcar.in.ua
macresource.co.uk mearns-tractors.co.uk
macrotek.ru medicinacom.ru
magazin-pics.ru megavolt.net.ua
mail.allnews24.in members.ghanaweb.com
makemoneyonline.com menetie.ru
mamasuper.prom.ua mikrobiologies.ru
manve.info mindeyegames.com
mararoom.ru minecraft-neo.ru
marketingtechniques.info misandesign.se
matchpal-a.akamaihd.net missclub.info
medictube.ru mmoguider.ru
medizinreisen.de mmostrike.ru
medmajor.ru mobile-appster.ru
mega-bony2017.pl mobplayer.ru
member-quiz.com modenamebel.ru
memberty.com moinozhki.com
metascephe.com monclercheap.xyz
metaxalonevstizanidine.blogspot.com monclercoats.xyz
microstatic.pl monclerjacketsoutlet.win
mikozstop.com moscow.online-podarki.com
mir-business-24.ru mototsikl.org
mncrftpcs.com moviemail-online.co.uk
moneytop.ru mp3ringtone.info
monsterdivx.tv mrsdalloways.com
mp3downloadhq.com multgo.ru
music.utrolive.ru mundoaberrante.com
musicspire.online musflashtv.com
muz-tracker.net mustwineblog.com
muzaporn.com muycerdas.xxx
mvpicton.co.uk my-floor.in.ua
myblogregistercm.tk mydirtyhobby.com
mylesosibirsk.ru mydownlodablefiles.com
mylida.org nailsimg.com
myliveblog.ru narutonaruto.ru
mypets.by
napalm51.nut.cc
naturalshair.site
naval.jislaaik.com naval.jislaaik.com
negociosdasha.com netpics.org
neobux-bg.info nfljerseys.online
networkad.net nfvsz.com
newstraveller.ru nippon-bearings.ru
nfljerseysforsalewholesaler.com nootrino.com
nlfjjunb5.ru novosti-hi-tech.ru
nudo.ca o.light.d0t.ru
o-o-8-o-o.com obesidadealgarve.com
o-o-8-o-o.ru obiavo.net
o333o.com oklogistic.ru
obuv-kupit.ru okroshki.ru
oivcvx.website olvanto.ru
omsk.xrus.org oneclickfiles.com
onclickpredictiv.com online-hd.pl
online-hit.info online-templatestore.com
onlinebay.ru onlinefilmz.net
onload.pw onlineku.com
onlythegames.com palvira.com.ua
ons-add.men pammik.ru
ontargetseo.us partner-trustworthy.men
openstat.com partnersafe.men
ordernorxx.com perederni.net
orenburg-gsm.ru photorepair.ru
orgasmatrix.com picanalyzer.data-ox.com
orhonit.com
osnova3.ru
outpersonals.com
ovirus.ru
partybunny.ru
pestomou.info
peugeot-club.org
phobia.us
photo-clip.ru
photochki.com
picquery.com
picsfair.com
picturesmania.com picturesmania.com
pierrehardysale.online picurams.pw
pk-services.ru pizdeishn.com
platinumdeals.gr pl-vouchers.com
playtap.us poem-paying.gq
pochemychka.net poisk-zakona.ru
podshipniki-nsk.ru poiskzakona.ru
pogosh.com popunder.net
pontiacsolstice.info pornokorol.com
pornofeuer.com posible.net
pornogratisdiario.com pospr.waw.pl
pornoinn.com
portadd.men
potoideas.us potoideas.us
pozdrawleniya.com pozdrawleniya.ru
pozvonim.com pravoholding.ru
pp-budpostach.com.ua princeadvantagesales.com
printdirectforless.com privatefx-in.ru
priora-2.com privatov-zapisi.ru
privatbank46.ru prlog.ru
privetsochi.ru pro-okis.ru
progonrumarket.ru prom23.ru
projectforte.ru promodj.com
pufip.com pst2017.onlinewebshop.net
puteshestvennik.com pulseonclick.com
puzo2arbuza.ru pussyspace.net
pxhdwsm.com q-moto.ru
quebec-bin.com qualitymarketzone.com
questionmarque.ch quickcashlimited.com
ranksays.com quizzitch.net
rankscanner.com randki-sex.com
rapidokbrain.com rangjued.com
rasteniya-vs-zombi.ru ranksignals.com
real-time-analytics.com razvratnoe.org
realmonte.net rcb101.ru
recipedays.ru regionshop.biz
redirlock.com registry-clean-up.net
regdefense.com
registrysweeper.com
remmling.de
remont-mobile-phones.ru remont-mobile-phones.ru
replica-watch.ru remybutler.fr
restorator-msk.ru reward-survey.net
reyel1985.webnode.fr rockingclicks.com
rniaeba.ga rubanners.com
robotixix.com rus-teh.narod.ru
rossanasaavedra.net rusvideos.su
rus-pornuha.com sad-torg.com.ua
rusexy.xyz sammyweaver.com
russintv.fr saugatuck.com
ryetaw.com sbf441.com
sack.net sbornik-zakonov.ru
salmonfishingsacramentoriver.com scanner-julia.top
sape.top scopich.com
sawin.beth.webd.pl sdsjweb.com
scanner-brian.top
scanner-jess.top
scanner-mark.top
scansafe.net
screpy.com
search-goo.com search-goo.com
searchaddis.com secret.xn--oogle-wmc.com
searchinterneat-a.akamaihd.net see-your-website-here.com
semaltmedia.com serialsway.ucoz.ru
semxiu.com sexfreepornoxxx.com
seoboxes.com
sexflirtbook.com
sexpornotales.net sexpornotales.net
sextracker.be sextracker.de
shakhtar-doneck.ru ship-marvel.co.ua
sherlock.se
shoppingjequiti.com.br shoppingjequiti.com.br
sim-service.net shtaketniki.ru
simplepooltips.com simul.co
sims-sims.ru site-speed-check.site
sitiz.club sitopreferito.it
sladkoevideo.com skachat-besplatno-obrazcy.ru
slayerlife.com skatestick.bid
slim.sellany.ru snworks.com
slujbauborki.ru social-s-iii.xyz
smstraf.ru soietvousmaime.fr
socialbuttons.xyz solicita.info
soundfrost.org sovinsteel.ru
spasswelt.xyz spb.afora.ru
sprttrack.com sports-supplements.us
stackthatbucks.com spy-app.info
start.myplaycity.com stairliftsarea.com
startufa.ru stard.shop
stretchingabuckblog.com stariy-baku.com
superinterstitial.com statistici.ro
svetodiodoff.ru steamoff.net
t-bygg.com steebook.com
tattooha.com sterva.cc
streetfooduncovered.com
stromerrealty.com
stroy-portal22.ru
studworks.org
sugvant.ru
superiends.org
superlist.biz
swinger-mobil.net
telsis.com telsis.com
tgtclick.com terrafootwear.us
the-trader.net
theheroes.ru theheroes.ru
tilido.com titslove.yopoint.in
tizanidine4mg.blogspot.com tizanidineoral.blogspot.com
tizanidineandcipro.blogspot.com tizanidinevscyclobenzaprine.blogspot.com
tizanidinerecreationaluse.blogspot.com tomatis.gospartner.com
tizanidinestreetvalue.blogspot.com topcasinoratings.ru
tomck.com torrents-tracker.com
top10-way.com trackmedia101.com
torrentred.games tsstcorpcddvdwshbbdriverfb.aircus.com
toyota.7zap.com tula.howotorg.ru
trafficreceiver.club tupper-posuda.ru
trainoffend.ml turbodsp.com
triplepanda.xyz
trkur.com
trustedmaccleaner.com
tryrating.com
tupper-shop.ru
tversvet.ru
twitlinks.com twitlinks.com
twu.com.ua txxx.com
ultramart.biz tyumen.xrus.org
umg-stroy.ru ustion.ru
uptime-alpha.net vapomnoncri.tk
urlcut.ru
usdx.us
userequip.com
uytmaster.ru
uzporno.mobi
vabasa.inwtrade.com
vacuumscleaner.com
valueclick.com
vashsvet.com
vbikse.com
vektorpress.ru
velen.io velen.io
ventelnos.com vepad.com
vesnatehno.ru
via-energy-acquistare.com via-energy-acquistare.com
via-gra.webstarts.com viagraneggrx.com
video-chat.in videochat.ph
video-production.com videochat.world
videochat.cafe vinylvault.co.uk
videochat.tv.br vip2ch.com
villakohlanta.nu visa-pasport.ru
virus-schutzmasken.de vksaver-all.ru
voditeltrezviy.ru vostoktrade.info
vucms.com vozbujdenie.com
w7s.ru vremya.eu
wanker.us vuclip.com
watchmygf.net wcb.su
weblo.com wdfdocando.com
webalan.ru
webcamtalk.net
webmonetizer.net webmonetizer.net
website-analyzer.info
website-stealer.nufaq.com
wedding-salon.net wedding-salon.net
weddingdresses.xyz weprik.ru
whiteproduct.com whatistizanidine2mg.blogspot.com
wholesalecheapjerseysfree.com whatistizanidinehclusedfor.blogspot.com
wholesalejerseys-cheapest.com wholesalejerseychinashop.com
wholesalejerseysgaa.com wingsoffury2.com
wineration.com winner7777.net
wladimirpayen.com wormix-cheats.ru
wnoz.de www.ehscloud.cn
wonderfulflowers.biz www.solartek.ru
word-vorlagen.net xboxster.ru
writersgroup580.web.fc2.com xn----7sbaaabaei0cc8aj5bj0bncejx.xn--p1ai
writingservices17.blogspot.ru xn----ctbigni3aj4h.xn--p1ai
x-mix.info xn--1-8sbcpb0bdm8k6a.xn--p1ai
x69ty.ru xn--80atua3d.xn--p1ai
xaylapdiendanang.com xn--c1acygb.xn--p1ai
xcombear.ru xsfetish.org
xitjw.info xtubeporno.net
xn----7sbahjd3btneuw1joc.xn--p1ai xxx-cam.webcam
xn----9sbebi2bvzr7h.xn--p1ai xzlive.com
xn----dtbndd4ae7eub.top yachts-cruise.info
xn--80aebbcbcdemfkhba4byaehoejh8dza3v.xn--p1ai yogamatsexpert.com
xn--80ajbshivpvn2i.xn--p1ai yougetsignal.com
xn--e1aggki3c.xn--80adxhks youjizz.vc
xrus.org youporn-forum.ga
xuki.us youresponsive.com
yaminecraft.ru yousense.info
yebocasino.co.za youtube-downloader.savetubevideo.com
yellowads.men zakonobosago.ru
yellowstonesafaritours.com zapatosenventa.info
yellowstonevisitortours.com zawyna.ua
yougotanewdomain.com zdesoboi.com
zahvat.ru zmoda.hostreo.com
zaimhelp.ru zolotoy-lis.ru
zarepta.com zone-kev717.info
zatjmuzu.info zoominfo.com
zeikopay.com zootoplist.com
zeroredirect1.com zx6.ru
zeroredirect11.com
zojirushi-products.ru
zoogdisany.com
zootravel.com
zryydi.com
zscalerone.net
zskdla.site

484
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
1001desert.com 20pascals.nl
40cg.com 2ads.co.uk
8jn.dating-app.ru adbetclickin.pink
DomainStatsBot adobereader-free.ru
aanapa.ru aibolita.com
abclauncher.com albamargroup.com
admitad.com albuteroli.com
afmuseum.com amigobulls.com
aghanyna.com anlimebel.kiev.ua
aktobe.xkaz.org anniemation.com
albumsuper.info anticrawler.org
alienwheels.de apartmentbay.ru
allforminecraft.ru apessay.com
allnews.md apiadanaknet-a.akamaihd.net
apartment.ru apiqualitinknet-a.akamaihd.net
apiportalmorecom-a.akamaihd.net arabseyes.com
approved.su asdfg.pro
autoblogger24.info asrvvv-a.akamaihd.net
aviapanda.ru asscenihotosidea.blogspot.co.za
aviav.co asseenontv.ru
ayakino.net atout-energie-69.com
azadnegar.com azlex.uz
bag77.ru
barbourjackets.online barbourjackets.online
battleforupdating.site bashtime.ru
bcmp.org beauty-clinic.ru
besthomemadepornsites.com best-mam.ru
bestwrinklecreamnow.com bingoporno.com
biographiya.com bioca.org
bioinnovate.co biznesschat.net
biteg.xyz bjgugu.net.ua
biznesrealnost.ru bloglag.com
bloggerads.net bloke.com
bonux.nextview.ru bolitgorlo.net
brakehawk.com bookhome.info
bumskontakte.org bradleylive.xyz
buqayy0.livejournal.com brainboosting.club
braindaily.xyz
brandimensions.com
bumble.cheapwebsitehoster.com
businesxxl.com
buytizanidineonlinenoprescription.blogspot.com buytizanidineonlinenoprescription.blogspot.com
callawaygolfstore.online bycontext.com
camdolls.com c-gainsbourg.com
cars-modification.net c1.onedmp.com
canyougethighofftizanidine.blogspot.com
cardsdumps.com
cartierlovebracelet.xyz cartierlovebracelet.xyz
cheapmoncler.win cashkitten-a.akamaihd.net
clicksotrk.com cete.ru
clipartnew.com cherrypointplace.ca
co.lumb.co choosecuisine.com
compliance-mark.top citysecurity.nu
countbertwistdisp26.soup.io compliance-fred.xyz
crackguru.tk compliance-jessica.top
crystalslot.com concordexoticrentals.com
datsun-do.com cookie-law-enforcement-ee.xyz
defenderxtactical.com creditmoney.com.ua
delta-line.men crest3d.ru
detmebel.su cutt.us
digital-sale.su cvety24.by
dipstar.org dalavia.ru
doktoronline.no decoratinghomes.ga
dokumentalkino.net deshevo-nedorogo.ru
dorratex.tn digitalassetmanagement.site
download-walpaperhd.blogspot.com donna7753191.ru
downloadeer.net dostavka-v-ukrainu.ru
drugs-no-rx.info downvids.net
dvr.biz.ua dyshagi.ru
eandsgallery.com e2click.com
earl-brown.info elektronischezi.livejournal.com
ec-file.info erotag.com
ekspertmed.com erotik0049.com
elektrischezigaretten2.beeplog.com estelight.ru
elektroniksigaraankara.info f012.de
elektronischezigarettekaufen1.myblog.de facecup.top
elektrozigarettekaufen2.blogse.nl
esnm.ru
european-torches.ru
exci.ru
facebook-mobile.xyz
feel-planet.com
fefo.gdn
fickblock18.com
fickenprivat.info
figuringmoneyout.com
finejewelryshop.ru
fitfloponline.store fitfloponline.store
fitness-video.net foodcrafts.website
flytourisme.org fortevidyoze.net
free-stock-illustration.com freeuploader.com
freshsuperbloop.com fuckingawesome.com
fuckmill.com fun2cell.net
futbolkisales.ru get-free-traffic-now.com
fym.com.ru getridofstretchmarks.org
galaxy-family.ru ghazel.ru
game-mmorpg.net girlgamerdaily.com
gepezz.info goldadpremium.com
globalscam.ga googlefeud.com
glogow.pl gox.com.ua
go2mike.ru gratuitxblcodes.com
gsasearchengineranker.space gribokstop.com
gwagka.com grupografico-pilar.com.ar
hdimagelib.com hatdc.org
hikesearch.net havepussy.com
historichometeam.com hidefiles.org
hjaoopoa.top himazin.info
hkladys.com hobbyhuren-datenbank.com
hollywoodactress.info hobbyhuren24.net
hotloans.ru hotxnights.info
https-legalrc.biz ikearugs.xyz
iideaidekonkatu.info ilbe.club
ilikevitaly.com img.wallpaperstock.net
imgarit.pw instabid.tech
int.search.mywebsearch.com intellektmedia.at
interfucks.net isitwp.com
iplusbit.blogspot.co.za istmira.ru
itbc.kiev.ua ivanovo.zrus.org
itch.io ivearchenceinflu.cf
itservicesthatworkforyou.com izismile.com
j33x.com jabimgo.pw
jav-fetish.com jagg.info
javatex.co.id javrip.net
javstock.com justdating.online
jerseychinabizwholesale.us justkillingti.me
jerseyswholesalechinalimited.com kaac.ru
justprofit.xyz kabbalah-red-bracelets.com
kidskunst.info kazinogames.lv
kino2018.cc kchaxton.com
konkursowo-24.pl kino-key.info
legionalpha.com kinobest.pl
letsart.ru kiwe-analytics.com
lexiangwan.com knowyournextmove.com
libertybilisim.com koronirealestate.gr
littleberry.ru languagecode.com
loftdigital.eu lashstudia.ru
losangeles-ads.com lechenie-gemorroya.com
mararoom.ru lightinghomes.net
marketingtechniques.info linkrr.com
matchpal-a.akamaihd.net lion.misba.us
metalonly.info lkbennettoutlet.online
michaelkorsoutlet.store luxmagazine.cf
mobstarr.com madot.onlinewebshop.net
moesonce.com mainlinehobby.net
moetomnenie.com manve.info
mrlmedia.net masterseek.com
msfsaar.de maxthon.com
muz-baza.net mbiologi.ru
nalogovyykodeks.ru mikozstop.com
naturalbreakthroughsresearch.com minet.club
new-apps.ru mmoguider.ru
new7ob.com monthlywinners.com
newhairstylesformen2014.com mossmesi.com
noclegonline.info musflashtv.com
nucia.biz.ly musicspire.online
okeinfo.online mybackgroundlandscape.blogspot.com
okmedia.sk narutonaruto.ru
online-podarki.com net-radar.com
onlinedomains.ru njkmznnb.ru
origin-my.ru notaria-desalas.com
oxotl.com notfastfood.ru
panchro.co.uk nyfinance.ml
partner-high.men o-o-8-o-o.com
partnerworkroom.men obiavo.com
perm.xrus.org one-gear.com
picmoonco.pw online.ktc45.ru
pillscheap24h.com online247.ml
onlineserialy.ru
openfrost.com
pavlodar.xkaz.org
picsfair.com
porno-chaman.info
pornofiljmi.com
pornohd1080.online pornohd1080.online
profolan.pl pornosemki.info
prohoster.info pornosexrolik.com
pukaporn.com postclass.com
rczhan.com pozdrawleniya.ru
recipedays.ru pricheski-video.com
rff-cfal.info prlog.ru
produkto.net
pron.pro
propranolol40mg.blogspot.com
ptr.ruvds.com
qcstrtvt.bloger.index.hr
real-time-analytics.com
realmonte.net
realting-moscow.ru
rep-am.com
restaurantlescampi.com
ringtonepartner.com
robertefuller.com robertefuller.com
royalvegascasino.com rp9.ru
rumamba.com ruex.org.ua
runstocks.com rus-teh.narod.ru
russia-tao.ru russkoe-zdorovie.ru
santasgift.ml rvtv.ru
sarm.tk scanmarine.info
saveriopiazza.it scanner-marwin.top
sdsjweb.com semalt.com
searchimpression.com senger.atspace.co.uk
seksotur.ru sibdevice.ru
semrush.com sitevaluation.com
sexfreepornoxxx.com sitevaluation.org
sexs-foto.com skladvaz.ru
shillyourcoins.com smstraf.ru
shop-electron.ru snjatie-geroinovoy-lomki.ru
si-unique.com soundfrost.org
simul.co spin2016.cf
sitechecker.pro spinnerco.ca
siteexpress.co.il sps-shop.com
siteonomy.com sptslmtrafms.com
skutecznetabletkinaporostwlosow.pl statustroll.com
smailik.org storehouse.ua
snip.to strana-krasoty.ru
soft-program.com stroilka.info
solicita.info students-cheapskate.ml
sotechco.co sukirgenk.dvrlists.com
sovetogorod.ru svarkagid.com
soviet-portal.do.am taqplayer.info
speechfoodie.com tastyfoodideas.com
spywareit.com teenfuck.tv
spywarenuker.com thecounter.com
stroyplus.ru thedownloadfromwarez.blogspot.com
swagbucks.com thefds.net
tabletkinaodchudzanie.com.pl tivolibasket.it
tam-gde-more.ru tmearegion26.com
tattooha.com toon-families.com
teastory.co tourismvictoria.com
tetracsaudi.com trichizobswiv.agddns.net
thebestweightlosspills.ovh trk-4.net
thebluffs.com tx41tclega.ru
thepornsex.org typimga.pw
tiandeural.ru ultimatesetnewfreeallsoftupgradesystems.pw
tizanidine4mg.blogspot.com ultramart.biz
tizanidine4mgstreetprice.blogspot.com unece.org
tizanidinepills.blogspot.com uvozdeckych.info
tizanidinesideeffects.blogspot.com vapeface.club
tjkckpytpnje.com vclicks.net
top10-online-games.com velen.io
topclickguru.com velpanex.ru
topdownloads.ru videochat.ph
topkarkas.com vipsiterip.org
tuckpointingmasonrysystems.com womama.ru
universals.com.ua worldhistory.biz
url-img.link worldinternetauthority.com
userequip.com
vacances-voyages.info
vepad.com
viagengrarx.com
video--production.com
videonsk.com
vipsexfinders.com
voluumtrk.com
websites-reviews.com
wfb.hatedriveapart.com
worldmusicfests.com
woweb.com.ua woweb.com.ua
ws.ampower.me x-diesel.org
x-diesel.com xclicks.net
xn--80aaagvmjabrs1aoc9luc.xn--p1ai xep.info
xn--80abgj3a5acid6ghs.top xerox-douglas.cf
xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai
xn--q1a.xn--b1aube0e.xn--c1acygb.xn--p1ai xn--q1a.xn--b1aube0e.xn--c1acygb.xn--p1ai
xnxx699.com xvideosporn.biz
xsion.net y8games-free.com
xtubeporno.net youdao.com
xxxdatinglocal.us zaapplesales.blogspot.com
yourporn.com zaidia.xhost.ro
yousense.info zeroredirect1.com
youtubedownload.org zoogdisany.com
ytmnd.com zzbroya.com.ua
zaloro.com
zigarettenonlinekaufen.tumblr.com
zynax.ua

10
dev-tools/test_units/ratelimittest-beta.txt
View file

@ -581,14 +581,4 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html> </html>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.31 0; 10.1.0.29 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.10.3384 ### Version: V4.2022.10.3385
### Updated: Sat Oct 1 22:01:18 UTC 2022 ### Updated: Sun Oct 2 10:03:32 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.10.3384 ### Version: V4.2022.10.3385
### Updated: Sat Oct 1 22:01:18 UTC 2022 ### Updated: Sun Oct 2 10:03:32 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##