vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-05 20:20:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.07.3210 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-07-05 22:07:26 +00:00
parent 45ff8abee7
commit 375cf822ea
47 changed files with 102926 additions and 102936 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.07.3209 #### Version: V4.2022.07.3210
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.07.3209 #### Version: V4.2022.07.3210
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.07.3209 #### Version: V4.2022.07.3210
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12730
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.07.3209 #### Version: V4.2022.07.3210
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12734
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12734
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

12734
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12734
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 1; 10.1.0.174 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 0; 10.1.0.174 0;
127.0.0.1 0; 127.0.0.1 0;

12734
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 1; 10.1.0.174 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 0; 10.1.0.174 0;
127.0.0.1 0; 127.0.0.1 0;

12734
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 1; 10.1.0.174 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 0; 10.1.0.174 0;
127.0.0.1 0; 127.0.0.1 0;

12734
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3209 3210

12734
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

12734
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.10.31.49 0; 104.43.209.169 0;

12734
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 1; 10.1.0.174 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 0; 10.1.0.174 0;
127.0.0.1 0; 127.0.0.1 0;

12734
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 1; 10.1.0.174 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 0; 10.1.0.174 0;
127.0.0.1 0; 127.0.0.1 0;

12734
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 1; 10.1.0.174 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 0; 10.1.0.174 0;
127.0.0.1 0; 127.0.0.1 0;

12734
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.10.31.49 0; 104.43.209.169 0;

12734
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.10.31.49 0; 104.43.209.169 0;

12734
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 1; 10.1.0.174 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Alexibot Craftbot
InternetSeer Custo
Lftp Discoverybot
Linkbot Hloader
LinkpadBot IstellaBot
Masscan JikeSpider
Nimbostratus SpaceBison
Shodan Swiftbot
SiteSnagger TurnitinBot
Zade muhstik-scan

320
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
ADmantX 404checker
ALittle Client AIBOT
ASPSeek
Abonti Abonti
Aboundexbot Acunetix
AfD-Verbotsverfahren Anarchy
AhrefsBot
Aipbot
Alligator
Anarchy99 Anarchy99
Anthill
Aspiegel Aspiegel
Atomseobot
AwarioRssBot AwarioRssBot
AwarioSmartBot BackStreet
BDCbot BackWeb
Backlink-Ceck
Badass Badass
Bandit Bandit
Barkrowler Barkrowler
BatchFTP
Bigfoot
Black Hole
Blackboard Blackboard
Blow Brandprotect
BlowFish Buck
Boardreader
Bolt
Buddy Buddy
CCBot BuiltBotTough
CODE87 BunnySlippers
Cegbfeieh BuzzSumo
CazoodleBot
Chlooe Chlooe
Citoid Cliqzbot
Cogentbot Cloud mapping
Collector Copier
Copyscape Crawling at Home Project
Cosmos
Craftbot
CrazyWebCrawler CrazyWebCrawler
Crescent Crescent
Curious
DBLBot DBLBot
DIIbot DIIbot
DataCha0s DSearch
DTS Agent
Demon Demon
Deusu
Devil Devil
DigitalPebble Digincore
Dirbuster
Disco Disco
Dispatch Discoverybot
DnBCrawler-Analytics DittoSpyder
DomainCrawler DomCopBot
DomainStatsBot DomainAppender
Domains Project Domains Project
Dotbot
Drip Drip
EMail Siphon
EasyDL
Ecxi Ecxi
EirGrabber EirGrabber
Exabot Express WebPictures
ExtLinksBot ExtractorPro
Extractor Ezooms
Extreme Picture Finder FDM
FHscan FHscan
FlashGet Fimap
FyberSpider Foobot
Go!Zilla GalaxyBot
Gotit GermCrawler
Grabber GetWeb
Grafula Getintent
GrapeFX Gigabot
GrapeshotCrawler
GridBot
HEADMasterSEO HEADMasterSEO
HaosouSpider HTTP::Lite
Harvest HTTrack
Havij Haansoft
HybridBot Hloader
IDBot Humanlinks
IDBTE4M
IRLbot IRLbot
Iblog Id-search
InfoNaviRobot IlseBot
InfoTekies Image Sucker
Intelliseek Indy Library
InternetSeer InterGET
Iria Jbrofuzz
JetCar Jetty
JikeSpider JikeSpider
Jorgee JustView
Keybot Translation-Search-Machine Kozmosbot
Keyword Density
Kinza
LNSpiderguy LNSpiderguy
LeechGet Larbin
Leap
Lftp
LibWeb
LieBaoFast
Lightspeedsystems Lightspeedsystems
Likse LinkScan
LinkWalker LinqiaMetadataDownloaderBot
LinkpadBot
LinqiaRSSBot LinqiaRSSBot
Litemage_walker Lipperhey
Ltx71 Lipperhey Spider
MFC_Tear_Sample Lmspider
MIDown tool
MQQBrowser MQQBrowser
MSFrontPage
MTRobot MTRobot
Magnet Mail.RU_Bot
Mass Downloader Majestic12
MauiBot MarkMonitor
Mb2345Browser Mata Hari
MeanPath Bot MegaIndex.ru
Meanpathbot
MicroMessenger MicroMessenger
Minefield
Moblie Safari
MolokaiBot MolokaiBot
Morfeus Fucking Scanner Morfeus Fucking Scanner
Mozlila
Mr.4x3 Mr.4x3
Msrabot Musobot
Name Intelligence Name Intelligence
Navroad Nameprotect
Needle
Nessus Nessus
Net Vampire Net Vampire
NetLyzer
Netcraft Netcraft
Nibbler Nettrack
Nimbostratus NextGenSearchBot
Nuclei Nikto
Octopus Ninja
Offline Navigator Not
OnCrawl Nutch
Offline Explorer
OpenLinkProfiler
OpenVAS OpenVAS
Openfind
Openvas Openvas
OrangeBot OrangeSpider
OutfoxBot Page Analyzer
PHPCrawl PageGrabber
POE-Component-Client-HTTP PageScorer
PageThing.com
Papa Foto Papa Foto
Pavuk
PeoplePal PeoplePal
Petalbot
Pi-Monster
Picsearch Picsearch
PictureFinder PictureFinder
Pixray Pixray
Pockey Pu_iN
ProPowerBot
Probethenet
Psbot
PxBroker PxBroker
PyCurl RSSingBot
RankActiveLinkBot RankActive
RankFlex
RankingBot2 RankingBot2
Recorder Rankivabot
RedesScrapy ReGet
RealDownload
RepoMonkey
RocketCrawler RocketCrawler
Rogerbot Rogerbot
SBIder SBIder
SEOkicks SEOlyticsCrawler
SMTBot SEOstats
SISTRIX
SalesIntelligent SalesIntelligent
Scanbot ScanAlert
ScoutJet
ScreenerBot ScreenerBot
ScrepyBot Searchestate
Semrush SemanticJuice
SemrushBot SemrushBot
SentiBot SentiBot
Seomoz Seomoz
Shodan SiteExplorer
Site Sucker SlySearch
SiteCheckerBotCrawler SmartDownload
SiteLockSpider Snake
SiteSnagger Snapbot
SiteSucker Sosospider
Snoopy SpaceBison
Sociscraper Spammen
Spanner Sqlmap
Sqlworm
Steeler
Sucuri Sucuri
Surfbot SuperBot
Swiftbot SuperHTTP
Szukacz SurveyBot
T0PHackTeam
Teleport
Telesphoreo Telesphoreo
Telesphorep Thumbor
TheNomad
TightTwatBot TightTwatBot
Titan Titan
Toata
Toweyabot Toweyabot
Turingos
TurnitinBot
TwengaBot TwengaBot
Typhoeus
URLy Warning URLy Warning
V-BOT UnisterBot
Upflow
VCI
Vacuum Vacuum
Vagabondo VeriCiteCrawler
VidibleScraper VidibleScraper
Virusdie Virusdie
VoidEYE
Voil Voil
WASALive-Bot WBSearchBot
WEBDAV WISENutbot
WPScan WPScan
WWW-Collector-E WWW-Collector-E
WWW-Mechanize Wallpapers
WWW::Mechanize
Wallpapers/3.0
WallpapersHD WallpapersHD
WeSEE
Web Auto Web Auto
Web Collage Web Collage
Web Fetch
Web Pix
Web Sucker Web Sucker
WebCollage
WebCopier WebCopier
WebEnhancer
WebFetch
WebFuck WebFuck
WebGo IS WebGo IS
WebImageCollector WebSauger
WebLeacher Website Quester
Whatweb WebsiteExtractor
Woobot WebsiteQuester
Zade Wonderbot
Xaldon WebSpider
Xaldon_WebSpider
Xenu
YoudaoBot
Zauba Zauba
Zermelo
Zitebot Zitebot
ZmEu ZmEu
ZoomBot
ZumBot ZumBot
adscanner
archive.org_bot archive.org_bot
arquivo-web-crawler
arquivo.pt
autoemailspider autoemailspider
backlink-check
cah.io.community
check1.exe
clark-crawler
cognitiveseo cognitiveseo
crawler.feedback com.plumanalytics
dataforseo.com crawler4j
domainsproject.org domainsproject.org
eCatch
evc-batch evc-batch
facebookscraper gopher
ips-agent heritrix
lwp-trivial
meanpathbot
mediawords mediawords
muhstik-scan muhstik-scan
oBot oBot
polaris version pcBrowser
s1z.ru s1z.ru
seoscanners seobility
seostar
serpstatbot
sexsearcher sexsearcher
sitechecker.pro spyfu
sp_auditbot sysscan
webgains-bot trendiction.com
webmeup-crawler trendiction.de
webprosbot ubermetrics-technologies.com
x09Mozilla x09Mozilla
zauba.io

302
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,162 +1,168 @@
01h4x.com ADmantX
404enemy
AIBOT AIBOT
Acunetix Aboundex
AfD-Verbotsverfahren Aboundexbot
AhrefsBot
AiHitBot AiHitBot
AlphaBot Aipbot
Apexoo AllSubmitter
AwarioSmartBot Anarchie
BDCbot Anarchy99
BDFetch Ankit
Backlink-Ceck BBBike
BLEXBot
BackStreet
Badass Badass
Black Hole
BlackWidow BlackWidow
Blow
BlowFish
Boardreader Boardreader
Brandwatch BotALot
Buck Brandprotect
Buddy Buddy
Bullseye
BunnySlippers BunnySlippers
CCBot CCBot
CSHttp
CazoodleBot CazoodleBot
CensysInspect
CheeseBot
CherryPicker CherryPicker
Chlooe Chlooe
Citoid Citoid
Cliqzbot Cliqzbot
Cosmos Cloud mapping
Craftbot Cogentbot
CrazyWebCrawler Copier
CrunchBot CopyRightCheck
Custo Copyscape
CyotekWebCopy CyotekWebCopy
DBLBot DBLBot
DIIbot DSearch
Demon
Deusu
Devil Devil
Digincore Disco
DigitalPebble
Dirbuster
Discobot Discobot
Discoverybot
DittoSpyder DittoSpyder
DomainAppender DnyzBot
DomainCrawler DomCopBot
Domains Project DomainStatsBot
Dotbot EMail Siphon
Dragonfly EMail Wolf
ECCP/1.0
EasyDL EasyDL
Ecxi
EirGrabber EirGrabber
EroCrawler Exabot
Express WebPictures
ExtLinksBot
ExtractorPro ExtractorPro
Fimap EyeNetIE
FlashGet Ezooms
FHscan
FemtosearchBot
Flunky Flunky
Freeuploader Freeuploader
FrontPage FrontPage
Fuzz
FyberSpider FyberSpider
Fyrebot Fyrebot
G-i-g-a-b-o-t GalaxyBot
GT::WWW GetRight
Genieo GetWeb
Go-Ahead-Got-It GoZilla
Gotit
GrabNet GrabNet
GrapeFX Grabber
Grafula
GrapeshotCrawler GrapeshotCrawler
HMView
HTMLparser
HTTP::Lite
Haansoft Haansoft
HaosouSpider Havij
Heritrix Heritrix
Hloader Humanlinks
HybridBot Image Fetch
IDBTE4M InfoNaviRobot
Iblog InfoTekies
IlseBot
IndeedBot
Indy Library
Intelliseek
InterGET InterGET
InternetSeer Internet Ninja
Iria
Iskanie Iskanie
JOC Web Spider
JamesBOT JamesBOT
Jbrofuzz JennyBot
JetCar Joomla
Jetty
Jorgee Jorgee
Kinza Kenjin Spider
Lanshanbot Lanshanbot
Larbin LeechFTP
Leap LeechGet
Lftp Libwhisker
LibWeb
LieBaoFast LieBaoFast
Lightspeedsystems Lightspeedsystems
Likse
LinkScan LinkScan
LinkWalker LinkWalker
Linkbot
LinkextractorPro LinkextractorPro
LinkpadBot LinkpadBot
LinqiaMetadataDownloaderBot
Ltx71 Ltx71
MIDown tool MFC_Tear_Sample
MJ12bot MJ12bot
MQQBrowser MQQBrowser
MSIECrawler MSIECrawler
Mail.RU_Bot Mag-Net
Majestic SEO Majestic-SEO
Majestic12
MarkMonitor
MarkWatch MarkWatch
Masscan
MeanPath Bot MeanPath Bot
Meanpathbot Mediatoolkitbot
MegaIndex.ru Minefield
Metauri Mojeek
MicroMessenger Mojolicious
Microsoft URL Control
Moblie Safari
Morfeus Fucking Scanner Morfeus Fucking Scanner
Mozlila Mr.4x3
Msrabot
NICErsPRO
Nameprotect
NearSite
NetLyzer NetLyzer
Netvibes
Nibbler Nibbler
Niki-bot
Nikto
NimbleCrawler NimbleCrawler
Nimbostratus Nmap
Ninja Offline Explorer
Octopus OnCrawl
OrangeBot OpenLinkProfiler
Openvas
OutclicksBot
OutfoxBot OutfoxBot
PHPCrawl POE-Component-Client-HTTP
PageAnalyzer
PageGrabber
PageScorer
PageThing.com
Pandalytics Pandalytics
Papa Foto
Pavuk Pavuk
PeoplePal
Petalbot
Pi-Monster Pi-Monster
Picscout Picscout
Piepmatz Picsearch
PleaseCrawl
ProPowerBot ProPowerBot
ProWebWalker ProWebWalker
Probethenet PxBroker
RankingBot Re-re
RankingBot2 Reaper
RankurBot RepoMonkey
ReGet SBIder
RebelMouse SEOkicks
RedesScrapy
Ripper
RocketCrawler
SEOkicks-Robot
SEOlyticsCrawler
SEOprofiler SEOprofiler
SISTRIX SEOstats
Scrapy ScoutJet
Screaming Screaming
ScreenerBot
ScrepyBot
SearchmetricsBot SearchmetricsBot
SentiBot SemanticJuice
SemrushBot
SeobilityBot
Shodan Shodan
Site Sucker Site Sucker
SiteExplorer SiteExplorer
@ -164,87 +170,81 @@ SiteLockSpider
SiteSucker SiteSucker
Sitebeam Sitebeam
Siteimprove Siteimprove
Snoopy SmartDownload
Sociscraper SocialRankIOBot
SpaceBison Spammen
SpankBot SpankBot
Spbot Spinn3r
Steeler Sqlmap
SuperBot Sucuri
SuperHTTP SurveyBot
Surfbot T0PHackTeam
Swiftbot TheNomad
Szukacz
TeleportPro
The Intraformant
TightTwatBot TightTwatBot
Toweyabot Toata
Trendiction Turingos
TurnitinBot TurnitinBot
URLy Warning Twice
URLy.Warning URLy.Warning
UnisterBot UnisterBot
Upflow
V-BOT V-BOT
Vagabondo VeriCiteCrawler
Virusdie Virusdie
Voil Voil
WASALive-Bot WASALive-Bot
WPScan WWW-Collector-E
WWW-Mechanize
WWW::Mechanize WWW::Mechanize
WeSEE Wallpapers
Web Enhancer Web Collage
Web Fetch Web Fetch
Web Pix Web Sauger
Web Sucker WebAuto
WebCollage
WebEnhancer
WebFetch WebFetch
WebFuck
WebImageCollector WebImageCollector
WebLeacher WebLeacher
WebPix
WebReaper
WebStripper
WebSucker WebSucker
WebWhacker Webshag
WebZIP Website Quester
Webalta
WebmasterWorldForumBot
WebsiteExtractor WebsiteExtractor
WebsiteQuester Webster
Whatweb Whacker
Who.is Bot Who.is Bot
Widow
Wonderbot Wonderbot
Woobot Woobot
Wotbox Wotbox
Xaldon WebSpider Xaldon_WebSpider
Xenu Xenu
Zauba
Zermelo Zermelo
Zeus
ZmEu ZmEu
ZoomBot
ZumBot ZumBot
arquivo-web-crawler ZyBorg
archive.org_bot
backlink-check
cah.io.community
cognitiveseo cognitiveseo
com.plumanalytics com.plumanalytics
crawl.sogou.com
crawler.feedback
dataforseo.com
demandbase-bot
domainsproject.org domainsproject.org
eCatch
facebookscraper
gopher gopher
heritrix instabid
internetVista monitor ips-agent
iubenda-radar lwp-trivial
linkdexbot page scorer
meanpathbot
muhstik-scan
netEstate NE Crawler
polaris version polaris version
ripz probe-image-size
s1z.ru s1z.ru
scalaj-http scalaj-http
seoscanners seostar
sp_auditbot serpstatbot
trendiction.de siteripz
webgains-bot sogouspider
x09Mozilla voyagerx.com
webmeup-crawler
zauba.io
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
alert-fdm.xyz aa625d84f1587749c1ab011d6f269f7d64.com
codq.info advertex.info
energydiet-info.ru headpharmacy.com
front.to letslowbefast.site
myseoconsultant.com pavlodar.xkaz.org
nagdak.ru pihl.se
pony-business.com pornonik.com
unitygame3d.com pornosok.ru
x-mix.info thedownloadfromwarez.blogspot.com
xmlinde.com website-analyzer.info

914
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
000free.us 00it.com
0daymusic.org 00webcams.com
1-99seo.com
1008.su
101billion.com
108shot.com
1qingdao.com 1qingdao.com
1webmaster.ml 202ch.com
24-ak.ru 2drittel.de
4kpics.rocks 40cg.com
4ureyesonly.com 4inn.ru
55wmz.ru 57883.net
5forex.ru
5ws.dating-app.ru 5ws.dating-app.ru
991mostfm.co.id 6128786.com
aa9bd78f328a6a41279d0fad0a88df1901.com 7search.com
aarbur.com 83net.jp
abwa.tk 888.com
8gold.com
8kisses.eu
999webdesign.com
aaa38852e886ac4af1a3cff9b47cab6272.com
aab94f698f36684c5a852a2ef272e031bb.com
aad1f4acb0a373420d9b0c4202d38d94fa.com
academia-nsk.org
acc.eu.org
acheterviagrafr24.com
addtoadd.men
adf.ly
adidas.frwebs.fr adidas.frwebs.fr
adloads.com admatic.com.tr
admeasures.com adserver-e7.com
admitad.com adsfresh.men
adsref.men adultnet.in
akusherok.ru advancedmassagebysara.com
allfinweb.com aksonural.ru
allwidewallpapers.com albuteroli.com
alpinism.ru alessandraleone.com
alvtank.se alkoravto.ru
anlimebel.kiev.ua allboard.xobor.de
anti-virus-removal.info allcredits.su
apibatbrowsecom-a.akamaihd.net allforminecraft.ru
apiholdingmypage-a.akamaihd.net ameblo.jp
apxeo.info analnoeporno.tv
arenda-avtoprokat-krasnodar.ru ananumous.ru
arraty.altervista.org animalia-life.club
articlesdirectoryme.info anjalika.co.in
artpicso.com anonymizeme.pro
astrochicks.com apartamentwroclaw.eu
app-ready.xyz
apptester.tk
asophoto.com
asseenontvonline.ru
atdedinotuho.tk atdedinotuho.tk
autoloans.com australia-opening-times.com
autonew.biz autoblogger24.info
autowebmarket.com.ua autowebmarket.com.ua
aviva-limoux.com azartniy-bonus.com
babespcs.com azbaseimages.net
babieca.com b-styles.xyz
beachfix.co backuperwebcam.weebly.com
bear.gotcher.us banan.tv
best-seo-offer.com bang-hotties.com
best-seo-solution.com barnfurnituremart.com
bestchoice.cf beachtoday.ru
benchmarkcommunications.co.uk
berlininsl.com
best-businessman.ru
best-mam.ru
bestdooz.com bestdooz.com
bestempresas.es bezcmexa.ru
bestpornuha.com bgviagrachrx.com
bestpriceninja.com bioscorp.ru
bidbuy.co.kr blagovest-med.ru
bif-ru.info blockworld.ru
bikini-image.com blog-hits.com
binaryoptionscops.info bloggers.nl
biographiya.com bluesman.nu
bizcheapjerseyswholesalechina.com bobba.dzaba.com
bizfly.info boom-celebs.com
bizlist.com.de bristolhotel.com.ua
blogs.rediff.com btc4u.ru
bmsco.co btvn.ru
bookmark4you.com.biz buehne-fuer-menschenrechte.de
brainsandeggs.blogspot.com buyclomidonlaine.com
breastaugmentation.co.za buytizanidineonline.blogspot.com
bugof.gq buzzurl.jp
buigas.00it.com cablecar.us
buybest1.biz calc-for-credit.ru
c-gainsbourg.com carabela.com.do
c1.onedmp.com cartierlove2u.com
callawaygolfoutlet.online cartierlovestore.xyz
candelluxsklep.pl cartierreplica.win
candlehandmade.com cbb1smartlist12.click
canyougethighofftizanidine.blogspot.com celejihad.info
cartierbracelet.xyz
cartierjusteunclou.xyz
cartierlovestore.com
casablancamanor.co.za
caulderwoodkennels.com
caveavins.fr
cbcseward.com
cellfun.mobi
cenoval.ru cenoval.ru
centraletermice.us
ceotrk.com ceotrk.com
chatroulette.video chatseo.com
cheap-pills-norx.com
cheapbarbour.online cheapbarbour.online
cheapestjerseys-wholesale.com cheatcode-lita12.rhcloud.com
cheapjerseysap.com cherubinimobili.it
christianlouboutinshoes.xyz chidporn.com
chuckguilford.com chimiver.info
cibpenonptib.flu.cc chinacheapelitejerseys.com
cigarpass.com choice-credit.ru
citetick.com cialischsrx.com
clips.ua.ac.be cntravelre.com
club-samodelkin.ru compliance-andrew.xyz
cmd.kz compliance-barak.top
coderstate.com compliance-jane.top
companies-catalog.com compliance-olga.top
compiko.info connectingsingles.com
compliance-alexa.top conocer-sanabria.com
compliance-walter.top contentsexpress.com
cookie-law-enforcement-ee.xyz counciltally.com
cookie-law-enforcement-hh.xyz counterbot.com
coolyarddecorations.com cpajunkies.com
copblock.org customerpromos-a.akamaihd.net
crest3d.ru danashop.ru
custom-product-labels.com danschawbel.com
dairyindia.in darkages.info
data1.scopich.com dengi-pod-zalog-nedvizhimosti.ru
dating-app.ru diamond-necklace.info
datract.com diesel-parts28.ru
davebestdeals.com digitalfaq.com
db.speedup.gdn dikqlyremy.info
dealighted.com dneprsvet.com.ua
deanmoore.ie dogbreedspicture.net
dear-diary.ru domznaniy.ru
decenttools.com dostavimvdom.ru
defenderxtactical.com draniki.org
delayreferat.ru dstroy.su
demian.kz dtnlyss.com
deutschland123.de dverimegapolis.ru
dildofotzen.net
directrev.com
divatraffic.com
djonwatch.ru
dktr.ru
dolohen.com
domaincrawler.com
domaineaneblanc.com
donna7753191.ru
download-of-the-warez.blogspot.com
doyouknowtheword-flummox.ml
droidlook.net
drpornogratisx.xxx
drugs-no-rx.info
e-biznes.info e-biznes.info
easycommerce.cf earl-brown.info
edge.sharethis.com econom.co
editmedios.com ekn-art.se
eduinfosite.com ekspertmed.com
ekspresihati.info elektronischezigarettekaufen1.beeplog.com
elektrischeziga.livejournal.com elektrozigarette2.webs.com
elektrischezigarette2.devhub.com
elektrozigarette2.wordpressy.pl elektrozigarette2.wordpressy.pl
eloconcream.blogspot.com elvenmachine.com
empathica.com emerson-rus.ru
encodable.com engines-usa.com
escortplius.com escort-russian.com
escortslet.net esnm.ru
estelight.ru essay-writing.work
eurosamodelki.ru
extener.org extener.org
extlinks.com
extremez.net extremez.net
ezigarettekaufen2.blox.pl eyes-on-you.ga
ezigarettenkaufen1.pagina.gr ezigarettekaufen.myblog.de
f0815.de face.hostingx.eu
fanpagerobot.com factorynightclub.com
fashion-stickers.ru favoritemoney.ru
fettefrauen.net feel-planet.com
filmi-onlain.info feeriaclub.ru
fenoyl.batcave.net
fickenbumsen.net
fikasound.tk
filmetricsasia.com
findacheaplawyers.com findacheaplawyers.com
findanysex.com
finejewelryshop.ru
flashlarevista.com flashlarevista.com
fm-upgrade.ru flirt4free.com
foodgid.net floridamobilebillboards.com
footbalive.org flowersbazar.com
formulaantiuban.com forminecrafters.ru
forpackningsutveckling.se
forsex.info
forum20.smailik.org
forzeronly.com forzeronly.com
foxweber.com foto-weinberger.at
free-gluten.ru free-share-buttons.com
freesitetest.com fres-news.com
freevpn.space funnel.co.za
freza-sverlo.ru fusoradio.info
fungamelands.com gamblingnerd.com
gallictures.com gamerextra.com
game-top.su gamingspark.com
gamedayhouse.com garciniaxt.us
geilehausfrauen.net geile-lelly.eu
getmiro.com geileweiber.tk
girls-ufa.ru get-free-traffic-now.com
globetrotting-culture.ru getpopunder.com
glogow.pl glass-msk.ru
go2jump.org glopages.ru
goforexvps.com gomusix.com
goodhousekeeping.com greatzip.com
goodly.pro gsasearchenginerankerdiscount.com
goodwriterssales.com handicapbathtubarea.com
googglet.com hdhc.ru
gopro-online.info healgastro.com
goroda-vsego-mira.ru heartofbeijing.blogspot.com
grupografico-pilar.com.ar hem.passagen.se
guardlink.com highstairs-a.akamaihd.net
hannasolution.ru hit-kino.ru
hao123.com
hatdc.org
hd720kino.ru
herehloadibs.cf
high-speed1.net
hitcpm.com hitcpm.com
hmmm.cz homedecoguide.info
homegardenlova.com hotblognetwork.com
homik.pw
hotsocialz.com hotsocialz.com
hoverboard360.se hoverboard360.at
hscsscotland.com huimin764128.com
ictizanidinehcl4mg.blogspot.com humanelydrew.com
ideibiznesa2015.ru hustoon.over-blog.com
imabase.com iccornacircri.cf
imgarcade.com igru-xbox.net
incest-ru.com ikritikimou.gr
indo-export.ru ilovevitaly.xyz
inet-traffic.com indiakino.net
insurple.com insomniagamingfestival.com
interfucks.net instakink.com
iptool.xyz interferencer.ru
iradiology.ru investsuccess.org
iscblog.info isotoner.com
isoveti.ru ispaniya-costa-blanca.ru
it-max.com.ua istizanidineanopiate.blogspot.com
izhevsk.xrus.org istock-mebel.ru
jaxcube.info itservicesthatworkforyou.com
jennyfire.ru jabimgo.pw
jerseyschinabizwholesale.us jagg.info
kakablog.net jav-way.site
kakadu-interior.com.ua jimmychoosale.online
kazan.xrus.org johnnyhaley.top
kidskunst.info jumptap.com
kambasoft.com
karachev-city.ru
karadene.com
kaz.kz
kerei.ru
keywordspay.com
kinosed.net kinosed.net
konica.kz kinostorm.net
koral.se kiskinhouse.com
kndxbkdx.bloger.index.hr
knowyournextmove.com
kolotiloff.ru
krafte.ru krafte.ru
kredit-pod-zalog-krasnodar.ru kraljeva-sutjeska.com
ks1234.com ku6.com
lacapilla.info kursy-ege.ru
ladsblue.com
lafriore.ru
lanadelreyfans.us
landscaping.center landscaping.center
laparfumotec.com lankarns.com
leftofcentrist.com laulini.soclog.se
letmacworkfaster.world le-clos-des-alouettes.com
lider-zhaluzi.kiev.ua leadn.pl
ledis.top
lernur.net
light.ifmo.ru
limads.men
linkdebrideur.xyz
linkpulse.com
linkrr.com
lion.snapmap.us lion.snapmap.us
liquimondo.com liran-locks.com
livingroomdecor.info livingcanarias.com
locatellicorretor.com.br locksmith.jp
lookingglassemb.com lombia.com
lordzfilmz.me lostaruban.com
love-baby.cz m.b00kmarks.com
luxup.ru macotool.com
m292.info magnetic-bracelets.ru
m5home.ru mailemedicinals.com
macresource.co.uk marcowebonyodziezowe.pl
magento-crew.net marmitaco.cat
magicdiet.gq masserect.com
makemoneyonline.com mastershef.club
mamasuper.prom.ua max-p.men
manualterap.roleforum.ru maxthon.com
markjaybeefractal.com mebelcomplekt.ru
marmotstore.online medanestesia.ru
mcar.in.ua megaindex.ru
mearns-tractors.co.uk
mebel-alait.ru
medkletki.ru
meds-online24.com
megahdporno.net
megavolt.net.ua megavolt.net.ua
member-quiz.com mensandals.xyz
merryhouse.co.uk message-warning.net
methodsmarketing.com mezaruk.info
midst.eu midst.eu
milblueprint.com
millionare.com
mini.7zap.com
misswell.net
mnogabukaff.net mnogabukaff.net
mobot.site moesen-ficken.com
mole.pluto.ro moesonce.com
moscow-region.ru moscow.online-podarki.com
mosrif.ru mrbitsandbytes.com
mossmesi.com
mrpornogratis.xxx
msfsaar.de
msk.afora.ru
mturkcontent.com mturkcontent.com
mustat.com muschisexbilder.com
mustwineblog.com musezone.su
mvpicton.co.uk musicspire.online
my-big-family.com my-aladin.com
myanyone.net myftpupload.com
mybackgroundlandscape.blogspot.com nac-bearings.ru
mydirtystuff.com natali-forex.com
mymercy.info nfljerseys.online
myonigroup.com niroo.info
na15.ru nosecret.com.ua
nacap.ru nrjmobile.fr
nalogovyykodeks.ru o-dachnik.ru
naturalbreakthroughsresearch.com oakridgemo.com
naturalshair.site obiavo.net
new-apps.ru odoratus.net
nextlnk12.com oklogistic.ru
nhl17coinsforps3.gratisblog.biz onemactrckr.com
nicefloor.co.uk onlinebay.ru
nippon-bearings.ru openfrost.com
no-rx.info openmediasoft.com
nodup.ru orion-v.com
noscrapleftbehind.co osb.se11.ru
nsatc.net ostroike.org
o.light.d0t.ru ovirus.ru
obiavo.kz paceform.com
okroshki.ru pacificair.com
oneminutesite.it paclitor.com
online-sbank.ru paparazzistudios.com.au
online-templatestore.com papasdelivery.ru
online-x.ru park.above.com
onlywoman.org partner-print.men
orel-reshka.net pdamods.ru
otbelivanie-zubov.com pearlisland.ru
outrageousdeal-a.akamaihd.net perl.dp.ua
page2rss.com
palvira.com.ua
panchro.xyz
parajumpersstore.online
parlament.biz
partner-pop.men
pc-services.ru
pensplan.com
perso.wanadoo.es
petrovka-online.com
pflexads.com
photokitchendesign.com photokitchendesign.com
phuketscreen.com physfunc.ru
pic-re.blogspot.com picscout.com
pixelrz.com pills24h.com
pk-services.ru plastweb.ru
placid-rounded-coast.glitch.me platezhka.net
ploenjitmedia.azurewebsites.net play.leadzupc.com
pobeiranie.pl playtap.us
porn555.com podshipniki-ntn.ru
porngalleries.top popmarker.com
pornhub-forum.ga popup-hgd.xyz
pornmania.pl porn-w.org
porno-video-chati.ru pornogad.com
pornofeuer.com pornosemki.info
pornolook.net pornotubexxx.name
portadd.men pornotubs.com
powitania.pl pozdravleniya-c.ru
pp-budpostach.com.ua predmety.in.ua
pravoholding.ru princeadvantagesales.com
privatov-zapisi.ru pro-tec.kz
propranolol40mg.blogspot.com profitfx.online
psoriasis-file.trade push-ad.com
pukaporn.com pussyfleet.com
purchasepillsnorx.com quickloanbank.com
puzo2arbuza.ru quit-smoking.ga
q-moto.ru rapidokbrain.com
razvratnoe.org
rcpmda.ikan1080.xyz
rczhan.com rczhan.com
recipedays.ru realitykings.com
redirlock.com reckonstat.info
registratciya-v-moskve.ru reklamuss.ru
remontgruzovik.ru remont-mobile-phones.ru
rep-am.com
rethinkwasteni.info rethinkwasteni.info
rimedia.org ric.info
rock-cafe.info rockma.se
roma-kukareku.livejournal.com romhacking.ru
rubanners.com rootandroid.org
rukino.org rossanasaavedra.net
rus-pornuha.com rus-teh.narod.ru
rusenvironmental.net rusprostitute.com
russkie-gorki.ru rustic-quiver.win
rvtv.ru sack.net
s1z.ru sape.top
saecsa.co sayyoethe.blogspot.co.za
safe-app.net scanhub.ru
sakhboard.ru scanner-john.top
sampleletters.net scansafe.net
santasgift.ml scenarii-1-sentyabrya.uroki.org.ua
sarafangel.ru schoolfiles.net
sbdl.no securityallianceservices.com
scalerite.co.za seeresultshub-a.akamaihd.net
scanner-andrew.top
search.pch.com
sei80.com sei80.com
sell-fb-group-here.com selectads.men
semp.net seo-platform.com
seo-2-0.com seo-smm.kz
seorankinglinks.us seokicks.de
servicecenter.co.ua service.adtech.us
sex-tracker.com shareyards.com
sexpornotales.com shemalegalls.blogporn.in
sexsaoy.com shijian.ac.cn
shariki-zuma-lines.ru shitting.pro
shiksabd.com shlyahten.ru
shtora66.ru shmetall.com.ua
si-unique.com shop.acim.org
silktide.com shoppingmiracles.co.uk
sisiynas.ru shymkent.xkaz.org
site-speed-checker.site sibdevice.ru
sitechecker.pro silvercash.com
skuteczna-dieta.co.pl simply.net
sledstvie-veli.net sinel.info
slujbauborki.ru siteonomy.com
smichovbike.cz sklad-24.ru
socas.pluto.ro skutecznetabletkinaporostwlosow.pl
social-s-iii.xyz skylta.com
socialmonkee.com skypasss.com
sockshares.tv slavia.info
sovetogorod.ru slim.sellany.ru
sloopyjoes.com
smart-scripts.com
sneakyboy.com
social-s-ggg.xyz
socialbuttons.xyz
soft1.ru
songplanet.ru
soserfis.com
sotkal.lark.ru
spb.ru spb.ru
sportbetfair.com spbchampionat.ru
starwars.wikia.com speedup-my.site
sterva.cc sr-rekneskap.no
superiends.org srecorder.com
supermesta.ru sribno.net
superoboi.com.ua sstroy44.ru
statoutlook.info
staynplay.net
sticken.co
stromerrealty.com
stroymonolit.su
stuffpride.com
sucsesofinspiration.com
svensk-poesi.com
symphonyintegratedhealthcare.com
taboola.com
tastyfoodideas.com
tattoo33.ru
td-l-market.ru
tds-advert005.info
techart24.com
tecspb.ru tecspb.ru
terrafootwear.us thecoolimages.net
themestotal.com thecounter.com
theporndude.com thetoiletpaper.com
thewomenlife.com tivolibasket.it
thfox.com tizanidineoral.blogspot.com
timecrimea.ru tmtrck.com
tizanidineforopiatewithdrawal.blogspot.com topads.men
tonivedu.it topcasinoratings.ru
topcar-krasnodar.ru totu.info
topquality.cf traffpartners.com
toys.erolove.in trichizobswiv.agddns.net
trafficcentr.xyz tuckpointingmasonrysystems.com
trafficnetzwerk.de tula.mdverey.ru
traxdom.ru
tri-slona.org
truck-addzilla.life
tsyndicate.com
tvory.predmety.in.ua tvory.predmety.in.ua
tvteleport.ru twelvevisionspartyofcolorado.com
twincitiescarservice.com twiclub.in
twodollarshows.com
tzritel.tk
ua.tc ua.tc
uchebavchehii.ru
ucoz.ru ucoz.ru
uggbootsoutletsale.us
uhdtv.website
ultimatesetnewfreeallsoftupgradesystems.pw ultimatesetnewfreeallsoftupgradesystems.pw
utrolive.ru upstore.me
varikoz24.com ural-buldozer.ru
url-img.link
vegan-foods.us
vekzdorov.ru
vereo.eu
vertaform.com
vezuviy.su vezuviy.su
vgoloveboli.net vgoloveboli.net
via-energy-order.com
viagraneggrx.com viagraneggrx.com
viddyoze.com video-hollywood.ru
videochat.life vikistars.com
vipms.ru vip.51.la
vkak.ru vipsiterip.org
vkonche.com vkonche.com
vriel.batcave.net voloslove.ru
vut.com.ru vremya.eu
vzlomtw.com warning.or.kr
w7s.ru watchinf.com
webshoppermac.com websiteaccountant.de
website-analyzer.info weekes.biz.tc
website-audit.com.ua whatsupinfoley.com
websites-reviews.com wheelchairliftsarea.com
whereverdesperate.gq
whiteelephantwellington.com
whiteproduct.com
wholesalecheapjerseysfree.com
wholesalejerseyscheapjerseys.us.com wholesalejerseyscheapjerseys.us.com
wordpress-crew.net winner7777.net
worldhistory.biz wladimirpayen.com
woweb.com.ua worldinternetauthority.com
wpsecurity.website wtsindia.in
wufak.com wyniki-lista.pl
xblognetwork.com x-lime.com
xchat26.myfreecams.com xn-----8kcatubaocd1bneepefojs1h2e.xn--p1ai
xclicks.net xn----8sbarihbihxpxqgaf0g1e.xn--80adxhks
xep.info xn----btbdvdh4aafrfciljm6k.xn--p1ai
xerox-douglas.cf xn--80adaggc5bdhlfamsfdij4p7b.xn--p1ai
xev.ru xn--h1ahbi.com.ua
xjrul.com xn--hxazdsfy.blogspot.com
xn----7sbbagbq7bd5aheftfllo4m.xn--p1ai xvideospornoru.com
xn----ctbbcjd3dbsehgi.xn--p1ai xz618.com
xn----ctbigni3aj4h.xn--p1ai yellowfootprints.com
xn--24-glceagatoq7c2a6ioc.xn--p1ai yhit.press
xn--80aebbcbcdemfkhba4byaehoejh8dza3v.xn--p1ai yogamatsexpert.com
xn--e1aggki3c.xn--80adxhks youbloodyripper.com
xn--l1aengat.xn--p1ai youdao.com
xxart.ru
xxx-cam.webcam
xxxdatinglocal.us
xxxtubesafari.com
yachts-cruise.info
youjizz.com youjizz.com
yourtemplatefinder.com yourtemplatefinder.com
zaobao.com.sg yuweng.info
zigarettenonl.canalblog.com zeroredirect10.com
zigzog.ru zeroredirect5.com
zlatnajesen.com zoodrawings.com
zoogdisany.com zooggames.com
zoosexart.com zoolubimets.ru
zvetki.ru
zx6.ru

480
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
0lovespells0.blogspot.com 007angels.com
2daytrendingnews.com 1webmaster.ml
66cpwgln.space 273-fz.ru
a14download.com 2nt.ru
aad1f4acb0a373420d9b0c4202d38d94fa.com 2pxg8bcf.top
adviceforum.info 4senses.co
advocatemsk.ru abcdeg.xyz
agroeconom.kz adf.ly
aimayangzhi.com ads-cool.pro
amazon-seo-service.com adssafeprotected.com
anime.dougasouko.com advokateg.xyz
apessay.com agapovdg.ru
apidigidocketnet-a.akamaihd.net alert-jdh.xyz
apparel-offer.com alive-ua.com
architecturebest.com all4wap.ru
as5000.com allforminecraft.ru
azartniy-bonus.com appfixing.space
azazaporn.com appleid-verification.com
backiee.com applyneedy.xyz
beauby.ru autodo.info
besthoro.ru autoloans.com
bezcmexa.ru aviav.ru.com
bigcareer.info bestadbid.com
bio-japan.net bestbuy.ca
bpro1.top bestcalovebracelet.cn
bestofpicture.com
bgviagrachrx.com
bioca.org
bjanshee.ru
bkns.vn
bluesman.nu
bongacams.com
breget74.ru
btnativenav.com
buytizanidineonline.blogspot.com
c1.onedmp.com c1.onedmp.com
chocoslim.pro cardsharp1.ru
cialisndbrx.com carstrends2015.com
cindymatches.com cartierlovebraceletreplica.xyz
clevernt.com cartierreplica.top
coderstate.com cb.iphantom.com
codysbbq.com cfcl.co.uk
community.allhiphop.com cgi2.nintendo.co.jp
compliance-john.top cheapwebsitehoster.com
cvety24.by checkhit.com
cherubinimobili.it
chinacheapelitejerseys.com
cienum.fr
cleanmypc.com
clicksotrk.com
clips.ua.ac.be
co.lumb.co
codq.info
comeondog.info
cometorussia.net
compliance-don.xyz
connectionstrenth.com
constantaservice.net
cosmeticswomens-womensports.rhcloud.com
countercrazy.com
covid-schutzmasken.de
crackzplanet.com
credit-online.ws
custom-electric-guitar.com
d2jsp.org d2jsp.org
day-news.info dame-ns.kz
diesel-parts28.ru dev33.dioniqlabb.se
digest-project.ru divku.ru
digilander.libero.it docs4all.com
diusyawiga.tk doktoronline.no
dnsrsearch.com domainanalyzing.xyz
doeco.ru domination.ml
donna7753191.ru doyouknowtheword-flummox.ml
drupa.com dscaas.website
dstroy.su duawitchrarato.tk
dyt.net efkt.jp
e-avon.ru election.interferencer.ru
ecommerce-seo.com elkacentr.ru
elektir.ru emerson-rus.ru
eloconcream.blogspot.com erotik0049.com
enbersoft.com evehemming.blogspot.com.au
eropornosex.ru ewebarticle.info
eshop4u.jp executehosting.com
ezigarettekaufen1.hpage.com extrabot.com
ezigarettenkaufen2.dreamwidth.org eyes-on-you.ga
falcon-images.blogspot.com facialporntube.com
factorynightclub.com
fellowshipoftheminds.com
femmesdenudees.com
ferretsoft.com
fickluder69.com fickluder69.com
fifa-coins.online findacheaplawyers.com
figuringmoneyout.com finstroy.net
filmbokep69.com firstdrugmall.ru
filmi-v.online
finuse.com
fishtauto.ru fishtauto.ru
flagcounter.me forums.d2jsp.org
fm-upgrade.ru forums.toucharcade.com
forumprofi.de fototravel.eu
foto-sisek.porngalleries.top free411games.com
fotzen-ficken.com frequiry.com
freelifetimefuckbook.com friendflnder.com
fxtips.ru fullgirl.ru
fym.com.ru fungirlsgames.net
galaxyflowers.ru gael-s.ru
gameplexcity.com galleryawesome.com
generic-pills-online.com gay.adultgalls.com
germes-trans.com ghernnqr.skyrock.com
getaclueamerica.com glass-msk.ru
getlaid-xxxhookupdirect.com go2mike.ru
goblacked.com goldandcard.ru
gomusix.com good-mummy.ru
growmyfunds.ca gpirate.com
gsasearchengineranker.site greatfind-a.akamaihd.net
gsasearchengineranker.space guildebzh.info
gunsvicceadadebt.tk hanwei.us
hdxnxxtube.mobi hdxnxxtube.mobi
headpress.ru healing-dysplasia.ru
heatpower.ru herehloadibs.cf
hexpilot.com hostingclub.lk
hostsshop.ru hotloans.ru
hotblognetwork.com hystersister.com
hothot.ru imagui.eu
hotkeys.com inboxdollars.com
house.sieraddns.com infobabki.ru
iccornacircri.cf infogame.name
iframe-toloka.com intermesh.net
imediadesk.com invivo.hu
incest-ru.com iqbazar.ru
infospot.pt j-times.ru
infoupdate.org j33x.com
ix20.ru jagg.info
janavibekken.no javcoast.com
justdating.online jo24news.com
justprofit.xyz kaidalibor.de
kaac.ru keywords-monitoring-your-success.com
kakablog.net kinogonew.ru
karting196.ru konteiner24.com
kinozapas.org kosova.de
kiprinform.com krasula.pp.ua
kndxbkdx.bloger.index.hr ks1234.com
knowyournextmove.com laulini.soclog.se
konoplisemena.com letmacwork.world
krasivoe-hd.net livefixer.com
ktotut.net loadingpages.me
labafydjxa.su
lacloop.info
languagecode.com
laserpen.club
lasvegaslockandsafe.com
law-check-eight.xyz
layola.biz.tc
limtu.ifmo.ru
link.ac
lion.misba.us
lionking-1994.blogspot.com
lkbennettstore.online
lockerz.com lockerz.com
luckyday.world long-beach-air-conditioning.com
magento-crew.net louboutinreplica.pw
make-money-online.com lovi-moment.com.ua
manve.info masterseek.com
mebel-ekb.com matrixalchemy.com
mebel-vstroika.ru medicines-choice.com
medicovi.com medosmotr-ufa.ru
members.ghanaweb.com mexicotravelnet.com
merryhouse.co.uk multgo.ru
mindbox.co.za musicpro.monster
mirobuvi.com.ua myhealthcare.com
mmog-play.ru mylida.org
monarhs.info mzdish.site
monclercheap.xyz nelc.edu.eg
money-for-placing-articles.com netallergy.ru
moneyzzz.ru netoil.no
mp3downloadhq.com nhl09.ru
mrinsidesales.com nikhilbahl.com
mrlmedia.net nl.netlog.com
muschisexbilder.com nrjmobile.fr
my-bc.ru ntic.fr
my-big-family.com odywpjtw.bloger.index.hr
my-floor.in.ua oivcvx.website
mycaf.it olgacvetmet.com
mydirtystuff.com ontargetseo.us
myplaycity.com
myshopmatemac.com
offtime.ru
online-hit.info
onlyforemont.ru
orenburg-gsm.ru
outshop.ru
ownshop.cf ownshop.cf
oxotl.com pandarastore.top
partner-high.men partnerads.men
patterntrader-en.com pastaleads.com
photo.houseofgaga.ru pliks.pl
picanalyzer.data-ox.com pochemychka.net
picturesboss.com polybuild.ru
pillscheap24h.com pornosmola.info
pixell.club potoideas.us
pony-business.com predmety.in.ua
popup-fdm.xyz preventheadacheguide.info
pornhubforum.tk
pornoelita.info
pornoplen.com
portside.cc
printingpeach.com printingpeach.com
priscilarodrigues.com.br prointer.net.ua
professionalwritingservices15.blogspot.ru promodj.com
psa48.ru putevka24.ru
puserving.com pyramidlitho.webs.com
pxhdwsm.com questionmarque.ch
rapidsites.pro razorweb-a.akamaihd.net
razyboard.com recinziireale.com
real-time-analytics.com ric.info
realnye-otzyvy.info ru-dety.ru
rent2spb.ru russkie-gorki.ru
research.ifmo.ru samo-soznanie.ru
retreatia.com saneitconsulting.com
rialp.getenjoyment.net scanner-jack.top
roofers.org.uk scat.porn
rus-pornuha.com semxiu.com
screentoolkit.com sendearnings.com
sdsjweb.com seorankinglinks.xyz
secretscook.ru sexcamamateurchat.com
seozoom.it sexkrasivo.net
sex-tracker.de sexy.babes.frontend-stack.top
shariki-zuma-lines.ru share-buttons-for-free.com
ship-marvel.co.ua sildenafilcitratemed.com
shtora66.ru skladvaz.ru
signx.info skutecznetabletkinaporostwlosow.pl
silverage.ru smichovbike.cz
snow.nvr163.com social-buttons.com
socialtrade.biz spamnuker.com
space4updating.win staynplay.net
statustroll.com stromerrealty.com
store-rx.com stroyplus.ru
strongholdsb.ru supernew.org
stroy-portal22.ru
stuff-about-money.com
tamada69.com
taqplayer.info taqplayer.info
tastyfoodideas.com techart24.com
tizanidinezanaflex.blogspot.com terrafootwear.us
toon-families.com thecoolimages.net
tootoo.to thefarmergame.com
trackmedia101.com thegioixekhach.com
tilido.com
timecrimea.ru
titan-cloud.life
trackzapper.com trackzapper.com
tracxn.com trafficmp.com
tsstcorpcddvdwshbbdriverfb.aircus.com trkur.com
tvoystartup.ru truck-land.life
tvteleport.ru tuberkulezanet.ru
typer.one tvory.predmety.in.ua
uhod-za-sobakoj.ru ultimatesetnewfreeallsoftupgradesystems.pw
ural-buldozer.ru umityangin.net
url2image.com unrealcommander.com
vacuumcleanerguru.com ustion.ru
vapeface.club videochat.bz
varikozdok.ru videochaty.ru
videogamesecrets.com videosbox.ru
videooko.weebly.com warning.or.kr
villacoloniale.com webtherapy.ru
vip.51.la wildnatureimages.com
visa-china.ru wma-x.com
volgograd.xrus.org wordkeyhelper.com
vostoktrade.info wyniki-lista.pl
websiteexplorer.info xn-----6kcaabbafhu7cskl7akvongwpo7hvjj.xn--p1ai
wedding0venues.tk xn----ctbbcjd3dbsehgi.xn--p1ai
williamrobsonproperty.com xxxdatinglocal.us
womensplay.net y8games-free.com
works.if.ua yougame.biz
xgftnlrt.bloger.index.hr yurgorod.ru
xn------7cdbapdecfd4ak1bn0amjffj7afu3y.xn--p1ai z-master.ru
xn----7sbaphztdjeboffeiof6c.xn--p1ai zawyna.ua
xn----dtbndd4ae7eub.top zocaparj.kz
xn--80aebbcbcdemfkhba4byaehoejh8dza3v.xn--p1ai
xn--80aikhbrhr.net
xn--e1afanlbnfckd7c3d.xn--p1ai
yebocasino.com
youresponsive.com
zaloro.com
zscaler.net
zynax.ua

12
dev-tools/test_units/ratelimittest-beta.txt
View file

@ -1,21 +1,11 @@
<!DOCTYPE html>
<html> <html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><html>
<head><title>503 Service Temporarily Unavailable</title></head> <head><title>503 Service Temporarily Unavailable</title></head>
<body> <body>
<center><h1>503 Service Temporarily Unavailable</h1></center> <center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center> <hr><center>nginx/1.18.0 (Ubuntu)</center>
</body> </body>
</html> </html>
<html> ml><html>
<head><title>503 Service Temporarily Unavailable</title></head> <head><title>503 Service Temporarily Unavailable</title></head>
<body> <body>
<center><h1>503 Service Temporarily Unavailable</h1></center> <center><h1>503 Service Temporarily Unavailable</h1></center>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.135 0; 10.1.0.174 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.07.3209 ### Version: V4.2022.07.3210
### Updated: Tue Jul 5 10:03:30 UTC 2022 ### Updated: Tue Jul 5 22:01:32 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.07.3209 ### Version: V4.2022.07.3210
### Updated: Tue Jul 5 10:03:30 UTC 2022 ### Updated: Tue Jul 5 22:01:32 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##