vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-02 18:50:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.01.2839 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-01-08 22:09:21 +00:00
parent 44bcb9eb95
commit 0fe2d38ee5
46 changed files with 74952 additions and 74952 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.01.2838 #### Version: V4.2022.01.2839
#### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.01.2838 #### Version: V4.2022.01.2839
#### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -13,7 +13,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.01.2838 #### Version: V4.2022.01.2839
#### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

9232
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.01.2838 #### Version: V4.2022.01.2839
#### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

9236
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9236
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

9236
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9236
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 1; 10.1.0.109 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 0; 10.1.0.109 0;
127.0.0.1 0; 127.0.0.1 0;

9236
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 1; 10.1.0.109 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 0; 10.1.0.109 0;
127.0.0.1 0; 127.0.0.1 0;

9236
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 1; 10.1.0.109 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 0; 10.1.0.109 0;
127.0.0.1 0; 127.0.0.1 0;

9236
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2838 2839

9236
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

9236
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.83.109.36 0; 168.62.58.134 0;

9236
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 1; 10.1.0.109 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 0; 10.1.0.109 0;
127.0.0.1 0; 127.0.0.1 0;

9236
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 1; 10.1.0.109 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 0; 10.1.0.109 0;
127.0.0.1 0; 127.0.0.1 0;

9236
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 1; 10.1.0.109 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 0; 10.1.0.109 0;
127.0.0.1 0; 127.0.0.1 0;

9236
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.83.109.36 0; 168.62.58.134 0;

9236
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.83.109.36 0; 168.62.58.134 0;

9236
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 1; 10.1.0.109 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Crawling at Home Project Copier
HTTrack DomainSigmaCrawler
HybridBot Evil
InfoTekies LieBaoFast
LinqiaRSSBot RealDownload
Magnet SiteCheckerBotCrawler
Rogerbot SiteSucker
Suzuran SmartDownload
TheNomad Website Quester
Tracemyfile spyfu

312
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com
360Spider
404checker 404checker
ADmantX ADmantX
AIBOT AIBOT
ALittle Client Abonti
ASPSeek Aboundex
Aboundexbot Aboundexbot
Acunetix
AfD-Verbotsverfahren AfD-Verbotsverfahren
AhrefsBot
Aipbot
Alexibot Alexibot
Alligator AllSubmitter
Anarchy Aspiegel
Apexoo
Asterias
AwarioSmartBot
BBBike BBBike
BDCbot BDCbot
BDFetch BLEXBot
BackWeb BackDoorBot
BacklinkCrawler BackStreet
BatchFTP Backlink-Ceck
Bigfoot Badass
Bitacle Barkrowler
Battleztar Bazinga
Black Hole Black Hole
Blackboard
Blow Blow
Boardreader BlowFish
Bolt
Brandprotect
Brandwatch Brandwatch
Buddy Buck
Bullseye BuzzSumo
CODE87
CSHttp CSHttp
CazoodleBot CheeseBot
CensysInspect Cocolyzebot
CheTeam
Cliqzbot
Copier Copier
Copyscape
Craftbot
CrazyWebCrawler CrazyWebCrawler
CyotekWebCopy DBLBot
DIIbot
DTS Agent DTS Agent
DatabaseDriverMysqli Demon
Deusu
Devil Devil
Digincore
DigitalPebble DigitalPebble
Disco
Discoverybot Discoverybot
DomCopBot DittoSpyder
DomainAppender DomainCrawler
DomainStatsBot DomainSigmaCrawler
Download Wonder Domains Project
Dragonfly
Drip Drip
ECCP/1.0 ECCP/1.0
EMail Siphon
EasyDL EasyDL
Ebingbong Ebingbong
Ecxi EirGrabber
EroCrawler
Evil
Express WebPictures
ExtLinksBot
EyeNetIE EyeNetIE
FHscan FDM
Fuzz Freeuploader
FyberSpider FyberSpider
Fyrebot
GT::WWW
Genieo Genieo
Go!Zilla Go-Ahead-Got-It
GoZilla
Gotit Gotit
Grafula Grafula
GridBot GrapeFX
HMView
HTMLparser HTMLparser
HTTrack HTTP::Lite
Haansoft HaosouSpider
Harvest Heritrix
Havij HybridBot
Hloader IDBTE4M
Id-search IRLbot
IlseBot
Indy Library Indy Library
InfoTekies InfoNaviRobot
Iria
IstellaBot
JOC Web Spider JOC Web Spider
JetCar Joomla
Keyword Density Jorgee
Kozmosbot Jyxobot
LNSpiderguy
Lanshanbot Lanshanbot
LexiBot LexiBot
Lftp LibWeb
Libwhisker
Lightspeedsystems
Likse
LinkScan
LinkWalker LinkWalker
LinkextractorPro
LinksManager
LinqiaMetadataDownloaderBot
LinqiaRSSBot LinqiaRSSBot
Lipperhey
Lipperhey Spider
Lmspider
MFC_Tear_Sample MFC_Tear_Sample
MIDown tool
MJ12bot
MQQBrowser MQQBrowser
MSIECrawler
MTRobot MTRobot
Mag-Net Mag-Net
Mail.RU_Bot Majestic-SEO
Majestic12 Majestic12
MauiBot MarkWatch
MeanPath Bot Mb2345Browser
Meanpathbot
MegaIndex.ru MegaIndex.ru
Metauri Microsoft Data Access
MicroMessenger Microsoft URL Control
Minefield Minefield
Mojeek Mojolicious
Morfeus Fucking Scanner Mr.4x3
Msrabot
Navroad
NearSite NearSite
NetLyzer Nessus
Net Vampire
NetAnts
Netcraft
Netvibes
NextGenSearchBot NextGenSearchBot
Niki-bot Niki-bot
NimbleCrawler Nikto
Nimbostratus
Ninja Ninja
Not
Nuclei Nuclei
Nutch Nutch
Octopus Octopus
Offline Explorer
Offline Navigator Offline Navigator
OnCrawl
OpenLinkProfiler OpenLinkProfiler
OpenVAS OpenVAS
Openfind Openfind
Openvas PHPCrawl
OrangeBot Page Analyzer
OutclicksBot PageGrabber
PECL::HTTP PageScorer
POE-Component-Client-HTTP Papa Foto
PageAnalyzer Pavuk
PageThing.com PeoplePal
Panscient
Pi-Monster Pi-Monster
Picscout Picscout
Picsearch Picsearch
Pimonster Pixray
PleaseCrawl ProPowerBot
Pu_iN Probethenet
Pump PyCurl
PxBroker QueryN Metasearch
RSSingBot RankActive
RankActiveLinkBot
RankFlex
RankingBot RankingBot
RankingBot2 Recorder
Rankivabot
RankurBot
Re-re
ReGet
RealDownload
RedesScrapy
RepoMonkey
Ripper
RocketCrawler
Rogerbot
SEOkicks SEOkicks
SEOlyticsCrawler SEOkicks-Robot
SEOstats SEOstats
ScanAlert SMTBot
Scanbot SalesIntelligent
Scrapy
Screaming Screaming
ScreenerBot ScreenerBot
Semrush Siphon
SemrushBot
SentiBot
SeoSiteCheckup
Seomoz
Site Sucker
SiteCheckerBotCrawler SiteCheckerBotCrawler
SiteLockSpider SiteLockSpider
SiteSnagger SiteSnagger
SiteSucker Sitevigil
Sitebeam
Siteimprove
SlySearch
SmartDownload SmartDownload
SocialRankIOBot Snake
Sqlmap Snapbot
Sqlworm Sociscraper
Steeler Sosospider
SuperBot Sottopop
Spbot
Spinn3r
Stripper
Surfbot Surfbot
SurveyBot Suzuran
Teleport Teleport
Telesphoreo
Telesphorep
The Intraformant The Intraformant
TheNomad TheNomad
Toata
Toweyabot Toweyabot
Tracemyfile Tracemyfile
Trendiction Trendiction
Trendictionbot
True_Robot True_Robot
Turingos TurnitinBot
TwengaBot TwengaBot
Twice Twice
Typhoeus URLy Warning
URLy.Warning URLy.Warning
UnisterBot VB Project
Upflow VelenPublicWebCrawler
Vagabondo VeriCiteCrawler
WBSearchBot Voltron
WEBDAV WEBDAV
WPScan WWW-Collector-E
WWWOFFLE WWW::Mechanize
Wallpapers/3.0 Wallpapers
Web Auto
Web Collage
Web Fetch
Web Fuck
Web Sauger Web Sauger
Web Sucker WebBandit
WebEnhancer WebCollage
WebFuck WebImageCollector
WebSauger WebSauger
WebStripper WebWhacker
WebZIP WebZIP
WebmasterWorldForumBot Webshag
Website Quester WebsiteExtractor
WebsiteQuester
Webster Webster
Whack Whack
Whacker Whatweb
Who.is Bot WiseGuys Robot
Widow
WinHTTrack
Woobot Woobot
Wotbox Wotbox
Xaldon WebSpider
Xaldon_WebSpider Xaldon_WebSpider
Xenu
Zade Zade
adscanner Zauba
backlink-check Zeus
check1.exe ZmEu
ZoominfoBot
autoemailspider
clark-crawler clark-crawler
com.plumanalytics crawl.sogou.com
crawler.feedback crawler.feedback
crawler4j dataforseo.com
demandbase-bot
eCatch
evc-batch
gopher
heritrix heritrix
isitwp.com isitwp.com
meanpathbot lwp-request
netEstate NE Crawler magpie-crawler
pcBrowser muhstik-scan
page scorer
plumanalytics plumanalytics
satoristudio.net probe-image-size
scan.lol
seobility
seocompany.store seocompany.store
seoscanners
seostar
sitechecker.pro sitechecker.pro
siteripz sysscan
sogouspider trendiction.com
sp_auditbot
tAkeOut
ubermetrics-technologies.com ubermetrics-technologies.com
voyagerx.com voyagerx.com
webgains-bot webprosbot
zauba.io
zgrab

320
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
404enemy 404checker
AIBOT ADmantX
ALittle Client ASPSeek
Acunetix Abonti
AhrefsBot Aboundex
Aboundexbot
Aipbot Aipbot
AllSubmitter Alexibot
Alligator Anarchy
Anarchie Anarchy99
Ankit
Apexoo
Aspiegel Aspiegel
AwarioSmartBot Attach
BDCbot BDCbot
BackDoorBot BLEXBot
BatchFTP Backlink-Ceck
BlackWidow Barkrowler
BlowFish Bigfoot
Boardreader Black Hole
Blackboard
Bolt Bolt
BotALot Brandprotect
Brandwatch
Buddy Buddy
BuiltBotTough BuiltWith
BuzzSumo
CCBot CCBot
CSHttp CODE87
CazoodleBot
CensysInspect CensysInspect
CheTeam
CherryPicker
ChinaClaw ChinaClaw
Chlooe
Citoid Citoid
Cocolyzebot Claritybot
Cogentbot
Copier
Copyscape
Cosmos Cosmos
Craftbot Craftbot
Crawling at Home Project
CrazyWebCrawler
CrunchBot CrunchBot
Curious
CyotekWebCopy
DBLBot DBLBot
DTS Agent DTS Agent
DatabaseDriverMysqli Demon
Deusu Deusu
Digincore Devil
DigitalPebble DigitalPebble
Dirbuster Dirbuster
Disco Disco
Discobot
Dispatch Dispatch
DittoSpyder DittoSpyder
DnyzBot
DomCopBot DomCopBot
Domains Project DomainAppender
Dotbot Dragonfly
Drip Drip
EMail Siphon EMail Siphon
EMail Wolf EMail Wolf
EasyDL Evil
Ebingbong ExtLinksBot
EroCrawler ExtractorPro
Express WebPictures EyeNetIE
Extractor Ezooms
Extreme Picture Finder
FDM
FHscan FHscan
FemtosearchBot Firefox/7.0
FlashGet Fyrebot
Freeuploader
FyberSpider
Genieo Genieo
GetRight
Go-Ahead-Got-It Go-Ahead-Got-It
GoZilla GoZilla
Gotit Gotit
GrabNet Grabber
Grafula
GrapeFX GrapeFX
GridBot GrapeshotCrawler
HEADMasterSEO HEADMasterSEO
HTMLparser
HTTP::Lite
HTTrack HTTrack
Haansoft Harvest
HaosouSpider
Havij Havij
HybridBot Heritrix
IDBot Hloader
Humanlinks
IRLbot IRLbot
Iblog IlseBot
Id-search Image Sucker
Image Fetch IndeedBot
InfoNaviRobot InfoNaviRobot
Intelliseek
InterGET
InternetSeer
IstellaBot IstellaBot
JennyBot JOC Web Spider
JetCar Jbrofuzz
Jetty Joomla
JustView Kenjin Spider
Jyxobot
Keyword Density Keyword Density
Leap Kinza
LeechGet LWP::Simple
LexiBot LibWeb
Lftp Libwhisker
LieBaoFast LieBaoFast
LinkWalker Lightspeedsystems
Likse
Linkdexbot Linkdexbot
LinkpadBot
LinksManager
LinqiaMetadataDownloaderBot
LinqiaRSSBot LinqiaRSSBot
LinqiaScrapeBot LinqiaScrapeBot
MJ12bot Lipperhey
MSFrontPage Lmspider
MSIECrawler MQQBrowser
MTRobot Magnet
Mag-Net
Mail.RU_Bot Mail.RU_Bot
Majestic SEO MarkMonitor
Mass Downloader
MauiBot
MegaIndex.ru MegaIndex.ru
MicroMessenger
Microsoft URL Control
Minefield Minefield
Moblie Safari Mister PiX
Mojolicious
Morfeus Fucking Scanner
Mozlila Mozlila
Navroad Mr.4x3
Musobot
Nameprotect
NetSpider
NetZIP
Netcraft Netcraft
Netvibes Nettrack
NextGenSearchBot NextGenSearchBot
Nibbler Niki-bot
NimbleCrawler NimbleCrawler
Nmap Nimbostratus
Not Not
Nuclei Octopus
Nutch Offline Explorer
OpenLinkProfiler
OpenVAS OpenVAS
OrangeSpider Openvas
POE-Component-Client-HTTP OrangeBot
PageGrabber OutclicksBot
OutfoxBot
PECL::HTTP
Page Analyzer
Pandalytics Pandalytics
PeoplePal PeoplePal
Petalbot
Picscout
Picsearch Picsearch
PictureFinder
Pimonster
Pixray Pixray
ProPowerBot PleaseCrawl
Probethenet Pockey
ProWebWalker
Pu_iN
QueryN Metasearch
RankActive RankActive
RealDownload RankFlex
Ripper RankingBot
RocketCrawler Re-re
Rogerbot SBIder
SEOlyticsCrawler SEOlyticsCrawler
SEOstats
SMTBot SMTBot
Scanbot Scanbot
ScoutJet Screaming
Scrapy
Searchestate
SemanticJuice
SemrushBot SemrushBot
SentiBot
SeoSiteCheckup SeoSiteCheckup
SeobilityBot SeobilityBot
Seomoz Seomoz
Siphon Siphon
SiteSnagger Site Sucker
SlySearch SiteCheckerBotCrawler
Sosospider Sitebeam
Snake
Snoopy
SocialRankIOBot
Sottopop Sottopop
Spammen Spammen
Spanner SpankBot
Spbot
SputnikBot
Sqlmap Sqlmap
Sqworm Sqworm
Steeler
SuperBot
SuperHTTP SuperHTTP
SurveyBot
Suzuran
Swiftbot Swiftbot
TeleportPro Szukacz
T0PHackTeam
Teleport
Telesoft Telesoft
Telesphoreo
TightTwatBot
Titan Titan
Trendiction True_Robot
Twice Turingos
Typhoeus Turnitin
URLy Warning TwengaBot
Vagabondo URLy.Warning
V-BOT
VCI
Vacuum
VelenPublicWebCrawler VelenPublicWebCrawler
VeriCiteCrawler
Virusdie
Voil Voil
WASALive-Bot Voltron
WBSearchBot
WEBDAV WEBDAV
WISENutbot
WWW-Collector-E
WWW-Mechanize
WWW::Mechanize WWW::Mechanize
WWWOFFLE WWWOFFLE
WeSEE Wallpapers
Web Auto Wallpapers/3.0
Web Collage Web Enhancer
Web Fetch Web Sucker
Web Fuck WebBandit
WebCollage WebCollage
WebCopier WebLeacher
WebImageCollector
WebPix WebPix
WebReaper
WebStripper WebStripper
WebmasterWorldForumBot WebSucker
Webster WebWhacker
WebZIP
Webshag
Website Quester
WebsiteExtractor
Whack Whack
Whatweb
Who.is Bot Who.is Bot
Widow WinHTTrack
WiseGuys Robot Woobot
Wonderbot
Wotbox
Wprecon
Xaldon WebSpider Xaldon WebSpider
Zauba Xaldon_WebSpider
YoudaoBot
Zade
Zeus
Zitebot
ZoomBot ZoomBot
ZoominfoBot ZumBot
adscanner adscanner
autoemailspider arquivo-web-crawler
cah.io.community arquivo.pt
check1.exe coccocbot-web
clark-crawler
com.plumanalytics com.plumanalytics
crawl.sogou.com crawler.feedback
crawler4j crawler4j
dataforseo.com domainsproject.org
evc-batch lwp-trivial
heritrix muhstik-scan
isitwp.com netEstate NE Crawler
magpie-crawler serpstatbot
page scorer sexsearcher
probe-image-size sogouspider
s1z.ru
satoristudio.net
scalaj-http
seocompany.store
seostar
sitechecker.pro
siteripz
sp_auditbot sp_auditbot
spyfu
sysscan sysscan
trendiction.de voyagerx.com
ubermetrics-technologies.com
webmeup-crawler webmeup-crawler
webpros.com
webprosbot webprosbot
x09Mozilla
zauba.io
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
customergrowthsystems.com cobaltpro.ru
extremepornos.net ihtec2019.org
filkhbr.com macrotek.ru
fivedwld.cf manifestation.betteroffers.review
gemara.com o-o-6-o-o.com
howmuchdoestizanidinecost.blogspot.com ons-add.men
kometars.xyz relayblog.com
monthlywinners.com rubbed.us
scanner-irvin.top surffoundation.nl
video-chat.in turizm.bz

920
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
20pascals.nl 00webcams.com
4istoshop.com 1-99seo.com
6128786.com 123cha.com
6hopping.com 14b.info
7ooo.ru 273-fz.ru
7search.com 28n2gl3wfyb0.ru
adconscious.com 3th.co.in
addbin.men abovetherivernc.com
adminshop.com absolute-s.ru
acc.eu.org
actionnooz.com
adanih.com
adclickthru.net
admanaerofoil.com
adpremium.org
adprotect.net
adsfresh.men adsfresh.men
adssafeprotected.com adult3dgames.com
adultactioncam.com adultactioncam.com
advancedcleaner.com adultfullhd.com
advertisingtag.net advokat-grodno.by
agroeconom.kz alessandraleone.com
ahmedabadwebs.com all4invest.info
alfa-medosmotr.ru allfinweb.com
alfabot.xyz allforminecraft.ru
alice.it alvtank.se
alive-ua.com amazingpic.net
alldownload.pw ample-awards-today.us
allkrim.com animenime.ru
am-se.com antons-transporte.de
anapa-inns.ru apioutoboxnet-a.akamaihd.net
apartmentratings.com artel116.ru
apiadanaknet-a.akamaihd.net artparquet.ru
apibatbrowsecom-a.akamaihd.net ascat.porn
apiluckyleapnet-a.akamaihd.net asia-forum.ru
approvedlocal.co.za astana.xxxkaz.org
apps-analytics.net atmovs.com
arewater.com avon-severozapad.ru
arkkivoltti.net awaybird.ru
asociatia-tipografilor-transilvania.ro babespcs.com
asrv-a.akamoihd.net babyfactory.fr
asrvrep-a.akamaihd.net babyguns.ru
asscenihotosidea.blogspot.com backiee.com
atdedinotuho.tk backuperwebcam.weebly.com
ausergrubhof.info bagsonsale.online
avek.ru
avtolombard-krasnodar.com
avtolombard-krasnodar.ru
babosas.com
bagcionderlab.com
baixar-musicas-gratis.com
balois.worldbreak.com
bambi.ck.ua
banki76.ru
bankiem.pl bankiem.pl
banksrf.ru bannerpower.com
bedcapdealers.com bazaronline24.ru
beget.tech beachfix.co
bfz.biz beetpics.pw
bjanshee.ru belinka.com.ua
blackle.com beremenyashka.com
blavia.00author.com berrymall.ru
bloggerads.net bestofferswalkmydogouteveryday.gq
blogrankers.com bestsexyblog.com
boeuklad.com bestwaystogetpaid.us
bonvillan.com bigcities.org
bookhome.info billiard-classic.com.ua
brandov.ru biznesschat.net
brothers-smaller.ru blackbabesporn.com
blackcurranthumidifiers.site
bloggedporn.com
boleznikogi.com
bonzbuddy.com
bristolhotel.com
browserprotecter.com browserprotecter.com
budpost.com.ua buigas.00it.com
bumble.cheapwebsitehoster.com buntube.net
buscarfoto.com buron.pw
business-made-fun.com businescoop.men
butstrap.space buttons-for-website.com
buyessaynow.biz buyparajumpers.online
buytizanidineonline.blogspot.com buytizanidineonline.blogspot.com
buyviagraa.blogspot.com buzzzg.men
bvlgariring.xyz bytimedance.ru
calc-for-credit.ru cablecar.us
cam-kontakte.org candy-glam-hp.com
candytiner.com canyougethighofftizanidine.blogspot.com
canrole.com cartechnic.ru
cartierloveringreplica.xyz cartierreplica.win
cctva.tv casinorewards.info
cdn.walkme.com cejewelry.xyz
changde.58.com cercacamion.it
chcu.net cheapestjerseysonwholesale.com
cheapcigarettesc.info cheapjerseysbizwholesale.us
cheapsergiorossi.online check-this-out-now.online
chee-by.biz
chelyabinsk.xrus.org
cherrypointplace.ca cherrypointplace.ca
chinacheapelitejerseys.com chinacheapelitejerseys.com
christianlouboutinoutlet.win christianlouboutinoutlet.win
christianlouboutinreplica.win christianlouboutinshoes.xyz
civilwartheater.com cialgenisrx.com
classiquebijoux.ru cialis-samples.com
claytransformations.info cleanpcnow.com
cleanallspyware.com clicksor.com
clippingphotoindia.com clippingphotoindia.com
cobaltpro.ru co.lumb.co
coindirect.io coinsspb.com
colehaanoutlet.store compliance-andrew.xyz
communisave.co.za compliance-irvin.xyz
compliance-mark.top contentsexpress.com
conciergegroup.org
cookie-law-enforcement-ff.xyz
coolgamechannel.com
costablue.xyz
counciltally.com counciltally.com
country-chic.ru counter.bloke.com
covadhosting.biz covid-schutzmasken.de
craftinsta.ru cracksplanet.com
credit-respect.ru crafthubs.com
customerguru.in crystalslot.com
cyprusbuyproperties.com
cyse.tk cyse.tk
db.speedup.gdn d0t.ru
ddsoldes.fr dating-time-now.com
detoxmed24.ru davidovic.info
deadlinkchecker.com
deanmoore.ie
dellalimov.com
demon-tweeks.com
detki-opt.ru
detskie-zabavi.ru
dev-seo.blog
dfwu1013.info dfwu1013.info
digitalassetmanagement.site digital-sale.su
digitalfaq.com dignitasdata.se
dinosaurus.site dildofotzen.net
dms-sw.ru diminishedvalueoforegon.com
dnepropetrovsk.mistr-x.org dkvorota.ru
documentbase.net dogbreedspicture.net
doeco.ru domaincheck.io
domcran.net downloader12.ru
domznaniy.ru driving.kiev.ua
dpihatinh.gov.vn drivotracker.com
drunkenstepfather.com dvd-famille.com
e-buyeasy.com e705.net
e-poker-2005.com earl-brown.info
e2click.com elektir.ru
election.interferencer.ru elektrischezigaretten2.beeplog.com
eleimgo.pw elektroniksigaraankara.info
elektrischezigarette1.blog.pl elektronischezi.livejournal.com
en.home-task.com elektronischezigarettekaufen1.beeplog.com
energy-ua.com elektronischezigarettekaufen1.myblog.de
eng-lyrics.com enbersoft.com
envaseslotusama.com erotag.com
etur.ru essayassist.com
europages.com.ru essayservicewriting.org
evidencecleanergold.com etenininrade.ga
expertblog.info everypony.ru
extads.net exmasters.com
extstat.com exonline.info
eyes-on-you.ga expolicenciaslatam.co
ezigaretteshop2.vefblog.net expresstoplivo.ru
fable.in.ua ezigarettenkaufen2.dreamwidth.org
fakehandbags.xyz falcon-images.blogspot.com
familyholiday.ml falllow.gq
fast-wordpress-start.com fandlr.com
fetishinside.com faz99.com
fifa-coins.online feeriaclub.ru
filesclub.net fickenprivat.info
filmbokep69.com finteks.ru
filmetricsasia.com floridahuntingfishingadventures.com
finemanteam.com forum-engineering.ru
fiuxy.com forums.toucharcade.com
fivedwld.cf foto-weinberger.at
flipper.top fr-bearings.ru
flooringinstallation-edmonton.com freelinkbuilding.website.tk
formseo.com freshmac.space
forum.doctissimo.fr funnypica.com
forum69.info fz139.ttk.ru
freesitetest.com gael-s.ru
freshsuperbloop.com galeon.com
fuckmill.com galeria-zdjec.com
fyl.com.ru gallerylisting.com
gambarkatabaru.com
gamblingnerd.com
gayxperience.com
gazobeton-p.com.ua
gdebestkupit.ru gdebestkupit.ru
gilbertbanda.net geile-lelly.eu
glavprofit.ru getpopunder.com
globalsurfari.com gk170.ru
goldadpremium.com glcomputers.ru
gombita.info gogalleryawesome.com
goodly.pro googglet.com
gox.com.ua googlefeud.com
greatzip.com googlsucks.com
gpirate.com
gsasearchengineranker.xyz gsasearchengineranker.xyz
guge.io guruofcasino.com
gwhwpxbw.bloger.index.hr
hamptonoaks.ca hamptonoaks.ca
hatdc.org hd-filmy.net
hausfrauensex18.com
hchha.com
hd720kino.ru hd720kino.ru
hdapp1008-a.akamaihd.net healgastro.com
hdimagelib.com healing-dysplasia.ru
heartofbeijing.blogspot.com heartofbeijing.blogspot.com
heatpower.ru hem.passagen.se
helpmymacfaster.trade hifidesign.ru
hermesbelts.xyz hitcpm.com
hermesreplica.pw hjaoopoa.top
heroz.fr hmmm.cz
histock.info homik.pw
home-task.com hornymatches.com
homeandhealth.ru host-protection.com
hotblog.top hptwaakw.blog.fc2.com
houtings.xyz hspline.com
hyhj.info i-service.kz
hyiphunter.org icaseclub.ru
ictizanidinehcl4mg.blogspot.com idealtits.net
igru-xbox.net iflycapetown.co.za
igtools.club images-free.net
ihtec2019.org imagez.co
ikearugs.xyz imguramx.pw
ilbe.club imk.com.ua
ilikevitaly.com inboundlinks.win
ilovevitaly.ru infobabki.ru
iminent.com infodocsportal.com
in-tandem.co intermesh.net
inome.com.ua internetartfair.com
investmac.com islamtoday.co.za
iplogger.org istock-mebel.ru
iqupdatetmz.win
itrevolution.cf itrevolution.cf
ivearchenceinflu.cf izoll.ru
iweblist.info jerseyschinabizwholesale.us
jav-fetish.com jerseywholesalechinabiz.com
javrip.net joyceblog.top
jennyfire.ru karting196.ru
jofucipiku.tk katadhin.co
jose.mulinohouse.co kemerovo.zrus.org
justdating.online keywordglobal.co.za
kaac.ru keywordsdoctor.com
keywordblocks.com kladrus.ru
keywordsuggests.com kosmetyki.tm.pl
kinobaks.com kovesszucs.atw.hu
kleine-titten.biz kraljeva-sutjeska.com
klosetkitten.com
koversite.info
krasivye-devushki.net
krasnodar.xrus.org krasnodar.xrus.org
kretpicf.pw lafourmiliaire.com
kurbappeal.info lafriore.ru
laexotic.com landinez.co
landmania.ru lapitec.eu
landscaping.center leadn.pl
laptop-4-less.com leadwayau.com
lenvred.org lexixxx.com
lida-ru.com
light.ifmo.ru light.ifmo.ru
liky.co.ua liky.co.ua
lineavabit.it lincolntheatre.com
linksharingt.com lineshops.biz
littleberry.ru liver-chintai.org
lkbennettoutlet.online locationdesaison.com
llastbuy.ru locimge.pw
loadingpages.me lomb.co
lombardfinder.ru
lottospring.com
louboutinreplica.pw louboutinreplica.pw
louisvuittonoutletstore.net luxup.ru
luckyday.world m3gadownload.pl
macresource.co.uk manuscript.su
mactechinfo.info mararoom.ru
mailemedicinals.com marcogrup.com
manifestyourmillion.com marmotstore.online
marcowebonyodziezowe.pl mathgym.com.au
maslenka.kz mebel-ekb.com
masserect.com members.ghanaweb.com
mebeldekor.com.ua mexicotravelnet.com
medi-fitt.hu michaelkorsoutletstore.net
member-quiz.com michellblog.online
merryhouse.co.uk minecraft-neo.ru
mmstat.com mir-business-24.ru
monclercoats.xyz misandesign.se
moneyviking-a.akamaihd.net mobile-appster.ru
moscow.xrus.org mobioffertrck.com
musirc.com mockupui.com
my-bc.ru moinozhki.com
my-big-family.com monarchfind-a.akamaihd.net
my-cash-bot.co montazhnic.ru
myblogregistercm.tk moz.com
mydeathspace.com mrvideospornogratis.xxx
myfreemp3.eu mturkcontent.com
mymobilemoneypages.com mural.co
mypornfree.ru musicspire.online
myyour.eu muz-tracker.net
nac-bearings.ru myfreetutorials.com
naj-filmy24.pl mykings.pw
nalogovyykodeks.ru na-telefon.biz
narco24.me namenectar.com
nardulan.com nash-krym.info
natali-forex.com naturalbreakthroughsresearch.com
net-profits.xyz nextlnk12.com
nlfjjunb5.ru nextrent-crimea.ru
no-fuel.org nfljerseyscheapbiz.us
nomuos.it noclegonline.info
nudepatch.net novgorod.xrus.org
ntic.fr
nufaq.com nufaq.com
oakleyglassesonline.us ok.ru
oconto.ru okeinfo.online
oddamzadarmo.eu onlinadverts.com
ofanda.com online-templatestore.com
offers.bycontext.com onlinewritingjobs17.blogspot.ru
okonich.com.ua onstrapon.purplesphere.in
omsk.xrus.org openlibrary.org
oneclickfiles.com openmediasoft.com
ontargetseo.us opinionreelle.com
ooo-olni.ru orenburg-gsm.ru
open-odyssey.org outshop.ru
ostrovtaxi.ru owathemes.com
ozas.net oyster-green.com
panchro.co.uk panchro.co.uk
paphoselectricianandplumber.com paparazzistudios.com.au
pastaleads.com paperwritingservice17.blogspot.ru
pawli.eu partners-ship.pro
paydayonlinecom.com pateaswing.com
pdn-4.com penisvergrotendepillennl.ovh
photo-clip.ru pflexads.com
phuketscreen.com picsforkeywordsuggestion.com
pillscheap24h.com pillscheap24h.com
pio.polytopesexempt.com pixell.club
pisanieprac.info pixshark.com
pizza-imperia.com pizda.lol
pkr1hand.com pk-services.ru
pl-top.pl play-movie.pl
pl.id-forex.com poisk-zakona.ru
pod-muzyku.club polybuild.ru
podshipniki-nsk.ru pornhive.org
pony-business.com pornobrazzers.biz
porndairy.in pornophoto.xyz
porndl.org powenlite24.ru
porno-transsexuals.ru pr0fit-b0x.com
porno-video-chati.ru pregnant.guru
pornoelita.info
pornohd1080.online
pornoklad.net
pornoxxx.com.mx
porodasobak.net
pozdrawleniya.com
prchecker.info
preconnubial.usuby.site
printingpeach.com printingpeach.com
privatbank46.ru prize44.com
producm.ru prod2016.com
profitsport.club prodess.ru
pronekut.com profolan.pl
pshare.biz promgirldresses.xyz
psoriasis-file.trade prostitutki-astana.org
pwwysydh.com quick-seeker.com
raavidesigns.com rankia.com
rada.ru rarbg.to
rangoman.date razamicroelectronics.com
ranking2017.ga referencemoi.com
rapidgator-porn.ga relayblog.com
realting-moscow.ru remont-komputerov-notebook.ru
reklama1.ru resultshub-a.akamaihd.net
relax.ru rewardpoll.com
remedyotc.com ricorsogiustizia.org
remont-ustanovka-tehniki.ru rocketchange.ru
remybutler.fr
restaurantlescampi.com
reversing.cc
reward-survey.net
ringporno.com
rome2rio.com rome2rio.com
ruscopybook.com rostov.xrus.org
rusenvironmental.net s.lollypopgaming.com
rutor.group s8-nowy-wygraj.comli.com
sa-live.com sanatorrii.ru
salmonfishingsacramentoriver.com sandhillsonline.com
samo-soznanie.ru saneitconsulting.com
sampleletters.net sanjosestartups.com
santasgift.ml sc-specialhost.com
sarahmilne.top scanner-alex.top
sardinie.us scanner-irvin.top
saveriopiazza.it scanner-jess.top
savetubevideo.com scanner-jessica.top
sbt-aqua.ru scanner-john.top
scanner-jack.top se.bnt-team.com
scanner-julianna.top searchaddis.com
school-diplomat.ru
searchinterneat-a.akamaihd.net
searchlock.com searchlock.com
seeingmeerkat.com searchtooknow-a.akamaihd.net
seo-tools-optimizing.com seemoreresultshu-a.akamaihd.net
service-core.ru seksotur.ru
serving.adbetclickin.pink seobility.net
sevendays.com.ua seocheki.net
sex.hotblog.top seoexperimenty.ru
sexspornotub.com seolab.top
sharpchallenge.com sexreliz.com
sherlock.se sexystrippe.info
shmetall.com.ua sharebutton.net
signoredom.com shlyahten.ru
simplesite.com shopfishing.com.ua
skatestick.bid sildenafilcitratemed.com
sklad-24.ru simplepooltips.com
smallseotools.com site.ru
smartbalanceworld.com siteheart.net
snomer1.ru sitiz.club
social-button.xyz skutecznetabletkinaporostwlosow.pl
softomix.com skypasss.com
softtor.com sloopyjoes.com
sovinsteel.ru smailik.org
spb.afora.ru snjatie-geroinovoy-lomki.ru
spin2016.cf sockshare.net
spitfiremusic.com soda.media
sodexo.com
soft-program.com
soochi.co
soviet-portal.do.am
spabali.org
space-worry.ml
spasswelt.xyz
spidtest.space
sport-video-obzor.ru
sportbetfair.com
sports-supplements.us sports-supplements.us
stal-rulon.ru statoutlook.info
start.myplaycity.com stats-collector.org
startwp.org stopnarco.ru
stathat.com
stats-public.grammarly.io
stauga.altervista.org
sterva.cc
stillmiracle.com
storehouse.ua
streamin.to streamin.to
stretchingabuckblog.com streha-metalko.si
styro.ru sundrugstore.com
supernew.org superkanpo.com
supermodni.com.ua
surcentro.com
sweepstakes.rewardit.com sweepstakes.rewardit.com
syndicate.fun tds-advert005.info
talant-factory.ru tdsing.ru
tarad.com
teastory.co
technopellet.gr
teencastingporn.com teencastingporn.com
thetoiletpaper.com terraclicks.com
tiens2010.ru texbaza.by
timer4web.com textads.men
tizanidineforopiatewithdrawal.blogspot.com the-usa-games.blogspot.com
thecoolimages.net
thegioixekhach.com
thejournal.ru
thenetinfo.com
thewomenlife.com
tiandeural.ru
tizanidineinteractions.blogspot.com
tizanidineoral.blogspot.com
tizanidinestreetprice.blogspot.com tizanidinestreetprice.blogspot.com
toolsky.com tongkatmadura.info
toon-families.com topappspro.com
top10registrycleaners.com topclickguru.com
torrents-tracker.com toxicwap.com
torture.ml tracxn.com
tracksurf.daooda.com trafaret74.ru
trackzapper.com trafficmonetize.org
traffic2money.com traffictrade.life
tsyndicate.com tsan.net
tt-ipd.info tupper-posuda.ru
turn-up-life.life
tvand.ru
tvnewsclips.info tvnewsclips.info
tvorozhnaja-zapekanka-recept.ru ua.tc
tvory.predmety.in.ua ugogo.info
twittruth.com ukkelberg.no
ucsol.ru ultimatesetnewfreeallsoftupgradesystems.pw
uhodzalijami.ru ultramart.biz
uptime-alpha.net umnovocaminho.com
urzedowski.eu unimodemhalfduplefw.pen.io
valkiria-tk.ru unpredictable.ga
valuado.com upupa.net
vduplo.ru usdx.us
v720hd.ru
vashsvet.com
veerotech.com
video-hollywood.ru
videochat.cafe
videochat.life videochat.life
videochaty.ru videochat.tv.br
videosbox.ru
vids18.site vids18.site
vintontech.info vikistars.com
vip-parfumeria.ru viktoria-center.ru
virus-respirators.com
visa-china.ru
vkmusics.ru vkmusics.ru
vod.com.ua vkontarkte.com
volgograd.xrus.org voluumtracker1.com
voloo.ru vpnhowto.info
vrotike.ru vriel.batcave.net
vvon.co.uk vuclip.com
vzlomsn.org wallpapersist.com
walkme.com watchmyfb.pl
warningwar.ru
we-are-gamers.com we-are-gamers.com
webads.co.nz weblibrary.win
webmasterhome.cn webmonetizer.net
webtherapy.ru webscouter.net
weburok.com westsextube.com
wheelchairliftsarea.com westvilletowingservices.co.za
whereiskentoday.com wetgames.ru
whereverdesperate.gq wholesalejerseychinashop.com
windowssearch-exp.com wholesalejerseysgaa.com
wineration.com wladimirpayen.com
wnoz.de wnoz.de
wordkeyhelper.com womensplay.net
wtsindia.in world-mmo.com
www.solartek.ru wovis.site
wygraj-teraz.com ww2awards.info
x-mix.info www.bookmaker-bets.com
xchat26.myfreecams.com x-diesel.com
xclicks.net x-diesel.org
xedserver.com xchangetrak.com
xlovecam.com
xn-----6kcaabbafhu7cskl7akvongwpo7hvjj.xn--p1ai
xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai
xn----7sbaphztdjeboffeiof6c.xn--p1ai
xn----9sbebi2bvzr7h.xn--p1ai xn----9sbebi2bvzr7h.xn--p1ai
xn--80ab4aa2g.xn--p1ai xn----itbeirbjbi7bc6bh2d.xn--p1ai
xn--80adaggc5bdhlfamsfdij4p7b.xn--p1ai xn--h1ahbi.com.ua
xn--80ahvj9e.xn--p1ai xtube.com
xn--hxazdsfy.blogspot.com yellowads.men
xxlargepop.com yhit.press
xxx-cam.webcam yourtemplatefinder.com
xxxasianporn.net youtubedownload.org
xxxdatinglocal.us zagadki.in.ua
xxxguitars.com zaidia.xhost.ro
you-shall-not-pass.is74.ru zarepta.com
yougotanewdomain.com zatjmuzu.info
zaobao.com.sg zed21.net
zastenchivosti.net zeroredirect8.com
zazagames.org zetgie.com.pl
zigarettenonlinekaufen1.bloog.pl zscalerone.net
zigarettenonlinekaufen1.blox.pl zzlgxh.com
zirondelli.it
znakomstvaonlain.ru
zona-aqua.ru
zone-kev717.info
zoogdinsney.com
zvetki.ru

488
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
1-free-share-buttons.com 00author.com
1001watch.com.ua 1kdailyprofit.me
2girls1cup-free.com 7search.com
3rbseyes.com aa625d84f1587749c1ab011d6f269f7d64.com
5000-cotydzien.com aa8b68101d388c446389283820863176e7.com
5kstore.com abovetherivernc.com
aa81bf391151884adfa3dd677e41f94be1.com adobereader-free.ru
addbin.men adtech.fr
ads.gold advocatemsk.ru
adserver-e7.com alert-jdh.xyz
adservme.com alfabot.xyz
adtech.us alldubai.biz
aktivator-windows10.blogspot.com amazingninja.com
albumsuper.info androids-store.com
all4invest.ru animaltoplist.com
apartamentwroclaw.eu apelsinnik.website
api.stathat.com apifasterlightin-a.akamaihd.net
arendovalka.xyz arabsexxxtube.com
artdeko.info arenanews.com.ua
asrvrep-a.akamaihd.net asmxsatadriverin.aircus.com
aviav.org ausmepa.org.au
ayanaspa.com automatic-seo.com
azbukadiets.ru avtointeres.ru
azlex.uz
b-styles.xyz b-styles.xyz
bag77.ru bannerswap.com
baltgem.com bfz.biz
bard-real.com.ua bimatoprost-careprost.com
battle.net biographiya.com
bestbuy.ca bjanshee.ru
bestkfiledxd.cf bloggedporn.com
betterhdporn.com boom-celebs.com
betwinservice.com break-the-chains.com
bioscorp.ru buqyxa.rincian.info
blog.koorg.ru buyessayonline19.blogspot.ru
bmsco.co c-english.ru
bolegapakistan.com
bongacams.com
brainboostingsupplements.org
brains2.biz
buybest1.biz
bvlgaribracelet.xyz
cakesplus.com.au
calcularpagerank.com calcularpagerank.com
cardul.ru canoncdriverq3.pen.io
cartierring.xyz cartierreplica.pw
caveavins.fr ccbill.com
cheatcode-lita12.rhcloud.com cctva.tv
ciekawinki.pl chcu.net
coindirect.io cinemaenergy-hd.ru
confib.ifmo.ru citizenclsdriveri7.pen.io
counter.yadro.ru colobit.biz
darmebel.com.ua compliance-elena.xyz
dataloading.net compliance-ivan.top
date-withme.com compliance-walt.top
delitime.info conocer-sanabria.com
deniven.1bb.ru copyrightclaims.org
dev.citetick.com covadhosting.biz
dipstar.org creativehutindia.com
dosugrostov.site credit-online.ws
dotnetdotcom.org culad.com
downloadmefiranaratb1972.xpg.com.br cwetochki.ru
dprtb.com daptravel.com
dscaas.website decenttools.com
ekspertmed.com decorazilla.com
englate.com degerlund.net
eyes-on-you.ga deluxedumps.com
ezigarettenkaufen2.dreamwidth.org dentfidemountpreach.tk
falllow.gq devochki.top
famix.xyz diusyawiga.tk
fantasticpixcool.com dmmspy.com
fearcrow.com dojki-hd.com
fetroshok.ru domcran.net
fickkontakte.org drugs-no-rx.info
filkhbr.com dyt.net
financetip.eu earncash.com.ua
forex-procto.ru egovaleo.it
forsex.info ereko.ru
forzeronly.com erotik-kostenlos.net
foto-weinberger.at evening-dating-club.info
frbizlist.com expresstoplivo.ru
freshwallpapers.info feel-planet.com
funtoonez.com fetlifeblog.com
gameplexcity.com fillmewithhappiness.com
gelezki.com firma-legion.ru
germes-trans.com flashahead.info
getmyads24.com freewebs.com
gifspics.com frvo.alptandem.ru
globatur.ru fsakhalin.ru
globetrotting-culture.ru ftns.ru
goldandcard.ru fym.com.ru
google-liar.ru gay-site.store
hackers-crackers.tk getpopunder.com
haikuware.com ggiaro.com
hao123.com glall.ru
hausfrauensex18.com glavtral.ru
headpress.ru gobongo.info
hermesbirkinhandbagoutlets.com gojiberriess.apishops.ru
homeandhealth.ru goodwinmetals.co
hostsshop.ru gorabagrata.ru
hotxnights.info gsasearchengineranker.space
i4track.net guod.me
ictizanidinehcl4mg.blogspot.com hanink.biz.ly
ifmo.ru hard-porn.mobi
ilmexico.com harmonyglen.us
images-free.net hetmanship.xyz
imitex-plus.ru homedo.fabpage.com
insta-add.pro hornymatches.com
investzalog.ru hoverboard360.at
iqoption-bin.com hoverboard360.de
iskalko.ru iminent.com
ispac.org internet-apteka.ru
ixora.pro iqoption.pro
jagg.info iron-age.info
jikoman.info jackwolfskinoutlet.online
johnnyhaley.top jolic2.com
kazka.ru journalhome.com
kino-key.info jumptap.com
klitimg.pw kamagragelusa.net
klosetkitten.com keki.info
knigonosha.net keywordsuggest.org
koronirealestate.gr kinofak.net
krasula.pp.ua kosynka-games.ru
kredit-blog.ru kvartir-remont.biz
kunstaktien.info lanadelreyfans.us
law-check-seven.xyz letmacworkfaster.world
lechtaczka.net libertybilisim.com
levaquin750mg.blogspot.com lirunet.ru
limtu.ifmo.ru localmatchbook.com
lostaruban.com
lrac.info lrac.info
macdamaged.tech luchshie-topcasino.ru
magda-gadalka.ru mail.allnews24.in
magicdiet.gq makedo.ru
makemoneyonline.com maladot.com
marketingtechniques.info managerpak204.weebly.com
marmotstore.online marblestyle.ru
me-ke.com mature.free-websites.com
mediawhirl.net
medicineseasybuy.com
medkritika.ru medkritika.ru
meltwater.com megagrabber.ru
mexicotravelnet.com microsoftportal.net
midst.eu mikozstop.com
mlf.hordo.win minyetki.ru
mmofreegames.online monclerjacketsoutlet.pw
mobilemedia.md morlat.altervista.org
mobplayer.ru mrpornogratis.xxx
mockupui.com mrvideospornogratis.xxx
monarhs.info muizre.ru
monclerjacketsoutlet.win muschisexbilder.com
mowser.com mystats.xyz
moyakuhnia.ru naperehresti.info
mp3s.club narkologiya-belgorod.ru
msk.afora.ru nfljerseyscheapchinabiz.com
mybackgroundlandscape.blogspot.com nonews.co
mydownlodablefiles.com nrv.co.za
mylovelibrabry.com obiavo.su
myshopmatemac.com ohmyrings.com
naj-filmy24.pl ons-add.men
national-today-winning-winner.club openmediasoft.com
nevansk.ru orsonet.ru
nicovideo.jp oxford-book.com.ua
nvformula.ru p-business.ru
obesidadealgarve.com paydayloanslocal.com
opinionreelle.com peekyou.com
optom-deshevo.ru pigrafix.at
ourville.info piski.top
panchro.co.uk playfortuna-play.ru
popugauka.ru ploenjitmedia.azurewebsites.net
pornodojd.ru plohaya-kreditnaya-istoriya.ru
pornofeuer.com poem-paying.gq
pornonik.com popserve.adscpm.net
portside.xyz poquoson.org
privat-girl.net porno-play.net
producm.ru privatamateure.com
profitsport.club profitsport.club
promoforum.ru prombudpostach.com.ua
pulse33.ru puzo2arbuza.ru
py100.ru rabotaetvse.ru
r-control.ru refads.pro
rankscanner.com renewablewealth.com
rasteniya-vs-zombi.ru rogervivierforsale.com
rednise.com rollercoin.com
reimageplus.com rospromtest.ru
remedyotc.com royalcar-ufa.ru
remont-fridge-tv.ru rubanners.com
rent2spb.ru saveriopiazza.it
rep-am.com scanner-jane.top
reward-survey.net scrapinghub.com
rp9.ru seo18.su
rrutw.com seobility.net
running-line.ru seonetwizard.com
sarafangel.ru seorankinglinks.com
scanner-barak.top sexsaoy.com
searchimage.co sharpchallenge.com
searchkut.com shell-pmr.ru
service.adtech.us shinikiev.com.ua
sexphoto.site ship-marvel.co.ua
shop.xz618.com shivafurnishings.com
sideeffectsoftizanidine.blogspot.com shmetall.com.ua
site-speed-check.site sibvitr.ru
sluganarodu.ru sildenafilcitratemed.com
smart-balancewheel.com silvermature.net
snyatie-lomki-v-stacionare.ru sisiynas.ru
sockshare.net siteheart.net
softomix.net slonechka.ru
songplanet.ru socialbookmarksubmission.org
solmarket.by
soundfrost.org soundfrost.org
splendorsearch-a.akamaihd.net souvenirua.com
stackthatbucks.com stretchingabuckblog.com
stopnarco.ru stroydetali.ru
stroiminsk.org superiends.org
stroymonolit.su surflinksmedical.com
students-cheapskate.ml takethatad.com
studiofmp.com taraz.xkaz.org
syndicate.fun telesvoboda.ru
t-bygg.com thebestweightlosspills.ovh
tiens2010.ru thetoiletpaper.com
tizanidinehcl4mgtablet.blogspot.com tmtrck.com
tizanidineonline.blogspot.com torrents.cd
tk-assortiment.ru tour-line.net
tmearegion26.com tpu.ru
tooplay.com track-rankings.online
topcasinoratings.ru trkdf.com
toys.erolove.in truck-land.life
tracksz.co truebeauty.cc
trahic.ru ttsq.fr
treasuretrack-a.akamaihd.net uhod-za-sobakoj.ru
trubywriting.com unblocksit.es
um-razum.ru uptimebot.net
unlimitdocs.net upupa.net
uprour.com urldelivery.com
uytmaster.ru viandpet.com
veerotech.com vkgaleria.com
vgoloveboli.net vykupavto-krasnodar.ru
video-woman.com vzlom-na-zakaz.com
vip-parfumeria.ru wallpaperstock.net
vseigru.one wildnatureimages.com
works.if.ua windowssearch-exp.com
wovis.site wnhjavlhezp.gq
www.jbetting.com woodyguthrie.se
xmronta.com ws.ampower.me
xn-----7kceclhb4abre1b4a0ccl2fxch1a.xn--p1ai wstroika.ru
xn----7sbifcamovvfggw9d.xn--p1ai x-diesel.info
xn----9sbubg3ambdfl1j.xn--p1ai xev.ru
xolodremont.ru xn-----6kcaabbafhu7cskl7akvongwpo7hvjj.xn--p1ai
yellowproxy.net xn----7sbbagbq7bd5aheftfllo4m.xn--p1ai
zaloadi.ru xn----ctbbcjd3dbsehgi.xn--p1ai
zapatosenventa.info xn--24-glceagatoq7c2a6ioc.xn--p1ai
zastroyka.org xn--80aodinpgi.xn--p1ai
zazagames.org xtube.com
zigarettenonl.canalblog.com xuki.us
xxxguitars.com
yourtemplatefinder.com
zebradudka.com
zhorapankratov7.blogspot.com

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.176 0; 10.1.0.109 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.01.2838 ### Version: V4.2022.01.2839
### Updated: Sat Jan 8 10:03:26 UTC 2022 ### Updated: Sat Jan 8 22:03:36 UTC 2022
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.01.2838 ### Version: V4.2022.01.2839
### Updated: Sat Jan 8 10:03:26 UTC 2022 ### Updated: Sat Jan 8 22:03:36 UTC 2022
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### Version Information ## ### Version Information ##