vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-03 19:20:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.11.2712 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-11-13 10:09:05 +00:00
parent ca817a6e4e
commit 0b616e7062
46 changed files with 63859 additions and 63859 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.11.2711 #### Version: V4.2021.11.2712
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [626](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [626](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.11.2711 #### Version: V4.2021.11.2712
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [626](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [626](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -11,7 +11,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2021.11.2711 #### Version: V4.2021.11.2712
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [626](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [626](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

7846
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2021.11.2711 #### Version: V4.2021.11.2712
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [626](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [626](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

7850
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

7850
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

7850
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

7850
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 1; 10.1.1.56 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 0; 10.1.1.56 0;
127.0.0.1 0; 127.0.0.1 0;

7850
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 1; 10.1.1.56 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 0; 10.1.1.56 0;
127.0.0.1 0; 127.0.0.1 0;

7850
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 1; 10.1.1.56 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 0; 10.1.1.56 0;
127.0.0.1 0; 127.0.0.1 0;

7850
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2711 2712

7850
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

7850
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.114.51.105 0; 13.92.182.20 0;

7850
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 1; 10.1.1.56 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 0; 10.1.1.56 0;
127.0.0.1 0; 127.0.0.1 0;

7850
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 1; 10.1.1.56 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 0; 10.1.1.56 0;
127.0.0.1 0; 127.0.0.1 0;

7850
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 1; 10.1.1.56 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 0; 10.1.1.56 0;
127.0.0.1 0; 127.0.0.1 0;

7850
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.114.51.105 0; 13.92.182.20 0;

7850
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.114.51.105 0; 13.92.182.20 0;

7850
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 1; 10.1.1.56 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Brandprotect CODE87
Cosmos FemtosearchBot
Iskanie FyberSpider
Musobot Heritrix
PictureFinder InfoNaviRobot
Psbot PyCurl
Rogerbot SEOstats
Wonderbot Spammen
sexsearcher Trendiction
x22Mozilla WWW-Collector-E

324
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,242 +1,241 @@
01h4x.com
360Spider 360Spider
404enemy ADmantX
80legs ALittle Client
AIBOT Abonti
Aboundexbot Acunetix
AiHitBot AllSubmitter
AlphaBot AlphaBot
Anarchy Anarchy99
Ankit Ankit
Anthill Anthill
AwarioRssBot Aspiegel
Attach
AwarioSmartBot AwarioSmartBot
BBBike BBBike
BLEXBot BDCbot
BackDoorBot
BackStreet
BackWeb BackWeb
Badass Backlink-Ceck
BacklinkCrawler
Barkrowler Barkrowler
BetaBot
Bigfoot Bigfoot
Bitacle Bitacle
Blackboard Blackboard
Blow BlowFish
Buck Boardreader
Buddy BotALot
Brandwatch
BuiltWith BuiltWith
Bullseye Bullseye
BuzzSumo CCBot
CODE87
CazoodleBot CazoodleBot
Cegbfeieh CheeseBot
CensysInspect
CherryPicker
Claritybot
Cliqzbot
Cocolyzebot Cocolyzebot
CopyRightCheck Collector
Crawling at Home Project Copyscape
Custo Cosmos
CyotekWebCopy CrazyWebCrawler
Crescent
CrunchBot
DBLBot
DIIbot DIIbot
DTS Agent
DataCha0s DataCha0s
Dirbuster Deusu
Discobot DigitalPebble
Disco
Dispatch
DomCopBot
DomainAppender
DomainCrawler DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
DomainStatsBot DomainStatsBot
Domains Project Domains Project
Dotbot Dragonfly
Evil Drip
Express WebPictures ECCP/1.0
ExtLinksBot EMail Siphon
Extractor EMail Wolf
EasyDL
Ecxi
EirGrabber
EroCrawler
Exabot
ExtractorPro
EyeNetIE
Ezooms Ezooms
FDM
FHscan
FemtosearchBot FemtosearchBot
Fimap
Firefox/7.0
FlashGet FlashGet
Fyrebot Flunky
Fuzz
G-i-g-a-b-o-t G-i-g-a-b-o-t
GT::WWW Gigabot
GalaxyBot Go!Zilla
GermCrawler
GetRight
GoZilla
Grafula Grafula
GridBot
HEADMasterSEO HEADMasterSEO
HMView HTMLparser
Haansoft HTTP::Lite
HaosouSpider
Harvest Harvest
Havij
Humanlinks
IDBTE4M IDBTE4M
IDBot IDBot
IRLbot IRLbot
Image Sucker IlseBot
IndeedBot Image Fetch
InfoNaviRobot Indy Library
InfoTekies
Intelliseek
Internet Ninja Internet Ninja
InternetSeer IstellaBot
Iskanie Joomla
Jbrofuzz
JetCar
JikeSpider
JustView
Kinza
LNSpiderguy
LWP::Simple LWP::Simple
Leap Lanshanbot
Libwhisker Larbin
Lightspeedsystems LexiBot
LinkextractorPro LibWeb
LinksManager LieBaoFast
LinqiaRSSBot Linkdexbot
LinqiaScrapeBot Lipperhey Spider
Lipperhey MIDown tool
Litemage_walker
MFC_Tear_Sample
MIIxpc MIIxpc
MJ12bot MTRobot
MSIECrawler Mail.RU_Bot
Mag-Net
Majestic SEO Majestic SEO
Masscan Mass Downloader
MauiBot Mata Hari
MegaIndex.ru MeanPath Bot
Metauri Metauri
Microsoft Data Access
Mister PiX Mister PiX
Moblie Safari Moblie Safari
Morfeus Fucking Scanner Mojeek
Mozlila Mr.4x3
Msrabot
NICErsPRO
NPbot NPbot
Nameprotect
Navroad Navroad
Needle Needle
Nessus Net Vampire
NetMechanic
NetSpider NetSpider
NetZIP
Nettrack
NextGenSearchBot NextGenSearchBot
NimbleCrawler OpenVAS
Offline Navigator PECL::HTTP
OpenLinkProfiler PageGrabber
Openvas PageScorer
OutfoxBot
PHPCrawl
POE-Component-Client-HTTP
Page Analyzer
PageThing.com
Papa Foto Papa Foto
Pavuk
PeoplePal PeoplePal
Pi-Monster Pi-Monster
Picscout Picscout
Picsearch Piepmatz
PictureFinder PleaseCrawl
ProWebWalker
Probethenet Probethenet
Pu_iN Psbot
PxBroker RankingBot
PyCurl RankurBot
QueryN Metasearch
RSSingBot
RankActive
RankActiveLinkBot
RankFlex
Re-re
ReGet ReGet
RealDownload Reaper
RebelMouse
Recorder Recorder
RepoMonkey RepoMonkey
Ripper RocketCrawler
SEOlyticsCrawler Rogerbot
SEOprofiler
SMTBot
SalesIntelligent SalesIntelligent
Scrapy Scanbot
ScrepyBot ScreenerBot
Searchestate
SearchmetricsBot SearchmetricsBot
SemrushBot
SeoSiteCheckup
SeobilityBot
Seomoz Seomoz
Siphon Siphon
Site Sucker SiteSnagger
SiteExplorer Sitevigil
Sitebeam SlySearch
Snapbot SmartDownload
Snoopy Snake
Sociscraper
Sosospider Sosospider
Spammen
SpankBot SpankBot
Spbot Spanner
Spinn3r Sqworm
Steeler Stripper
Sucuri
SuperBot
SurveyBot
Swiftbot Swiftbot
Szukacz Telesphoreo
T0PHackTeam Telesphorep
TeleportPro The Intraformant
TheNomad TightTwatBot
True_Robot Toweyabot
TwengaBot Turingos
Twice Twice
URLy.Warning Typhoeus
URLy Warning
UnisterBot
Upflow Upflow
V-BOT V-BOT
VB Project Vacuum
Vagabondo Vagabondo
VelenPublicWebCrawler VeriCiteCrawler
VidibleScraper
Virusdie
VoidEYE VoidEYE
Voil Voltron
WASALive-Bot
WBSearchBot
WISENutbot
WPScan WPScan
WWW-Mechanize WWW::Mechanize
Wallpapers
WeSEE WeSEE
Web Enhancer Web Auto
Web Collage
Web Fuck
Web Pix Web Pix
Web Sauger Web Sucker
WebAuto
WebBandit WebBandit
WebCollage
WebCopier
WebEnhancer WebEnhancer
WebFuck WebFetch
WebPix WebReaper
WebWhacker
WebZIP WebZIP
WebmasterWorldForumBot Whack
Website Quester WiseGuys Robot
Who.is Bot Wonderbot
Widow Woobot
Wotbox
Wprecon Wprecon
Xenu Xaldon WebSpider
Xaldon_WebSpider
YoudaoBot YoudaoBot
Zade
Zauba Zauba
Zitebot Zeus
ZoomBot ZmEu
ZoominfoBot adscanner
ZumBot
archive.org_bot archive.org_bot
arquivo-web-crawler
arquivo.pt
backlink-check
cah.io.community cah.io.community
check1.exe clark-crawler
crawl.sogou.com crawler4j
crawler.feedback
dataforseo.com dataforseo.com
demandbase-bot domainsproject.org
facebookscraper facebookscraper
instabid gopher
ips-agent ips-agent
isitwp.com isitwp.com
plumanalytics magpie-crawler
mediawords
muhstik-scan
netEstate NE Crawler
pcBrowser
polaris version polaris version
ripz ripz
scan.lol
seobility
seoscanners
seostar seostar
serpstatbot sexsearcher
sitechecker.pro sitechecker.pro
siteripz siteripz
sogouspider sogouspider
@ -244,7 +243,8 @@ sp_auditbot
spyfu spyfu
tAkeOut tAkeOut
trendiction.com trendiction.com
webmeup-crawler ubermetrics-technologies.com
webpros.com voyagerx.com
webprosbot x22Mozilla
x09Mozilla xpymep1.exe
zgrab

314
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
404checker 01h4x.com
ADmantX 404enemy
80legs
AIBOT
ALittle Client
Aboundexbot Aboundexbot
Acunetix
AfD-Verbotsverfahren
Aipbot
Alexibot Alexibot
Anarchy AllSubmitter
Alligator
Anarchy99 Anarchy99
Anthill
Apexoo Apexoo
Aspiegel Asterias
BackDoorBot Attach
AwarioRssBot
AwarioSmartBot
BBBike
BDFetch
BLEXBot
BackStreet BackStreet
BacklinkCrawler BackWeb
Backlink-Ceck
Badass Badass
BetaBot Bandit
Barkrowler
Bigfoot Bigfoot
Bitacle
Black Hole Black Hole
BlackWidow
Blackboard Blackboard
Bolt BlowFish
Boardreader
BotALot BotALot
Brandprotect Brandprotect
Brandwatch Brandwatch
CATExplorador BuiltBotTough
BuiltWith
CCBot CCBot
CODE87 Calculon
CSHttp
CazoodleBot CazoodleBot
CensysInspect CensysInspect
CheeseBot CheeseBot
Cliqzbot CherryPicker
Cogentbot Copier
Collector
Cosmos Cosmos
CrazyWebCrawler Craftbot
Crawling at Home Project
Crescent
CrunchBot CrunchBot
DSearch CyotekWebCopy
DTS Agent DBLBot
Deusu DIIbot
Digincore Digincore
Dirbuster Disco
Discobot Dispatch
Discoverybot DnyzBot
DomainAppender DomCopBot
DomainCrawler DomainSigmaCrawler
Dragonfly DomainStatsBot
Dotbot
Drip Drip
ECCP/1.0
EasyDL EasyDL
EirGrabber Ecxi
EroCrawler Evil
Express WebPictures ExtLinksBot
ExtractorPro EyeNetIE
Ezooms Ezooms
FDM
FHscan FHscan
FemtosearchBot FlashGet
Fimap
Flunky Flunky
Freeuploader Foobot
FrontPage Fyrebot
G-i-g-a-b-o-t G-i-g-a-b-o-t
GalaxyBot GalaxyBot
Genieo Genieo
GetRight
GetWeb GetWeb
Getintent
Gigabot Gigabot
Go!Zilla GoZilla
Go-Ahead-Got-It
Gotit
GrabNet GrabNet
GrapeFX GrapeshotCrawler
GridBot HTMLparser
HMView HTTP::Lite
Haansoft HTTrack
HaosouSpider HaosouSpider
Harvest Humanlinks
Heritrix HybridBot
IDBTE4M
IDBot
Iblog Iblog
IlseBot Image Sucker
Image Fetch IndeedBot
Indy Library InfoNaviRobot
InterGET Intelliseek
Internet Ninja InternetSeer
Iskanie Iria
JOC Web Spider
Jbrofuzz Jbrofuzz
JennyBot JennyBot
Joomla Joomla
Jorgee Jorgee
Kenjin Spider Kozmosbot
Kinza
LWP::Simple LWP::Simple
Lanshanbot Lanshanbot
Leap Larbin
LeechFTP LeechFTP
LexiBot LeechGet
Libwhisker Libwhisker
Lightspeedsystems LinkWalker
LinkScan Linkbot
LinksManager LinksManager
Litemage_walker LinqiaMetadataDownloaderBot
Lmspider Lipperhey Spider
Ltx71
MIIxpc MIIxpc
MQQBrowser MQQBrowser
MSFrontPage MTRobot
Majestic-SEO Majestic12
MarkMonitor MauiBot
MarkWatch MeanPath Bot
Mass Downloader Metauri
Mata Hari
Mb2345Browser
Mediatoolkitbot
MegaIndex.ru
MicroMessenger
Microsoft Data Access Microsoft Data Access
Mojeek Moblie Safari
Morfeus Fucking Scanner Morfeus Fucking Scanner
Mozlila
Mr.4x3 Mr.4x3
NICErsPRO Musobot
NPbot
Navroad Navroad
NearSite NearSite
Nessus
Net Vampire
NetAnts NetAnts
NetLyzer NetSpider
NetMechanic NetZIP
Nettrack Netvibes
NextGenSearchBot
Niki-bot Niki-bot
Ninja Nikto
NimbleCrawler
Nmap Nmap
Octopus
Offline Explorer Offline Explorer
Offline Navigator Offline Navigator
OnCrawl OnCrawl
Openvas Openfind
OrangeBot
OrangeSpider OrangeSpider
OutclicksBot
OutfoxBot OutfoxBot
PECL::HTTP
Page Analyzer Page Analyzer
PageAnalyzer PageAnalyzer
Panscient PageScorer
Pandalytics
Papa Foto
Pavuk Pavuk
PeoplePal PeoplePal
ProPowerBot Petalbot
Picscout
PictureFinder
ProWebWalker ProWebWalker
Psbot Quick-Crawler
QueryN Metasearch
RSSingBot
RankActiveLinkBot
RankingBot
RankingBot2 RankingBot2
Re-re ReGet
RealDownload RealDownload
Reaper
Recorder Recorder
Ripper SEOprofiler
RocketCrawler
SEOstats SEOstats
SISTRIX SalesIntelligent
Screaming ScoutJet
ScreenerBot Searchestate
SemrushBot SearchmetricsBot
SentiBot Semrush
SeobilityBot SeobilityBot
SiteLockSpider Shodan
SiteCheckerBotCrawler
SiteExplorer
SiteSnagger SiteSnagger
SiteSucker Sociscraper
Sitebeam
Siteimprove
Snapbot
Sosospider Sosospider
Sottopop Sottopop
SpaceBison SpaceBison
Spammen Spanner
SpankBot Spinn3r
Spbot Sqlworm
SputnikBot Sqworm
Sqlmap Sucuri
Steeler
SuperBot
SuperHTTP SuperHTTP
SurveyBot
TeleportPro
Telesoft Telesoft
Telesphoreo TheNomad
Telesphorep Toata
TightTwatBot
Titan
Toweyabot
Trendiction
Trendictionbot Trendictionbot
Turingos Turingos
Turnitin Typhoeus
TurnitinBot Upflow
UnisterBot
VB Project VB Project
VCI VCI
Vacuum VelenPublicWebCrawler
Vagabondo VeriCiteCrawler
VidibleScraper
Virusdie
VoidEYE
Voil
WASALive-Bot WASALive-Bot
WWW::Mechanize WISENutbot
WWWOFFLE WWWOFFLE
Wallpapers/3.0
WallpapersHD
WeSEE
Web Enhancer
Web Fetch Web Fetch
Web Fuck Web Fuck
WebCollage WebAuto
WebCopier WebEnhancer
WebFetch WebLeacher
WebFuck WebsiteQuester
WebPix
WebReaper
WebSucker
WebWhacker
WebZIP
Webalta
Webshag
Website Quester
WebsiteExtractor
Webster Webster
Whack Whack
Whacker Whatweb
WinHTTrack Widow
Woobot WiseGuys Robot
Wprecon Wonderbot
Xaldon WebSpider
Xenu
Zade
Zauba Zauba
Zeus Zeus
Zitebot Zitebot
ZoominfoBot ZyBorg
archive.org_bot archive.org_bot
arquivo-web-crawler arquivo-web-crawler
autoemailspider
backlink-check
cah.io.community
check1.exe check1.exe
cognitiveseo
crawl.sogou.com crawl.sogou.com
crawler4j crawler.feedback
eCatch domainsproject.org
instabid facebookscraper
lwp-request gopher
lwp-trivial internetVista monitor
plumanalytics magpie-crawler
netEstate NE Crawler
pcBrowser
probe-image-size
ripz
s1z.ru s1z.ru
seocompany.store scalaj-http
serpstatbot siteripz
spyfu
sysscan sysscan
trendiction.com
ubermetrics-technologies.com ubermetrics-technologies.com
x22Mozilla webmeup-crawler
x09Mozilla
xpymep1.exe
zauba.io zauba.io
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
bjetjt.com am-se.com
cezartabac.ro artdestshop.eu
compliance-fred.xyz brains2.biz
foodgid.net courtshipgift.com
hotshoppymac.com foto-weinberger.at
metascephe.com hdxnxxtube.mobi
runtnc.net kw21.org
segol.tv lulea-auktionsverk.se
trafficreceiver.club trafficgenius.xyz
unimodemhalfduplefw.pen.io u-cheats.ru

914
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
0912701309f8ce.com 03e.info
1001watch.com.ua 11pikachu.ru
1008.su 1314dh.com
100searchengines.com 14b.info
123movies.love 2020iscoming.info
178evakuator178.ru 24videos.tv
1millionusd.xyz 2girls1cup-free.com
1webmaster.ml 3rbseyes.com
1xbet4you.com 3th.co.in
28n2gl3wfyb0.ru 404.mn
4k-player.pl 4ureyesonly.com
8jn.dating-app.ru 5u.com
999webdesign.com 9med.net
aa9bd78f328a6a41279d0fad0a88df1901.com a96527gi.beget.tech
absolute-s.ru aa14ab57a3339c4064bd9ae6fad7495b5f.com
absugars.com aac500b7a15b2646968f6bd8c6305869d7.com
abusora.com abctoppictures.net
ad-words.ru academiacsmendoza.org
adhome.biz acheterviagrafr24.com
adservme.com adminshop.com
adsland.men admitad.com
adultfullhd.com
affiliate-fr.com
agapovdg.ru agapovdg.ru
alborzan.com agreda.pluto.ro
alert-hgd.xyz aimayangzhi.com
alfa-medosmotr.ru alienwheel.es
all4invest.info aliexpresscashback.club
allknow.info allbizne.co.ua
amazingninja.com allboard.xobor.de
alt-servis.ru
amateurlivechat.org
ameblo.top ameblo.top
apccargo.com animebox.com.ua
apibatbrowsecom-a.akamaihd.net anti-virus-removal.info
apidigidocketnet-a.akamaihd.net antons-transporte.de
apparel-offer.com ap.senai.br
appfixing.space apioasisspacenet-a.akamaihd.net
applicationg29.com apishops.ru
applyneedy.xyz apispringsmartne-a.akamaihd.net
approvedlocal.co.za arendatora.ru
architecturebest.com artparquet.ru
arendas.net astana.xxxkaz.org
arewater.com atelielembrancaqueencanta.com.br
arquivo.pt auto-moto-elektronika.cz
arvut.org aversis.net
astrochicks.com ayeartoforget.com
atmagroup.ru azartmsl.com
aviapanda.ru b00kmarks.com
azbukadiets.ru backgroundpictures.net
bagcionderlab.com baersaratov.ru
bangla-chat-uk-paralud.ga bag77.ru
bannerspace.com bankcrediti.ru
barbourjackets.online bannerconnect.net
basedecor.ru bannerpower.com
beclean-nn.ru bdsmgalls.net
beget.tech beatifullandscape.co
bestbrainprod.win bedandbreakfast.com
bestempresas.es beetpics.pw
best-mam.ru
bestplacetobuyeriacta.jw.lt bestplacetobuyeriacta.jw.lt
bharatdefencekavach.com bestprofits-there.com
bimmerpost.com bestwrinklecreamnow.com
bitnote.co beta.hotkeys.com
bjetjt.com beyan.host.sk
blackplanet.com bg6s0.com
bidbuy.co.kr
bigcities.org
binaryoptionscops.info
biznesrealnost.ru
bjanshee.ru
blackhatworth.com
blackwitchcraft.ru blackwitchcraft.ru
bluejays-jerseys.us blagovest-med.ru
bmwhighperformers.com blog.f00kclan.de
bonzbuddy.com blogrankers.com
bonzybuddy.com blogs.rediff.com
blpmovies.com
bolegapakistan.com
boole.onlinewebshop.net boole.onlinewebshop.net
boostmyppc.com braindaily.xyz
brains2.biz bristolhotel.com.ua
brus-vsem.ru brokergid.ru
burger-imperia.com bscodecs.com
buscarfoto.com bumble.cheapwebsitehoster.com
business-suggested.tk buyk.host.sk
c-english.ru buynorxx.com
californianews.cf buzzurl.jp
callejondelpozo.es bvlgariring.xyz
carmuffler.net calc-for-credit.ru
carsplate.com canacopegdl.com
cartierlovestore.xyz candy-glam-hp.com
cayado.snn.gr candytiner.com
celebsopics.com capture-room.com
cenoval.ru cashonads.com
centraletermice.us chatroulette.online
ceotrk.com cheapjerseysap.com
cfjump.com chelnytruck.ru
chastnoeporno.com chocoslim.pro
cheapestjerseysonwholesale.com cienum.fr
cheapmoncler.xyz cityadspix.com
cheapsergiorossi.online
cialiswithoutadoctor.net
cinemaenergy-hd.ru
claytransformations.info claytransformations.info
cleanmypc.com clmforexeu.com
clickbank.net cocyq.inwtrade.com
clips.ua.ac.be com-supportcenter.website
coaching-netz.info cometorussia.net
coldfilm.ru compliance-checker.info
communisave.co.za compliance-jessica.top
conboy.us compliance-mark.top
conocer-sanabria.com compucelunlock.net
contentlook.co concordexoticrentals.com
contentsexpress.com
contextualyield.com contextualyield.com
cookingmeat.ru cool-wedding.net
cougfan.info coolingoods.com
counterbot.com coolwallpapers-hd.com
craftburg.ru copypaste.traderzplanet.in
customerpromos-a.akamaihd.net cordstrap.cc
ddsoldes.fr cscwtalkto.site
dealitright.click custodieva.ru
dealwifi.com cutalltheshit.com
dentfidemountpreach.tk d2jsp.org
dailystrength.org
data.vtc.pw
db.speedup.gdn
derevesendeco.com derevesendeco.com
destinationrealestate.com dispo.de
detective01.ru divatraffic.com
digital-video-processing.com dktr.ru
dimkino.ru
dogbreedspicture.net
dokumentalkino.net
dolohen.com dolohen.com
donna7753191.ru domain2008.com
donvito.unas.cz dorratex.tn
downloadkakaotalk.com dotnetdotcom.org
doyouknowtheword-flummox.ml drugstoreforyou.com
dptaughtme.com drupa.com
draniki.org dynainbox.com
droidlook.net e-kwiaciarz.pl
eas-seo.com eas-seo.com
easync.io eldiariodeguadalajara.com
ekaterinburg.xrus.org electro-prom.com
ekspertmed.com elektronischezigarettekaufen1.myblog.de
encodable.com elitepcgames.com
energydiet24.ru englate.com
englishdictionaryfree.com enpolis.ru
eropornosex.ru envaseslotusama.com
escort-russian.com eredijovon.com
eupornstar.info ereko.ru
evening-dating-club.info europages.com.ru
evogarage.com everypony.ru
ewebarticle.info excitacion.info
exmasters.com exlarseva.webblog.es
exoneration-project.us exoneration-project.us
eyemagination.com
eyes-on-you.ga
eywords-monitoring-your-success.com
ezigarettekaufen.myblog.de ezigarettekaufen.myblog.de
ezigarettenkaufen1.pagina.gr ezigarettekaufen2.yolasite.com
f00kclan.de facialporntube.com
factorynightclub.com faracontrol.ir
falopicm.pw fashion.stellaconstance.co
fanoboi.com favornews.com
farm26.ru fighrofacciufreesig.ga
farmprofi.net
fast-torrent.ru
fast-wordpress-start.com
femmesdenudees.com
fickenprivat.info
filmania-x.ru
filmgo.ru
find1friend.com find1friend.com
fiverr.com findpics.pw
fitnesspiks.com
flex4launch.ru flex4launch.ru
floridamobilebillboards.com flowwwers.com
foodcrafts.website
footballfarrago.com
forpostlock.ru forpostlock.ru
forsex.info franecki.net
fototravel.eu free-social-buttons.com
free-fbook-traffic.com freefoto.ca
free-today.com freewebs.com
freemaintenancesysforpcandmac.top fres-news.com
fuck-paid-share-buttons.xyz front.ru
fuckbuddybestgilf.info front.to
fullfileaccess.com game-mmorpg.net
fxund.us game300.ru
galaxy-family.ru games.kolossale.ru
gallery.rennlist.com
gallictures.com
gamerscorps.com
gamevalue7.weebly.com gamevalue7.weebly.com
gaming-journal.com gate5.co.za
gateway.zscalerone.net gavuer.ru
gdcentre.ru gay.adultgalls.com
gepezz.info gazoblok.net.ua
getmiro.com getoutofdebtfree.org
getmyads24.com gheus.altervista.org
goforexvps.com given2.com
goldenggames.com gne8.com
gomusix.com gojiberriess.apishops.ru
goo.ne.jp goldpanningtools.com
goodwriterssales.com goodwinmetals.co
googglet.com gopixdatabase.com
google-liar.ru gotcher.us
goosefishpost.bid gourcy.altervista.org
gosmeb.ru gox.com.ua
gosreg.amchs.ru gq-catalog.gq
graphicwe.org graphicwe.org
greatgrace.ru
green-tea.tv
gsasearchengineranker.xyz gsasearchengineranker.xyz
guardlink.com
gwagka.com
gwhwpxbw.bloger.index.hr gwhwpxbw.bloger.index.hr
hamptonoaks.ca hd-filmy.net
hccoder.info headpress.ru
hdhc.ru herehloadibs.cf
hdimagegallery.net hotblog.top
helicalpile.us houdom.net
hermesbelts.xyz hoverboard360.nl
hledejvshopech.cz hplaserjetpdriver8y.pen.io
home.myplaycity.com ideashome.id
hotdl.in igadgetsworld.com
housemilan.ru imabase.com
hoverboard360.at immobiliaremassaro.com
i-service.kz in-tandem.co
iideaidekonkatu.info ipchicken.com
ilovevitaly.xyz ipornox.xxx
incanto.in.ua ispac.org
indetiske.ya.ru istanbulit.com
indiakino.net itsdp3.com
infospot.pt jacago.com
intermesh.net jaxcube.info
iplogger.org jerseyssportsshop.com
iscblog.info kakadu-interior.com.ua
istripper.com kalb.ru
iwantedmoney.com kenaba.su
izhevsk.xrus.org keywordbasket.com
juliadiets.com keywordsking.com
jwcialislrt.com kichenaid.ru
jwss.cc
karusel-market.ru
katadhin.co
kazan.zrus.org
kerwinandcariza.com
keywordsuggest.org
kiinomaniak.pl kiinomaniak.pl
kinohall.ru kimcurlrvsms.com
komputers-best.ru kino-key.info
kostenloser-sex.com kino2018.cc
krasnodar.ru kinohit1.ru
krasula.pp.ua kit-opt.ru
kreativperlen.ch kongoultry.net
ku6.com konkursov.net
laudit.ru kredit-blog.ru
ledpolice.ru kvartir-remont.biz
letmacworkfaster.world lacapilla.info
lampokrat.ws
laxdrills.com
lechtaczka.net
ledis.top
letslowbefast.today letslowbefast.today
levaquin750mg.blogspot.com letsrepair.in
lion.conboy.us lezbiyanki.net
lion.playtap.us life-instyle.com
liran-locks.com likesdesign.com
live-sexchat.ru liky.co.ua
liveshoppersmac.com linkdebrideur.xyz
ljusihus.se lion.misba.us
lolnex.us lirunet.ru
lonely-mature.com liverpool.gsofootball.com
lonfon.xyz loftdigital.eu
longlifelomilomi.info luciddiagnostics.in
low-format.ru lutherstable.org
luxup.ru m292.info
m.b00kmarks.com mackeeper-center.club
magicplayer-s.acestream.net mahnwachen-helfen.info
mailemedicinals.com
make-money-online.com
manifestyourmillion.com manifestyourmillion.com
marcogrup.com manve.info
marmitaco.cat maridan.com.ua
masthopehomes.com mebeldekor.com.ua
matchpal-a.akamaihd.net
matrixalchemy.com
max-p.men
medicovi.com medicovi.com
merryhouse.co.uk mega-bony-2017.pl
message-warning.net
metaxalonevstizanidine.blogspot.com
mhtr.be
mikrobiologies.ru
mil-stak.com
milblueprint.com milblueprint.com
militarysale.pro minecraft-neo.ru
mobstarr.com minharevisao.com
moesonce.com misandesign.se
monthlywinners.com missis.top
moomi-daeri.com mmstat.com
mobifunapp.weebly.com
money-every-day.com
moneyteam24.com
morepoweronmac.trade
morpicert.pw morpicert.pw
mowser.com moviezbonkerssk.cf
mrmoneymustache.com mtmtv.info
musezone.ru musflashtv.com
mwtpludn.review musicspire.online
myphotopipe.com muz-shoes.ru
myra.top mvpicton.co.uk
myfreetutorials.com
nalogovyykodeks.ru
napalm51.nut.cc
narutonaruto.ru
nerudlogistik.ru
netallergy.ru netallergy.ru
networkcheck.xyz niki-mlt.ru
nfvsz.com nmrk.ru
nicefloor.co.uk notfastfood.ru
nikhilbahl.com nottyu.xyz
nisuturnetdgu.tk novatech.vn
nosecret.com.ua novosibirsk.xrus.org
noumeda.com nufaq.com
nzfilecloud.weebly.com o-dachnik.ru
oddamzadarmo.eu obiavo.kz
offf.info obiavo.net
ochistka-stokov.ru
oecnhs.info
offer.camp
okel.co okel.co
onlainbesplatno.ru online-sbank.ru
onlinebay.ru
onlinemeetingnow.com
onlineporno.site onlineporno.site
ontargetseo.us ooomeru.ru
oz-offers.com ops.picscout.com
paparazzistudios.com.au orenburg-gsm.ru
paphoselectricianandplumber.com orgasmatrix.com
ourville.info
outshop.ru
ozas.net
panicatack.com
partner-cdn.men
partner-print.men partner-print.men
patol01.pw peugeot-club.org
paydayloanslocal.com pflexads.com
pelfind.me pictures-and-images.com
perfectpracticeweb.com pinsdaddy.com
perm-profnastil.ru
perso.wanadoo.es
pestomou.info
petrovka-online.com
photo-clip.ru
picquery.com
picscout.com
picswe.com
picture-group.com
picurams.pw
pinup-casino1.ru
pinwallpaper.top pinwallpaper.top
piter.xrus.org pixgood.com
piuminiita.com plastgran.com
pivka.xyz platesauto.com
popunder.ru podshipniki-ntn.ru
popup-jdh.xyz poisk-zakona.ru
porn555.com pornmania.pl
pornhub-ru.com pornoforadult.com
porno-dojki.net pornoklad.ru
porno-raskazy.ru porodasobak.net
porno-video-chati.ru
pornobrazzers.biz
pornotubs.com
portal-eu.ru
portside.xyz
pourvous.info pourvous.info
preparevideosafesystem4unow.space pr-ten.de
presleycollectibles.com praisong.net
pricheskaonline.ru preparevideosafesystem4unow.site
purplesphere.in privacylocationforloc.com
puserving.com projectforte.ru
questionmarque.ch prostitutki-almata.org
quizzitch.net psa48.ru
putitin.me
puzo2arbuza.ru
quickchange.cc
quickloanbank.com
raavidesigns.com raavidesigns.com
ranking2017.ga redirect.trafficreceiver.club
rating-casino2021.ru remontbiz.ru
realizmobi.com research.ifmo.ru
reelheroes.net resellerclub.com
remont-mobile-phones.ru reward-survey.net
retailwith.com rezeptiblud.ru
rialp.getenjoyment.net rfid-locker.co
ribieiendom.no rfserial.net
royal-betting.net riders.ro
rozalli.com rockma.se
rp9.ru royalads.net
rus-teh.narod.ru sack.net
russian-postindex.ru saltspray.ru
s.lollypopgaming.com santasgift.ml
saecsa.co saratov.xrus.org
salmonfishingsacramentoriver.com sayyoethe.blogspot.co.za
salutmontreal.com sbetodiodnye-lampy.ru
saneyes.com
sawin.beth.webd.pl
sbwealthsolutions.ca sbwealthsolutions.ca
scanner-elena.top sc-specialhost.com
scanner-mary.top scalerite.co.za
scat.porn scenicmissouri.us
searchlock.com seksvideoonlain.com
secret.xn--oogle-wmc.com sel-hoz.com
seminarykansas904.ml seminarykansas904.ml
seo-prof1.xyz seofirmreviewsus.info
seo18.su serdcenebolit.com
seonetwizard.com sex-pr.net
service-core.ru sextracker.com
seryeznie-znakomstva.ru sexuria.net
sex-foto.pw shell-pmr.ru
sexcamamateurchat.com shitmovs.com
sexobzor.info shop.xz618.com
sexystrippe.info shopfishing.com.ua
shitting.pro simple-share-buttons.com
shop-electron.ru site-speed-check.site
shopcheermakeup.info siteexpress.co.il
shoppingjequiti.com.br sitevaluation.com
sildenafilcitratemed.com skincrate.net
sitechecker.pro slomm.ru
slavkokacunko.de smartshoppymac.com
sobecjvuwa.com.ru sobecjvuwa.com.ru
socialsignals24.com social-search.me
sophang8.com soundfrost.org
souvenirua.com
sovinsteel.ru
spb.afora.ru spb.afora.ru
spb.ru sptslmtrafms.com
speedup-my.site
spidtest.space
srdrvp.com srdrvp.com
stackthatbucks.com stanthonyscatholicchurch.org
staff.prairiesouth.ca stard.shop
stairliftsarea.com staynplay.net
start.myplaycity.com stickers-market.ru
statustroll.com stjamesschool.info
stillmiracle.com stmassage.ru
stroyplus.ru streetfire.net
suggest-keywords.com stroiminsk.org
swinginwithme.ru stroydetali.ru
tabakur77.com superlist.biz
telegramdownload10.com surflinksmedical.com
terraclicks.com surgut.zrus.org
swimpool.ca
taraz.xkaz.org
tastyfoodideas.com
tattooreligion.ru
tds-advert002.info
teastory.co
telefonsexi.com
telesvoboda.ru
teslathemes.com teslathemes.com
theautoprofit.ml testingads.pro
themeforest.net thegalerie.eu
thewebsitetemplate.info theheroes.ru
tiandeural.ru thejournal.ru
timer4web.com threecolumnblogger.com
titan-cloud.life timetorelax.biz
tizanidine6mg.blogspot.com titelhelden.eu
tjkckpytpnje.com tizanidinedruginteractions.blogspot.com
tnaionline.org tizanidineduringpregnancy.blogspot.com
toloka.hurtom.com tizanidinehcl4mgtabinfo.blogspot.com
tonerbox.kz tizanidinenarcotic.blogspot.com
tootoo.to tksn.ru
topbestgames.com topanasex.com
totu.info topflownews.com
toursmaps.com topkarkas.com
toyota.7zap.com topsiteminecraft.com
traffique.net torrent-to-magnet.com
trion.od.ua torrents.cd
truck-rece.life toxicwap.com
trudogolik.net tradgardspartner.se
traffic-club.info
traffic2cash.org
traffictrade.life
traffpartners.com
trainoffend.ml
try-rx.com
tube8.com tube8.com
tupper-shop.ru tubeline.biz
turizm.bz tuckermktg.com
turn-up-life.life tula.howotorg.ru
turvgori.ru tvand.ru
tvorozhnaja-zapekanka-recept.ru tvorozhnaja-zapekanka-recept.ru
twu.com.ua twelvevisionspartyofcolorado.com
typer.one u-cheats.ru
u17795.netangels.ru
ublaze.ru ublaze.ru
uhdtv.website ukkala.xyz
uhod-za-sobakoj.ru umityangin.net
ukr-lit.com uptime.com
ukrobstep.com urccvfmc.bloger.index.hr
unimodemhalfduplefw.pen.io urll.eu
unrealcommander.com usbggettwku.ga
uprour.com vacuumcleanerguru.com
usswrite.com ventelnos.com
uytmaster.ru verymes.xyz
vektorpress.ru vezuviy.su
videochat.bz videochat.cafe
vip2ch.com videochat.world
vipps.com.my videochaty.ru
visa-pasport.ru videokrik.net
viteonlusarezzo.it videooko.weebly.com
viveresaniesnelli.it videos-for-your-business.com
vladimir.zrus.org vigrx-original.ru
vucms.com vikistars.com
wallpaperaccess.com villacoloniale.com
wallpaperstock.net vip-file.com
warningwar.ru vipsiterip.org
webextract.profound.net virtualbb.com
wechatdownload10.com vitalads.net
westsextube.com vkmusics.ru
whatsappdownload10.com vkontaktemusic.ru
whatsupinfoley.com vostoktrade.info
whereverdesperate.gq vzube.com
whiteelephantwellington.com warningzscaler.heraeus.com
wieseversa.no waterefficiency.co
wineitudes.wordpress.com weboptimizes.com
wpsecurity.website webpromotion.ae
wpthemedetector.co.uk webshoppermac.com
wsgames.ru website-analytics.online
wstroika.ru website-datenbank.de
www.event-tracking.com websiteaccountant.de
www.kabbalah-red-bracelets.com wildworld.site
www.labves.ru wowas31.ucoz.ru
x-diesel.info wrona.it
xbaboon.com www.arenda-yeisk.ru
x-stars.ru
xblognetwork.com
xboxster.ru xboxster.ru
xn-----7kceclhb4abre1b4a0ccl2fxch1a.xn--p1ai xn----8sbhefaln6acifdaon5c6f4axh.xn--p1ai
xn----7sbaaabaei0cc8aj5bj0bncejx.xn--p1ai xn----itbeirbjbi7bc6bh2d.xn--p1ai
xn----7sbbagbq7bd5aheftfllo4m.xn--p1ai xn--b1addnj3cah.xn--p1ai
xn--90acenikpebbdd4f6d.xn--p1ai xn--b1ag5cfn.xn--p1ai
xxx-treker.ru xn--d1abj0abs9d.in.ua
xxxtube69.com xtraffic.plus
yellowads.men xxart.ru
yginekologa.com xxxdatinglocal.us
yogamatsexpert.com yaderenergy.ru
yottos.com yottos.com
yougame.biz youjizz.com
yoursite.com youtubedownload.org
z-master.ru zigzog.ru
za-fun-offer.com zlatnajesen.com
zacreditom.ru zs2vm.top
zaidia.xhost.ro
zarabotok--doma.ru
zeleznobeton.ru
zirondelli.it
zone-kev717.info
zygophyceous.womanstars.site
zzbroya.com.ua

478
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
1000n1.ru 4replicawatch.net
18ps.ru 8xv8.com
1bet.com a96527gi.beget.tech
1hwy.com aanapa.ru
2kata.ru adventureparkcostarica.com
abclauncher.com aero2.ru
abctoppictures.net aghanyna.com
adultwebhosting.info alekseevec.ru
albuteroli.com alessandraleone.com
alive-ua.com allcredits.su
amateurgalls.com angkortours.vn
amyfoxfitness.com apioutoboxnet-a.akamaihd.net
anlimebel.kiev.ua arclk.net
apelsinnik.website arendovalka.xyz
apiluckyleapnet-a.akamaihd.net asscenihotosidea.blogspot.co.za
apispringsmartne-a.akamaihd.net avon-ukraine.com
apiwoodensealcom-a.akamaihd.net azadnegar.com
apps-analytics.net bbsoldes.fr
ascat.porn bg6s0.com
atlant-auto.info biglistofwebsites.com
ayanaspa.com bioinnovate.co
baltgem.com blockety.co
bazaronline24.ru blog.koreadaily.com
beauty-clinic.ru bonux.nextview.ru
best-seo-solution.com boobsimge.com
bidr.trellian.com brandehk.dk
binomo.com brothers-smaller.ru
binomo.kz buttons-for-website.com
bioscorp.ru buytizanidineonlinenoprescription.blogspot.com
bizcheapjerseyswholesalechina.com camdenmemorials.com
browsepulse-a.akamaihd.net camsex.xxx-cam.webcam
bycontext.com candcstuccoandstone.com
camel-beach.com checkhit.com
candelluxsklep.pl
cartechnic.ru
cdn.walkme.com
cete.ru
chatseo.com
cheapjerseysap.com
chinacheapelitejerseys.com chinacheapelitejerseys.com
choosecuisine.com cibpenonptib.flu.cc
citizenclsdriveri7.pen.io compliance-jess.top
cityadspix.com
clayvasedesigns.tk
cleansearch.net
clickso.com
compliance-jessica.top compliance-jessica.top
compliance-walt.top compliance-john.top
cookie-law-enforcement-ff.xyz confib.ifmo.ru
costablue.xyz contextualyield.com
covetnica.com cookie-law-enforcement-aa.xyz
ddpills.com cougfan.info
de.zapmeta.com cpabegins.ru
dellalimov.com customergrowthsystems.com
dev33.dioniqlabb.se customsua.com.ua
djonwatch.ru czat.wp.pl
dandiyabeats.in
datsun-do.com
dev78.dioniqlabb.se
divatraffic.com
djstools.com
dktr.ru
dkvorota.ru
dneprsvet.com.ua dneprsvet.com.ua
dosugrostov.site
doublepimp.com doublepimp.com
dpihatinh.gov.vn downtuptv.gq
drunkmoms.net e-avon.ru
dumuelave.xyz e-commerce-seo.com
ec-file.info e-commerce-seo1.com
eloconcream.blogspot.com ebonyporn.site
emailaccountlogin.co ebooknovel.club
enternet.ee ednorxmed.com
essaypro.com ekto.ee
essayservicewriting.org electrik-avenue.com
evehemming.blogspot.com.au energydiet24.ru
ezigarettenshop2.myblog.de ezigarettenshop1.yolasite.com
f-online.de
f1nder.org
falopicm.pw falopicm.pw
faz99.com familienzahnaerzte.com
farmingworm.com
femdom.twiclub.in
fettefrauen.net fettefrauen.net
fickluder69.com filkhbr.com
find1friend.com fitness-video.net
floating-share-buttons.com forex-indextop20.ru
floridamhca.org forum20.smailik.org
flowwwers.com foxtechfpv.com
foodcrafts.website free-share-buttons.blogspot.com
foxweber.com funponsel.com
free-fb-traffic.com futbolkisales.ru
fsakhalin.ru gateway.zscalertwo.net
galaxys6manual.info gearcraft.us
gameplexcity.com germetiki.com.ua
gate5.co.za getlamborghini.ga
gavuer.ru globalsurfari.com
gerhardhealer.com go2album.com
gtopstats.com gok-kasten.net
guge.io gonextmedia.com
guidefs.ru goo.ne.jp
healing-dysplasia.ru goodly.pro
heatpower.ru gov.yanao.ru
himgaws.pw hamilton.ca
hol.es help.tpu.ru
homeinns.com hostsshop.ru
houdom.net hoverboardforsaledirect.com
houseofgaga.ru hubbble.com
houtings.xyz immobiliaremassaro.com
howlongdoestizanidinestayinyoursystem.blogspot.com import-sales.com
hoztorg-opt.ru in-tandem.co
humbmosquina.tk intellego.info
hunthillfarmtrust.org intellekt21.ru
hystersister.com investzalog.ru
informatiecentro.be irunfar.com
innodgfdriverhm.aircus.com islandminingsupply.wordpress.com
iomoio.net isotoner.com
iqs.biz.ua istizanidineanopiate.blogspot.com
jagg.info janerikholst.se
javcoast.com jjbabskoe.ru
kamorel.com kinohit1.ru
ketrzyn.pl kiskinhouse.com
keywordglobal.co.za kosynka-games.ru
konteiner24.com koversite.info
ktotut.net kreativperlen.ch
lapolis.it kutikomi.net
letolove.ru kvartir-remont.biz
liky.co.ua laxdrills.com
littlecity.ch lechtaczka.net
liveinternet.ro lescinq.com
localflirtbuddies.com lexaprogeneric.link
lollypopgaming.com libertybilisim.com
lostfilm-online.ru littlesunraiser.com
louisvuittonoutletstore.net lombia.co
lutherstable.org lonfon.xyz
lyngdalhudterapi.no lyngdalhudterapi.no
m0r0zk0-krava.ru m3gadownload.pl
managerpak204.weebly.com macfix.life
mcnamaratech.com machicon-akihabara.info
mebel-alait.ru manualterap.roleforum.ru
meble-bogart.info matchpal-a.akamaihd.net
medi-fitt.hu matpre.top
meet-flirt-dating.com meendo-free-traffic.ga
mesto-x.com missis.top
mistr-x.org
mmofreegames.online mmofreegames.online
mobile.ok.ru mojowhois.com
mojaocena.com
monclerboots.xyz
money-for-placing-articles.com
morefastermac.trade
moroccosurfadventures.com moroccosurfadventures.com
myanyone.net mountainstream.ms
myonlinepayday.co muizre.ru
na15.ru my-cash-bot.co
nakozhe.com myindospace.com
naphukete.ru myra.top
netoil.no mysex21.com
newstudio.tv mysexpics.ru
nina.az national-today-winning-winner.club
noscrapleftbehind.co nikkiewart.ru
nudo.ca niroo.info
nvformula.ru nomuos.it
offers.bycontext.com nonameread45.live
okroshki.ru oz-offers.com
okuos.com paradontozanet.ru
olympescort.com
parajumpersoutlet.online
park.above.com park.above.com
partner-cdn.men photosaga.info
partybunny.ru pinkduck.ga
pastaleads.com pinsdaddy.com
paydayonlinecom.com pintattoos.com
pb-dv.ru polska-poezja.com
pinwallpaper.xyz pooleroadmedicalcentre.co.uk
pixell.club pornogratisdiario.com
plastjulgranar.se pornoreino.com
play.leadzupc.com pornosexrolik.com
playlott.com pozdrawleniya.com
pokemon-go-play.online profitwithalex.info
porn-w.org pussyspace.net
pornhub-ru.com puteshestvennik.com
pornmania.pl qcstrtvt.bloger.index.hr
pornoinn.com quickbuck.com
powc.r.ca.d.sendibm2.com randalljhoward.com
priceg.com rapevideosmovies.com
profitfx.online rednise.com
prosmibank.ru registry-clean-up.net
prostitutki-kiev.org remybutler.fr
pyrodesigns.com.au riralmolamsaca.tk
radiodigital.co runofilms.ru
redirlock.com sabizonline.com
remorcicomerciale.ro saddiechoua.com
renewablewealth.com saltspray.ru
rentalcarnavi.info sanjosestartups.com
research.ifmo.ru scanner-don.top
rightenergysolutions.com.au scanner-elena.top
robot-forex.biz sdsjweb.com
rockma.se search.1and1.com
rukino.org search.xtconnect.com
salutmontreal.com seokicks.de
sape.top serdcenebolit.com
scanner-jane.top sex.hotblog.top
searchinquire.com sharebutton.to
sei80.com shemale-sex.net
seotoolsagency.com shopfishing.com.ua
shinikiev.com.ua sizeplus.work
shopcheermakeup.info skatestick.bid
simply.net slavic-magic.ru
sims-sims.ru smokewithrabbits.com
sisi-go.ru socialmadesimple.com
sivs.ru spaceshipad.com
snsdeainavi.info spitfiremusic.com
solitaire-game.ru stillmiracle.com
soundfrost.org susanholtphotography.com
sovinsteel.ru svarbit.com
space4updating.win the-trader.net
spasswelt.net thexart.club
spravka-medosmotr.ru thfox.com
srecorder.com thruport.com
stroilka.info tizanidineandgabapentin.blogspot.com
support.nopeas.sk tizanidinehydrochloride2mg.blogspot.com
svtrd.com tizanidineingredients.blogspot.com
the-torrent-tracker.blogspot.com tizanidinepain.blogspot.com
thedownloadfromwarez.blogspot.com tizanidinevscyclobenzaprine.blogspot.com
theprofitsmaker.net tonivedu.it
tizanidineinfo.blogspot.com top1-seo-service.com
topclickguru.com topappspro.com
topmira.com torrnada.ru
tpu.ru trailer.cinemaflix.website
traffic100.com trion.od.ua
tsstcorpcddvdwshbbdriverfb.aircus.com triumf-realty.ru
tuberkuleznik.ru trustaffs.com
unitexindia.com ttrraacckkrr.com
uptimechecker.com turvgori.ru
urdoot.win unmaroll.ya.ru
urldelivery.com uptime-gamma.net
uzporno.mobi us-america.ru
vapeface.club v720hd.ru
veles.shop vegascosmetics.ru
veopornogratis.xxx via-energy-cumpara.com
vkonche.com video-chat.love
vkontarkte.com visa-china.ru
vladimir.xrus.org vitanail.ru
vse-pesni.com vladhistory.com
waycash.net vriel.batcave.net
website-speed-up.top vuclip.com
welck.octopis.com web-betting.ru
whatsappdownload10.com webcamdevochka.com
x-rates.ru webix.biz
xclicks.net weburlopener.com
xlovecam.com wildnatureimages.com
xn--80aeahghtf8ac5i.xn--p1ai win-spy.com
xxlargepop.com winwotgold.pl
xxxdatinglocal.us woman-tampon.ru
xxxguitars.com www.pinnacle-bets.com
www.souvenirua.com
www.wohnkabinen-shop.de
wygraj-skiny.win
xdoza.com
xjlottery.com
xn--80ajjbdhgmudixfjc8c5a9df8b.xn--p1ai
y8games-free.com y8games-free.com
yaderenergy.ru yousense.info
yoursearch.me z-master.ru
zeroredirect9.com zooggames.com
zirondelli.it zs2vm.top
zskdla.site

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.13 0; 10.1.1.56 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.11.2711 ### Version: V4.2021.11.2712
### Updated: Fri Nov 12 22:01:13 UTC 2021 ### Updated: Sat Nov 13 10:03:18 UTC 2021
### Bad Bot Count: 626 ### Bad Bot Count: 626
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.11.2711 ### Version: V4.2021.11.2712
### Updated: Fri Nov 12 22:01:13 UTC 2021 ### Updated: Sat Nov 13 10:03:18 UTC 2021
### Bad Bot Count: 626 ### Bad Bot Count: 626
################################################### ###################################################
### Version Information ## ### Version Information ##