vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-03 19:20:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.09.2594 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-09-16 10:09:23 +00:00
parent e512dd9815
commit 4029a12f8c
46 changed files with 82013 additions and 82013 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.09.2593 #### Version: V4.2021.09.2594
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.09.2593 #### Version: V4.2021.09.2594
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -11,7 +11,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2021.09.2593 #### Version: V4.2021.09.2594
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

10118
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2021.09.2593 #### Version: V4.2021.09.2594
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

10122
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

10122
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

10122
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

10122
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 1; 10.1.1.164 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 0; 10.1.1.164 0;
127.0.0.1 0; 127.0.0.1 0;

10122
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 1; 10.1.1.164 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 0; 10.1.1.164 0;
127.0.0.1 0; 127.0.0.1 0;

10122
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 1; 10.1.1.164 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 0; 10.1.1.164 0;
127.0.0.1 0; 127.0.0.1 0;

10122
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2593 2594

10122
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

10122
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.183.34.99 0; 52.173.94.180 0;

10122
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 1; 10.1.1.164 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 0; 10.1.1.164 0;
127.0.0.1 0; 127.0.0.1 0;

10122
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 1; 10.1.1.164 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 0; 10.1.1.164 0;
127.0.0.1 0; 127.0.0.1 0;

10122
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 1; 10.1.1.164 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 0; 10.1.1.164 0;
127.0.0.1 0; 127.0.0.1 0;

10122
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.183.34.99 0; 52.173.94.180 0;

10122
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.183.34.99 0; 52.173.94.180 0;

10122
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 1; 10.1.1.164 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Ecxi Bullseye
Indy Library Iblog
NPbot Morfeus Fucking Scanner
Page Analyzer Piepmatz
Screaming Snapbot
SearchmetricsBot SurveyBot
WebZIP Telesphorep
Whack WPScan
ZyBorg Zeus
crawl.sogou.com facebookscraper

296
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
360Spider 360Spider
404checker
404enemy 404enemy
ADmantX ADmantX
AIBOT
Aboundex Aboundex
Aboundexbot
Acunetix Acunetix
AfD-Verbotsverfahren AhrefsBot
AiHitBot AiHitBot
AllSubmitter Alligator
AlphaBot
Anarchy Anarchy
Anthill
Apexoo
Aspiegel Aspiegel
Asterias
Attach Attach
BLEXBot BBBike
BackDoorBot BDCbot
BDFetch
BackStreet BackStreet
Backlink-Ceck
Bandit Bandit
BetaBot Barkrowler
BatchFTP
Black Hole Black Hole
Blow
BlowFish BlowFish
Boardreader Boardreader
Bolt
BotALot BotALot
Brandprotect Buck
Brandwatch Buddy
BuiltBotTough
BuiltWith BuiltWith
Bullseye
BunnySlippers BunnySlippers
CCBot BuzzSumo
CODE87 CODE87
CSHttp CazoodleBot
CensysInspect Cegbfeieh
CherryPicker
Chlooe Chlooe
Claritybot
CopyRightCheck
Copyscape Copyscape
Craftbot
Crawling at Home Project Crawling at Home Project
CrazyWebCrawler
CrunchBot CrunchBot
CyotekWebCopy
DBLBot
DIIbot DIIbot
DTS Agent
DataCha0s DataCha0s
Demon Demon
DigitalPebble
Dirbuster Dirbuster
Disco DnyzBot
Discobot DomainAppender
DittoSpyder DomainCrawler
DomainSigmaCrawler Dragonfly
DomainStatsBot
EMail Siphon
EMail Wolf EMail Wolf
EasyDL Evil
EirGrabber Exabot
Express WebPictures Express WebPictures
Extractor Extractor
FDM ExtractorPro
EyeNetIE
FHscan FHscan
Fimap
Foobot Foobot
FyberSpider Freeuploader
Genieo FrontPage
GermCrawler Fuzz
Getintent GetWeb
Go-Ahead-Got-It Go!Zilla
GrabNet Gotit
Grabber
Grafula Grafula
GrapeFX GrapeFX
GridBot HEADMasterSEO
HTMLparser HTTP::Lite
Haansoft Haansoft
Heritrix Heritrix
IRLbot Humanlinks
HybridBot
Iblog Iblog
IlseBot IlseBot
Image Fetch Indy Library
InfoNaviRobot
InfoTekies
Intelliseek Intelliseek
InternetSeer Iria
Iskanie IstellaBot
JOC Web Spider Jbrofuzz
JetCar Jetty
JikeSpider
Jorgee
Jyxobot Jyxobot
Kenjin Spider Larbin
Keyword Density
Kinza
LNSpiderguy
LWP::Simple
Leap Leap
LeechGet LeechGet
Lftp LexiBot
Libwhisker Likse
LinkScan LinkWalker
Linkbot Linkbot
Linkdexbot LinksManager
LinqiaMetadataDownloaderBot Lipperhey Spider
Lipperhey Litemage_walker
Lmspider MFC_Tear_Sample
Ltx71
MIDown tool MIDown tool
MIIxpc MQQBrowser
MJ12bot
MSFrontPage
MSIECrawler
Mag-Net
Magnet Magnet
MauiBot Majestic12
Mass Downloader
Masscan
Mb2345Browser
MeanPath Bot
Mediatoolkitbot
MegaIndex.ru
Metauri Metauri
MicroMessenger MicroMessenger
Microsoft Data Access Microsoft URL Control
Mister PiX
Moblie Safari Moblie Safari
Mojeek Mojolicious
NICErsPRO NPbot
Navroad Nameprotect
NearSite NearSite
Nessus NetAnts
NetLyzer NetMechanic
NextGenSearchBot NetZIP
Niki-bot Nettrack
NimbleCrawler Nibbler
Nimbostratus
Ninja Ninja
Nmap Octopus
Nuclei OpenLinkProfiler
Offline Explorer Openfind
OnCrawl
Openvas Openvas
PHPCrawl PECL::HTTP
PageGrabber POE-Component-Client-HTTP
Page Analyzer
PageAnalyzer
PageScorer
PageThing.com PageThing.com
Petalbot Papa Foto
Picscout
Picsearch Picsearch
PictureFinder
Piepmatz Piepmatz
Pixray Pixray
Pockey Pockey
Probethenet
Psbot
Pu_iN
Pump
PyCurl PyCurl
QueryN Metasearch
Quick-Crawler
RSSingBot RSSingBot
RankActive RankActive
RankActiveLinkBot
Rankivabot Rankivabot
RealDownload
Reaper Reaper
RedesScrapy
RepoMonkey RepoMonkey
Ripper
RocketCrawler RocketCrawler
Rogerbot
SEOkicks
SEOkicks-Robot
SEOlyticsCrawler SEOlyticsCrawler
SEOprofiler
SEOstats
SMTBot SMTBot
ScoutJet
Scrapy Scrapy
ScreenerBot ScreenerBot
SearchmetricsBot
Semrush
SentiBot
SeoSiteCheckup SeoSiteCheckup
SeobilityBot Shodan
Seomoz
Siphon Siphon
SiteCheckerBotCrawler
SiteExplorer
Site Sucker Site Sucker
Sitevigil SiteExplorer
SiteLockSpider
SiteSnagger
SiteSucker
SlySearch SlySearch
SocialRankIOBot Snapbot
Spammen Sosospider
Spbot Sottopop
Spinn3r Spinn3r
SputnikBot Sqlmap
Sqworm Sqworm
Steeler
Sucker Sucker
Sucuri T8Abot
Suzuran TeleportPro
Swiftbot
Szukacz
T0PHackTeam
Telesoft Telesoft
The Intraformant
Thumbor
TightTwatBot TightTwatBot
Titan Titan
Toata Toweyabot
Trendiction
Turingos
TurnitinBot
TwengaBot TwengaBot
Twice
Typhoeus
URLy Warning
URLy.Warning
UnisterBot UnisterBot
Upflow Upflow
V-BOT V-BOT
VCI Vagabondo
WPScan VeriCiteCrawler
WBSearchBot
WEBDAV
WISENutbot
WWW-Mechanize WWW-Mechanize
Wallpapers Wallpapers/3.0
WallpapersHD
Web Enhancer Web Enhancer
Web Fetch
Web Pix Web Pix
Web Sauger Web Sauger
WebAuto Web Sucker
WebBandit WebEnhancer
WebCopier
WebFetch WebFetch
WebImageCollector WebFuck
WebReaper
WebSauger WebSauger
WebSucker WebSucker
WebmasterWorldForumBot Webalta
Webshag
Website Quester Website Quester
WebsiteExtractor
WebsiteQuester
Whack
Whatweb Whatweb
Wonderbot Wonderbot
Wprecon Woobot
Wotbox
Xaldon WebSpider Xaldon WebSpider
Zade
Zauba Zauba
Zermelo ZoomBot
ZumBot archive.org_bot
ZyBorg arquivo.pt
autoemailspider cah.io.community
backlink-check
check1.exe check1.exe
cognitiveseo cognitiveseo
crawl.sogou.com crawl.sogou.com
crawler.feedback
demandbase-bot demandbase-bot
evc-batch eCatch
facebookscraper facebookscraper
instabid gopher
internetVista monitor internetVista monitor
magpie-crawler lwp-request
meanpathbot lwp-trivial
netEstate NE Crawler mediawords
polaris version muhstik-scan
oBot
pcBrowser
plumanalytics
ripz ripz
scan.lol s1z.ru
seobility scalaj-http
seoscanners
sexsearcher sexsearcher
sitechecker.pro siteripz
spyfu sp_auditbot
trendiction.com sysscan
trendiction.de webprosbot
voyagerx.com
x09Mozilla x09Mozilla
xpymep1.exe x22Mozilla
zauba.io zgrab

284
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
ADmantX 360Spider
AIBOT 404checker
ASPSeek 404enemy
80legs
ALittle Client
Abonti
AfD-Verbotsverfahren
Aipbot Aipbot
Alexibot
Alligator
Anarchy Anarchy
Ankit Anthill
Apexoo
Aspiegel
Asterias Asterias
Attach
AwarioRssBot AwarioRssBot
BDCbot
BDFetch
BLEXBot BLEXBot
BackDoorBot Badass
Backlink-Ceck BatchFTP
Battleztar Bazinga
BetaBot BetaBot
Bigfoot Black Hole
BlackWidow
Blackboard Blackboard
Blow
BlowFish BlowFish
BotALot Boardreader
Brandprotect Brandprotect
Buck
BuiltBotTough BuiltBotTough
Bullseye
BuzzSumo
CATExplorador CATExplorador
CCBot
CODE87
CSHttp
Calculon Calculon
CazoodleBot CensysInspect
CheTeam CheTeam
CheeseBot CheeseBot
CherryPicker CherryPicker
ChinaClaw ChinaClaw
Chlooe Chlooe
Claritybot
Cloud mapping Cloud mapping
Cocolyzebot
Cogentbot Cogentbot
Collector Copier
Copyscape Copyscape
Cosmos CrunchBot
Craftbot Custo
Crescent
Curious
CyotekWebCopy CyotekWebCopy
DBLBot DIIbot
DSearch
DTS Agent
Demon
Deusu
Devil Devil
DigitalPebble
Dirbuster
Disco Disco
Discobot Discobot
Dispatch Discoverybot
DomainSigmaCrawler DomainSigmaCrawler
DomainStatsBot DomainStatsBot
Dotbot
Download Wonder Download Wonder
Dragonfly Drip
EMail Siphon
Ebingbong
Ecxi Ecxi
EirGrabber EirGrabber
Exabot EyeNetIE
Extreme Picture Finder Firefox/7.0
Ezooms Foobot
FDM
Freeuploader Freeuploader
FrontPage FrontPage
FyberSpider
Fyrebot
Genieo Genieo
GermCrawler
GetWeb
Getintent
Gigabot Gigabot
Gotit Go!Zilla
GrapeFX
GrapeshotCrawler
HEADMasterSEO
HMView
HTMLparser
HTTP::Lite HTTP::Lite
Humanlinks Hloader
HybridBot Indy Library
IDBot InfoNaviRobot
Iblog InterGET
IlseBot IstellaBot
IndeedBot JOC Web Spider
Intelliseek JennyBot
Internet Ninja Jetty
Iria JikeSpider
JamesBOT Jorgee
JetCar
JustView JustView
Keyword Density Keyword Density
Larbin Kozmosbot
Lftp Lanshanbot
LibWeb Leap
LinkScan LinkextractorPro
Linkdexbot LinkpadBot
LinqiaMetadataDownloaderBot
LinqiaScrapeBot LinqiaScrapeBot
Lipperhey Lipperhey Spider
Litemage_walker
Ltx71 Ltx71
MQQBrowser
MSFrontPage
MTRobot MTRobot
Magnet Magnet
Majestic SEO Mail.RU_Bot
Majestic-SEO Majestic-SEO
MarkMonitor MarkMonitor
Mass Downloader Mata Hari
MauiBot MauiBot
MeanPath Bot Mediatoolkitbot
Meanpathbot MicroMessenger
MegaIndex.ru Microsoft Data Access
Moblie Safari Microsoft URL Control
Mr.4x3 Mister PiX
Mojeek
Mojolicious
Msrabot
Musobot Musobot
NPbot NPbot
Navroad Name Intelligence
Nameprotect
NearSite
Nessus Nessus
Net Vampire
NetAnts NetAnts
Netcraft
Nettrack Nettrack
Niki-bot Netvibes
NextGenSearchBot
Nimbostratus Nimbostratus
Ninja
Nmap Nmap
Nuclei
Nutch
Octopus Octopus
Offline Explorer Offline Explorer
OnCrawl Offline Navigator
OrangeBot OpenLinkProfiler
Openvas
OutclicksBot OutclicksBot
Page Analyzer PHPCrawl
POE-Component-Client-HTTP
PageGrabber PageGrabber
PageScorer PageThing.com
Pandalytics Panscient
Papa Foto
Pavuk Pavuk
Petalbot PeoplePal
Pi-Monster Picscout
Picsearch
PictureFinder PictureFinder
Pixray
PleaseCrawl PleaseCrawl
Pockey Pockey
ProWebWalker ProWebWalker
Psbot
QueryN Metasearch QueryN Metasearch
Quick-Crawler Quick-Crawler
RankActiveLinkBot RankActive
RankFlex
RankingBot
RankingBot2 RankingBot2
RankurBot Re-re
ReGet ReGet
RebelMouse SEOkicks-Robot
Recorder
RocketCrawler
Rogerbot
SEOlyticsCrawler SEOlyticsCrawler
SEOstats
SISTRIX SISTRIX
SMTBot
SalesIntelligent SalesIntelligent
ScanAlert ScanAlert
Scrapy ScoutJet
Screaming Screaming
ScreenerBot
ScrepyBot ScrepyBot
SentiBot Searchestate
Seomoz Semrush
Shodan SeoSiteCheckup
Site Sucker SeobilityBot
SiteCheckerBotCrawler SiteExplorer
SiteLockSpider SiteLockSpider
SiteSnagger SiteSnagger
Sitebeam Siteimprove
Sitevigil
SmartDownload SmartDownload
Snake SpaceBison
Snoopy SpankBot
Sosospider Steeler
Spammen Sucuri
Spbot SuperHTTP
Sqlmap
Sqworm
SuperBot
Surfbot
SurveyBot SurveyBot
Suzuran Suzuran
T0PHackTeam TeleportPro
Telesoft Thumbor
Telesphoreo Titan
Toata Toata
Toweyabot
Trendiction
Turingos Turingos
Turnitin Turnitin
Typhoeus URLy Warning
UnisterBot
Upflow
VB Project
VCI VCI
VeriCiteCrawler Vacuum
Voil Vagabondo
WWW-Mechanize VelenPublicWebCrawler
WWW::Mechanize WEBDAV
WWWOFFLE WWW-Collector-E
WallpapersHD Wallpapers
Web Auto
Web Enhancer Web Enhancer
Web Sauger Web Sauger
WebAuto Web Sucker
WebBandit
WebEnhancer WebEnhancer
WebFetch
WebFuck WebFuck
WebGo IS WebGo IS
WebLeacher WebLeacher
WebPix
WebReaper WebReaper
WebSauger WebSauger
WebStripper WebStripper
WebWhacker Webalta
WebmasterWorldForumBot Webshag
WebsiteExtractor Whack
WebsiteQuester
Webster
Whacker Whacker
Widow Whatweb
Wotbox Xenu
Xaldon WebSpider
Zade Zade
Zauba
Zermelo Zermelo
Zitebot
ZmEu ZmEu
ZyBorg ZyBorg
archive.org_bot
arquivo-web-crawler arquivo-web-crawler
arquivo.pt arquivo.pt
autoemailspider autoemailspider
check1.exe check1.exe
crawler.feedback crawl.sogou.com
crawler4j
domainsproject.org domainsproject.org
eCatch gopher
heritrix
lwp-request lwp-request
lwp-trivial
magpie-crawler magpie-crawler
meanpathbot meanpathbot
mediawords
muhstik-scan muhstik-scan
page scorer
probe-image-size probe-image-size
ripz ripz
s1z.ru s1z.ru
serpstatbot scalaj-http
sitechecker.pro scan.lol
siteripz seobility
seostar
sogouspider sogouspider
sp_auditbot tAkeOut
spyfu trendiction.de
trendiction.com ubermetrics-technologies.com
voyagerx.com
webmeup-crawler webmeup-crawler
webprosbot
x22Mozilla
xpymep1.exe xpymep1.exe
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
1qingdao.com blog.koreadaily.com
bagsonsale.online domain-tracker.com
dlya-android.org domain2008.com
jumptap.com doxysexy.com
kerei.ru e-buyeasy.com
monsterdivx.tv mydownloadengine.com
onlinemeetingnow.com notfastfood.ru
orion-code-access.net office2web.com
orion-v.com thecounter.com
pricheskaonline.ru ytmnd.com

924
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
1688.com 00it.com
1millionusd.xyz 101lesbian.xyz
4inn.ru 108shot.com
4rent.ru 10bet.com
4senses.co 123kuma.com
7minuteworkout.com 14b.info
883zy.com 1flag.co.za
98oi.ru 1pamm.ru
aaa38852e886ac4af1a3cff9b47cab6272.com 1xbet4you.com
abcdeg.xyz 2015god.org
accessoires-mode.in 2345.com
4pp13.com
51.la
7a2rnuey1tw9ar.ru
7search.com
acortarurl.es
adclickservice.com
addray.pro
addtoadd.men
adelly.bg adelly.bg
adnotbad.com adhome.biz
adtech.us adman.se
adultnet.in adprotect.net
advertisingtag.net ads.tremorhub.com
aerogo.com adverster.com
afslankpillen2017nl.eu advertex.info
ahmedabadwebs.com advokat-grodno.by
algerianembassy.co.in aksonural.ru
alive-ua.com albuteroli.com
all-number.com allblogroll.com
alltheviews.com allknow.info
alphahoverboards.com allproblog.com
alphavisions.net alot.com
amateurgalls.com alpharma.net
ampower.me alveris.ru
amyfoxfitness.com amazingninja.com
anaseracresar.tk ameblo.jp
android4fun.org amospalla.es
angigreene.com anal-acrobats.com
apartment.ru apartamentwroclaw.eu
apccargo.com api.stathat.com
apidiamondatanet-a.akamaihd.net apiappenableinfo-a.akamaihd.net
apiitsthirteende-a.akamaihd.net apilinkswiftco-a.akamaihd.net
apple.com-cleaner.systems apiwebwebgetcom-a.akamaihd.net
appsaurus.com apple.com-webbrowsing-security.review
artdestshop.eu arate.ru
artparquet.ru asrvvv-a.akamaihd.net
aucoinhomes.com audiobangout.com
aussie-prizes.com avitocars.ru
autoplate.info avkzarabotok.info
availit.weebly.com awaybird.ru
avek.ru awency.com
avtovolop.ru badavit.com.ua
azazaporn.com banki76.ru
b-styles.xyz bannertesting.com
bag77.ru batanga.net
baixar-musicas-gratis.com bbtec.net
bankcrediti.ru bestserials.com
bdsmgalls.net big-cash.net
beremenyashka.com
best-mam.ru
beyan.host.sk
bigpictures.club bigpictures.club
bingo8888.com binaryoptionscops.info
bizcheapjerseyswholesalechina.com bitporno.sx
bizru.info bjorkbacken.nu
bjgugu.net.ua blavia.00author.com
blackcurranthumidifiers.site
blackhatworth.com
blobar.org
blockety.co
blog.remote-computer.de
blogig.org blogig.org
boazpower.com blogs.rediff.com
bosefux.esy.es bloke.com
bradleylive.xyz bluerobot.info
braip.com.br bmw.afora.ru
buchananshardware.com bolezniorganov.ru
buckspar.gq bonky.biz
buqayy0.livejournal.com brillianty.info
buqyxa.rincian.info businesn.men
bus-offer.com butstrap.space
buypanicdisorderpill.com buyessay3.blogspot.ru
buytizanidineonline.blogspot.com buyfriend.ru
callejondelpozo.es buyk.host.sk
cardiosport.com.ua callstevens.com
cartierreplica.pw calstaterealty.us
cashonads.com cam-kontakte.org
cayado.snn.gr carabela.com.do
centraletermice.us cardul.ru
cfacarrosserie74.com cartechnic.ru
cheap-pills-norx.com casinorewards.info
cheapcigarettesc.info cb.iphantom.com
cheapmoncler.xyz cbb1smartlist12.click
chiblackhawks-jerseys.com cdnanalytics.xyz
chimiver.info cdncash.com
cialiswithoutadoctor.net cdncash.org
cleanmypc.com chocoslim.pro
clickhype.com christianlouboutinsaleonline.us
coaching-netz.info cialisovercounteratwalmartusa.com
coldfilm.ru clipartnew.com
com-onlinesupport.site clksite.com
cmhomestayagency.com
commoncrawl.org
communisave.co.za
compliance-andrew.top compliance-andrew.top
compliance-jessica.top compliance-barak.xyz
compliance-mary.top compliance-viktor.top
cottageofgrace.com construmac.com.mx
courtshipgift.com cookielawblog.wordpress.com
cubs-jerseys.us copenergo.ru
customcatchcan.com coprofam.org
darmebel.com.ua covid-schutzmasken.de
davidovic.info crackguru.tk
demon-tweeks.com culmimg.pw
dentfidemountpreach.tk cv.wallhade.co
deplim.com cypernhuset.se
dfiles.me dame-ns.kz
danceuniverse.ru
dcj-nn.ru
deadlinkchecker.com
demenageur.com
dentalpearls.com.au
design-ideas.info
diamond-necklace.info
digest-project.ru
digital-sale.su
dirtpics.pw
discovertreasure-a.akamaihd.net discovertreasure-a.akamaihd.net
diusyawiga.tk dms-sw.ru
div.as dnepropetrovsk.mistr-x.org
divatraffic.com doctissimo.fr
divku.ru drpornogratisx.xxx
doctormakarova.ru earncash.com.ua
doeco.ru ekn-art.se
domain.webkeyit.com ekto.ee
donna7753191.ru elektrozigarette2.wordpressy.pl
downvids.net eluxer.net
drugstoreforyou.com elvel.com.ua
dtm-spain.com elvenar.com
dumpsmania24.com emailaccountlogin.co
dvr.biz.ua
dvrlists.com
dyt.net
e-commerce-seo.com
eachdayisagift.review
earthmagic.info
eas-seo.com
ed-shop01.ru
edudocs.net
eets.net
election.interferencer.ru
elitedollars.com
elmifarhangi.com
emergencyneeds.org emergencyneeds.org
enginebay.ru empirepoker.com
envaseslotusama.com energy-ua.com
erotik-kostenlos.net enge-fotzen.info
erotische-geschichten-xxl.com eorogo.top
etm-consult.de eralph.tk
eyemagination.com erectile.bid
ezofest.sk erotikstories.ru
face.hostingx.eu esnm.ru
falcon-images.blogspot.com etenininrade.ga
fastfixing.tech eurosamodelki.ru
fealq.com evening-dating-club.info
fighrofacciufreesig.ga extremepornos.net
figuringmoneyout.com extremez.net
filesmonster.porn f-loaded.de
financepoints.eu feeriaclub.ru
femmesdenudees.com
fettefrauen.net
fickenbumsen.net
fickkontakte.org
fidelityfunding.com
fitnesspiks.com
flash4fun.com
flashahead.info
flipper.top
floridahuntingfishingadventures.com
floridamobilebillboards.com floridamobilebillboards.com
flowwwers.com fordsonmajbor.cf
footbalive.org forensicpsychiatry.ru
forex-procto.ru formseo.com
forum.doctissimo.fr formularz-konkurs.tk
forum.poker4life.ru fotopop.club
fr.netlog.com free-share-buttons.blogspot.com
franch.info free-share-buttons.top
franecki.net freelinkbuilding.website.tk
freenode.info
freeseedsonline.com
freewebs.com freewebs.com
freshmac.space freshdz.com
frivgame250.com fusoradio.info
fullfileaccess.com galaxy-family.ru
funnymama.com galaxyflowers.ru
fx-brokers-review.com gamezblox.com
gagrasector.ru gazobeton-p.com.ua
game300.ru gcup.ru
gamebackyard.com gdcentre.ru
gaygalls.net generalporn.org
germetiki.com.ua genericlowlatencyasiodriverhq.aircus.com
getresponse.com get-free-social-traffic.com
gg-arena.ru getmyads24.com
ggiaro.com getyourimage.club
gigapeta.com ghostvisitor.com
goatse.ru gk170.ru
goodwriterssales.com glass-msk.ru
googlefeud.com goldandcard.ru
googleseo.com.tr golden-catalog.pro
gosreg.amchs.ru gosreg.amchs.ru
graphics8.info gotorussia.com
greatdealshop.com gratuitxblcodes.com
greatzip.com
greendream.com.ua
greenidesign.co
grupografico-pilar.com.ar
guardlink.com guardlink.com
guod.me gwagka.com
hawaiisurf.com gwhwpxbw.bloger.index.hr
hdapp1008-a.akamaihd.net gymi.name
hamacapty.com
heartofbeijing.blogspot.com heartofbeijing.blogspot.com
hebr.myddns-flir.com hermesbracelets.xyz
help.tpu.ru hermesreplica.pw
herehloadibs.cf hiwibyh.bugs3.com
hitcpm.com hmywwogw.bloger.index.hr
hjaoopoa.top hotblog.top
homemade.gq hotdl.in
homemature.net houseofrose.com
hostcritique.com hulfingtonpost.com
hotelcrocenzi.sm i4track.net
hotloans.ru ido3.com
hoztorg-opt.ru incest-ru.com
ideoworld.org
ie.57883.net
igrovyeavtomaty777.ru
ilmen.net
incomekey.net
indianmedicaltourismshop.com indianmedicaltourismshop.com
int.search.mywebsearch.com infoupdate.org
intelhdgraphicsgtdrive6w.metroblog.com innodgfdriverhm.aircus.com
iopeninghours.co.uk insta-add.pro
iqoption.com instabid.tech
iqoption.pro intellego.info
iskalko.ru intellekt21.ru
iplogger.org
irkutsk.zrus.org
itag.pw itag.pw
izhevsk.xrus.org janavibekken.no
jovencitas.gratis jasonpartington.com
javidol.site
jerseychinabizwholesale.us
jerseyssportsshop.com
johannesburgsingles.co.za
justucalling32211123456789.tk justucalling32211123456789.tk
karaganda.xkaz.org jyrxd.com
kareliatobacco.ru kambasoft.com
kartiniresto.com kazrent.com
katjimej.blog.fc2.com kevblog.top
kino-rating.ru
kino-rf.ru kino-rf.ru
kino2018.cc kiwe-analytics.com
konica.kz klumba55.ru
konturkrasoty.ru kmgamex.cf
krasnodar.ru kndxbkdx.bloger.index.hr
krasnodar.xrus.org komp-pomosch.ru
krynica.info konpax.com
ku6.com kopihijau.info
krasivoe-hd.net
krassh.ru
kumuk.info
kustanay.kz kustanay.kz
ladov.ru kw21.org
lakomka.com.ua ladsup.com
lancheck.net lampokrat.ws
landmania.ru laparfumotec.com
landscapebackgrounds.blogspot.com larger.io
languagecode.com laxob.com
law-check-seven.xyz letolove.ru
lego4x4.ru
lezbiyanki.net
lignofix.ua lignofix.ua
lion.bolegapakistan.com lion.snapmap.us
live-sexchat.ru live-cam6.info
livingroomdecor.info liveinternet.ro
locatellicorretor.com.br livesex-amateure.info
lmrauction.com
lockerz.com lockerz.com
lombia.com lovasszovetseg.hu
longadventure.com luckyshop.net.ua
lookingglassemb.com lunchrock.co
love-baby.cz m-google.xyz
lubetube.com mac-shield.com
lumb.co mactechinfo.info
lunamedia.co magazintiande.ru
m0r0zk0-krava.ru magda-gadalka.ru
macrotek.ru maladot.com
maju.bluesalt.co mapquestz.us
manualterap.roleforum.ru maranbrinfo.com.br
marmitaco.cat marketland.ml
matsdale.com mathgym.com.au
meet-flirt-dating.com matrixalchemy.com
mega-bony2017.pl mebel-vstroika.ru
meble-bogart.info
mediaoffers.click
meetlocalchicks.com
mega-bony-2017.pl
meltwater.com
metalonly.info
metaxalonevstizanidine.blogspot.com metaxalonevstizanidine.blogspot.com
mir-betting.ru microstatic.pl
missclub.info minet.club
mmoguider.ru minharevisao.com
mmostrike.ru mir-business-24.ru
mister-shop.com
mixx.com
mmofreegames.online
mobplayer.net mobplayer.net
mobplayer.ru moivestiy.biz
mobstarr.com monclerjacketsoutlet.pw
moi-glazki.ru monetizer.com-01.site
monclercheap.xyz morf.snn.gr
monclervests.xyz moskva.nodup.ru
money-for-placing-articles.com
monitorwebsitespeed.com
monsterdivx.com
moxo.com moxo.com
mturkcontent.com mp3s.club
muabancantho.info mrvideospornogratis.xxx
myblogregistercm.tk music7s.me
myghillie.info mwtpludn.review
mylovelibrabry.com myhealthcare.com
mymobilemoneypages.com myhitmp3.club
myonigroup.com myhydros.org
myplaycity.com mylesosibirsk.ru
mysex21.com nakrutka.cc
nakozhe.com nalogovyy-kodeks.ru
namenectar.com national-today-winning-winner.club
narco24.me naturalpharm.com.ua
neks.info negociosdasha.com
news-speaker.com negral.pluto.ro
nextbackgroundcheck.gq net-radar.com
netanalytics.xyz
netoil.no
newstraveller.ru
next-dentists.tk
nextlnk12.com nextlnk12.com
nikitabuch.com nfvsz.com
nippon-bearings.ru ntic.fr
nofreezingmac.click obiavo.ru
novatech.vn odoratus.net
nowtorrents.com offtopic.biz
oballergiya.ru okel.co
offer.wpsecurity.website olvanto.ru
office2web.com onclickpredictiv.com
okonich.com.ua online-zaymy.ru
okroshki.ru onlineporno.site
old-rock.com optibuymac.com
onescreen.cc osb.se11.ru
onlineku.com owohho.com
onlinemeetingnow.com oxford-book.com.ua
onload.pw painting-planet.com
onlythegames.com paramountmarble.co.uk
optom-deshevo.ru parlament.biz
osagonline.ru partnerads.men
osoznanie-narkotikam.net pavlodar.xkaz.org
p-business.ru petsblogroll.com
palvira.com.ua peugeot-club.org
patol01.pw pflexads.com
pdns.cz phormchina.com
petedrummond.com photo-clip.ru
pewit.pw photosaga.info
physfunc.ru phpdevops.com
piuminiita.com picmoonco.pw
pixgood.com picsforkeywordsuggestion.com
pl-top.pl pigrafix.at
playboyfiles.xblog.in piulatte.cz
pk-services.ru
pl-vouchers.com
podshipniki-nsk.ru
podshipniki-ntn.ru
popugaychiki.com popugaychiki.com
popup-fdm.xyz porn555.com
pornofeuer.com pornhub-forum.ga
pornonik.com pornhub-ru.com
pornosexrolik.com pornoklad.ru
pornoted.com
pornozhara.com
portnoff.od.ua
portside.cc portside.cc
pp-budpostach.com.ua pourvous.info
pr-ten.de pozdrawleniya.ru
priceg.com preparevideosafesystem4unow.site
proprostatit.com primedice.com
prostitutki-tolyatti.org privatefx.all4invest.info
prostitutki-yaroslavlya.org prom23.ru
psa48.ru prospekt-st.ru
pulseonclick.com pyramidlitho.webs.com
pushdata.sendpulse.com questionmarque.ch
pussyfleet.com quickloanbank.com
py100.ru quit-smoking.ga
quick-offer.com r-e-f-e-r-e-r.com
razyboard.com rankexperience.com
redirlock.com recipedays.ru
reeyanaturopathy.com remontvsamare.su
registrysweeper.com remote-dba.de
remont-fridge-tv.ru robot-forex.biz
remont-mobile-phones.ru rogervivierforsale.com
rep-am.com roma-kukareku.livejournal.com
rimedia.org rossanasaavedra.net
robertefuller.com russian-postindex.ru
rock-cafe.info sack.net
roll123.com saddiechoua.com
rospromtest.ru samoiedo.it
rusprostitute.com sandhillsonline.com
russkie-gorki.ru scanner-alexa.top
sajatvelemeny.com scanner-donald.top
sale-japan.com
sbetodiodnye-lampy.ru
scanner-george.top
scanner-john.top scanner-john.top
search.xtconnect.com scanner-mark.top
searchencrypt.com sdelai-prosto.ru
secretscook.ru seadragonherbery.com
senger.atspace.co.uk
seo-platform.com seo-platform.com
seoheap.com
seolab.top
seomarketings.online
seonetwizard.com
seoprofiler.com
servethis.com
servicecenter.co.ua
servisural.ru servisural.ru
sexcamamateurchat.com sex.hotblog.top
sexxdate.net sexvporno.ru
shama-rc.net sherlock.se
shinikiev.com.ua shodanhq.com
silvercash.com shopsellcardsdumps.com
simon3.ru signal03.ru
sirpornogratis.xxx simplepooltips.com
sisiynas.ru sinel.info
site-speed-check.site site.ru
sizeplus.work site5.com
skanninge.se slayerlife.com
sklad-24.ru sledstvie-veli.net
slujbauborki.ru smartbalanceworld.com
smallseotools.com soblaznu.net
smstraf.ru socialmadesimple.com
snip.tw
snjack.info
snowplanes.com
sobecjvuwa.com.ru
social-s-hhh.xyz
socialmonkee.com socialmonkee.com
sofit-dmd.ru sogimlecal.tk
spy-app.info space2019.top
spylog.com spitfiremusic.com
srdrvp.com spywareit.com
star61.de stard.shop
stathat.com
staynplay.net staynplay.net
stopnarco.ru steame.ru
store-rx.com steebook.com
stpolice.com stop-zavisimost.com
strana-krasoty.ru strag-invest.ru
stroiminsk.org streetfire.net
stroyhelp-dv.ru stroy-portal22.ru
surcentro.com strv.se
studiokamyk.com.pl
sudexpert66.ru
suhanpacktech.com
superfish.com
superinterstitial.com
surfbuyermac.com surfbuyermac.com
surintech.ac.th surintech.ac.th
tackletarts.co swinger-mobil.net
technopellet.gr swsociety.se
taboola.com
taihouse.ru
tanieaukcje.com.pl
tattomedia.com
teguh.info teguh.info
teletype.in the-trader.net
templates.franklinfire.co thetoiletpaper.com
texbaza.by thewebsitetemplate.info
thenews-today.info timdreby.com
thepornsex.org tizanidinedosageforsleep.blogspot.com
theprofitsmaker.net tizanidineduringpregnancy.blogspot.com
thesmartsearch.net tizanidineoral.blogspot.com
thewomenlife.com tizanidinerecreationaluse.blogspot.com
titslove.yopoint.in tonivedu.it
tizanidinedruginteractions.blogspot.com toonfamilies.net
tizanidinerestlesslegsyndrome.blogspot.com topappspro.com
tizanidinevscyclobenzaprine.blogspot.com
tizanidinewithdrawalsymptoms.blogspot.com
tizanidinezanaflex.blogspot.com
token-lab.org
toondinsey.com
top10-way.com
topads.men
topkarkas.com
torture.ml torture.ml
tourcroatia.co.uk touchmods.fr
track112.site traffic2cash.org
trafaret74.ru traffmonster.info
tramadolandtizanidine.blogspot.com trustl.life
truck-land.life tv-spoty.info
trucri.me
trumpetedextremes.com
tupper-posuda.ru
tvand.ru
tvoystartup.ru tvoystartup.ru
uac.net.au tx41tclega.ru
unblocksit.es umg-stroy.ru
undergroundcityphoto.com umnovocaminho.com
unionmarkt.de unisexjewelry.org
upupa.net unlimitdocs.net
urzedowski.eu vacuumcleanerguru.com
vacances-voyages.info vancleefreplica.pw
valmetrundan.se
vapeface.club
vclicks.net
vereo.eu
via-energy-acquistare.com via-energy-acquistare.com
video-hollywood.ru video-chat.in
videochat.ph videogamesecrets.com
vietimgy.pw vids18.site
vinylvault.co.uk virtuagirl.com
viteonlusarezzo.it
vitoriacabos.com
vmnmvzsmn.over-blog.com vmnmvzsmn.over-blog.com
vod.com.ua vod.com.ua
vodaodessa.com voloomoney.com
voditeltrezviy.ru vremya.eu
voluumtracker1.com webmasters.stackexchange.com
vysigy.su websiteexplorer.info
wait3sec.org websites-reviews.com
warezkeeper.com wedding-salon.net
warningwar.ru
weblo.com
weekes.biz.tc weekes.biz.tc
welcomeauto.ru westermarkanjou.se
wemedinc.com wildcattube.com
winterclassichockeyjerseys.com workle.website
wiosenny-bon-1500.pl wstroika.ru
wordpress-crew.net xcc24.pl
www.arenda-yeisk.ru xmladserver.com
www.solartek.ru xmlinde.com
www.timer4web.com xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai
wzgyyq.com xn----7sbbahaq9bb5afgiqfliv4m.xn--p1ai
x-musics.com xn----9sbubg3ambdfl1j.xn--p1ai
xboxster.ru xn----dtbndd4ae7eub.top
xn----7sbahjd3btneuw1joc.xn--p1ai xn--80ab4aa2g.xn--p1ai
xn----7sbifcamovvfggw9d.xn--p1ai xn--e1afanlbnfckd7c3d.xn--p1ai
xn----8sbarihbihxpxqgaf0g1e.xn--80adxhks xtube.com
xn----itbkqkfiq.xn--p1ai xxxguitars.com
xn--80aodinpgi.xn--p1ai xxxmania.top
xn--h1aakne2ba.xn--p1ai y8games-free.com
xn--h1ahbi.com.ua yachts-cruise.info
xn--l1aengat.xn--p1ai yatrk.xyz
xtraffic.plus yellocloud.be
yeartwit.com you-shall-not-pass.is74.ru
yellowstonevisitortours.com yousense.info
yottos.com zarabiaj-dzis.pl
youtoner.it zastenchivosti.net
zahvat.ru zeroredirect5.com
zetmaster.ru znakom.sibtest.ru
zhacker.net zoodrawings.com
znakomstva-moskva77.ru
znaniyapolza.ru
zocaparj.kz
zoompegs.com
zryydi.com
zzbroya.com.ua

484
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
101flag.ru 03e.info
1314dh.com
1bet.com
2020iscoming.info
34.gs 34.gs
abiente.ru 45en.ru
acmebtn.ml 5elementov.ru
adman.gr 70casino.online
admitad.com above.com
adzpower.com abovetherivernc.com
aktobe.xkaz.org adanih.com
akuhni.by adexprts.com
alienwheels.de ads.tremorhub.com
afmuseum.com
alarmobninsk.ru
alf-img.com
allornamenti.com allornamenti.com
allsilver925.co.il alphacarolinas.com
allwomen.info amazingpic.net
ameblo.jp amigobulls.com
animaltoplist.com anime.dougasouko.com
anticrawler.org applicationg29.com
apimegabrowsebiz-a.akamaihd.net aquarium-pro.ru
apispringsmartne-a.akamaihd.net asia-forum.ru
ascat.porn asmxsatadriverin.aircus.com
asophoto.com ausergrubhof.info
asrvvv-a.akamaihd.net auto-zapchasti.org
autowebmarket.com.ua awency.com
aviav.co ayakino.net
azartmix.com babs.com.ua
azte.ch beget.tech
b2b-lounge.com bget.ru
baixar-musicas-gratis.com bharatdefencekavach.com
baltgem.com biglistofwebsites.com
battleforupdating.site bimmerpost.com
beachfix.co bkgr.se
bet-prognoz.com blogqpot.com
bitporno.sx bloke.com
blog.yam.com bongacams.com
boc.kz bonky.biz
bolezniorganov.ru browsepulse-a.akamaihd.net
brains2.biz butstrap.space
btnativenav.com buyscabiescream.com
buy-forum.ru bycontext.com
buydissertation.net calc-for-credit.ru
candlehandmade.com cand.jp
cdn.walkme.com cartierjusteunclou.xyz
cete.ru cartierlovestore.com
cheapmarmot.online cartierreplica.pw
cicaki.net cartiertrinity.xyz
cl.s7.exct.net caveavins.fr
clash-clans.ru chelnytruck.ru
coaching-netz.info cibpenonptib.flu.cc
coin-hive.com cjmc.info
com-secure.download clubfashionista.com
cornomase.win colobit.biz
crest3d.ru compliance-brian.xyz
cypernhuset.se cookie-law-enforcement-hh.xyz
danschawbel.com corta.co
data1.scopich.com cpabegins.ru
dailystrength.org
datadepths.com datadepths.com
deadlinkchecker.com daydream-studio.ru
dekorkeramik.ru defenderxtactical.com
diesel-parts28.ru demian.kz
directivepub.com demon-tweeks.com
discountliv.com den-noch24.ru
dneprsvet.com.ua design-lands.ru
dodlive.mil dichvuvesinhhanoi.com
dosugrostov.site dirtpics.pw
easytuningshop.ru domain2008.com
ebonyporn.site domcran.net
education-cz.ru downloadkakaotalk.com
elektrischezigarettekaufen2.cowblog.fr eachdayisagift.review
elvel.com.ua elektroniksigaraankara.info
en.altezza.travel elitesportsadvisor.com
essay-edu.biz emediate.eu
etm-consult.de etur.ru
f012.de everytalk.tv
fantasticpixcool.com ez8motelseaworldsandiego.com
farmprofi.net ferieboligkbh.dk
filesclub.net fermersovet.ru
filesdatabase.net filerockstar298.weebly.com
findsexguide.com filmfanatic.com
fishtauto.ru florida-tourism.net
forensicpsychiatry.ru focalink.com
fortunejack.com foodcrafts.website
frameimage.org forexgb.ru
freewhatsappload.com fortevidyoze.net
funnel.co.za freevpn.space
funnymama.com fun-mobi.pl
gabeshop.ru fym.com.ru
galeria-zdjec.com gaypornmovie.net
gayxperience.com gemgrab-a.akamaihd.net
getlamborghini.ga generic-pills-online.com
ghostvisitor.com gepezz.info
gidonline.one glasof.es
gk170.ru globatur.ru
glastecfilms.com.my glogow.pl
gne8.com grupografico-pilar.com.ar
h2monline.com hdfreeporno.net
hol.es hem.passagen.se
home.myplaycity.com herokuapp.com
homeart.space hildinghr.se
hornymatches.com hoporno.com
hosting-tracker.com hostingclub.lk
hothot.ru hplaserjetpdriver8y.pen.io
hotsocialz.com hptwaakw.blog.fc2.com
hreade.com hscsscotland.com
ico.re hundejo.com
ifirestarter.ru igru-xbox.net
iloveitaly.ru igtools.club
imitex-plus.ru iideaidekonkatu.info
inet-traffic.com incest-ru.com
infowarcraft.ru innoslicon.com
insurple.com iqupdatetmz.win
itag.pw irkutsk.zrus.org
itch.io iskalko.ru
javxxx18.com iwantmyfreecash.com
jerseyssportsshop.com janettabridal.com
kalb.ru jerseywholesaleelitestore.com
kandidos.com junglenet-a.akamaihd.net
kazinogames.lv kazan.xrus.org
ketrzyn.pl kazrent.com
kinoduh.ru keywordsuggests.com
kopihijau.info klikbonus.com
koronirealestate.gr labvis.host.sk
kountrylife.com ladsup.com
krasnodar-avtolombards.ru laexotic.com
laserpen.club lampokrat.ws
lida-ru.com laudit.ru
linkbux.ru letslowbefast.site
linkpulse.com liumimgx.pw
lion.playtap.us lockerz.com
lionking-1994.blogspot.com magazin-pics.ru
liveshoppersmac.com magicalfind-a.akamaihd.net
livingcanarias.com maju.bluesalt.co
lombardfinder.ru marcoislandvacations.net
luchshie-topcasino.ru matpre.top
magento-crew.net medictube.ru
makis.nu megagrabber.ru
marketingvici.com mezaruk.info
matsdale.com microstatic.pl
mebelcomplekt.ru
members.ghanaweb.com
michaelkorsoutletstore.net
microsearch.ru
militarysale.pro
mirmedinfo.ru mirmedinfo.ru
mjchamonix.org missclub.info
monclerboots.xyz missvietnam.org
more-letom.ru mojpregled.com
mozello.ru mojpreskumanie.com
mrwhite.biz moneyviking-a.akamaihd.net
musirc.com moonci.ru
mydearest.co mrbojikobi4.biz
mydownloadengine.com msk-diplomat.com
mykings.pw musezone.ru
na15.ru muz-baza.net
naphukete.ru muznachas-service.ru
netanalytics.xyz mydownlodablefiles.com
nhl17coins.exblog.jp myghillie.info
nlfjjunb5.ru mysex21.com
nodding-passion.tk nalogovyy-kodeks.ru
nsatc.net nextrent-crimea.ru
okmedia.sk nippon-bearings.ru
optom-deshevo.ru nmrk.ru
p-business.ru obiavo.by
painting-planet.com obiavo.su
pandarastore.top oda.as
partner-stop.men online-hit.info
parvezmia.xyz osagonline.ru
patol01.pw otbelivanie-zubov.com
pearlisland.ru paidonlinesites.com
pechikamini.ru partners-ship.pro
phobia.us pay2me.pl
physfunc.ru pc-test.net
picmoonco.pw petedrummond.com
pinkduck.ga popunder.net
pivka.xyz porno-dojki.net
pizdopletka.club pornstartits.xblog.in
pochemychka.net positive2b.ru
popup-jdh.xyz printdirectforless.com
pornowarp.info profolan.pl
pornpost.in pushdata.sendpulse.com
posible.net pyramidlitho.webs.com
potoideas.us quick-offer.com
prodvigator.ua reimageplus.com
promgirldresses.xyz remont-ustanovka-tehniki.ru
pufip.com rennlist.com
puteshestvennik.com rfd-split.hr
quality-traffic.com rossanasaavedra.net
rada.ru royal-investments.net
rank-checker.online sa-live.com
redirectingat.com sajatvelemeny.com
reklamuss.ru sampleletters.net
remontbiz.ru sankt-peterburg.nodup.ru
ros-ctm.ru scanner-don.top
s8-nowy-wygraj.comli.com seadragonherbery.com
samoiedo.it seccioncontrabajo.com
sarf3omlat.com seksotur.ru
satoristudio.net seo-tools-optimizing.com
scanner-jane.top seofied.com
score-ads.men seorussian.ru
se-welding.ru shopwme.ru
semaltmedia.com sildenafil-tadalafil.info
seo18.su similardeals.net
sex-znakomstva.online siteexpress.co.il
sexkontakteao.info snow.nvr163.com
shmetall.com.ua soc-proof.su
sierraapps.com sovetogorod.ru
skanninge.se sstroy44.ru
smokewithrabbits.com stard.shop
social-search.me strangeduckfilms.com
socialsignals24.com sugarkun.com
soochi.co suggest-keywords.com
sotechco.co
spaceshipad.com
ssconstruction.co
suchenindeutschland.com
support.nopeas.sk support.nopeas.sk
suture.co sureone.pro
swiped.su susanholtphotography.com
takeprofitsystem.com swimpool.ca
tattomedia.com the-torrent-tracker.blogspot.com
thebluenoodle.com thecoral.com.br
tiandeural.ru thefarmergame.com
traffmonster.info thewomenlife.com
timetorelax.biz
tizanidinedrugtest.blogspot.com
tizanidinerestlesslegsyndrome.blogspot.com
trafaret74.ru
tsan.net
tupper-posuda.ru
tytoona.com
uggbootsoutletsale.us
uginekologa.com uginekologa.com
uprour.com uk-zheu20.ru
unpredictable.ga
uptime-alpha.net uptime-alpha.net
urdoot.win urldelivery.com
uroffer.link varbergsvind.se
us-america.ru varikozdok.ru
vizag.kharkov.ua ventelnos.com
weblo.com vepad.com
welck.octopis.com video-hollywood.ru
wikes.20fr.com voronezh.xrus.org
wrz0iuebwhp5fg.freeddns.com wladimirpayen.com
ws.ampower.me worldoffiles.ru
www.bookmaker-bets.com www.888.com
www.jbetting.com www.pinnacle-bets.com
xjrul.com x-stars.ru
xlolitka.com
xn----itbkqkfiq.xn--p1ai xn----itbkqkfiq.xn--p1ai
xn--80aaafbn2bc2ahdfrfkln6l.xn--p1ai xn--80aaajbdbddwj2alwjieei2afr3v.xn--p1ai
xn--e1aggki3c.xn--80adxhks xn--80atua3d.xn--p1ai
xnxx699.com xtube.com
xwatt.ru xxxmania.top
yeartwit.com yuarra.pluto.ro
yebocasino.com zoogdiesney.com
yellocloud.be
yottos.com
zaidia.xhost.ro
zaloadi.ru
zigarettenonlinekaufen2.bloog.pl
zocaparj.kz

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.164 0; 10.1.1.164 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.09.2593 ### Version: V4.2021.09.2594
### Updated: Wed Sep 15 22:01:18 UTC 2021 ### Updated: Thu Sep 16 10:03:28 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.09.2593 ### Version: V4.2021.09.2594
### Updated: Wed Sep 15 22:01:18 UTC 2021 ### Updated: Thu Sep 16 10:03:28 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##