vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-03 11:10:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.08.3323 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-08-31 10:09:22 +00:00
parent 37fce0885d
commit cf2d0ea451
46 changed files with 83097 additions and 83097 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.08.3322 #### Version: V4.2022.08.3323
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.08.3322 #### Version: V4.2022.08.3323
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.08.3322 #### Version: V4.2022.08.3323
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

10252
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.08.3322 #### Version: V4.2022.08.3323
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

10256
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

10256
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

10256
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

10256
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 1; 10.1.0.220 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 0; 10.1.0.220 0;
127.0.0.1 0; 127.0.0.1 0;

10256
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 1; 10.1.0.220 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 0; 10.1.0.220 0;
127.0.0.1 0; 127.0.0.1 0;

10256
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 1; 10.1.0.220 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 0; 10.1.0.220 0;
127.0.0.1 0; 127.0.0.1 0;

10256
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3322 3323

10256
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

10256
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.232.152.41 0; 40.121.133.119 0;

10256
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 1; 10.1.0.220 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 0; 10.1.0.220 0;
127.0.0.1 0; 127.0.0.1 0;

10256
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 1; 10.1.0.220 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 0; 10.1.0.220 0;
127.0.0.1 0; 127.0.0.1 0;

10256
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 1; 10.1.0.220 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 0; 10.1.0.220 0;
127.0.0.1 0; 127.0.0.1 0;

10256
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.232.152.41 0; 40.121.133.119 0;

10256
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.232.152.41 0; 40.121.133.119 0;

10256
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 1; 10.1.0.220 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
CSHttp Barkrowler
CherryPicker CrazyWebCrawler
Cloud mapping EMail Siphon
Curious Nimbostratus
Go-Ahead-Got-It RebelMouse
HEADMasterSEO Sqworm
Image Fetch Toata
MQQBrowser Trendictionbot
Moblie Safari linkdexbot
mediawords muhstik-scan

294
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com
360Spider 360Spider
404enemy 80legs
Abonti
Aboundexbot Aboundexbot
Acunetix
AfD-Verbotsverfahren
AhrefsBot AhrefsBot
Aipbot Anarchie
Alexibot
AllSubmitter
Alligator
Anarchy
Ankit Ankit
Anthill
Apexoo
Aspiegel Aspiegel
Atomseobot Asterias
AwarioSmartBot Attach
BBBike BBBike
BDCbot BDCbot
BDFetch BDFetch
BLEXBot BLEXBot
BackDoorBot
BackStreet BackStreet
BackWeb Backlink-Ceck
BacklinkCrawler BacklinkCrawler
Barkrowler Barkrowler
BatchFTP BatchFTP
BetaBot BetaBot
Bitacle
Black Hole Black Hole
Blow Blackboard
Bolt Bolt
BotALot BotALot
Brandprotect Brandprotect
Brandwatch Brandwatch
BuiltWith
Bullseye Bullseye
BunnySlippers CODE87
BuzzSumo CSHttp
CCBot Calculon
CazoodleBot
Cegbfeieh Cegbfeieh
CensysInspect
CheTeam
CheeseBot
CherryPicker CherryPicker
ChinaClaw ChinaClaw
Citoid Chlooe
Claritybot
Cocolyzebot
Cogentbot
Copier Copier
Copyscape
Craftbot Craftbot
Crawling at Home Project Crawling at Home Project
Crescent CrazyWebCrawler
CrunchBot
Curious Curious
Custo Custo
CyotekWebCopy
DBLBot DBLBot
DataCha0s DTS Agent
DatabaseDriverMysqli
Demon
Deusu Deusu
Dirbuster Dirbuster
Disco
Discobot
Discoverybot Discoverybot
Dispatch
DittoSpyder DittoSpyder
DomainCrawler DnBCrawler-Analytics
Domains Project Domains Project
Download Wonder Dotbot
Dragonfly Dragonfly
EMail Wolf
Ecxi
EirGrabber EirGrabber
EroCrawler EroCrawler
Express WebPictures Exabot
ExtLinksBot
ExtractorPro ExtractorPro
EyeNetIE EyeNetIE
Ezooms FDM
FemtosearchBot FHscan
Fimap Firefox/7.0
Foobot
Fuzz
GT::WWW GT::WWW
GalaxyBot
GermCrawler
GetRight GetRight
GetWeb GrapeFX
Gigabot GridBot
Go-Ahead-Got-It HTTrack
Gotit
GrabNet
GrapeshotCrawler
HMView
HTMLparser
HTTP::Lite
Haansoft Haansoft
HaosouSpider Harvest
Havij Heritrix
Humanlinks Hloader
HybridBot IRLbot
IDBot
Iblog
Id-search Id-search
InfoNaviRobot Image Fetch
InfoTekies
Iria Iria
JOC Web Spider IstellaBot
Jbrofuzz JamesBOT
JetCar Jorgee
Jyxobot Kenjin Spider
Keybot Translation-Search-Machine
Lanshanbot Lanshanbot
Larbin Larbin
LeechFTP
LeechGet
LexiBot
LibWeb LibWeb
Libwhisker LinksManager
Likse Lmspider
LinkWalker
LinkextractorPro
Lipperhey
Litemage_walker
MFC_Tear_Sample MFC_Tear_Sample
MTRobot MQQBrowser
MSIECrawler
Mag-Net
Magnet Magnet
Mail.RU_Bot
Majestic-SEO Majestic-SEO
Majestic12 Majestic12
MarkMonitor MarkMonitor
MeanPath Bot MauiBot
Mb2345Browser
Meanpathbot Meanpathbot
MegaIndex.ru MicroMessenger
Microsoft URL Control Moblie Safari
Mister PiX Morfeus Fucking Scanner
Mojeek Mozlila
NPbot Msrabot
Name Intelligence Musobot
Nameprotect
Navroad Navroad
Needle Net Vampire
NetAnts
NetLyzer NetLyzer
Netcraft Nikto
Netvibes
Nibbler
Niki-bot
NimbleCrawler NimbleCrawler
Nmap Ninja
Not
Nuclei
Nutch Nutch
Offline Explorer Octopus
OnCrawl OnCrawl
OpenLinkProfiler OpenVAS
Openfind OrangeBot
OrangeSpider
OutclicksBot OutclicksBot
OutfoxBot OutfoxBot
PECL::HTTP Pandalytics
PageGrabber Papa Foto
Pavuk Petalbot
PeoplePal Picscout
Pi-Monster Pimonster
PictureFinder Probethenet
Piepmatz Psbot
PleaseCrawl PyCurl
ProWebWalker
QueryN Metasearch QueryN Metasearch
RankActive RSSingBot
RankActiveLinkBot RankActiveLinkBot
RankingBot RankFlex
Rankivabot RankingBot2
Re-re
ReGet ReGet
RedesScrapy RealDownload
Reaper
RebelMouse
RepoMonkey RepoMonkey
Ripper Ripper
Rogerbot
SEOkicks-Robot SEOkicks-Robot
SISTRIX ScanAlert
SalesIntelligent
Scanbot Scanbot
ScoutJet
Scrapy Scrapy
Searchestate ScreenerBot
Seekport Seekport
SemanticJuice SemanticJuice
SemrushBot
SeobilityBot
Site Sucker Site Sucker
SiteExplorer SiteExplorer
SiteLockSpider SiteLockSpider
SiteSucker
Sitebeam
Sitevigil Sitevigil
SlySearch SlySearch
Snapbot SmartDownload
SocialRankIOBot SocialRankIOBot
Sociscraper
Spammen
SpankBot SpankBot
Spanner Sqlworm
Spbot Suzuran
Spinn3r
SputnikBot
Sucuri
SuperHTTP
Szukacz Szukacz
T0PHackTeam TeleportPro
Telesoft TightTwatBot
Telesphorep Titan
The Intraformant Toata
TheNomad
Thumbor
Toweyabot Toweyabot
Tracemyfile Turnitin
True_Robot
TurnitinBot TurnitinBot
TwengaBot Typhoeus
Upflow Vagabondo
V-BOT
VCI
VeriCiteCrawler
Virusdie
VoidEYE
Voil
Voltron Voltron
WASALive-Bot WASALive-Bot
WEBDAV WISENutbot
WPScan WPScan
WWW::Mechanize WWWOFFLE
Wallpapers/3.0 Wallpapers
WeSEE WallpapersHD
Web Fetch Web Fetch
Web Fuck Web Pix
Web Sucker
WebCollage WebCollage
WebCopier
WebEnhancer WebEnhancer
WebFuck WebLeacher
WebPix WebSauger
WebStripper
WebSucker WebSucker
Webalta Webalta
WebmasterWorldForumBot
Website Quester
WebsiteExtractor
WebsiteQuester WebsiteQuester
Whack
Who.is Bot Who.is Bot
Widow
WiseGuys Robot WiseGuys Robot
Zeus Woobot
ZoomBot Xaldon WebSpider
Xaldon_WebSpider
Xenu
YoudaoBot
Zauba
Zitebot
ZmEu
ZumBot ZumBot
ZyBorg ZyBorg
arquivo.pt
autoemailspider autoemailspider
backlink-check
cah.io.community
check1.exe
clark-crawler clark-crawler
coccocbot crawl.sogou.com
cognitiveseo
crawler.feedback
dataforseo.com
demandbase-bot
eCatch eCatch
gopher evc-batch
heritrix heritrix
linkdexbot ips-agent
meanpathbot meanpathbot
mediawords
oBot
page scorer page scorer
polaris version polaris version
scan.lol ripz
seocompany.store scalaj-http
sexsearcher seoscanners
sitechecker.pro sitechecker.pro
siteripz
sogouspider
sp_auditbot
sysscan sysscan
webgains-bot tAkeOut
trendiction.de
webmeup-crawler webmeup-crawler
webprosbot zauba.io
xpymep1.exe
zgrab

302
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
404checker 01h4x.com
80legs AIBOT
ADmantX ALittle Client
Acunetix Abonti
AhrefsBot Alexibot
Anarchy AllSubmitter
Anarchie
Anarchy99
Ankit Ankit
Anthill Aspiegel
Apexoo
Asterias Asterias
Attach Atomseobot
AwarioRssBot AwarioRssBot
AwarioSmartBot BLEXBot
BBBike
BackDoorBot BackDoorBot
BackStreet BackStreet
Backlink-Ceck
Badass
Bandit Bandit
Barkrowler Barkrowler
BatchFTP BatchFTP
Battleztar Bazinga
BetaBot BetaBot
Blackboard Bitacle
Boardreader Black Hole
Bolt Brandprotect
BotALot Brandwatch
BuiltBotTough Buck
BuiltWith BuiltWith
BunnySlippers Bullseye
BuzzSumo BuzzSumo
CCBot
Cegbfeieh Cegbfeieh
CheeseBot
CherryPicker
Chlooe Chlooe
Cloud mapping Citoid
Cocolyzebot Claritybot
Cogentbot Copier
CopyRightCheck CopyRightCheck
Copyscape
Cosmos
Crawling at Home Project
CrazyWebCrawler CrazyWebCrawler
Crescent Custo
Curious
CyotekWebCopy CyotekWebCopy
DSearch Discobot
DataCha0s Dispatch
Demon DittoSpyder
DigitalPebble DnBCrawler-Analytics
Disco DomainSigmaCrawler
DomCopBot Domains Project
DomainAppender ECCP/1.0
Dotbot
Dragonfly
EMail Siphon EMail Siphon
Ebingbong EMail Wolf
EirGrabber
Exabot
Express WebPictures
ExtLinksBot ExtLinksBot
Extractor
Ezooms
FHscan FHscan
Fimap FemtosearchBot
Flunky Flunky
Freeuploader Foobot
G-i-g-a-b-o-t FrontPage
GermCrawler FyberSpider
GalaxyBot
Genieo
GetRight
GetWeb GetWeb
Go-Ahead-Got-It
GoZilla
GrabNet
Grafula Grafula
GrapeshotCrawler GrapeshotCrawler
HEADMasterSEO
HMView
HTMLparser
Haansoft Haansoft
Havij HaosouSpider
Heritrix
HonoluluBot HonoluluBot
Humanlinks
HybridBot HybridBot
IDBTE4M Iblog
IDBot
Id-search
IlseBot IlseBot
Image Sucker IndeedBot
Indy Library Iria
InfoTekies Iskanie
InterGET JOC Web Spider
InternetSeer JamesBOT
JennyBot Jbrofuzz
JetCar JetCar
JustView JikeSpider
Kenjin Spider Kenjin Spider
LWP::Simple Keyword Density
Lanshanbot Kozmosbot
Lftp Larbin
Likse Leap
LinkScan LeechGet
LinkWalker LieBaoFast
Linkbot Linkbot
LinkextractorPro
LinkpadBot LinkpadBot
LinksManager
Lipperhey Lipperhey
Litemage_walker Litemage_walker
MFC_Tear_Sample
MJ12bot MJ12bot
MSIECrawler MQQBrowser
Mail.RU_Bot
Majestic SEO Majestic SEO
MarkMonitor Majestic12
MarkWatch
Mass Downloader Mass Downloader
Masscan Masscan
Mata Hari Mata Hari
MeanPath Bot MauiBot
MegaIndex.ru MegaIndex.ru
MicroMessenger Metauri
Microsoft Data Access Minefield
Mister PiX Mojeek
Mojolicious
Morfeus Fucking Scanner Morfeus Fucking Scanner
Mr.4x3
Msrabot
Nameprotect Nameprotect
Navroad
NearSite NearSite
Needle Net Vampire
NetZIP NetLyzer
NetMechanic
NetSpider
Netcraft Netcraft
NextGenSearchBot Nikto
Niki-bot Nmap
Ninja Nuclei
Offline Explorer Offline Explorer
OnCrawl
OpenLinkProfiler
OpenVAS OpenVAS
Openvas
OrangeBot OrangeBot
OrangeSpider
OutclicksBot OutclicksBot
OutfoxBot
PHPCrawl PHPCrawl
POE-Component-Client-HTTP Page Analyzer
PageGrabber PageGrabber
PageScorer
PageThing.com PageThing.com
Panscient
Papa Foto
PeoplePal PeoplePal
Petalbot
Picsearch Picsearch
PictureFinder PictureFinder
Piepmatz
Pimonster Pimonster
Pixray
PleaseCrawl PleaseCrawl
Pockey
ProPowerBot
ProWebWalker
Probethenet Probethenet
Psbot Pump
PyCurl PyCurl
QueryN Metasearch QueryN Metasearch
Quick-Crawler Quick-Crawler
RankingBot
RankingBot2 RankingBot2
Rankivabot
ReGet
RebelMouse RebelMouse
Recorder
RepoMonkey RepoMonkey
Ripper
Rogerbot
SEOlyticsCrawler
SEOprofiler SEOprofiler
Scanbot SEOstats
ScoutJet
Scrapy
ScreenerBot ScreenerBot
SentiBot SemanticJuice
SeoSiteCheckup
SeobilityBot SeobilityBot
Seomoz Siphon
SiteCheckerBotCrawler
SiteExplorer SiteExplorer
SiteLockSpider
SiteSnagger SiteSnagger
SiteSucker Sitebeam
Siteimprove Sitevigil
Snapbot SlySearch
Sociscraper SmartDownload
Snake
SocialRankIOBot
Sosospider
Sottopop Sottopop
Spanner SpaceBison
Spbot SpankBot
Steeler Sucker
Stripper
SuperHTTP SuperHTTP
Surfbot
SurveyBot
Suzuran
T0PHackTeam
T8Abot T8Abot
TeleportPro TeleportPro
Telesoft Telesoft
Telesphorep Telesphoreo
TheNomad The Intraformant
TightTwatBot
Titan Titan
Toata
Toweyabot Toweyabot
Trendiction True_Robot
Turingos Turnitin
TurnitinBot Twice
URLy Warning
UnisterBot
V-BOT
VB Project VB Project
VeriCiteCrawler
Virusdie Virusdie
VoidEYE VoidEYE
Voil
WEBDAV WEBDAV
WPScan WISENutbot
WWW-Collector-E WWW-Collector-E
WWWOFFLE WWWOFFLE
Wallpapers Wallpapers
WallpapersHD
Web Auto
Web Fetch Web Fetch
Web Pix Web Sucker
WebCopier WebBandit
WebFetch WebPix
WebGo IS WebSauger
WebImageCollector Webshag
WebReaper Website Quester
WebStripper WebsiteExtractor
Webalta Whatweb
WebsiteQuester
WiseGuys Robot
Woobot
Xaldon_WebSpider Xaldon_WebSpider
Xenu
YoudaoBot YoudaoBot
Zade Zade
Zauba Zitebot
Zermelo
Zeus
ZoomBot ZoomBot
ZumBot ZumBot
arquivo-web-crawler adscanner
backlink-check arquivo.pt
clark-crawler coccocbot
crawl.sogou.com com.plumanalytics
crawler.feedback crawler.feedback
crawler4j crawler4j
dataforseo.com demandbase-bot
eCatch domainsproject.org
evc-batch gopher
instabid instabid
internetVista monitor ips-agent
isitwp.com isitwp.com
linkdexbot
lwp-trivial
magpie-crawler
meanpathbot meanpathbot
page scorer oBot
plumanalytics plumanalytics
polaris version
ripz ripz
satoristudio.net s1z.ru
scalaj-http
scan.lol scan.lol
seocompany.store seoscanners
serpstatbot
sexsearcher sexsearcher
sysscan sitechecker.pro
siteripz
sogouspider
spyfu
tAkeOut tAkeOut
trendiction.com
voyagerx.com voyagerx.com
webgains-bot x09Mozilla
webmeup-crawler
webpros.com
webprosbot
zauba.io
zgrab zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
articlesdirectoryme.info 8558l.jobs.net
cscwtalkto.site aac500b7a15b2646968f6bd8c6305869d7.com
dotnetdotcom.org chocolateslim-original.com
exci.ru errorfixing.space
iboss.com iqs.biz.ua
keywordbasket.com musicvidz.ru
macresource.co.uk video-chat.in
myanyone.net wzgyyq.com
parajumpersstore.online xblognetwork.com
tizanidineandcipro.blogspot.com xn--80ajjbdhgmudixfjc8c5a9df8b.xn--p1ai

930
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
101flag.ru 00webcams.com
123movies.love 1008.su
13tabs.com
1kdailyprofit.me
1qingdao.com
24-ak.ru 24-ak.ru
4pp13.com 256bit.by
51.la 4inn.ru
7a2rnuey1tw9ar.ru 4kpics.rocks
7yue.org 4senses.co
8jn.dating-app.ru 4ureyesonly.com
51unlim.ru
5u.com
72-news.com
7ooo.ru
8558l.jobs.net
991mostfm.co.id 991mostfm.co.id
a1security.com.ua aa8780bb28a1de4eb5bff33c28a218a930.com
aa08daf7e13b6345e09e92f771507fa5f4.com abusora.com
aa8b68101d388c446389283820863176e7.com
aab94f698f36684c5a852a2ef272e031bb.com
acc.eu.org acc.eu.org
actices.com adrenalin-stk.ru
actionnooz.com
actulite.com
adamoads.com
adanih.com
adclickthru.net
addtoadd.men
admitad.com
adtology.com adtology.com
adultwebhosting.info adultfullhd.com
advertisingtag.net adultgalls.com
affrh2015.com afmuseum.com
alevinefesleri.com akuhni.by
alfa9.com ald2014.org
algerianembassy.co.in alekseevec.ru
allwidewallpapers.com alfabot.xyz
alphaforum.ru alfapro.ru
alpinism.ru alldubai.biz
ameblo.top allkrim.com
android-vsem.org anal-acrobats.com
analnoeporno.tv
android4fun.org
angigreene.com angigreene.com
anonymizeme.pro apelsinnik.website
anonymousfox.co applyneedy.xyz
antons-transporte.de architecturebest.com
apartment.ru artparquet.ru
apessay.com asdfg.pro
apisecretsaucebi-a.akamaihd.net asrv-a.akamaihd.net
approvedlocal.co.za atofilms.com
arewater.com atout-energie-69.com
arquivo.pt autobrennero.it
aruplighting.com automobile-spec.com
arvut.org
asseenontvonline.ru
autotrends.today
avtochehli.by avtochehli.by
awency.com
azbuka-mo.ru azbuka-mo.ru
azbukafree.com babyfactory.fr
backgroundpictures.net backuperwebcam.weebly.com
bamo.xsl.pt balook.com
bankiem.pl battle.net
bannerswap.com bayareaaeroclub.org
baoxaydung.com.vn bear.gotcher.us
basedecor.ru besthoro.ru
beauby.ru betterhdporn.com
begalka.xyz betune.onlinewebshop.net
beget.tech biglistofwebsites.com
bestfortraders.com biketank.ga
bizcheapjerseyswholesalechina.com bingoporno.com
bizlist.com.de binomo.com
bolezniorganov.ru bitcoinremote.com
bookmark4you.com bitcoins-live.ru
brandehk.dk bitnote.co
browsepulse-a.akamaihd.net blackle.com
browserprotecter.com blackplanet.com
btnativenav.com boeuklad.com
bukleteg.xyz bonux.nextview.ru
burger-imperia.com brains2.biz
buy-cheap-online.info braintobe.win
buyessaynow.biz brainzpod.win
buyessayonline19.blogspot.ru brandedleadgeneration.com
buyk.host.sk brandimensions.com
cakemediahq.com.au break-the-chains.com
camsex.xxx-cam.webcam bristolhotel.com
candiceloves.us buchananshardware.com
candypeople.se bugof.gq
carson.getenjoyment.net butstrap.space
cartierreplica.pw buy-meds24.com
cartierreplica.top buyclomidonlaine.com
catalogs-parts.com buydissertation.net
cbb1smartlist12.click buzz-porno.info
cbcseward.com buzzonclick.com
cfcl.co.uk callejondelpozo.es
changde.58.com calstaterealty.us
cheapmoncler.pw canyougethighofftizanidine.blogspot.com
checkm8.com carabela.com.do
cherrypointplace.ca chatroulette.world
children-learningreading.info chatseo.com
chinacheapelitejerseys.com cheap-trusted-backlinks.com
chinajerseyswholesalecoupons.com cheapbelstaff.online
ciekawinki.pl cheapjerseysap.com
clickbank.net christianlouboutinoutlet.win
clickintext.net ciceros.co
com-onlinesupport.host comeondog.info
cometorussia.net compliance-alex.xyz
comissionka.net
compliance-fred.xyz compliance-fred.xyz
compliance-margo.top compliance-george.top
compliance-mark.top crazyboost.pro
compliance-mary.top creditnation.ru
compliance-viktor.top cupday.com
compliance-walter.top customergrowthsystems.com
cornerstone-countertops.com cutalltheshit.com
cornomase.win cwetochki.ru
cougfan.info dailyfinancefix.com
counter.yadro.ru damedingel.ya.ru
cscwtalkto.site darkbooks.org
cvety24.by data1.scopich.com
danceuniverse.ru destinationrealestate.com
daretodonate.co detective01.ru
dating-app.ru dev.citetick.com
deda-moroza-zakaz.ru diesel-parts28.ru
delfin-aqua.com.ua digital-sale.su
derevesendeco.com
deutschehobbyhuren.net
dieswaene.com
diffbot.com
dimfour.com
dirtpics.pw
djonwatch.ru
djstools.com djstools.com
dlya-android.org domainsatcost.com
dnsrsearch.com domcran.net
doeco.ru dptaughtme.com
dogclothing.store draniki.org
domain-submit.info druzhininevgeniy63.blogspot.com
domain.webkeyit.com dumuelave.xyz
domashneeruporno.com e-biznes.info
dorratex.tn e-poker-2005.com
dotmass.top eandsgallery.com
dotnetdotcom.org easycommerce.cf
dustyorate.com edge.sharethis.com
earn-from-articles.com eduinfosite.com
ecxtracking.com elektrischezigarette1.blog.pl
editmedios.com elite-sex-finders.com
elektronischezigarettekaufen2.tumblr.com enskedesquashclub.se
elektrozigarette2.wordpressy.pl erotag.com
elektrozigaretten1.tumblr.com escortslet.net
enews.tech essaypro.com
englate.com evaashop.ru
ertelecom.ru excitacao.com
esnm.ru exoclick.com
eu-cookie-law.blogspot.com extlabs.io
familyholiday.ml extlinks.com
faracontrol.ir ezigarettekaufen2.mpbloggar.se
farmingworm.com ezigarettenkaufen1.pagina.gr
fashion-mk.net ezrvrentals.com
fastcrawl.com family1st.ca
fdzone.org fanrto.com
figuringmoneyout.com fapgon.com
fil.ru fellowshipoftheminds.com
filmidivx.com fickblock18.com
find1friend.com fidelityfunding.com
finejewelryshop.ru fillmewithhappiness.com
financeloan.us
fingerprintjs.com
firesub.pl
firstdrugmall.ru firstdrugmall.ru
flytourisme.org flowersbazar.com
focalink.com fok.nl
fordsonmajbor.cf forpackningsutveckling.se
forex-procto.ru fortevidyoze.net
formseo.com fotoxxxru.com
forsex.info free-social-buttons.xyz
foxweber.com free-stock-illustration.com
freegamesplay.online free-video-tool.com
freelifetimefuckbook.com freejabs.com
freetangodownload.com fxtips.ru
freeuploader.ml game-top.su
fuckmill.com gamedayassist.com
g33.org gamerextra.com
gabeshop.ru gamesprite.me
galaxy-family.ru gate5.co.za
gallerylisting.com gdebestkupit.ru
gavuer.ru ge0ip.com
gdzkurokam.ru geckoandfly.com
gelezki.com geile-lelly.eu
get-free-traffic-now.com genetworx.com
gg-arena.ru get-free-social-traffic.com
girls-ufa.ru getoutofdebtfree.org
girlsatgames.ru getridofstretchmarks.org
gettpromos.com
getyourimage.club
gigixo.com
glastecfilms.com.my glastecfilms.com.my
gogalleryawesome.com goldenggames.com
goodhousekeeping.com goodly.pro
goodwriterssales.com gosarhivrt.ru
googlsucks.com gotomontenegro.net
gourcy.altervista.org graphics8.info
gsbs.com.ua greatzip.com
gta-club.ru hdpixion.com
haikuware.com hdwalls.xyz
hamacapty.com hebr.myddns-flir.com
haveinc.xyz hobbyhuren24.net
hchha.com hotelcrocenzi.sm
hd720kino.ru hoterika.com
heliko.no hothot.ru
hildinghr.se hotsocialz.com
hit-kino.ru husky-shop.cz
hotdl.in igadgetsworld.com
howlongdoestizanidinestayinyoursystem.blogspot.com images-graphics-pics.com
hptwaakw.blog.fc2.com immigrational.info
humbmosquina.tk inmate-locator.us
iccornacircri.cf insurple.com
ico.re intelhdgraphicsgtdrive6w.metroblog.com
ideashome.id invitefashion.com
ihc2015.info io9.com
ikearugs.xyz
ilmexico.com
imagefinder.site
images.gyffu.com
imicrovision.com
insta-add.pro
internetproviderstucson.com
intimshop-fantasy.ru
investmac.com
investpamm.ru
iplogger.org iplogger.org
iradiology.ru islandminingsupply.wordpress.com
isitpaleo.info ispac.org
istizanidineanarcoticdrug.blogspot.com istmira.ru
istizanidineanopiate.blogspot.com iwantedmoney.com
itis4you.com izhevsk.zrus.org
ivanovo.zrus.org janettabridal.com
izismile.com jasonpartington.com
jerseysbizwholesalecheap.com jav-fetish.com
jewelryandfiligree.com javxxx18.com
jus0wil.pp.ua jerseywholesalechinabiz.com
kam-dom.ru jobgirl24.ru
keki.info johnrobertsoninc.com
kemerovo.zrus.org kabbalah-red-bracelets.com
kichenaid.ru kambasoft.com
kimcurlrvsms.com kazan.zrus.org
kinoduh.ru ketoanhanoi.info
kinomaniatv.pl keywordhut.com
keywordsuggest.org
kino-fun.ru
kinogolos.ru
kinopolet.net kinopolet.net
kirov.zrus.org
kliksaya.com kliksaya.com
konoplisemena.com klosetkitten.com
konteiner24.com komputernaya-pomosh-moscow.ru
kovrenok.ru koopilka.com
krasnodar-avtolombards.ru kostenloser-sex.com
krassh.ru kosynka-games.ru
kuzinsp.ru krasivye-devushki.net
laapp.com kskjco.club
landmania.ru labvis.host.sk
lasvegaslockandsafe.com laexotic.com
lecbter-relationships.ga lancheck.net
letolove.ru landoftracking.com
lincolntheatre.com lashstudia.ru
link.ac laudit.ru
linkbolic.com lenvred.org
lion.misba.us lexaprogeneric.link
lezbiyanki.net
liquimondo.com
lirunet.ru lirunet.ru
lkbennettoutlet.online liupis.com
loadopia.com lollypopgaming.com
lodki-pvh.dp.ua longgreen.info
lombia.com lostaruban.com
lonfon.xyz lotto6888.com
lovelycraftyhome.com louboutinreplica.xyz
lubetube.com lsex.xyz
luciddiagnostics.in luckyday.world
machicon-ueno.info luckyshop.net.ua
mackeeper-center.club lunamedia.co
macotool.com lyrster.com
mail.allnews24.in macfix.life
managerpak204.weebly.com magda-gadalka.ru
marcogrup.com magento-crew.net
maridan.com.ua magicdiet.gq
marktforschung-stuttgart.com magnetpress.sk
marwer.info manifestyourmillion.com
masserect.com marketingtechniques.info
maxxtor.eu marketingvici.com
mazda-roadsters.com marsgatan.com
mebeldekor.com.ua martlinker.com
mecash.ru mcadamssupplyco.com
mediawhirl.net medkletki.ru
member-quiz.com mesbuta.info
metabar.ru
metaxalonevstizanidine.blogspot.com metaxalonevstizanidine.blogspot.com
mhi-systems.ru meteostate.com
microstatic.pl mezaruk.info
mil-stak.com michaelkorsoutletstore.net
mindbox.co.za mirzonru.net
minecraft-neo.ru mobicover.com.ua
minet.club mobilierland.com
mixx.com moetomnenie.com
mncrftpcs.com moncleroutletonline.pw
monclervests.xyz monitorwebsitespeed.com
money-every-day.com montazhnic.ru
mowser.com montredemarque.nl
mozello.ru moomi-daeri.com
mrinsidesales.com morpicert.pw
mydeathspace.com moscow-clining.ru
mzdish.site moscow.online-podarki.com
movies-in-theaters.net
mtmtv.info
mustat.com
my-aladin.com
my-big-family.com
my-cash-bot.co
mylesosibirsk.ru
mysexpics.ru
nalogovyykodeks.ru nalogovyykodeks.ru
narco24.me namenectar.com
naval.jislaaik.com narkologiya-voronezh.ru
netvouz.com net-radar.com
newstaffadsshop.club nevansk.ru
newtechspb.ru new-post.tk
newyorkhotelsmotels.info nrv.co.za
nextrent-crimea.ru obiavo.by
niroo.info obiavo.uz
no-fuel.org oconto.ru
nonews.co onetravelguides.com
nvssf.com onko-24.com
o.light.d0t.ru onlinadverts.com
ok.ru ooomeru.ru
okmedia.sk openlibrary.org
omoikiri-japan.ru oralsexfilme.net
onefilms.net
onescreen.cc
onlinedomains.ru
onlinefilmz.net
onlinemeetingnow.com
onlineserialy.ru
onlinewritingjobs17.blogspot.ru
ons-add.men
openstat.com
optibuymac.com
orion-code-access.net
outshop.ru outshop.ru
parfusale.se pastaleads.com
pathwhelp.org patol01.pw
perederni.net pawli.eu
perfection-pleasure.ru paydayloanslocal.com
pcimforum.com
pdamods.ru
percin.biz.ly
perfectpracticeweb.com
phelissota.xyz phelissota.xyz
photorepair.ru picturesmania.com
picquery.com pio.polytopesexempt.com
pinkduck.ga pipki.r.acdnpro.com
pisanieprac.info pisanieprac.info
pkr1hand.com piter.xrus.org
placid-rounded-coast.glitch.me pixelrz.com
plastweb.ru plohaya-kreditnaya-istoriya.ru
playlott.com
pokerniydom.ru pokerniydom.ru
popup-jdh.xyz pony-business.com
pornofeuer.com poptool.net
pornoklad.ru poquoson.org
pornotubs.com porn-w.org
pornodojd.ru
pornohd1080.online
pornoinn.com
pornokajf.com
pornoklad.net
porodasobak.net porodasobak.net
privatefx.all4invest.info presleycollectibles.com
prizeestates.cricket pro-poly.ru
proftests.net proekt-gaz.ru
prointer.net.ua prointer.net.ua
prombudpostach.com.ua ptr.ruvds.com
pronorm.fr pufip.com
propranolol40mg.blogspot.com pussyspace.net
prostitutki-kiev.org putevka24.ru
pskcijdc.bloger.index.hr qwarckoine.com
rada.ru raavidesigns.com
randalljhoward.com randalljhoward.com
rczhan.com rankscanner.com
realizmobi.com reklamuss.ru
realtytimes.com relax.ru
rednise.com relayblog.com
refererx.com remedyotc.com
registrationdomainsite.com remont-komputerov-notebook.ru
restorator-msk.ru responsinator.com
rff-cfal.info reyel1985.webnode.fr
ribieiendom.no risparmiocasa.bz.it
roma-kukareku.livejournal.com rixpix.ru
royalvegascasino.com rniaeba.ga
russia-today-video.ru rollercoin.com
saddiechoua.com rutor.vip
sammlungfotos.online sack.net
sanatorrii.ru salmonfishingsacramentoriver.com
sbornik-zakonov.ru
scanhub.ru scanhub.ru
searchinterneat-a.akamaihd.net scanner-andrew.top
searchlock.com scanner-jessica.top
security60-e.com scanner-john.top
seeresultshub-a.akamaihd.net scanner-margo.top
seofied.com scanner-mark.top
seolab.top scansafe.net
scoopquest.com
screentoolkit.com
seansonline24.pl
search.xtconnect.com
segol.tv
seocheckupx.com
seomarketings.online
serdcenebolit.com serdcenebolit.com
servisural.ru serpstat.com
sex-foto.pw seryeznie-znakomstva.ru
sex-sex-sex5.com sex.hotblog.top
sex-tracker.de sexpornotales.com
sexobzor.info sexxdate.net
sexytrend.ru sexystrippe.info
shemalegalls.blogporn.in shijian.ac.cn
sherlock.se shtaketniki.ru
shillyourcoins.com site3.free-share-buttons.com
shlyahten.ru sky-mine.ru
shtora66.ru sloopyjoes.com
sientalyric.co
simon3.ru
sindragosa.comxa.com
smailik.org
small-game.com small-game.com
smallseotools.com sms2x2.ru
snsdeainavi.info snyatie-lomki-v-stacionare.ru
soc-econom-problems.ru social-s-hhh.xyz
social-fun.ru socialbuttons.xyz
socialmonkee.com socialmadesimple.com
softxaker.ru
solution4u.com
songplanet.ru songplanet.ru
soviet-portal.do.am soserfis.com
sovinsteel.ru sotkal.lark.ru
spasswelt.xyz sports-supplements.us
spywarenuker.com
sribno.net
stal-rulon.ru stal-rulon.ru
startwp.org starpages.net
stats-collector.org start.myplaycity.com
steebook.com strongholdsb.ru
strigkaomsk.ru studiofaca.com
stroilka.info suhanpacktech.com
studiokamyk.com.pl tacbelarus.ru
supermama.top tdsing.ru
svolze.com thebluffs.com
tacbibirfa.tk theendivechronicles.com
teastory.co thefarmergame.com
teenforporn.com theguardlan.com
teenporn18.net themestotal.com
texbaza.by thetardistimes.ovh
the-usa-games.blogspot.com tiandeural.ru
titan-ads.life titelhelden.eu
titslove.yopoint.in tizanidine4mgstreetvalue.blogspot.com
tizanidine4mguses.blogspot.com tizanidinedosageforsleep.blogspot.com
tizanidineformigraines.blogspot.com tizanidinefibromyalgia.blogspot.com
tjkckpytpnje.com tizanidinehcl4mgisitanarcotic.blogspot.com
tizanidineingredients.blogspot.com
tizanidinevssoma.blogspot.com
tkanorganizma.ru
tn811.us tn811.us
tomatis.gospartner.com topquality.cf
tongkatmadura.info toys.erolove.in
tootoo.to track112.site
top10-online-games.com
torrent-to-magnet.com
touchmods.fr
tradedeals.biz
traffic100.com
trafficjunky.net trafficjunky.net
traffmonster.info trafficnetzwerk.de
trustaffs.com traffictrade.life
tsyndicate.com traffixer.com
ttsq.fr tripper.de
tube8.com trudogolik.net
tubo360.com tubo360.com
tula.howotorg.ru twodollarshows.com
twinderbella.com tytoona.com
typer.one ultimateclassicrock.com
ublaze.ru undergroundcityphoto.com
ukrobstep.com uptimechecker.com
ukrtextbook.com ural-buldozer.ru
ultimatesetnewfreeallsoftupgradesystems.pw urlcut.ru
unisexjewelry.org usacasino.com
urccvfmc.bloger.index.hr uvozdeckych.info
urll.eu valsalud.com
us-america.ru vavilone.com
usadacha.net vbabule.net
vapsy.com venerologiya.com
vbikse.com via-energy-acquistare.com
veopornogratis.xxx videochat.tv.br
virtuagirl.com villacoloniale.com
virus-respirators.com vinsit.ru
vpnhowto.info vip2ch.com
vseigru.one vkontaktemusic.ru
wallpapersdesk.info voluumtracker1.com
wallpaperstock.net vroze.com
webshoppermac.com vsesubwaysurfers.com
website-datenbank.de vykup-avto-krasnodar.ru
websitevaluebot.com web-betting.ru
weekes.biz.tc webextract.profound.net
wellcome2slovenia.ru weclipart.com
wingsoffury2.com wejdz-tu.pl
winterclassichockeyjerseys.com westen-v.life
wma-x.com wikes.20fr.com
womama.ru wingsofrefuge.net
word-vorlagen.xyz worldinternetauthority.com
world-mmo.com worldmusicfests.com
woweb.com.ua ww1943.ru
wpsecurity.website www.event-tracking.com
www.solartek.ru www.jbetting.com
x5market.ru wzgyyq.com
xbaboon.com
xep.info xep.info
xfire.com xn-----6kcaabbafhu7cskl7akvongwpo7hvjj.xn--p1ai
xn--80ajbshivpvn2i.xn--p1ai xn--80adaggc5bdhlfamsfdij4p7b.xn--p1ai
xn--80ahvj9e.xn--p1ai
xn--80atua3d.xn--p1ai
xn--b1agm2d.net
xn--h1ahbi.com.ua xn--h1ahbi.com.ua
xn--l1aengat.xn--p1ai xxxdatinglocal.us
xxxsiterips.xyz xxxmania.top
xxxtube69.com xxxnatelefon.ru
yebocasino.com yellowfootprints.com
youdao.com yoshkarola.zrus.org
youjizz.com yougotanewdomain.com
youjizz.vc yourdesires.ru
youporn-forum.ga yourporngay.com
your-bearings.com yoursite.com
yun56.co youtubologia.it
yunque.pluto.ro zacreditom.ru
zaidia.xhost.ro zagadki.in.ua
zarepta.com zajm-pod-zalog-nedvizhimosti.ru
zed21.net zakazfutbolki.com
zeg-distribution.com zarabotok--doma.ru
zarenica.net
zazagames.org
zebradudka.com
zetgie.com.pl
zetmaster.ru zetmaster.ru
zhacker.net zlatnajesen.com
zigarettenonl.canalblog.com zzbroya.com.ua
zigarettenonlinekaufen1.bloog.pl zzlgxh.com
zirondelli.it
zoogdiesney.com
zootoplist.com

484
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
0c47f8422d3f.com 12bet.com
0lovespells0.blogspot.com 1qingdao.com
1adult.com
20pascals.nl
404.mn
4kplayer.pl 4kplayer.pl
4webmasters.org 4rent.ru
abclauncher.com 72-news.com
abiente.ru 777-club.ru
add-add.men 91abcw.com
addtoadd.men aa625d84f1587749c1ab011d6f269f7d64.com
adtology.com abovetherivernc.com
adultactioncam.com adexprts.com
adultfriendfinder.com adultfullhd.com
adzpower.com agardomains.com
allcredits.su alphacarolinas.com
amazingpic.net apccargo.com
animalia-life.club apiadanaknet-a.akamaihd.net
anlimebel.kiev.ua asscenihotosidea.blogspot.co.za
anonymousfox.co autoblogger24.info
anticrawler.org autogrep.ru
apibetweenlinesn-a.akamaihd.net b2b-lounge.com
apilinkswiftco-a.akamaihd.net baksman.com
appiq.mobi bala.getenjoyment.net
appmsr.org bestbrainprod.win
artefakct.com bestbuy.ca
autotrends.today bestempresas.es
avek.ru bingo8888.com
azartmsl.com bolegapakistan.com
bimatoprost-careprost.com brandedleadgeneration.com
bitcoin-ua.top buketeg.xyz
brandbucket.com butstrap.space
brandehk.dk buytizanidineonline.blogspot.com
breastaugmentation.co.za cascadelink.org
buy-forum.ru caulderwoodkennels.com
buy-loft.ru
calstaterealty.us
camdenmemorials.com
cartierloveringreplica.xyz
cartierreplica.xyz
cdnanalytics.xyz
cejewelry.xyz
certifywebsite.win certifywebsite.win
cherrypointplace.ca cheapcigarettesc.info
chinaelitecheapjerseys.com chlooe.com
clickso.com cialis-samples.com
coinsspb.com cleanallspyware.com
compliance-brian.top comissionka.net
compliance-nelson.top compliance-george.xyz
constantaservice.net compliance-julianna.top
cottageofgrace.com cookielawblog.wordpress.com
csgo4.win cool-wedding.net
dating-app.ru country-chic.ru
dating-time-now.com crackguru.tk
dayibiao.com cranly.net
detmebel.su customerpromos-a.akamaihd.net
deutschland123.de dailyfinancefix.com
dev33.dioniqlabb.se dailystrength.org
directivepub.com datract.com
dispo.de detsky-nabytek.info
distver.ru deutschehobbyhuren.net
dnepropetrovsk.mistr-x.org dev-seo.blog
dojki-hd.com dividendo.ru
domaineaneblanc.com djonwatch.ru
earthmagic.info docsportal.net
elektrischeziga.livejournal.com domainsatcost.com
elektronischezigarettekaufen1.myblog.de dostavka-v-krym.com
elektronischezigarettekaufen2.tumblr.com download-wallpaper.net
elidelcream.weebly.com earn-from-articles.com
elitedollars.com eldiariodeguadalajara.com
elmifarhangi.com elektronischezigarettekaufen1.beeplog.com
elvenar.com elektrozigarette2.wordpressy.pl
enginebay.ru eluxer.net
ero-advertising.com eorogo.top
erotikchat-24.com erotikstories.ru
erotische-geschichten-xxl.com excitacion.info
escort-russian.com expresstoplivo.ru
ewebarticle.info
extstat.com
eyes-on-you.ga eyes-on-you.ga
fanpagerobot.com ezigarettekaufen2.yolasite.com
ferrotodo.com facialporntube.com
fidelityfunding.com fishingwholesale.us
finejewelryshop.ru flipper.top
fotopop.club flowersbazar.com
fotosfotos.eu footbalive.org
free-fbook-traffic.com forensicpsychiatry.ru
freegamesplay.online forumprofi.de
galaxyflowers.ru foto-weinberger.at
gamebackyard.com free-video-tool.com
gamesprite.me freewlan.info
gaz-voshod.ru funcrushgames.com
geoads.com fx-brokers-review.com
getmiro.com gate5.co.za
girlgamerdaily.com getlaid-xxxhookupdirect.com
gnuetella.com golden-catalog.pro
go2mike.ru guruofcasino.com
goodnightjournal.com hd-filmy.net
greamimgo.pw help.tpu.ru
greenshop.su helvetia.com.ua
gribokstop.com homeinns.com
guildebzh.info hotcore.info
gymi.name housemilan.ru
hawaiisurf.com husky-shop.cz
hermesbirkinhandbagoutlets.com hyipmanager.in
hesteel.pl i4track.net
hexpilot.com iklysha.ml
highland-homes.com imagecoolpub.com
histats.com imitex-plus.ru
homik.pw interferencer.ru
hoverboard360.de japfm.com
howtoclean.club jav-fetish.site
hybrid.ru javstock.com
igadgetsworld.com jovencitas.gratis
iinstalll-fii1leis.jus0wil.pp.ua joyceblog.top
imagerydatabase.com keywordbasket.com
jose.mulinohouse.co keywordhouse.com
jyrxd.com kidd.reunionwatch.com
kadashihotel.com kino-filmi.com
kakablog.net kinoplen.ru
kambasoft.com knogg.net
kongruan.com kutikomi.net
konoplisemena.com labplus.ru
kozhniebolezni.com lawyersinfo.org
kvartira-sutochno.com le-clos-des-alouettes.com
ledis.top lego4x4.ru
littlecity.ch lider-zhaluzi.kiev.ua
littlesunraiser.com lonerangergames.com
livefixer.com louboutinreplica.pw
lowephotos.info low-format.ru
lumb.co lubetube.com
m5home.ru lyngdalhudterapi.no
marblestyle.ru mainlinehobby.net
mbiologi.ru makedo.ru
mebel-vstroika.ru marketingtechniques.info
megaindex.ru maslenka.kz
menstennisforums.com masturbate.co.uk
mini-modus.ru max-eclat.men
mjchamonix.org mezaruk.info
mnogabukaff.net monclercheap.xyz
mobplayer.ru moyakuhnia.ru
moneymaster.ru mp3s.club
morocco-nomad-excursions.com mtmtv.info
moroccosurfadventures.com mygameplus.ru
mrwhite.biz myseoconsultant.com
muz-baza.net nac-bearings.ru
nagdak.ru
naj-filmy24.pl
naperehresti.info
naphukete.ru
naturalpharm.com.ua naturalpharm.com.ua
nbsproject.ru nbsproject.ru
netpics.org nerudlogistik.ru
next-dentists.tk netcheckcdn.xyz
ningessaybe.me networkcheck.xyz
northfacestore.online nextrent-crimea.ru
o.light.d0t.ru officedocuments.net
offers.bycontext.com ok.ru
ok-ua.info
okroshki.ru
omsk.xrus.org
online-hd.pl online-hd.pl
outrageousdeal-a.akamaihd.net optom-deshevo.ru
paidonlinesites.com origin-my.ru
partner-pop.men osnova3.ru
partnersafe.men paparazzistudios.com.au
pastaleads.com paradontozanet.ru
pinsdaddy.com partners-ship.pro
pivka.xyz photostudiolightings.com
pk-services.ru piccdata.com
pkr1hand.com pics-group.com
pl.id-forex.com play-mp3.com
poffet.net pogruztehnik.ru
porno-home365.com polimga.pw
pourvous.info popander.mobi
priora-2.com poppen-nw.net
pro-okis.ru porndl.org
promoforum.ru pornoelita.info
prospekt-st.ru pornpost.in
prostitutki-rostova.org pospr.waw.pl
puteshestvennik.com pricheski-video.com
puzzleweb.ru prombudpostach.com.ua
ranksonic.com proxyradar.com
razorweb-a.akamaihd.net pst2017.onlinewebshop.net
realresultslist.com puzo2arbuza.ru
redbottomheels.xyz qexyfu.bugs3.com
relayblog.com quangcaons.com
riralmolamsaca.tk reklama1.ru
rockingclicks.com remontvsamare.su
rubanners.com renhacklids.tk
rentehno.ru
revistaindustria.com
richinvestmonitor.com
running-line.ru
rustic-quiver.win
rybalka-opt.ru
s.lollypopgaming.com s.lollypopgaming.com
saletool.ru samara.rosfirm.ru
sarafangel.ru sarahmilne.top
sarf3omlat.com satoristudio.net
search.xtconnect.com sbdl.no
seminarykansas904.ml scanner-alexa.top
seo-smm.kz scrnet.biz.ua
seobility.net selectads.men
seoboxes.com seomarketings.online
sethrollins.net seozoom.it
sexad.net sibtest.ru
shtora66.ru signal03.ru
sigmund-freud.co.uk silverage.ru
silvercash.com simple-share-buttons.com
simplesite.com smallseotools.com
sitevaluation.com smartadserver.com
sklad-24.ru snworks.com
slavic-magic.ru social-fun.ru
social-buttons.xyz sotechco.co
soda.media spaceshipad.com
space-worry.ml spravka130.ru
sribno.net streetfire.net
stackthatbucks.com stroiminsk.com
stretchmate.net surflinksmedical.com
stroilka.info teguh.info
surffoundation.nl thaismartloan.com
swimpool.ca topshef.ru
swingerseiten.com torontoplumbinggroup.com
tastyfoodideas.com torrent-newgames.com
telefonsexkostenlos.tk toys.erolove.in
telesvoboda.ru traffictrade.life
tizanidinehcl4mgtab.blogspot.com traffpartners.com
tizanidinemusclerelaxant.blogspot.com umityangin.net
torrentgamer.net unacittaconte.org
trafficnetzwerk.de underthesite.com
trustaffs.com validdomain.xyz
twincitiescarservice.com vapsy.com
txxx.com vids18.site
u17795.netangels.ru vintontech.info
uasb.ru vk-mus.ru
uchil.net vvon.co.uk
udayavani.com vzlomtw.com
urzedowski.eu w7s.ru
vavilone.com wait3sec.org
vedomstvo.net waterefficiency.co
via-energy-cumpara.com waterpurifier.club
vielporno.net webalan.ru
vilingstore.net websites-reviews.com
vipromoffers.com weburok.com
vozbujdenie.com wheelchairliftsarea.com
wikes.20fr.com wikes.20fr.com
wrona.it winwotgold.pl
ws.ampower.me workle.website
wyniki-lista.pl worst-sites.online
xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai ww2awards.info
xn--80aebbcbcdemfkhba4byaehoejh8dza3v.xn--p1ai x-rates.ru
xn--e1afanlbnfckd7c3d.xn--p1ai xboxster.ru
xn--oogle-wmc.com xclicks.net
xxxasianporn.net xn----7sbbahaq9bb5afgiqfliv4m.xn--p1ai
yottos.com xn----ctbbcjd3dbsehgi.xn--p1ai
yourporn.com xn--e1aggki3c.xn--80adxhks
zeroredirect1.com xtraffic.plus
zhacker.net xtubeporno.net
zolotoy-lis.ru xxxkaz.org
yoopsie.com
zaim-pod-zalog-krasnodar.ru
zeroredirect12.com
znakomstva-moskva77.ru
zojirushi-products.ru

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.140 0; 10.1.0.220 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.08.3322 ### Version: V4.2022.08.3323
### Updated: Tue Aug 30 22:01:15 UTC 2022 ### Updated: Wed Aug 31 10:03:34 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.08.3322 ### Version: V4.2022.08.3323
### Updated: Tue Aug 30 22:01:15 UTC 2022 ### Updated: Wed Aug 31 10:03:34 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##