vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-02 18:50:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.08.2547 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-08-24 22:07:06 +00:00
parent 73fcc0aa74
commit 1f4754cda7
46 changed files with 66597 additions and 66597 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.08.2546 #### Version: V4.2021.08.2547
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.08.2546 #### Version: V4.2021.08.2547
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -11,7 +11,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2021.08.2546 #### Version: V4.2021.08.2547
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

8190
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2021.08.2546 #### Version: V4.2021.08.2547
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

8194
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

8194
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

8194
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

8194
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 1; 10.1.0.104 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 0; 10.1.0.104 0;
127.0.0.1 0; 127.0.0.1 0;

8194
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 1; 10.1.0.104 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 0; 10.1.0.104 0;
127.0.0.1 0; 127.0.0.1 0;

8194
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 1; 10.1.0.104 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 0; 10.1.0.104 0;
127.0.0.1 0; 127.0.0.1 0;

8194
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2546 2547

8194
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

8194
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.98.69.251 0; 20.94.206.155 0;

8194
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 1; 10.1.0.104 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 0; 10.1.0.104 0;
127.0.0.1 0; 127.0.0.1 0;

8194
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 1; 10.1.0.104 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 0; 10.1.0.104 0;
127.0.0.1 0; 127.0.0.1 0;

8194
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 1; 10.1.0.104 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 0; 10.1.0.104 0;
127.0.0.1 0; 127.0.0.1 0;

8194
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.98.69.251 0; 20.94.206.155 0;

8194
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.98.69.251 0; 20.94.206.155 0;

8194
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 1; 10.1.0.104 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
ADmantX Bandit
BuiltWith CSHttp
CATExplorador Gotit
CrazyWebCrawler IDBTE4M
CrunchBot Image Sucker
Discobot MeanPath Bot
MTRobot NetZIP
Mister PiX Pandalytics
NetLyzer PeoplePal
Offline Navigator Snake

310
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
360Spider 360Spider
80legs
ADmantX
AIBOT AIBOT
Abonti Abonti
Aboundex Aboundex
Aboundexbot AiHitBot
Acunetix
AhrefsBot
Aipbot Aipbot
Anarchy Alexibot
Anarchy99 AllSubmitter
Ankit Alligator
Asterias AlphaBot
BDCbot Anthill
BackDoorBot AwarioRssBot
AwarioSmartBot
BDFetch
BackStreet
Backlink-Ceck Backlink-Ceck
Battleztar Bazinga Badass
BetaBot Bigfoot
Black Hole BlackWidow
Blackboard
Bolt Bolt
Brandprotect
Buddy
BunnySlippers
BuzzSumo BuzzSumo
CATExplorador CazoodleBot
CODE87 Cegbfeieh
Calculon
CensysInspect
CheeseBot CheeseBot
Claritybot ChinaClaw
Cliqzbot Cloud mapping
Cogentbot Cosmos
Collector Craftbot
CyotekWebCopy Curious
Custo
DBLBot DBLBot
DSearch
Demon
Deusu Deusu
Digincore Digincore
Disco Disco
Discoverybot DittoSpyder
Dispatch DnyzBot
DomainAppender DomainCrawler
DomainSigmaCrawler
DomainStatsBot
Domains Project
Dotbot Dotbot
Drip Drip
EMail Wolf ECCP/1.0
Evil EasyDL
Ebingbong
Ecxi
EirGrabber
Express WebPictures Express WebPictures
Extractor ExtLinksBot
EyeNetIE
FHscan FHscan
Fimap Fimap
FlashGet Flunky
Foobot Foobot
FrontPage FrontPage
Fuzz G-i-g-a-b-o-t
FyberSpider
GT::WWW GT::WWW
Genieo
GermCrawler GermCrawler
GetWeb Gigablast
Getintent
Gigabot
Go!Zilla Go!Zilla
Go-Ahead-Got-It GoZilla
Gotit Gotit
Grabber
Grafula Grafula
GrapeFX GrapeFX
GrapeshotCrawler GrapeshotCrawler
HEADMasterSEO
HMView HMView
HTTP::Lite
Haansoft Haansoft
HaosouSpider
Harvest
Hloader Hloader
Humanlinks Humanlinks
HybridBot HybridBot
IDBTE4M
Iblog Iblog
IlseBot IlseBot
Image Sucker Image Fetch
IndeedBot IndeedBot
InfoNaviRobot
Intelliseek
InterGET
Internet Ninja Internet Ninja
JamesBOT Iskanie
JetCar IstellaBot
Jbrofuzz
JennyBot
Jetty
Jorgee
Jyxobot
Kozmosbot Kozmosbot
LNSpiderguy LNSpiderguy
LWP::Simple LWP::Simple
Larbin Lanshanbot
Leap Leap
LeechGet LeechFTP
Libwhisker LexiBot
LinkScan
LinkWalker
Linkbot
Linkdexbot
LinkextractorPro LinkextractorPro
LinqiaRSSBot LinqiaMetadataDownloaderBot
Lmspider LinqiaScrapeBot
MIDown tool Lipperhey
MJ12bot Litemage_walker
MQQBrowser Ltx71
MFC_Tear_Sample
Mag-Net
Magnet Magnet
Mail.RU_Bot Mail.RU_Bot
Majestic-SEO MarkMonitor
Majestic12 MarkWatch
Mass Downloader
Masscan Masscan
Mata Hari
MauiBot
MeanPath Bot MeanPath Bot
Mediatoolkitbot MicroMessenger
Moblie Safari Moblie Safari
Morfeus Fucking Scanner Morfeus Fucking Scanner
Mr.4x3
Msrabot Msrabot
Musobot
NICErsPRO NICErsPRO
NPbot NPbot
NearSite Name Intelligence
Needle
Nessus
Net Vampire Net Vampire
NetAnts
NetLyzer NetLyzer
NetMechanic NetSpider
NetZIP Netvibes
Netcraft
Nettrack
NextGenSearchBot NextGenSearchBot
Niki-bot Nibbler
NimbleCrawler Nikto
Nimbostratus Ninja
Nmap Nmap
Nuclei
Offline Explorer Offline Explorer
OnCrawl Offline Navigator
OpenVAS
Openfind Openfind
Pandalytics Openvas
Panscient PHPCrawl
Picsearch Page Analyzer
PictureFinder PageGrabber
Pavuk
Picscout
Piepmatz Piepmatz
Pixray PleaseCrawl
Pockey
ProPowerBot
Probethenet Probethenet
Psbot PxBroker
QueryN Metasearch RankActive
RankingBot RankingBot2
Rankivabot Rankivabot
RankurBot Re-re
ReGet ReGet
Reaper RealDownload
Recorder Recorder
RedesScrapy RepoMonkey
SBIder Ripper
SEOkicks
SEOkicks-Robot
SEOprofiler
SEOstats SEOstats
SISTRIX
SMTBot
SalesIntelligent SalesIntelligent
ScoutJet Scrapy
Screaming ScrepyBot
Searchestate
SearchmetricsBot SearchmetricsBot
SeobilityBot SentiBot
Shodan
Site Sucker
SiteCheckerBotCrawler SiteCheckerBotCrawler
SlySearch SiteSnagger
Sitebeam
SmartDownload SmartDownload
Snake Sociscraper
Snapbot
Sosospider Sosospider
SpaceBison SpaceBison
Spammen
Spinn3r
Sqlmap
Sqlworm
Sqworm
Stripper Stripper
Sucker
Sucuri
SuperHTTP
Suzuran
Szukacz Szukacz
T0PHackTeam
Telesoft
Telesphoreo
The Intraformant The Intraformant
Thumbor TheNomad
Toata Titan
Tracemyfile
Trendiction Trendiction
Trendictionbot Turingos
True_Robot
TurnitinBot TurnitinBot
TwengaBot TwengaBot
Twice Typhoeus
URLy Warning URLy.Warning
UnisterBot UnisterBot
Upflow V-BOT
VB Project
VCI VCI
VidibleScraper Vagabondo
Virusdie WEBDAV
WASALive-Bot WISENutbot
WWW-Collector-E WWWOFFLE
WWW-Mechanize Wallpapers/3.0
Web Enhancer Web Auto
Web Fetch Web Collage
Web Fuck Web Sauger
Web Pix WebAuto
WebCopier WebCollage
WebEnhancer WebEnhancer
WebFetch
WebGo IS WebGo IS
WebReaper WebImageCollector
WebSucker WebLeacher
WebSauger
WebWhacker
WebZIP
Webalta
Webshag Webshag
Website Quester
WebsiteQuester WebsiteQuester
Webster
Whack Whack
Whacker
Whatweb Whatweb
Who.is Bot WiseGuys Robot
Wotbox
Wprecon Wprecon
ZmEu Xaldon_WebSpider
ZoomBot Xenu
Zauba
Zermelo
ZumBot ZumBot
archive.org_bot ZyBorg
arquivo-web-crawler
arquivo.pt arquivo.pt
backlink-check
check1.exe check1.exe
cognitiveseo com.plumanalytics
crawl.sogou.com crawl.sogou.com
eCatch demandbase-bot
evc-batch domainsproject.org
gopher
instabid instabid
internetVista monitor isitwp.com
magpie-crawler magpie-crawler
meanpathbot meanpathbot
muhstik-scan mediawords
oBot
page scorer page scorer
pcBrowser pcBrowser
probe-image-size plumanalytics
ripz
s1z.ru s1z.ru
satoristudio.net scan.lol
seoscanners seobility
serpstatbot
sexsearcher
sitechecker.pro sitechecker.pro
spyfu sp_auditbot
sysscan ubermetrics-technologies.com
tAkeOut voyagerx.com
trendiction.com xpymep1.exe
trendiction.de zauba.io
webmeup-crawler zgrab
webpros.com
webprosbot

286
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
360Spider
404checker
80legs 80legs
ASPSeek AIBOT
ALittle Client
Abonti
Aboundex Aboundex
Acunetix
AfD-Verbotsverfahren
AiHitBot AiHitBot
Aipbot Alexibot
AllSubmitter AllSubmitter
Alligator AlphaBot
Anarchie
Ankit Ankit
Apexoo
AwarioRssBot AwarioRssBot
AwarioSmartBot BBBike
BDCbot BDCbot
BLEXBot
BackDoorBot BackDoorBot
BackStreet
Backlink-Ceck Backlink-Ceck
Badass BacklinkCrawler
Bandit Bigfoot
Barkrowler
Black Hole Black Hole
BlackWidow
Boardreader
Bolt Bolt
BotALot
Brandwatch
Buddy Buddy
BunnySlippers BunnySlippers
BuzzSumo
CATExplorador
CCBot CCBot
Calculon
Cegbfeieh Cegbfeieh
CheeseBot CheeseBot
CherryPicker CherryPicker
ChinaClaw ChinaClaw
Chlooe Chlooe
Claritybot Cliqzbot
Cloud mapping Cocolyzebot
Cogentbot
Collector
Cosmos Cosmos
Crawling at Home Project Crawling at Home Project
CrunchBot CrunchBot
Custo
CyotekWebCopy CyotekWebCopy
DBLBot
DIIbot
DSearch DSearch
DTS Agent Deusu
Digincore Devil
Dirbuster DigitalPebble
Discobot Disco
Discoverybot
Dispatch Dispatch
DnyzBot DittoSpyder
DomCopBot
DomainStatsBot DomainStatsBot
Drip Dotbot
ECCP/1.0 ECCP/1.0
EMail Siphon EMail Siphon
EMail Wolf
EasyDL EasyDL
FDM Ecxi
FHscan EirGrabber
ExtLinksBot
Extreme Picture Finder
EyeNetIE
Ezooms
Fimap Fimap
Firefox/7.0 FlashGet
Flunky
FrontPage
Fuzz Fuzz
GermCrawler
GetRight
GetWeb GetWeb
Getintent Getintent
Gigabot
Go!Zilla Go!Zilla
Gotit
GrabNet GrabNet
GrapeshotCrawler Grabber
GridBot HaosouSpider
HTMLparser Havij
Haansoft
Heritrix Heritrix
Humanlinks Humanlinks
HybridBot IDBot
IRLbot Iblog
IlseBot Id-search
Image Fetch
Image Sucker Image Sucker
IndeedBot IndeedBot
Indy Library Indy Library
InfoNaviRobot
InterGET InterGET
IstellaBot InternetSeer
Iria
JOC Web Spider JOC Web Spider
JennyBot JamesBOT
JetCar Jbrofuzz
Joomla Joomla
Kozmosbot JustView
LNSpiderguy Jyxobot
LeechGet Kinza
Lftp LWP::Simple
Leap
LeechFTP
LexiBot
LibWeb
Libwhisker Libwhisker
LinkScan Lightspeedsystems
LinkWalker Likse
Linkbot
Linkdexbot Linkdexbot
LinkpadBot
LinqiaRSSBot LinqiaRSSBot
LinqiaScrapeBot LinqiaScrapeBot
Litemage_walker
Lmspider Lmspider
MFC_Tear_Sample MIDown tool
MQQBrowser MSFrontPage
Mail.RU_Bot Mail.RU_Bot
Majestic SEO
Majestic-SEO Majestic-SEO
MarkWatch MarkMonitor
Mass Downloader Masscan
Mb2345Browser
Meanpathbot Meanpathbot
MegaIndex.ru Mediatoolkitbot
Metauri Metauri
Microsoft URL Control Microsoft URL Control
Moblie Safari Mister PiX
Mojolicious Mojolicious
Mozlila Mozlila
Msrabot
Nameprotect Nameprotect
Navroad Navroad
NearSite NearSite
Needle
Nessus
Net Vampire Net Vampire
NetAnts
NetLyzer NetLyzer
NetMechanic NetMechanic
NetSpider NextGenSearchBot
Nettrack Nibbler
Netvibes NimbleCrawler
Niki-bot
Nimbostratus
Nmap
Nuclei Nuclei
Nutch Nutch
Octopus
Offline Explorer
Offline Navigator Offline Navigator
OrangeBot OpenLinkProfiler
OrangeSpider OutfoxBot
OutclicksBot
POE-Component-Client-HTTP
Page Analyzer
PageAnalyzer
PageGrabber PageGrabber
PageThing.com PageScorer
Panscient Pandalytics
Petalbot Petalbot
PictureFinder Picscout
Piepmatz
Pimonster
PleaseCrawl PleaseCrawl
ProPowerBot ProPowerBot
ProWebWalker Probethenet
Pump Pump
PxBroker PxBroker
PyCurl
QueryN Metasearch
Quick-Crawler Quick-Crawler
RSSingBot
RankActive
RankActiveLinkBot RankActiveLinkBot
RankFlex
Rankivabot Rankivabot
RankurBot RankurBot
Re-re Re-re
RealDownload RepoMonkey
RebelMouse
Recorder
RedesScrapy
RocketCrawler RocketCrawler
Rogerbot Rogerbot
SEOkicks SEOprofiler
SEOkicks-Robot ScanAlert
SEOstats ScoutJet
SISTRIX ScreenerBot
Scrapy
Screaming
ScrepyBot
Searchestate Searchestate
SearchmetricsBot
Semrush Semrush
SeobilityBot SeoSiteCheckup
Shodan Shodan
Siphon Siphon
SiteSnagger SiteSnagger
Sitebeam SiteSucker
Siteimprove Sitevigil
SlySearch SlySearch
Snapbot Snake
SocialRankIOBot SocialRankIOBot
Sogou web spider Sociscraper
Spanner SpaceBison
SputnikBot Spbot
Sqworm Sqlmap
Sqlworm
Stripper Stripper
Sucker Sucker
Swiftbot SurveyBot
Suzuran
Szukacz Szukacz
T0PHackTeam
T8Abot T8Abot
Teleport Telesoft
TeleportPro Thumbor
The Intraformant Toweyabot
TheNomad
Titan
Tracemyfile Tracemyfile
Trendiction True_Robot
Turingos Turingos
Turnitin Turnitin
URLy.Warning TwengaBot
UnisterBot URLy Warning
VCI V-BOT
VelenPublicWebCrawler Vacuum
VeriCiteCrawler
VidibleScraper
VoidEYE
Voltron Voltron
WBSearchBot
WISENutbot WISENutbot
WPScan WWWOFFLE
WWW::Mechanize
Wallpapers Wallpapers
WallpapersHD Wallpapers/3.0
Web Auto
Web Fetch Web Fetch
WebBandit Web Sucker
WebCopier WebCopier
WebEnhancer
WebFetch
WebFuck WebFuck
WebGo IS WebGo IS
WebWhacker WebLeacher
WebsiteExtractor WebPix
WebsiteQuester Webalta
WiseGuys Robot WebmasterWorldForumBot
Woobot Webshag
Xaldon_WebSpider Webster
Whatweb
Widow
YoudaoBot YoudaoBot
Zade Zeus
Zitebot Zitebot
ZmEu ZoomBot
ZoominfoBot
ZumBot ZumBot
autoemailspider arquivo.pt
com.plumanalytics backlink-check
cah.io.community
check1.exe
crawl.sogou.com crawl.sogou.com
crawler4j
eCatch eCatch
evc-batch instabid
facebookscraper
gopher
internetVista monitor internetVista monitor
ips-agent lwp-request
meanpathbot
mediawords mediawords
oBot netEstate NE Crawler
plumanalytics polaris version
ripz probe-image-size
satoristudio.net scalaj-http
seostar scan.lol
sexsearcher sexsearcher
spyfu siteripz
voyagerx.com sogouspider
sysscan
tAkeOut
trendiction.de
webmeup-crawler webmeup-crawler
webpros.com
x09Mozilla
x22Mozilla

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
activecampaign.dreamhosters.com 12u.info
bizfly.info domination.ml
braintobe.win gribokstop.com
cleansearch.net medictube.ru
culad.com nfljerseyscheapchinabiz.com
iphantom.com siteexpress.co.il
mere.host.sk tako3.com
sotechco.co vavilone.com
thecounter.com venta-prom.ru
zeikopay.com womensplay.net

916
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
00webcams.com 01apple.com
1001desert.com 10bet.com
101billion.com 1adult.com
108shot.com 1bet.com
1kinobig.ru 1kinobig.ru
1xbet4you.com 300richmond.co.nz
24videos.tv 4inn.ru
4k-player.pl 4senses.co
4webmasters.com
5-steps-to-start-business.com
5000-cotydzien.com 5000-cotydzien.com
999webdesign.com 5kstore.com
aa14ab57a3339c4064bd9ae6fad7495b5f.com 7yue.org
aab94f698f36684c5a852a2ef272e031bb.com 8jn.dating-app.ru
academia-nsk.org 8kisses.eu
actices.com aac52006ec82a24e08b665f4db2b5013f7.com
activecampaign.dreamhosters.com aadroid.net
activepr.ru absugars.com
adman.se abtasty.com
adprotect.net accessoires-mode.in
adrunnr.com adclickthru.net
adtech.us admanaerofoil.com
adtiger.tk adservme.com
advokateg.xyz adsland.men
allkrim.com adtech.de
afora.ru
ageofclones.com
air-edem.ru
allesohnegirls.net
alloysteel.ru
alltheviews.com alltheviews.com
alphaforum.ru
alphahoverboards.com
alpharma.net alpharma.net
alt-servis.ru amazon-adsystem.com
andpolice.com ananumous.ru
animali.deagostinipassion.it anapa-inns.ru
animebox.com.ua angkortours.vn
antons-transporte.de apiallgeniusinfo-a.akamaihd.net
apiluckyleapnet-a.akamaihd.net apibrowsesmartne-a.akamaihd.net
appleid-verification.com apiportalmorecom-a.akamaihd.net
apptester.tk apisecretsaucebi-a.akamaihd.net
arabseyes.com apple.com-webbrowsing-security.review
arclk.net appsecurityr.com
arquapetrarca.info aproposde.com
arquivo.pt ardimobileinfo.ml
artparquet.ru
asia-forum.ru
asrv-a.akamaihd.net
atdedinotuho.tk atdedinotuho.tk
atleticpharm.org auto-moto-elektronika.cz
atovh.local-finders.com avtointeres.ru
avkzarabotok.com ayakino.net
backgroundpictures.net azartmix.com
baksman.com azbukadiets.ru
balitouroffice.com b2b-lounge.com
banki76.ru babespcs.com
beachtoday.ru bannerspace.com
berdasovivan.ru bashtime.ru
bestempresas.es bcmp.org
bestssaker.com beachfix.co
bg6s0.com beachpics.com
bimmerpost.com beauby.ru
bio-market.kz beauty-bracelet.com
bitporno.sx beauty-things.com
bizru.info best-seo-offer.com
blackbabesporn.com bestdooz.com
blockety.co bestfortraders.com
bloggedporn.com besthoro.ru
bmsco.co bgviagrachrx.com
bobba.dzaba.com bimatoprost-careprost.com
boc.kz bizlist.com.de
bochemit.com.ua blogs.rediff.com
bonzibuddi.com bloke.com
bookmark4you.com bmwhighperformers.com
brainboostingsupplements.org boostmyppc.com
budilneg.xyz brainzpod.win
bukleteg.xyz brandedleadgeneration.com
buqyxa.rincian.info brandimensions.com
burger-tycoon.com bravegirlsclub.com
buyviagraa.blogspot.com buckspar.gq
californianews.cf buron.pw
candcstuccoandstone.com camdolls.com
canoncdriverq3.pen.io cand.jp
canrole.com cartierlove.xyz
cardiosport.com.ua casinopinup-wins.com
cardul.ru cctva.tv
carina-sy.de centraletermice.us
carstrends2015.com charmstroy.info
ccjp.eu chcu.net
cenoval.ru cheapjerseysap.com
cfcl.co.uk cheapjerseysfootballshop.com
chastnoeporno.com cheapmarmot.online
cheapjerseysa.com chelnytruck.ru
chelyabinsk.xrus.org chiptuninger.com
chinacheapelitejerseys.com chlooe.com
chinajerseyswholesalecoupons.com christianlouboutinsaleoutletonline.us
chocolateslim-original.com cigarpass.com
christianlouboutinshoes.xyz citetick.com
cialischmrx.com citysecurity.nu
cleanersoft.com clickbank.net
clickaider.com clickprotects.com
clickbanksites.info coin-hive.com
club-samodelkin.ru coindirect.io
com-onlinesupport.site
com-secure.download
comeondog.info
cometorussia.net cometorussia.net
compliance-ivan.top communisave.co.za
compliance-jane.top compliance-george.top
cool-wedding.net compliance-julianna.top
coolwallpapers-hd.com comsysnet.com
coprofam.org copenergo.ru
costablue.xyz coslab.club
country-chic.ru countbertwistdisp26.soup.io
cpabegins.ru creativehutindia.com
cs-passion.pl creditnation.ru
d0t.ru cubook.supernew.org
dangphoto.trade cupday.com
customchocolate.business-for-home.com
cv.wallhade.co
cvety24.by
daptravel.com
darkbooks.org
data-mining.tk data-mining.tk
decenttools.com decenttools.com
deluxedumps.com dikqlyremy.info
destinationrealestate.com diminishedvalueoforegon.com
detki-opt.ru documentsite.net
detskie-konstruktory.ru dojki365.online
detsky-nabytek.info domaineaneblanc.com
dfwu1013.info domainsphoto.com
diesel-parts28.ru dominateforex.ml
directrev.com doska-vsem.ru
domashneeruporno.com doyouknowtheword-flummox.ml
domination.ml drupa.com
dosugrostov.site duplicashapp.com
doublepimp.com dvrlists.com
downtuptv.gq e-commerce-seo1.com
downvids.net e-poker-2005.com
drillsaw.ru e705.net
droidlook.net e90post.com
dvervmoskvu.ru ecommerce-seo.org
e-biznes.info editors.choice6912650.hulfingtonpost.com
e-buyeasy.com eduserver.net
e2click.com efkt.jp
ecookna.com.ua efnor-ac.com
ekto.ee egvar.net
elektronischezi.livejournal.com el-nation.com
elvel.com.ua electronicadirect.com
en.altezza.travel elektronischezigarettekaufen1.beeplog.com
enskedesquashclub.se elitedollars.com
epngo.bz englishgamer.com
ero-advertising.com eralph.tk
erotik0049.com eropho.com
essayservicewriting.org es5.com
estelight.ru everypony.ru
eu-cookie-law.info
everflora.ru
everytalk.tv everytalk.tv
expertblog.info evidencecleanergold.com
extrabot.com
extractorandburner.com extractorandburner.com
ezigarettekaufen1.hpage.com f-online.de
f5mtrack.com fashionavenuegame.com
fable.in.ua fast-torrent.ru
fakehandbags.xyz fermersovet.ru
fantasticpixcool.com ff30236ddef1465f88547e760973d70a.com
fastfixing.tech filmi-v.online
feel-planet.com filunika.com.ru
filesdatabase.net finuse.com
fishtauto.ru flauntyoursite.com
fitness-video.net
fivedwld.cf
fix-website-errors.com
footbalive.org
forensicpsychiatry.ru forensicpsychiatry.ru
forex21.ru fortwosmartcar.pw
forum-engineering.ru
forum.doctissimo.fr
foto-sisek.porngalleries.top foto-sisek.porngalleries.top
franklinfire.co fotzen-ficken.com
free-floating-buttons.com foxinsocks.ru
freelotto.com fr.netlog.com
freemaintenancesysforpcandmac.top free-laptop-reward.com
freewhatsappload.com freejabs.com
fres-news.com freshberry.com.ua
freshdz.com freshdz.com
funcrushgames.com ftns.ru
fuckmill.com
funnel.co.za funnel.co.za
gallerylisting.com funnymama.com
gaypornmovie.net fusoradio.info
gaytube.com gallerily.com
gazobeton-p.com.ua gamebackyard.com
gamedayassist.com
gamesprite.me
gardene.ru
gate5.co.za
gavuer.ru
gay-file.com
get-your-social-buttons.info
getaclueamerica.com getaclueamerica.com
given2.com getlamborghini.ga
golden-catalog.pro getprismatic.com
gheus.altervista.org
glall.ru
glopages.ru
goblacked.com
golden-praga.ru
gopixdatabase.com
goroda-vsego-mira.ru
grand-chlen.ru
greatfind-a.akamaihd.net
greendream.com.ua greendream.com.ua
gsbs.com.ua grizzlysgrill.com
growboxbl.ru
gsasearchengineranker.site
gsasearchenginerankerhelp.com
guidefs.ru
hanwei.us hanwei.us
hausfrauensex18.com hauleddes.com
hd720kino.ru hawaiielectriclight.com
hd-film.pl
hdimagegallery.net hdimagegallery.net
health-medical-portal.info heartofbeijing.blogspot.com
heartofpayne.xyz heatpower.ru
helicalpile.us helpmymacfaster.trade
histock.info hermesbirkinhandbagoutlets.com
hoholikik.club hidefiles.org
homedecoguide.info historichometeam.com
host-protection.com hobbyhuren-datenbank.com
hoterika.com homegardenlova.com
hotshoppymac.com hopeonthestreet.co.uk
houdom.net hotloans.ru
hplaserjetpdriver8y.pen.io houseofgaga.ru
huhn.altervista.org howmuchdoestizanidinecost.blogspot.com
hydropump.su hspline.com
i4track.net hut1.ru
iboss.com
icaseclub.ru
idc.com.ua idc.com.ua
ihc2015.info ideashome.id
ikritikimou.gr ilovevitaly.ru
imagecoolpub.com imagerydatabase.com
images-free.net images-free.net
images.gyffu.com imagez.co
impresagaia.it insider.pro
increasewwwtraffic.info insomniagamingfestival.com
indianmedicaltourismshop.com inzn.ru
infostatsvc.com iscblog.info
infowarcraft.ru isitwp.com
inome.com.ua
insta-add.pro
instabid.tech
intellekt21.ru
investyb.com
iqoption.com
islamtoday.co.za
istanbulit.com istanbulit.com
ixora.pro istizanidinelikexanax.blogspot.com
jerseyswholesalechinalimited.com jackwolfskinoutlet.online
jetsli.de janerikholst.se
jewelryandfiligree.com javatex.co.id
job.icivil.ir jobgirl24.ru
johnnyhaley.top katushka.net
kamen-e.ru keywordsking.com
kevblog.top kidd.reunionwatch.com
keywesthideaways.co kiinomaniak.pl
keywordblocks.com kimcurlrvsms.com
keywordsuggest.org kinocccp.net
kichenaid.ru kleine-titten.biz
kinoflux.net knowsitall.info
kinoplen.ru konica.kz
klumba55.ru kountrylife.com
kndxbkdx.bloger.index.hr krasnodar.xrus.org
kozhniebolezni.com kumuk.info
kupit-adenu.ru kunstaktien.info
labafydjxa.su lapitec.eu
languagecode.com laptoper.net
lashstudia.ru laudit.ru
letmacworkfaster.site letsart.ru
letmacworkfaster.world lineshops.biz
letslowbefast.today linkdebrideur.xyz
life.biz.ua liumimgx.pw
lion.conboy.us lombia.co
liupis.com longadventure.com
livetsomudvekslingstudent.bloggersdelight.dk luciddiagnostics.in
lkbennettoutlet.online magicalfind-a.akamaihd.net
loadopia.com
lonerangergames.com
longlifelomilomi.info
m.b00kmarks.com
macnewtech.com
managerpak204.weebly.com managerpak204.weebly.com
marblestyle.ru mantramusic.ru
martlinker.com mastershef.club
marwer.info matchpal-a.akamaihd.net
mazda-roadsters.com mcadamssupplyco.com
mearns-tractors.co.uk megakino.net
medicine-4u.org meget.co.za
menetie.ru
merryhouse.co.uk merryhouse.co.uk
millionare.com mexicosleevegastrectomy.com
mirmedinfo.ru microsoftportal.net
missclub.info miniads.ca
mobile.ok.ru mlf.hordo.win
monclerjacketsoutlet.pw mnogolok.info
moneymaster.ru mobilemedia.md
monitorwebsitespeed.com monetizationking.net
morepoweronmac.trade morpicert.pw
moscow-region.ru moskva.nodup.ru
muschisexbilder.com mp3downloadhq.com
musezone.su mrlmedia.net
musirc.com musicstock.me
my-floor.in.ua myfreemp3.eu
myfreetutorials.com myonlinepayday.co
myonigroup.com myyour.eu
na15.ru nardulan.com
narkologiya-penza.ru needtosellmyhousefast.com
narosty.com news-speaker.com
nationalbreakdown.com nextconseil.com
negral.pluto.ro nfljerseyscheapchinabiz.com
nero-us.com ningessaybe.me
newtechspb.ru nl.netlog.com
nfljerseys.online nonameread45.live
nfljerseysforsalewholesaler.com
nikitsyringedrivelg.pen.io
nosecret.com.ua
npoet.ru
nrv.co.za
nudejapan.net nudejapan.net
obesidadealgarve.com obiavo.in
olgacvetmet.com ok.ru
olvanto.ru oneminutesite.it
ooo-olni.ru oneshotdate.com
ooomeru.ru
openlibrary.org openlibrary.org
optom-deshevo.ru
orhonit.com orhonit.com
origin-my.ru outshop.ru
orion-v.com owohho.com
ourville.info oxford-book.com.ua
oz-offers.com oz-offers.com
partnerads.men parajumpersoutlet.online
partnersafe.men
pateaswing.com pateaswing.com
pavlodar.xkaz.org pay2me.pl
pcads.ru pepperstyle.ru
petedrummond.com perm-profnastil.ru
pharmacyincity.com perosan.com
phelissota.xyz perso.wanadoo.es
photochki.com phpdevops.com
pictures-and-images.net pic-re.blogspot.com
pierrehardysale.online pic2fly.com
pinwallpaper.xyz picquery.com
pl.id-forex.com pictures-and-images.com
playmsn.com pijoto.net
popander.mobi pintattoos.com
pornodojd.ru piter.xrus.org
pornoelita.info piuminiita.com
pornogig.com pixelrz.com
pornohub.me plastjulgranar.se
pornoklad.ru play-mp3.com
pornoxxx.com.mx play.leadzupc.com
predominant-invent.tk ploenjitmedia.azurewebsites.net
printdirectforless.com pogosh.com
proftests.net porno-play.net
prombudpostach.com.ua pornokajf.com
pron.pro pregnant.guru
pronekut.com preparevideosafesystem4unow.site
prospekt-st.ru privat-girl.net
proxyelite.biz prizefestival.mobi
q-moto.ru profitwithalex.info
qwesa.ru prom23.ru
r-control.ru promoforum.ru
randki-sex.com prostitutki-astana.org
rankscanner.com prostitutki-novgoroda.org
rasteniya-vs-zombi.ru prostitutki-yaroslavlya.org
rating-bestcasino.com realting-moscow.ru
redbottomheels.xyz recipedays.com
refudiatethissarah.info refudiatethissarah.info
registrysweeper.com relatodelpresente.com.ar
remont-comp-pomosh.ru remont-mobile-phones.ru
rennlist.com remote-dba.de
resultshub-a.akamaihd.net replica-watch.ru
rethinkwasteni.info rfid-locker.co
rfd-split.hr
rialp.getenjoyment.net
ricorsogiustizia.org
rock-cafe.info
roll123.com
rollercoin.com rollercoin.com
rootandroid.org roofers.org.uk
royal-betting.net rozalli.com
royalads.net rvtv.ru
rvi.biz sa-live.com
sa-rewards.co.za safe-app.net
sammlungfotos.online sanyuprojects.com
sarahmilne.top sbornik-zakonov.ru
saveriopiazza.it scanner-donald.top
savingsslider-a.akamaihd.net
sawin.beth.webd.pl
sayyoethe.blogspot.co.za
sbdl.no
sbf441.com
scanner-alexa.top
scanner-john.top
scanner-margo.top scanner-margo.top
scanner-nelson.top
scanner-walter.top
scenicmissouri.us scenicmissouri.us
school-diplomat.ru sdsjweb.com
scrnet.biz.ua
sdrescher.net
searchaddis.com searchaddis.com
searchinterneat-a.akamaihd.net searchimage.co
securityallianceservices.com searchwebknow-a.akamaihd.net
seksotur.ru seoboxes.com
selectads.men seocheki.net
semaltmedia.com serptehnika.ru
seocdvig.ru service.adtech.us
seoexperimenty.ru seryeznie-znakomstva.ru
seogadget.ru sevendays.com.ua
seorankinglinks.xyz sexvporno.ru
servethis.com shell-pmr.ru
sex-pr.net
sexpartygirls.net
sextracker.de
shemalegalls.blogporn.in shemalegalls.blogporn.in
shop-electron.ru
shoppingmiracles.co.uk
si-unique.com si-unique.com
sildenafil-tadalafil.info sibvitr.ru
sivs.ru siteheart.net
sixcooler.de
slavic-magic.ru slavic-magic.ru
small-games.biz smichovbike.cz
smart-scripts.com socialsignals24.com
smartbalanceworld.com softomix.net
smsactivator.ru soochi.co
soft-program.com spaceshipad.com
softlinesolutions.me spinnerco.ca
sogimlecal.tk stardevine.com
souvenirua.com staticfs.host
spidtest.space stpicks.com
splendorsearch-a.akamaihd.net stpolice.com
spravka-medosmotr.ru
squidoo.com
stickers-market.ru
stillmiracle.com
storehouse.ua
stroiminsk.org
stroydetali.ru stroydetali.ru
studworks.org studentguide.ru
sugarlyflex.pw sukarame.net
suggest-keywords.com supermodni.com.ua
sumo.com survival.betteroffers.review
superlist.biz swingerseiten.com
swsociety.se syndicate.fun
syvertsen-da.no tackletarts.co
taihouse.ru taihouse.ru
tako3.com tattooreligion.ru
tanieaukcje.com.pl teencastingporn.com
tasteidea.com telefonsexi.com
td-33.ru tengohydar.tk
tecnoteakviareggio.it thedownloadfromwarez.blogspot.com
templates.franklinfire.co theendivechronicles.com
texbaza.by theplacetoupdating.pw
thecounter.com thexart.club
thelottosecrets.com timecrimea.ru
thfox.com titan-ads.life
tizanidine4mgprice.blogspot.com tizanidine4mgprice.blogspot.com
tizanidinefibromyalgia.blogspot.com tizanidine6mg.blogspot.com
top1-seo-service.com tizanidineandgabapentin.blogspot.com
topcasinoratings.ru tizanidinecapsules.blogspot.com
touchmods.fr tizanidinehydrochloride4mgstreetvalue.blogspot.com
trackzapper.com tkanorganizma.ru
trafficmp.com top-deal.com.pl
trafficstars.com topads.men
trk-4.net toproadrunner5.info
turist-strani.ru totu.us
twinderbella.com tourismvictoria.com
ucoz.ru toursmaps.com
uggbootsoutletsale.us toys.erolove.in
ukrtextbook.com tracfone.com
unrealcommander.org traffic-club.info
urlopener.blogspot.com.au trafficmania.com
valsalud.com traffixer.com
vavilone.com traffpartners.com
velobikestock.com triumf-realty.ru
vesnatehno.ru turizm.bz
videochat.mx twodollarshows.com
videokrik.net txxx.com
vids18.site tyumen.xrus.org
viven.host.sk u555u.info
vkonche.com unitygame3d.com
voluumtrk.com urengoy.pro
voronezh.xrus.org ustion.ru
vostoktrade.info vacuumscleaner.com
vpnmouse.com validccseller.com
vapomnoncri.tk
varikozdok.ru
viandpet.com
video-hollywood.ru
videospornogratisx.net
vielporno.net
virtualbb.com
virus-schutzmasken.de
vitoriacabos.com
vizag.kharkov.ua
volgograd.xrus.org
vseigry.fun
vzubah.com
wahicbefa31.soup.io
wait3sec.org
wapsite.me
wareseeker.com wareseeker.com
warningwar.ru wdfdocando.com
webcamtalk.net webadvance.club
webshoppermac.com webmasterhome.cn
website-analytics.online weboptimizes.com
website-stealer.nufaq.com webstatsdomain.org
wellcome2slovenia.ru whatistizanidinehclusedfor.blogspot.com
westermarkanjou.se wingsoffury2.com
westum.se woodyguthrie.se
whengirlsgowild.com word-vorlagen.net
wholesalejerseysgaa.com worldtraveler.world
williamrobsonproperty.com wowas31.ucoz.ru
writingservices17.blogspot.ru wpsecurity.website
wrz0iuebwhp5fg.freeddns.com writersgroup580.web.fc2.com
www.arenda-yeisk.ru ws.ampower.me
www.get-free-traffic-now.com wstroika.ru
x-diesel.com www.wohnkabinen-shop.de
xboxster.ru xcc24.pl
xcombear.ru xdoza.com
xfluro.com
xgames-04.com xgames-04.com
xkaz.org xn--80abgj3a5acid6ghs.top
xn-----6kcaabbafhu7cskl7akvongwpo7hvjj.xn--p1ai xn--hxazdsfy.blogspot.com
xn----dtbndd4ae7eub.top xtrafficplus.com
xn--80ajbshivpvn2i.xn--p1ai xxart.ru
xn--b1adccaf1bzj.xn--p1ai xxxhdvideo.site
xn--c1acygb.xn--p1ai xxxsiterips.xyz
xn--lifehacer-1rb.com
xxxguitars.com
xzlive.com
yachts-cruise.info
yoopsie.com yoopsie.com
youradulthosting.com youdesigner.kz
yurgorod.ru zagadki.in.ua
za-fun-offer.com zaobao.com.sg
zajm-zalog-krasnodar.ru zarepta.com
zambini.ru zawyna.ua
zastroyka.org zero1.it
zatjmuzu.info zeroredirect6.com
zeroredirect.com zkjovpdgxivg.ga
zetmaster.ru zlatnajesen.com
zigarettenonlinekaufen.tumblr.com znakomstvaonlain.ru
zigzog.ru zojirushi-products.ru
znaturaloriginal.com zx6.ru
zvooq.eu

490
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
101raccoon.ru 00it.com
123cha.com 03e.info
2daytrendingnews.com 1314dh.com
6hopping.com 2020iscoming.info
a.pr-cy.ru 2pxg8bcf.top
3w24.com
5forex.ru
6128786.com
aa9d046aab36af4ff182f097f840430d51.com aa9d046aab36af4ff182f097f840430d51.com
aasoldes.fr adf.ly
acc.eu.org adultmeetonline.info
actulite.com ahrntal.verymes.xyz
adanih.com amateurgalls.com
adidas.frwebs.fr anal-acrobats.hol.es
adsloads.com arewater.com
agahinameh.com asiengirls.net
allblogroll.com asociatia-tipografilor-transilvania.ro
amazingninja.com ausmepa.org.au
arendovalka.xyz auto.rusvile.lt
arius.tech autoplate.info
asiavirtualsolutions.com ayeartoforget.com
backgroundpictures.net ayerbo.xhost.ro
beauby.ru bang-hotties.com
best-businessman.ru banki76.ru
best-seo-solution.com bannerspace.com
bet-prognoz.com bensbargains.net
bg6s0.com betterscooter.com
biznesschat.net bfz.biz
blogos.kz blog.koreadaily.com
bookmark4you.com blpmovies.com
boole.onlinewebshop.net bristolhotel.com
bronzeaid-a.akamaihd.net bumble.cheapwebsitehoster.com
brothers-smaller.ru buyessay3.blogspot.ru
bupropion-sr-150-mg.us buzzsumo.com
burger-imperia.com bwlx.prepedu.cn
businesn.men canyougethighofftizanidine.blogspot.com
cablecar.us cartierreplica.pw
carina-sy.de cartierring.xyz
cityadspix.com cartiertrinity.xyz
clash-clans.ru cheap-pills-norx.com
classiquebijoux.ru chiptuninger.com
compliance-barak.xyz choice-credit.ru
compliance-george.xyz cjmc.info
coolwallpapers4k.info clarithromycin500mg.com
coprofam.org codysbbq.com
dame-ns.kz coindirect.io
dev.citetick.com compliance-alexa.xyz
dichvuvesinhhanoi.com compliance-don.xyz
directrev.com compliance-jessica.top
dividendo.ru compliance-john.top
dkvorota.ru conboy.us
domik-derevne.ru cowblog.fr
e-avon.ru credit-online.ws
crest-poloski.ru
cvety24.by
cyse.tk
daretodonate.co
dear-diary.ru
dikqlyremy.info
divci-hry.info
domain-tracker.com
dominateforex.ml
e-biznes.info
education-cz.ru education-cz.ru
elektir.ru eloconcream.blogspot.com
elektronischezigarettekaufen1.beeplog.com englate.com
elitedollars.com essaypro.com
escort-russian.com eugenevaultstorage.com
esopini.com executehosting.com
excaliburfilms.com ezigarettenshop2.myblog.de
exoneration-project.us facebook-mobile.xyz
ezigarettenshop1.yolasite.com familienzahnaerzte.com
farm26.ru faracontrol.ir
fast-wordpress-start.com findthe.pet
fearcrow.com fingerprintjs.com
ferretsoft.com free-gluten.ru
flowersbazar.com freecamdollars.com
flytourisme.org froggytube.com
forex.osobye.ru fxtips.ru
forsex.info gameprimary.com
forum-engineering.ru ge0ip.com
franklinfire.co gelezki.com
free-floating-buttons.com get-seo-domain.com
fres-news.com googlemare.com
gallerily.com graphicwe.org
gameplexcity.com greendream.com.ua
gearcraft.us greenidesign.co
getmyads24.com gyffu.com
ghernnqr.skyrock.com hamilton.ca
go2album.com handicapbathtubarea.com
gotorussia.com hasfun.com
hawaiisurf.com hdpixion.com
herehloadibs.cf hebr.myddns-flir.com
hunterboots.online heliko.no
immobiliaremassaro.com heroz.fr
in-tandem.co hitcpm.com
incolors.club homedecorpicture.us
int.search.mywebsearch.com hoverboard360.es
iqs.biz.ua ictizanidinehcl4mg.blogspot.com
irunfar.com ideoworld.org
iscblog.info ilovevitaly.com
ivearchenceinflu.cf inclk.com
javspace.net indo-export.ru
jerseysbizwholesalecheap.com itrevolution.cf
jestr.org izoll.ru
johnrobertsoninc.com jovencitas.gratis
kabbalah-red-bracelets.com joyceblog.top
kakablog.net juliadiets.com
kerwinandcariza.com justucalling32211123456789.tk
kidskunst.info keyhantercume.com
kinotorka.ru keywords-monitoring-your-success.com
knowyournextmove.com kometars.xyz
koleso24.com.ua komputernaya-pomosh-moscow.ru
kongoultry.net konpax.com
kurwa.win konteiner24.com
la-fa.ru kostenloser-sex.com
lashstudia.ru krasnodar-avtolombards.ru
letsart.ru kupiproday.com.ua
lifehacĸer.com laptop-4-less.com
locimge.pw liffa.ru
locustdesign.co lion.playtap.us
loginduepunti.it lollypopgaming.com
lombia.co lomza.info
mac-shield.com ltvperf.com
macfix.life luckyshop.net.ua
magazintiande.ru maranbrinfo.com.br
matematikus.info marmotstore.online
matrixalchemy.com masterseek.com
megakino.net mazda-roadsters.com
menetie.ru mediawhirl.net
migente.com mhtr.be
minneapoliscopiers.com minet.club
mobot.site mirzonru.net
mobplayer.net mixtapetorrent.com
monclercheap.xyz mrsdalloways.com
morefastermac.trade mycouponizemac.com
motherboard.vice.com mygameplus.ru
muz-shoes.ru mylida.org
mybackgroundlandscape.blogspot.com nastroyke.net
najaden.no naturalbreakthroughsresearch.com
nfvsz.com nero-us.com
njpalletremoval.com nhl09.ru
nw-servis.ru nicovideo.jp
obiavo.su nikitabuch.com
oddamzadarmo.eu nuker.com
oklogistic.ru o-o-6-o-o.ru
onlinadverts.com oconto.ru
onlineku.com ohmyrings.com
ontargetseo.us online247.ml
partnerworkroom.men oralsexfilme.net
pharmacyincity.com oren-cats.ru
picsearch.com ossmalta.com
pogosh.com oyster-green.com
poisk-zakona.ru patol01.pw
pony-business.com pensplan4u.com
poppen-nw.net perl.dp.ua
pornoplen.com pic-re.blogspot.com
picswe.com
pintattoos.com
pizdeishn.com
poffet.net
pogruztehnik.ru
pornowarp.info pornowarp.info
postclass.com pp-budpostach.com.ua
prchecker.info
princeadvantagesales.com princeadvantagesales.com
printie.com projectforte.ru
printingpeach.com
prospekt-st.ru
pussyspace.net pussyspace.net
quickloanbank.com quelle.ru
qwertty.net qwesa.ru
rank-checker.online rcpmda.ikan1080.xyz
rating-bestcasino.com reelheroes.net
realizmobi.com regdefense.com
rednise.com reimageplus.com
refads.pro relax.ru
replicalouboutin.xyz remmling.de
rethinkwasteni.info resultshub-a.akamaihd.net
rfserial.net risparmiocasa.bz.it
russia-tao.ru ruspdd.com
sale-japan.com sajatvelemeny.com
saltspray.ru scanmyphones.com
scanner-fred.top security60-e.com
scanner-irvin.top sell-fb-group-here.com
scansafe.net seoprofiler.com
scoopquest.com servingnotice.com
secret.xn--oogle-wmc.com sexy-pings.com
sei80.com sibdevice.ru
seksvideoonlain.com sivs.ru
selectads.men skytraf.xyz
seo-prof1.xyz slujbauborki.ru
seofied.com smichovbike.cz
sex-pr.net soblaznu.net
sex-sex-sex5.com social-search.me
sexblog.pw sodexo.com
sexfreepornoxxx.com soserfis.com
sexkvartal.com stal-rulon.ru
sexs-foto.top stefanbakosab.se
sexxdate.net stmassage.ru
signal03.ru stoki.ru
signoredom.com streetfire.net
sirpornogratis.xxx stroimajor.ru
site.ru suture.co
siteripz.net svarkagid.com
slomm.ru taboola.com
snjack.info takeprofitsystem.com
social-buttons.com taxi-v-eisk.ru
solicita.info tecnoteakviareggio.it
sonnikforme.ru tedy.su
speeddream.xyz thaisamkok.com
spinnerco.ca thedownloadfromwarez.blogspot.com
sport7777.net tizanidine4mg.blogspot.com
sportbetfair.com tizanidinecost.blogspot.com
spravka-medosmotr.ru tizanidinedosage.blogspot.com
spywarespy.com tizanidinehydrochloride4mgstreetvalue.blogspot.com
stair.registrydomainservices.com tizanidinevscyclobenzaprine.blogspot.com
surflinksmedical.com tmtrck.com
swinginwithme.ru topdownloads.ru
syndicate.fun topquality.cf
tarad.com topsy.com
tetracsaudi.com track-rankings.online
thetoiletpaper.com traderzplanet.in
thewomenlife.com ttrraacckkrr.com
tivolibasket.it tvorozhnaja-zapekanka-recept.ru
tizanidinehcl2mgtablet.blogspot.com tx41tclega.ru
tizanidinehcl4mgtabinfo.blogspot.com umg-stroy.ru
tizanidineingredients.blogspot.com uogonline.com
tizanidinepills.blogspot.com uptime-alpha.net
top-deal.com.pl ural-buldozer.ru
toyota.7zap.com urdoot.win
traffictrade.life usacasino.com
tryrating.com v720hd.ru
tsyndicate.com vepad.com
turizmus.us videonsk.com
uamtrk.com vids18.site
ucoz.ru webstatsdomain.org
ukrtextbook.com wholesalejerseys-cheapest.com
undergroundcityphoto.com wovis.site
uptimebot.net ww2awards.info
viagra.pp.ua x-diesel.com
video-woman.com xmlinde.com
vipsiterip.org xn--b1adccaf1bzj.xn--p1ai
vkak.ru yachts-cruise.info
vse-pesni.com yourads.website
wapsite.me yourserverisdown.com
warningzscaler.heraeus.com youthreaders.com
weboptimizes.com yugk.net
webs.com zaimhelp.ru
website-audit.com.ua zakonobosago.ru
wjgony.com zarepta.com
xcombear.ru zlatnajesen.com
xerox-douglas.cf zona-aqua.ru
xmnb.net
xn--80aeahghtf8ac5i.xn--p1ai
xvideospornoru.com
xxxguitars.com
yadro.ru
youjizz.vc
yubikk.info
za-fun-offer.com
zambini.ru
zeroredirect10.com
znakomstvaonlain.ru
zoomovies.org
zootravel.com
zreloeporno.tv
zvetki.ru

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.188 0; 10.1.0.104 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.08.2546 ### Version: V4.2021.08.2547
### Updated: Tue Aug 24 10:03:15 UTC 2021 ### Updated: Tue Aug 24 22:01:16 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.08.2546 ### Version: V4.2021.08.2547
### Updated: Tue Aug 24 10:03:15 UTC 2021 ### Updated: Tue Aug 24 22:01:16 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##