vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-06 04:30:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.06.3159 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-06-10 22:07:32 +00:00
parent e5bea15c93
commit de5ec13648
46 changed files with 96928 additions and 96928 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.06.3158 #### Version: V4.2022.06.3159
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.06.3158 #### Version: V4.2022.06.3159
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.06.3158 #### Version: V4.2022.06.3159
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

11980
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.06.3158 #### Version: V4.2022.06.3159
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

11984
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11984
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

11984
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11984
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 1; 10.1.0.99 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 0; 10.1.0.99 0;
127.0.0.1 0; 127.0.0.1 0;

11984
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 1; 10.1.0.99 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 0; 10.1.0.99 0;
127.0.0.1 0; 127.0.0.1 0;

11984
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 1; 10.1.0.99 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 0; 10.1.0.99 0;
127.0.0.1 0; 127.0.0.1 0;

11984
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3158 3159

11984
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

11984
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.225.149.163 0; 13.90.116.222 0;

11984
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 1; 10.1.0.99 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 0; 10.1.0.99 0;
127.0.0.1 0; 127.0.0.1 0;

11984
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 1; 10.1.0.99 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 0; 10.1.0.99 0;
127.0.0.1 0; 127.0.0.1 0;

11984
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 1; 10.1.0.99 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 0; 10.1.0.99 0;
127.0.0.1 0; 127.0.0.1 0;

11984
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.225.149.163 0; 13.90.116.222 0;

11984
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.225.149.163 0; 13.90.116.222 0;

11984
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 1; 10.1.0.99 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
HEADMasterSEO BotALot
HTTP::Lite IDBot
IDBTE4M NetAnts
Indy Library Nimbostratus
Magnet Rankivabot
RankingBot Telesphorep
Snapbot Web Collage
Spammen Web Enhancer
plumanalytics Whatweb
sp_auditbot webmeup-crawler

328
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
360Spider 360Spider
ADmantX AIBOT
ALittle Client ALittle Client
Abonti Abonti
Acunetix
AfD-Verbotsverfahren AfD-Verbotsverfahren
AiHitBot
Aipbot
Alexibot
AllSubmitter AllSubmitter
Anarchy99 Anarchy99
Anthill Asterias
Aspiegel Attach
AwarioRssBot AwarioRssBot
BLEXBot BLEXBot
BackDoorBot BackDoorBot
BackStreet
BackWeb
Backlink-Ceck
Barkrowler Barkrowler
BatchFTP Bigfoot
Bitacle BlackWidow
Blow BotALot
BlowFish
Boardreader
Bolt
Brandprotect
BuiltBotTough BuiltBotTough
BuiltWith BuzzSumo
Bullseye CSHttp
CCBot Calculon
CODE87 CheeseBot
CazoodleBot Cocolyzebot
Cegbfeieh
CensysInspect
CheTeam
CherryPicker
Chlooe
Citoid
Claritybot
Cogentbot
CopyRightCheck
Copyscape
Craftbot Craftbot
Crawling at Home Project
CrazyWebCrawler CrazyWebCrawler
Crescent Crescent
Custo
DBLBot
DIIbot
DataCha0s
Demon
Deusu Deusu
Devil Devil
Digincore
DigitalPebble
Dirbuster
Discobot Discobot
Discoverybot Discoverybot
DittoSpyder DittoSpyder
DomainCrawler DnBCrawler-Analytics
DnyzBot
DomainAppender
DomainSigmaCrawler DomainSigmaCrawler
Dragonfly Domains Project
Dotbot
EMail Wolf EMail Wolf
EirGrabber EasyDL
ExtLinksBot EroCrawler
Exabot
Extreme Picture Finder Extreme Picture Finder
FDM Ezooms
FHscan FemtosearchBot
Fimap
Firefox/7.0 Firefox/7.0
Flunky
Foobot
Freeuploader
Fuzz
Fyrebot Fyrebot
Genieo G-i-g-a-b-o-t
GermCrawler
Getintent
Gigabot Gigabot
Go!Zilla
GoZilla GoZilla
Gotit Grabber
GrabNet GrapeFX
GrapeshotCrawler HEADMasterSEO
HMView
HTTP::Lite HTTP::Lite
Heritrix Haansoft
HaosouSpider
Hloader Hloader
HonoluluBot HonoluluBot
Humanlinks IDBot
IRLbot Id-search
Iblog Image Fetch
Image Sucker Image Sucker
Internet Ninja InfoTekies
Intelliseek
InterGET
InternetSeer InternetSeer
Iskanie JetCar
JOC Web Spider JikeSpider
JamesBOT
Jbrofuzz
Jetty
Jorgee
JustView
Jyxobot Jyxobot
Keybot Translation-Search-Machine
Kinza
Kozmosbot
LWP::Simple LWP::Simple
Lanshanbot
Leap Leap
LeechFTP
LexiBot LexiBot
Libwhisker Lftp
Likse LibWeb
Linkbot LieBaoFast
Lightspeedsystems
LinkScan
LinkextractorPro
LinkpadBot
LinqiaRSSBot
Lipperhey Spider
Litemage_walker Litemage_walker
MFC_Tear_Sample Lmspider
Ltx71
MIDown tool MIDown tool
MJ12bot MIIxpc
MQQBrowser
MSFrontPage
MSIECrawler
Magnet Magnet
Majestic12 Mass Downloader
MarkMonitor MauiBot
Mb2345Browser MeanPath Bot
Mediatoolkitbot Meanpathbot
Moblie Safari MegaIndex.ru
NPbot Mister PiX
Navroad Mojeek
Mojolicious
Mr.4x3
Msrabot
Nessus Nessus
NetAnts Net Vampire
NetMechanic NetZIP
Niki-bot Netcraft
Nettrack
Nikto
NimbleCrawler NimbleCrawler
Nutch Nmap
Octopus Offline Explorer
OnCrawl Offline Navigator
OpenVAS
Openvas Openvas
OrangeBot OrangeBot
PECL::HTTP OutfoxBot
PHPCrawl Page Analyzer
POE-Component-Client-HTTP PageAnalyzer
PageThing.com
Panscient
Papa Foto Papa Foto
Picsearch Petalbot
PictureFinder Pimonster
Pixray PleaseCrawl
ProPowerBot ProPowerBot
ProWebWalker Psbot
Probethenet
Pu_iN
Pump Pump
PxBroker RSSingBot
PyCurl
Quick-Crawler
RankActive
RankActiveLinkBot
RankingBot
RankingBot2 RankingBot2
Rankivabot
RankurBot
Re-re Re-re
ReGet RebelMouse
Reaper RedesScrapy
RepoMonkey
Ripper
RocketCrawler
Rogerbot Rogerbot
SEOkicks-Robot SBIder
SEOprofiler SEOkicks
SEOstats SEOlyticsCrawler
SISTRIX SISTRIX
ScanAlert SalesIntelligent
Scanbot Scanbot
ScoutJet ScoutJet
ScreenerBot Scrapy
Screaming
ScrepyBot ScrepyBot
Searchestate Searchestate
SearchmetricsBot Semrush
SentiBot SeoSiteCheckup
Seomoz SeobilityBot
Shodan SiteCheckerBotCrawler
Site Sucker SiteLockSpider
Sitebeam Sitebeam
SlySearch Siteimprove
Snake Snake
Snapbot Snapbot
SocialRankIOBot Sociscraper
Sogou web spider Sosospider
SpaceBison
Spammen Spammen
SpankBot Spbot
Spanner
Spinn3r Spinn3r
SputnikBot
Sqlworm Sqlworm
Steeler
Stripper
SuperBot
Surfbot
Suzuran
T0PHackTeam T0PHackTeam
TeleportPro Telesphoreo
Telesoft
Telesphorep Telesphorep
The Intraformant TheNomad
Thumbor
TightTwatBot TightTwatBot
Titan
Toweyabot
Tracemyfile Tracemyfile
Trendiction Turnitin
TurnitinBot TurnitinBot
V-BOT Typhoeus
Vagabondo URLy.Warning
VeriCiteCrawler VelenPublicWebCrawler
VoidEYE VoidEYE
Voil Voil
Voltron
WASALive-Bot WASALive-Bot
WEBDAV
WISENutbot WISENutbot
WPScan WWW::Mechanize
WWW-Collector-E WeSEE
WWW-Mechanize Web Auto
Web Enhancer Web Sauger
WebAuto Web Sucker
WebBandit
WebEnhancer WebEnhancer
WebFuck WebImageCollector
WebLeacher
WebPix WebPix
WebReaper WebReaper
WebStripper WebStripper
WebZIP WebZIP
Webshag WebmasterWorldForumBot
WebsiteQuester Webster
Whack Whack
Whacker
WinHTTrack
Wonderbot Wonderbot
Woobot Xaldon_WebSpider
Wprecon Zauba
Xenu ZmEu
Zermelo ZoomBot
Zitebot ZoominfoBot
ZumBot
ZyBorg
adscanner adscanner
archive.org_bot cah.io.community
arquivo-web-crawler
autoemailspider
backlink-check
cognitiveseo cognitiveseo
com.plumanalytics
crawl.sogou.com crawl.sogou.com
crawler4j domainsproject.org
dataforseo.com facebookscraper
demandbase-bot gopher
eCatch heritrix
evc-batch
instabid
isitwp.com
iubenda-radar iubenda-radar
linkdexbot linkdexbot
lwp-request
lwp-trivial
magpie-crawler magpie-crawler
meanpathbot
mediawords mediawords
pcBrowser pcBrowser
s1z.ru probe-image-size
satoristudio.net scalaj-http
scan.lol sexsearcher
seobility
seocompany.store
seoscanners
seostar
serpstatbot
sogouspider sogouspider
trendiction.com sp_auditbot
trendiction.de spyfu
webgains-bot tAkeOut
webmeup-crawler webmeup-crawler
webpros.com x09Mozilla
webprosbot zauba.io
xpymep1.exe
zgrab

278
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 360Spider
404checker
404enemy
80legs 80legs
ADmantX ADmantX
Aboundex Acunetix
Aboundexbot
AiHitBot AiHitBot
Aipbot
Alligator Alligator
Anarchy AlphaBot
Anthill Anthill
Apexoo
Aspiegel
Asterias
Atomseobot
AwarioRssBot AwarioRssBot
BBBike AwarioSmartBot
BDCbot
BDFetch BDFetch
BLEXBot
BackStreet
BackWeb BackWeb
Backlink-Ceck
Badass Badass
Bandit
Barkrowler
BatchFTP BatchFTP
Battleztar Bazinga Battleztar Bazinga
Bigfoot Bigfoot
Bitacle Bitacle
Black Hole
BlackWidow
BlowFish BlowFish
Boardreader Boardreader
BotALot Brandwatch
Brandprotect
BuiltWith BuiltWith
Bullseye CSHttp
BunnySlippers CazoodleBot
BuzzSumo
CCBot
Calculon
CherryPicker CherryPicker
ChinaClaw
Chlooe
Claritybot
Cliqzbot
Collector Collector
Copier
CopyRightCheck CopyRightCheck
Copyscape
Cosmos Cosmos
CrazyWebCrawler Crescent
CrunchBot CrunchBot
Curious
Custo Custo
CyotekWebCopy DBLBot
DIIbot
DSearch DSearch
DataCha0s
Devil
Digincore Digincore
Dirbuster
Discobot Discobot
DittoSpyder DittoSpyder
DnBCrawler-Analytics
DnyzBot
DomCopBot
DomainCrawler DomainCrawler
ECCP/1.0 DomainSigmaCrawler
DomainStatsBot
Domains Project
EMail Wolf
EasyDL EasyDL
Ecxi
EirGrabber
Exabot Exabot
ExtractorPro
Extreme Picture Finder
Ezooms
FDM FDM
Firefox/7.0 FHscan
Freeuploader FemtosearchBot
Fimap
Flunky
FrontPage
Fuzz
FyberSpider FyberSpider
G-i-g-a-b-o-t Genieo
GetRight GoZilla
GetWeb GrabNet
Getintent
Gigabot
Go!Zilla
Go-Ahead-Got-It
Grabber Grabber
GrapeFX HEADMasterSEO
GrapeshotCrawler
GridBot
HMView HMView
HTMLparser
HTTP::Lite HTTP::Lite
HTTrack HTTrack
Haansoft Haansoft
HaosouSpider HaosouSpider
Harvest
Havij
Hloader
HonoluluBot
Humanlinks
IRLbot IRLbot
Id-search Iblog
IlseBot IlseBot
Image Sucker Image Sucker
IndeedBot IndeedBot
Internet Ninja Intelliseek
InternetSeer InterGET
Iskanie
IstellaBot IstellaBot
Jorgee
JustView JustView
Keybot Translation-Search-Machine Kenjin Spider
Keyword Density Keyword Density
Kinza
Kozmosbot Kozmosbot
Lanshanbot
Larbin Larbin
Leap
LeechGet LeechGet
LexiBot Libwhisker
LinksManager
LinqiaRSSBot LinqiaRSSBot
Lipperhey Lipperhey Spider
MIIxpc Litemage_walker
MJ12bot MJ12bot
MQQBrowser
MSIECrawler
MTRobot
Mag-Net Mag-Net
Magnet
Majestic SEO Majestic SEO
MarkWatch
Mass Downloader
MauiBot
MeanPath Bot MeanPath Bot
Meanpathbot Meanpathbot
Mediatoolkitbot Mediatoolkitbot
Metauri
Microsoft Data Access
Minefield Minefield
MolokaiBot
Morfeus Fucking Scanner
Mr.4x3 Mr.4x3
Msrabot Msrabot
Musobot Musobot
NICErsPRO Navroad
Nameprotect
NearSite
Needle Needle
NetLyzer NetLyzer
NetSpider NetMechanic
NetZIP
Netcraft Netcraft
Nettrack Niki-bot
Netvibes
NextGenSearchBot
Nibbler
Nikto Nikto
Nimbostratus Nimbostratus
Ninja Ninja
OnCrawl Not
Nuclei
Octopus
OpenLinkProfiler OpenLinkProfiler
Openfind
Openvas Openvas
OrangeSpider OrangeSpider
PECL::HTTP OutclicksBot
OutfoxBot
PHPCrawl PHPCrawl
Page Analyzer PageScorer
PageAnalyzer
PageThing.com PageThing.com
PeoplePal Pandalytics
Pi-Monster Papa Foto
Picsearch Pavuk
PictureFinder Pixray
Pockey PleaseCrawl
Psbot ProPowerBot
Probethenet
Pu_iN Pu_iN
Pump Quick-Crawler
PyCurl
RankActive RankActive
RankActiveLinkBot
RankingBot RankingBot
RankingBot2 RealDownload
Recorder
RedesScrapy
Ripper Ripper
RocketCrawler SBIder
SEOkicks SEOkicks
SEOkicks-Robot
SEOlyticsCrawler SEOlyticsCrawler
SEOstats SEOstats
SalesIntelligent
ScanAlert ScanAlert
Scanbot Scanbot
ScoutJet Scrapy
Screaming ScrepyBot
Seekport Seekport
SeobilityBot Seomoz
SiteCheckerBotCrawler Shodan
SiteLockSpider Siphon
SiteExplorer
SiteSucker
Sitebeam
SmartDownload SmartDownload
Snoopy Snoopy
Sosospider SpaceBison
Spammen
Spbot Spbot
Spinn3r Spinn3r
Sqlmap Sqworm
Sqlworm Sucuri
Steeler
Stripper
SuperHTTP SuperHTTP
Surfbot
Swiftbot Swiftbot
T8Abot
TeleportPro TeleportPro
Telesphoreo Telesoft
Telesphorep Telesphorep
TightTwatBot TheNomad
Titan
Tracemyfile
Turingos
Turnitin Turnitin
TwengaBot TurnitinBot
Twice Typhoeus
URLy Warning
UnisterBot UnisterBot
Vacuum V-BOT
VB Project
VCI
VelenPublicWebCrawler
VeriCiteCrawler VeriCiteCrawler
VidibleScraper Virusdie
VoidEYE VoidEYE
Voltron
WASALive-Bot WASALive-Bot
WBSearchBot Wallpapers
WPScan
WWW-Collector-E
WWW::Mechanize
WeSEE
Web Auto Web Auto
Web Collage
Web Enhancer Web Enhancer
Web Pix Web Sauger
Web Sucker
WebBandit WebBandit
WebCollage WebCollage
WebFetch WebCopier
WebFuck WebSauger
WebImageCollector WebSucker
WebLeacher WebWhacker
WebPix
WebStripper
WebZIP
Webalta
Website Quester Website Quester
WebsiteQuester WebsiteQuester
Whatweb Whatweb
Widow Widow
WinHTTrack Wotbox
Wonderbot Wprecon
Woobot Xaldon WebSpider
Xenu Xenu
Zade Zermelo
Zauba adscanner
Zeus arquivo.pt
ZyBorg
autoemailspider autoemailspider
coccocbot backlink-check
com.plumanalytics cah.io.community
cognitiveseo
crawler.feedback crawler.feedback
crawler4j
domainsproject.org domainsproject.org
eCatch
evc-batch evc-batch
heritrix
internetVista monitor internetVista monitor
ips-agent lwp-request
muhstik-scan lwp-trivial
meanpathbot
mediawords
netEstate NE Crawler netEstate NE Crawler
plumanalytics pcBrowser
polaris version probe-image-size
satoristudio.net ripz
scan.lol s1z.ru
scalaj-http
seobility seobility
seostar
serpstatbot
sitechecker.pro sitechecker.pro
siteripz sysscan
spyfu
trendiction.de
webprosbot webprosbot
zauba.io
zgrab zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
coolwallpapers-hd.com cbozhe.com
digitalfaq.com chorus.terakeet.com
gamingspark.com derevesendeco.com
girlspicsa.com gcup.ru
javcoast.com indetiske.ya.ru
jerseywholesalechinabiz.com konkursowo-24.pl
lazymanyoga.com mobilierland.com
physfunc.ru playlott.com
promalp-universal.ru telefonsexi.com
scrapy.org tizanidinehcl2mgtablet.blogspot.com

934
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
03e.info 00it.com
123cha.com 1000n1.ru
1688.com 1001desert.com
1bet.com 18ps.ru
1j7740kd.website 1flag.co.za
1stat.ru 24-ak.ru
20pascals.nl 2728fb936f0.com
2itech.ru 3dracergames.com
300richmond.co.nz 51unlim.ru
98oi.ru 5kstore.com
aa14ab57a3339c4064bd9ae6fad7495b5f.com aab94f698f36684c5a852a2ef272e031bb.com
aanapa.ru abctoppictures.net
above.com adamoads.com
acheterviagrafr24.com adinterax.com
adbetclickin.pink
adclickthru.net
add-add.men
admanaerofoil.com admanaerofoil.com
adultgalls.com ads-cool.pro
adultmeetonline.info adultfullhd.com
adultnet.in aerogo.com
albamargroup.com affrh2015.com
aktobe.xkaz.org
alert-fdm.xyz alert-fdm.xyz
algarveglobal.com
all-streaming-media.com all-streaming-media.com
alldubai.biz allblogroll.com
allforminecraft.ru amigobulls.com
alltheviews.com anal-acrobats.hol.es
amateurmatch.com android-systems.ru
ameblo.jp angkortours.vn
animal-drawings.com anjalika.co.in
anmysite.com apccargo.com
aosexkontakte.net
api.stathat.com api.stathat.com
apibatbrowsecom-a.akamaihd.net apibatbrowsecom-a.akamaihd.net
apioasisspacenet-a.akamaihd.net apimegabrowsebiz-a.akamaihd.net
arcadepages.com app5.letmacworkfaster.world
arkkivoltti.net appearance-cool.com
artavenuegardenstudios.com approved.su
asdfg.pro appsecurityr.com
asiengirls.net aquarium-pro.ru
astrochicks.com arabseyes.com
autonew.biz aramaicmedia.org
autorn.ru arcteryxsale.online
availit.weebly.com asia-forum.ru
aviapanda.ru asiavirtualsolutions.com
aviav.co asseenontvonline.ru
awency.com atleticpharm.org
azartmsl.com autoplate.info
babespcs.com autotrends.today
backgroundpictures.net autovideobroadcast.com
avtoarenda.by
ayeartoforget.com
azinoofficial777.ru
bablonow.ru
backlinkwatch.com
bag77.ru bag77.ru
balois.worldbreak.com banksrf.ru
baltgem.com bestserials.com
bangla-chat-uk-paralud.ga bezlimitko.xyz
battle.net bidbuy.co.kr
bayareaaeroclub.org bigpenisguide.com
beetpics.pw bjanshee.ru
begalka.xyz blackcurranthumidifiers.site
best-mam.ru bloggers.nl
bestprofits-there.com blogqpot.com
bharatdefencekavach.com blogsfunky672.weebly.com
bimmerpost.com book-bets.com
binaryoptionscops.info bosman.pluto.ro
biznesluxe.ru bradleylive.xyz
blackle.com
blackwitchcraft.ru
bmsco.co
bobinoz.com
brainsandeggs.blogspot.com brainsandeggs.blogspot.com
brandov.ru brakehawk.com
brianhenry.co brothers-smaller.ru
bugof.gq browserprotecter.com
bumascloset.com budpost.com.ua
burger-tycoon.com buy-forum.ru
burn-fat.ga buyclomidonlaine.com
bus-offer.com c-gainsbourg.com
buttons-for-website.com
byme.se
cacheimages.com cacheimages.com
cakemediahq.com.au callcafe.info
calvet.altervista.org
cam-kontakte.org
camel-beach.com camel-beach.com
capacitacionyaprendizaje.com candlehandmade.com
carsdined.org canzoni.ru
centre-indigo.org.ua carapics.com
cerev.info carloans.com
cfjump.com cartierlove2u.xyz
cheapjerseysap.com cartierlovebraceletreplica.xyz
cheapjerseysbizwholesale.us casite-513049.cloudaccess.net
cheapmoncler.win castingbank.ru
cheapmoncler.xyz caveavins.fr
cheapsergiorossi.online cdncash.net
cheatcode-lita12.rhcloud.com cdncash.org
cdnnetwok.xyz
cete.ru
cheapwebsitehoster.com
check-host.net
checktext.org
chee-by.biz
chinacheapelitejerseys.com chinacheapelitejerseys.com
cialis-samples.com cialiswithoutadoctor.net
clayvasedesigns.tk classiquebijoux.ru
cleansearch.net
clickpapa.com clickpapa.com
compliance-alex.top clmforexeu.com
compliance-alexa.top cognitiveseo.com
compliance-checker-7.info com-onlinesupport.host
compliance-julianna.top comissionka.net
compliance-nelson.top companies-catalog.com
computernetworksonline.com compliance-margo.top
contentsexpress.com consorzioilmosaico.org
cookie-law-enforcement-aa.xyz cookie-law-enforcement-aa.xyz
crd.clan.su copyrightclaims.org
credit-online.ws counciltally.com
culmimg.pw customerpromos-a.akamaihd.net
cymbaltaandtizanidine.blogspot.com customsua.com.ua
dailystrength.org d0t.ru
darmebel.com.ua dailyfinancefix.com
datadepths.com daptravel.com
datsun-do.com dating-time-now.com
davebestdeals.com davebestdeals.com
demian.kz ddsoldes.fr
dengi-pod-zalog-nedvizhimosti.ru dealwifi.com
deniven.1bb.ru decorationspcs.com
dieswaene.com detskie-konstruktory.ru
discountliv.com dirtpics.pw
dogsrun.net dkvorota.ru
domaincheck.io domainanalyzing.xyz
domainsatcost.com dotmass.top
domik-derevne.ru e-buyeasy.com
download-wallpaper.net eachdayisagift.review
drev.biz elektrozigarette1.dreamwidth.org
eandsgallery.com elvel.com.ua
ecookna.com.ua emerson-rus.ru
editmedios.com enews.tech
elektronischezigarette2.mex.tl erot.co
elektrozigarette2.wordpressy.pl erotik0049.com
elektrozigarettekaufen1.devhub.com
elektrozigaretten2.yn.lt
elitedollars.com
eloxal.ru
enternet.ee
envaseslotusama.com
ero-advertising.com
eropho.com
estibot.com
etotupo.ru
eu-cookie-law.blogspot.com
eu-cookie-law.info
eupornstar.info eupornstar.info
exchanges-bet.com europeanwatches.ru
executehosting.com exonline.info
ezigarettenkaufen1.deviantart.com fapgon.com
ezigaretteshop2.mywapblog.com faptitans.com
falllow.gq faracontrol.ir
feeriaclub.ru feel-planet.com
felizporno.com
ferieboligkbh.dk ferieboligkbh.dk
fidelityfunding.com fifa-coins.online
financetip.eu filesclub.net
filmci.pro
financeloan.us
finemanteam.com finemanteam.com
finstroy.net fingerprintjs.com
finuse.com forensicpsychiatry.ru
flashlarevista.com forex21.ru
flirt4free.com formseo.com
flowersbazar.com forums.d2jsp.org
flprog.com frcls.fr
forexunion.net free-floating-buttons.com
fotosfotos.eu free-gluten.ru
fotoxxxru.com free411games.com
foxjuegos.com fsakhalin.ru
free-video-tool.com fun-mobi.pl
freefoto.ca fun2cell.net
freelifetimefuckbook.com gamesprite.me
freewhatsappload.com gamingspark.com
frequiry.com gate5.co.za
ftns.ru gazoblok.net.ua
fungirlsgames.net ge0ip.net
funnymama.com
g.starmoe.xyz
gallictures.com
garciniaxt.us
gearcraft.us
geileweiber.tk geileweiber.tk
gerhardhealer.com gemara.com
getlaid-xxxhookupdirect.com gepezz.info
gettpromos.com getpopunder.com
gidonline.one gigapeta.com
gk-atlant.info gobongo.info
gkvector.ru gotorussia.com
goldadpremium.com gpms.org.my
goodwinmetals.co gratis-sexkontakte.com
gourcy.altervista.org greendream.com.ua
gta-club.ru gribokstop.com
gta-top.ru guarrasdelporno.xxx
gyffu.com hamptonoaks.ca
hahashka.ru handicapbathtubarea.com
hanwei.us
hasfun.com
hasshe.com hasshe.com
hatdc.org hatedriveapart.com
hauleddes.com hdapp1008-a.akamaihd.net
hebr.myddns-flir.com hem.passagen.se
house.sieraddns.com hentaiheroes.com
houston-vikings.com hermesreplica.pw
hscsscotland.com heroz.fr
hydropump.su himazin.info
ideibiznesa2015.ru histock.info
imabase.com horoshieokna.com
imgarcade.com hothor.se
imguramx.pw hubbble.com
immobiliaremassaro.com hunthillfarmtrust.org
imperia31.ru igrovyeavtomaty777.ru
inboxdollars.com ihc2015.info
incomekey.net ilovevitaly.com
inet-traffic.com imagefinder.site
infospot.pt imagerydatabase.com
insider.pro infodocsportal.com
int.search.mywebsearch.com infostatsvc.com
internetproviderstucson.com interferencer.ru
intervsem.ru investpamm.ru
investsuccess.org
iptool.xyz iptool.xyz
ispaniya-costa-blanca.ru iqoption.pro
istmira.ru irunfar.com
itsdp3.com ivanstroi.ru
itservicesthatworkforyou.com iwantmyfreecash.com
izhevsk.xrus.org j33x.com
jagg.info janettabridal.com
jav-idol.com jav-fetish.site
jo24news.com jav-way.site
kam-dom.ru jerseywholesalechinabiz.com
karaganda.xkaz.org job.icivil.ir
kareliatobacco.ru jobgirl24.ru
kaz.kz johnnyhaley.top
kemerovo.zrus.org jongose.ninja
ketoanhanoi.info juliadiets.com
keywordhouse.com jyrxd.com
khadastoafarde.tk jyvopys.com
khafre.us kamorel.com
kazka.ru
kfon.eu
kidd.reunionwatch.com
kimcurlrvsms.com
kinohit1.ru
kletkimehan.ru kletkimehan.ru
klikbonus.com kol-energo.ru
kolotiloff.ru koopilka.com
komputers-best.ru kosmetyki.tm.pl
koral.se kosynka-games.ru
kreativperlen.ch landscaping.center
kriokomora.info laulini.soclog.se
kvartir-remont.biz linkbolic.com
kwzf.net live-sexcam.tk
ladsup.com liveinternet.ru
landmania.ru liveshoppersmac.com
lapitec.eu llastbuy.ru
laserpen.club lomb.co
lazy-z.com mactechinfo.info
ledis.top mansparskats.com
letmacworkfaster.world marketingtechniques.info
letsrepair.in marmotstore.online
lifehacĸer.com maslenka.kz
lifeinsurancekb.com master-muznachas.ru
limads.men mecash.ru
link.ac
liupis.com
liverpool.gsofootball.com
liveu.infoteka.hu
machicon-akihabara.info
magicdiet.gq
mailemedicinals.com
manuscript.su
mararoom.ru
martlinker.com
mcar.in.ua
medizinreisen.de
members.ghanaweb.com members.ghanaweb.com
mensandals.xyz merryhouse.co.uk
michaelkorsoutlet.store mexicotravelnet.com
middlerush-a.akamaihd.net mezaruk.info
militarysale.pro mil-stak.com
millionare.com mini.7zap.com
mmofreegames.online minyetki.ru
mobicover.com.ua mixx.com
moivestiy.biz mlf.hordo.win
moncleroutletonline.pw mmstat.com
monetizationking.net mobile.ok.ru
money-every-day.com mobplayer.net
moneymaster.ru modnie-futbolki.net
monthlywinners.com morefastermac.trade
moonci.ru
mountainstream.ms mountainstream.ms
msk-diplomat.com moviemail-online.co.uk
mtmtv.info movies-in-theaters.net
muz-shoes.ru moviezbonkerssk.cf
muz-tracker.net mp3ritm.top
my-big-family.com mug-na-chas-moscow.ru
music.utrolive.ru
muznachas-service.ru
my-bc.ru
myanyone.net
mybackgroundlandscape.blogspot.com mybackgroundlandscape.blogspot.com
myblogregistercm.tk mycaf.it
mylovelibrabry.com myfreemp3.eu
myra.top myftpupload.com
nac-bearings.ru myhealthcare.com
natali-forex.com myhydros.org
neobux-bg.info mylida.org
nerudlogistik.ru nailsimg.com
net-profits.xyz nakozhe.com
netanalytics.xyz new-post.tk
nextconseil.com newtechspb.ru
nicovideo.jp no-fuel.org
normalegal.ru noclegonline.info
nodup.ru
november-lax.com november-lax.com
oakleyglassesonline.us nudo.ca
oballergiya.ru oklogistic.ru
obiavo.com oneshotdate.com
obiavo.kz onlinadverts.com
obsessionphrases.com onlinemeetingnow.com
of-ireland.info onstrapon.purplesphere.in
offtopic.biz open-odyssey.org
okuos.com opinionreelle.com
online-hit.info ozoz.it
onlinetvseries.me p-business.ru
openfrost.com
ostrovtaxi.ru
paclitor.com paclitor.com
paradontozanet.ru par-fallen.ga
pastaleads.com parlament.biz
perl.dp.ua partnerline.men
phobia.us pearlisland.ru
photokitchendesign.com pensplan4u.com
phrcialiled.com petsblogroll.com
picturesify.com pharmacyincity.com
pillscheap24h.com pic-re.blogspot.com
pistonclasico.com pigrafix.at
pizdopletka.club pinapchik.com
pkr1hand.com pizda.lol
plaff-go.ru pizza-imperia.com
pontiacsolstice.info pl.id-forex.com
pony-business.com plohaya-kreditnaya-istoriya.ru
popunder.ru popserve.adscpm.net
pornhive.org pornoblood.com
pornoklad.net pornoelita.info
pravoholding.ru pornosee.info
prefersurvey.net potoideas.us
privatefx-in.ru primedice.com
privatefx.all4invest.info printdirectforless.com
pro-okis.ru priora-2.com
produkto.net privat-girl.net
proftests.net privatbank46.ru
prostitutki-rostova.org privatov-zapisi.ru
proxyelite.biz progress-upakovka.ru
py100.ru prom23.ru
qwarckoine.com prostitutki-kharkova.org
radiogambling.com prostitutki-kiev.org
razleton.com prostitutki-odessa.org
rebelmouse.com pst2017.onlinewebshop.net
recinziireale.com puzo2arbuza.ru
reklamuss.ru rankings-analytics.com
remybutler.fr ranksays.com
renecaovilla.online razvratnoe.org
rentalcarnavi.info realitykings.com
resant.ru recipedays.com
reelheroes.net
reeyanaturopathy.com
registrationdomainsite.com
reining.lovasszovetseg.hu
rethinkwasteni.info
reversing.cc
reward-survey.net reward-survey.net
ricorsogiustizia.org ringporno.com
ros-ctm.ru romhacking.ru
rospromtest.ru rootandroid.org
running-line.ru roznica.com.ua
rus-teh.narod.ru rubbed.us
russia-tao.ru ruf777.com
rutor.group rutor.vip
s-iwantyou.com sanjosestartups.com
saletool.ru sanyuprojects.com
salutmontreal.com sarafangel.ru
sanidumps.com sardinie.us
sarm.tk sarf3omlat.com
sbetodiodnye-lampy.ru
sbprabooks.com
sbwealthsolutions.ca
sc-specialhost.com
scanmarine.info
scanner-george.top scanner-george.top
scanner-mary.top scanner-nelson.top
scottbywater.com screpy.com
search.pch.com scrnet.biz.ua
searchaddis.com search-goo.com
seasonvar.ru searchtooknow-a.akamaihd.net
sell-fb-group-here.com see-your-website-here.com
seo-platform.com semalt.com
seo-smm.kz seoexperimenty.ru
seo-traffic-ranking.info seojokes.net
seobility.net seonetwizard.com
seoprofiler.com sexphoto.site
service-core.ru sexs-foto.com
sex-sex-sex5.com sexsearch.com
sexfreepornoxxx.com sexstream.pl
sexkontakte-seite.com sexy.babes.frontend-stack.top
sharebutton.to shareyards.com
sharpchallenge.com shariki-zuma-lines.ru
sherlock.se shijian.ac.cn
shisha-swag.de shoppingjequiti.com.br
shtaketniki.kz shopwme.ru
sibvitr.ru silvermature.net
silvercash.com site-speed-checker.site
sinestesia.host.sk skanninge.se
sisiynas.ru skuteczna-dieta.co.pl
smartshoppymac.com skylta.com
socialbutton.xyz slayerlife.com
sockshare.net slowmac.tech
spaceshipad.com small-game.com
spb.afora.ru smart-balancewheel.com
special-porn.com smartpet.ru
smsactivator.ru
smstraf.ru
snip.tw
snjack.info
snyatie-lomki-v-stacionare.ru
social-search.me
socialbuttons.xyz
softonicads.com
sogimlecal.tk
spabali.org
specialfinanceoffers.com
spinnerco.ca
spl63.fr spl63.fr
spywarenuker.com spywarespy.com
squidoo.com ssconstruction.co
stair.registrydomainservices.com stats-collector.org
stard.shop statustroll.com
stariy-baku.com stpolice.com
sukarame.net stretchmate.net
susanholtphotography.com strfls.com
svbur.ru stroy-portal22.ru
symphonyintegratedhealthcare.com studentguide.ru
syndicate.fun sudexpert66.ru
tacbelarus.ru suggest-keywords.com
tandvardshuset.net superinterstitial.com
taxi-v-eisk.ru support.nopeas.sk
tds-advert005.info sureone.pro
tecnoteakviareggio.it svensk-poesi.com
teenforporn.com swiped.su
teesdaleflyballclub.co.uk takeflyte.com
telegraf.by taraz.xkaz.org
terraclicks.com tattooreligion.ru
texbaza.by tedxrj.com
theendivechronicles.com teguh.info
thefarmergame.com telesvoboda.ru
tiandeural.ru thebluenoodle.com
ticketsys.inetwd.com thegioixekhach.com
tilido.com thegolfclub.info
timhost.ru tizanidinehydrochloride4mgstreetvalue.blogspot.com
tizanidine4mguses.blogspot.com tizanidinezanaflex.blogspot.com
tizanidine6mg.blogspot.com tjkckpytpnje.com
tizanidineshowupondrugtest.blogspot.com topanasex.com
tizanidinevscyclobenzaprine.blogspot.com touchmods.fr
topquality.cf tour-line.net
torrent-newgames.com track.deriv.com
toy-shop.top trackmedia101.com
tracksurf.daooda.com tradgardspartner.se
trafaret74.ru traffic-club.info
trafficcentr.xyz trafficjunky.net
tube8.com traffpartners.com
tupper-shop.ru trainoffend.ml
tvory.predmety.in.ua triplepanda.xyz
twittruth.com triumf-realty.ru
ukkala.xyz trkdf.com
truck-land.life
tsyndicate.com
tt-ipd.info
typimga.pw
tzritel.tk
uginekologa.com
ukrtextbook.com ukrtextbook.com
unlimitdocs.net ultimatesetnewfreeallsoftupgradesystems.pw
unionmarkt.de
unrealcommander.biz unrealcommander.biz
urdoot.win unrealcommander.com
urll.eu url-img.link
valmetrundan.se usdx.us
vapsy.com v-doc.co
varbergsvind.se v24s.net
venerologiya.com valkiria-tk.ru
videochat.mx valueclick.com
videochat.world varikoz24.com
videooko.weebly.com vashsvet.com
videosbox.ru vbtracker.net
vinylvault.co.uk velobikestock.com
viteonlusarezzo.it vesnatehno.ru
viven.host.sk video-chat.cn
vkgaleria.com video-production.com
voloslove.ru videojam.tv
vzlom-na-zakaz.com villacoloniale.com
vzube.com vitanail.ru
wapsite.me vladimir.zrus.org
watch-movies.ru volgograd.xrus.org
webads.co.nz voluumtracker1.com
webix.biz vrnelectro.ru
webmasterhome.cn vzubah.com
wemarketing.se w7s.ru
weblibrary.win
weboptimizes.com
webs.com
webstatsdomain.org
welck.octopis.com
wellcome2slovenia.ru
wfb.hatedriveapart.com
whereverdesperate.gq whereverdesperate.gq
wholesalecheapjerseysfree.com womensterritory.ru
wholinkstome.com word-vorlagen.net
winterclassichockeyjerseys.com wovis.site
womensplay.net wsgames.ru
word-vorlagen.xyz
worldlovers.ru
wpthemedetector.co.uk
writersgroup580.web.fc2.com
writingservices17.blogspot.ru
wufak.com wufak.com
ww2awards.info
www.event-tracking.com
www.solartek.ru www.solartek.ru
www.wohnkabinen-shop.de xev.ru
wzgyyq.com xingzi-vision.com
xep.info xn------7cdbapdecfd4ak1bn0amjffj7afu3y.xn--p1ai
xn----7sbbagbq7bd5aheftfllo4m.xn--p1ai xn--80aaaks3bbhabgbigamdr2h.xn--p1ai
xn--80ahvj9e.xn--p1ai xn--80atua3d.xn--p1ai
xnxx699.com xn--e1aggki3c.xn--80adxhks
xportvusbdriver8i.snack.ws xxxmania.top
xsfetish.org yellowstonevisitortours.com
xvideospornoru.com yoopsie.com
xz618.com youdao.com
yaaknaa.info ytmnd.com
yachts-cruise.info yun56.co
yellowproxy.net yunque.pluto.ro
yuweng.info zarabotok--doma.ru
zahvat.ru zazagames.org
zaimhelp.ru zeleznobeton.ru
zajm-pod-zalog-nedvizhimosti.ru zirondelli.it
zdesoboi.com zoogdisany.com
zynax.ua zzlgxh.com

484
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
108shot.com 1000n1.ru
123any.com 100dollars-seo.com
12bet.com 1314dh.com
24-ak.ru 2345.com
34.gs 2ads.co.uk
4rent.ru activecampaign.dreamhosters.com
883zy.com adsland.men
a.pr-cy.ru adtiger.tk
abcguru.xyz adultmeetonline.info
admeasures.com affordablewebsitesandmobileapps.com
adultfullhd.com allwidewallpapers.com
afslankpillen2017nl.eu an-donut.com
agecheckadult.com angigreene.com
akvamaster.dp.ua
analytics-ads.xyz
animal-drawings.com
animaltoplist.com animaltoplist.com
anonymizeme.pro apimossnetinfo-a.akamaihd.net
apparel-offer.com apiportalmorecom-a.akamaihd.net
apple.com-cleaner.systems ardimobileinfo.ml
apxeo.info arenda-avtoprokat-krasnodar.ru
backiee.com aviav.eu
backlink4u.net avtoarenda.by
balitouroffice.com avtolombard-krasnodar.com
bambi.ck.ua bdsmgalls.net
benchmarkcommunications.co.uk begalka.xyz
bestcalovebracelet.cn bet-prognoz.com
beztuberkuleza.ru blogporn.in
bitcoin-ua.top bobinoz.com
bloggers.nl bonkers.name
boeuklad.com bpro1.top
brainboostingsupplements.org brains2.biz
bristolhotel.com brillianty.info
buzzsumo.com bvlgariring.xyz
cakesplus.com.au bzero1jewelry.net
chocolateslim-en-espana.com c-gainsbourg.com
chocolateslim-u-srbiji.com callstevens.com
choosecuisine.com cartierlovebracelet.xyz
click2pawn.com cartierloveringreplica.xyz
collegeessay19.blogspot.ru cbcseward.com
compliance-ivan.top cdncash.com
constantaservice.net check-host.net
cookielawblog.wordpress.com chiblackhawks-jerseys.com
credit-online.ws cialgenisrx.com
creditwell.ru citizenclsdriveri7.pen.io
crystalslot.com clipartnew.com
custom-electric-guitar.com compliance-irvin.top
cyse.tk conocer-sanabria.com
decenttools.com cookie-law-enforcement-hh.xyz
delfin-aqua.com.ua coolingoods.com
deplim.com copyrightclaims.org
discountliv.com culad.com
docs4all.com cxpromote.com
d2jsp.org
dailystrength.org
danschawbel.com
daretodonate.co
dating-time-now.com
db.speedup.gdn
delayreferat.ru
delta-line.men
digital-sale.su
discovertreasure-a.akamaihd.net
disruptingdinnerparties.com
dneprsvet.com.ua
dostavka-v-krym.com
dotmass.top dotmass.top
download-walpaperhd.blogspot.com e-poker-2005.com
elektrischezigarette1.blog.pl e90post.com
energydiet24.ru earl-brown.info
engines-usa.com eduinfosite.com
englate.com ege-russian.ru
escortslet.net elektrischezigarette1.onsugar.com
esnm.ru elektronischezigarettekaufen2.tumblr.com
etotupo.ru elitedollars.com
europeanwatches.ru eloconcream.blogspot.com
euroskat.ru empathica.com
evening-dating-club.info encodable.com
ewebarticle.info espaceinventoristes.com
expresstoplivo.ru eupornstar.info
ezigarettenshop1.yolasite.com exchangeit.gq
ezigaretteshop.webs.com exmasters.com
ezigaretteshop2.mywapblog.com extlabs.io
facebook-mobile.xyz eyes-on-you.ga
filesdatabase.net fanrto.com
fiverr.com favornews.com
fm-upgrade.ru fearcrow.com
footballfarrago.com fikasound.tk
fotoxxxru.com filesclub.net
free-laptop-reward.com firstsiteguide.com
freelinkbuilding.website.tk forms-mtm.ru
freetangodownload.com fortevidyoze.net
funnymama.com franklinfire.co
fym.com.ru freenode.info
fz139.ttk.ru freshberry.com.ua
g7m.pl friendflnder.com
generic-pills-online.com frivgame250.com
gentamicineyedrops.blogspot.com game-mmorpg.net
gorodservis.ru getpopunder.com
handsandlegs.ru gheus.altervista.org
hanink.biz.ly gk-atlant.info
hannasolution.ru gojiberriess.apishops.ru
havepussy.com goodwinmetals.co
hawaiielectriclight.com gsmtlf.ru
help.tpu.ru hdpixent.com
hentaiheroes.com headpharmacy.com
hermesbracelets.xyz
hitcpm.com
hotblog.top
hoterika.com
hoztorg-opt.ru
hptwaakw.blog.fc2.com hptwaakw.blog.fc2.com
images-free.net hreade.com
incanto.in.ua i4track.net
inet-traffic.com imagui.eu
isotoner.com impotentik.com
istizanidineanarcoticdrug.blogspot.com infogame.name
javxxx18.com invivo.hu
iplusbit.blogspot.co.za
iptool.xyz
iqoption.com
ispac.org
itronics.ca
itsdp3.com
jerseyschinabizwholesale.us
jerseywholesalebizchina.com
jerseywholesalechinabiz.com
jo24news.com jo24news.com
jobgirl24.ru jofucipiku.tk
journeydownthescale.info jrcigars.com
junglenet-a.akamaihd.net junketjuice.blogspot.com
justprofit.xyz
kabbalah-red-bracelets.com
kamagragelusa.net
kino-fun.ru kino-fun.ru
koopilka.com klin3952.ru
lasvegaslockandsafe.com koleso24.com.ua
lazy-z.com komp-pomosch.ru
legionalpha.com konkursov.net
letmacwork.world kunstaktien.info
le-clos-des-alouettes.com
legalrc.biz
lexiangwan.com lexiangwan.com
life-instyle.com linkbuddies.com
lkbennettoutlet.online lion.playtap.us
lombardfinder.ru mackeeper-land-672695126.us-east-1.elb.amazonaws.com
lonely-mature.com macrotek.ru
macresource.co.uk mall.uk
manipulyator-peterburg.ru malls.com
martlinker.com manualterap.roleforum.ru
max-eclat.men marmitaco.cat
mecash.ru maslenka.kz
mexicosleevegastrectomy.com mebelcomplekt.ru
mielec.pl mediaoffers.click
miniature.io meltwater.com
mistr-x.org metarip.ru
mobile.ok.ru micasainvest.com
monsterdivx.tv minegam.com
mostantikor.ru mobsfun.net
mozello.ru morpicert.pw
mvpicton.co.uk mototsikl.org
mydownloadengine.com nalogovyykodeks.ru
narutonaruto.ru naturalshair.site
net-radar.com net-profits.xyz
newstraveller.ru netanalytics.xyz
nightvision746.weebly.com news-speaker.com
nikkiewart.ru nvssf.com
nisuturnetdgu.tk oda.as
noclegonline.info ofanda.com
npoet.ru oren-cats.ru
nrjmobile.fr pensplan.com
nucia.biz.ly photosaga.info
nudo.ca pizdopletka.club
o-o-11-o-o.com poligon.com
oconto.ru porno-play.net
okmedia.sk pornofiljmi.com
okroshki.ru pornstartits.xblog.in
onclkads.com pozvonim.com
online-hd.pl preg.marketingvici.com
onload.pw printie.com
orgasmatrix.com pro-poly.ru
ownshop.cf propranolol40mg.blogspot.com
panouri-solare-acoperis.com rabotaetvse.ru
paydayloanslocal.com rating-casino2021.ru
petrovka-online.com realitykings.com
pisanieprac.info realresultslist.com
piuminiita.com recipedays.com
pivka.xyz rentalcarnavi.info
pizdeishn.com rogervivierforsale.com
pornoklad.net ruf777.com
porto.abuilder.net rusenvironmental.net
preconnubial.usuby.site sammlungfotos.online
preparevideosafesystem4unow.site scanner-brian.top
projefrio.com.br scanner-julianna.top
proposal-engine.com scanner-walt.top
putevka24.ru schalke04fc.info
q-moto.ru scmor.ilxc.cc
registratciya-v-moskve.ru sdi-pme.com
remont-komputerov-notebook.ru seeresultshub-a.akamaihd.net
rn-to-bsn.com segol.tv
rock-cafe.info seo-smm.kz
rusoft-zone.ru seorankinglinks.us
saddiechoua.com sexreliz.com
scrapinghub.com shopfishing.com.ua
sdelatmebel.ru siteaero.com
se-welding.ru sklad-24.ru
senger.atspace.co.uk softomix.com
sex-tracker.de softwaretrend.net
sexad.net songplanet.ru
sigmund-freud.co.uk spacash.com
silvercash.com speechfoodie.com
slimcdn.com srdrvp.com
slonechka.ru steamoff.net
smart-balancewheel.com strigkaomsk.ru
snjack.info strv.se
soda.media sukirgenk.dvrlists.com
spinazdrav.ru surintech.ac.th
stevemonsen.com taqywu51.soup.io
supermodni.com.ua techart24.com
superstarfloraluk.com
swiped.su
taihouse.ru
taraz.xkaz.org
td-l-market.ru
teenfuck.tv
terrafootwear.us
testbotprocessor44.com
tfxiq.com tfxiq.com
thebluffs.com thelottosecrets.com
thiegs.reco.ws themestotal.com
tizanidine4mg.blogspot.com tizanidinecost.blogspot.com
tizanidinehcl4mgtab.blogspot.com tizanidinehcl2mgsideeffects.blogspot.com
tizanidinestreetvalue.blogspot.com tizanidinenarcotic.blogspot.com
tizanidinezanaflex.blogspot.com tizanidinerestlesslegsyndrome.blogspot.com
topdownloads.ru tizanidinesideeffectsweightloss.blogspot.com
topflownews.com todohr.com
topkarkas.com toonfamilies.net
tramadolandtizanidine.blogspot.com totu.us
tryrating.com triplepanda.xyz
tula.howotorg.ru tube8.com
turbodsp.com umnovocaminho.com
tvand.ru usbggettwku.ga
ua.tc vacuumscleaner.com
uhodzalijami.ru varikozdok.ru
unmaroll.ya.ru via-energy-order.com
url-extractor.xyz wallpapers-best.com
urlopener.blogspot.com.au wdrake.com
us-america.ru westen-z.life
valueclick.com wildnatureimages.com
vavilone.com worldlovers.ru
vektorpress.ru x-diesel.info
vksex.ru xboxster.ru
vsexkontakte.net xn-----8kcatubaocd1bneepefojs1h2e.xn--p1ai
vtc.pw xn----8sbdbjgb1ap7a9c4czbh.xn--p1acf
wardreapptokone.tk xn----8sblgmbj1a1bk8l.xn----161-4vemb6cjl7anbaea3afninj.xn--p1ai
website-speed-checker.site xn----ctbigni3aj4h.xn--p1ai
wechatdownload10.com yaoguangdj.com
wesharepics.site youandcredit.ru
westsextube.com yourothersite.com
wnoz.de youtubologia.it
word-vorlagen.net youtuhe.com
woweb.com.ua yubikk.info
x-diesel.com zarabiaj-dzis.pl
xblognetwork.com zastenchivosti.net
xingzi-vision.com zoosexart.com
xn--80aeahghtf8ac5i.xn--p1ai
xn--oogle-wmc.com
yellowproxy.net
yellowstonesafaritours.com
youtube-downloader.savetubevideo.com
zacreditom.ru
zaimite.ru
zreloeporno.tv
zs2vm.top
zscaler.net
zscalerone.net
zvetki.ru zvetki.ru

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.144 0; 10.1.0.99 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.06.3158 ### Version: V4.2022.06.3159
### Updated: Fri Jun 10 10:03:43 UTC 2022 ### Updated: Fri Jun 10 22:01:42 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.06.3158 ### Version: V4.2022.06.3159
### Updated: Fri Jun 10 10:03:43 UTC 2022 ### Updated: Fri Jun 10 22:01:42 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##