vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-02 18:50:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.03.3015 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-03-31 22:10:31 +00:00
parent 541b1bbb0a
commit 5d1095cd4b
46 changed files with 96502 additions and 96502 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.03.3014 #### Version: V4.2022.03.3015
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.03.3014 #### Version: V4.2022.03.3015
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.03.3014 #### Version: V4.2022.03.3015
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

11928
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.03.3014 #### Version: V4.2022.03.3015
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

11932
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11932
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

11932
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11932
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 1; 10.1.0.129 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 0; 10.1.0.129 0;
127.0.0.1 0; 127.0.0.1 0;

11932
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 1; 10.1.0.129 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 0; 10.1.0.129 0;
127.0.0.1 0; 127.0.0.1 0;

11932
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 1; 10.1.0.129 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 0; 10.1.0.129 0;
127.0.0.1 0; 127.0.0.1 0;

11932
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3014 3015

11932
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

11932
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
168.61.190.140 0; 40.77.42.210 0;

11932
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 1; 10.1.0.129 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 0; 10.1.0.129 0;
127.0.0.1 0; 127.0.0.1 0;

11932
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 1; 10.1.0.129 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 0; 10.1.0.129 0;
127.0.0.1 0; 127.0.0.1 0;

11932
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 1; 10.1.0.129 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 0; 10.1.0.129 0;
127.0.0.1 0; 127.0.0.1 0;

11932
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
168.61.190.140 0; 40.77.42.210 0;

11932
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
168.61.190.140 0; 40.77.42.210 0;

11932
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 1; 10.1.0.129 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
BuiltBotTough Grabber
Dotbot IDBTE4M
Image Sucker InfoTekies
Mojolicious RebelMouse
Nikto RedesScrapy
Rogerbot TheNomad
Turingos VCI
WebSauger WebFuck
Xaldon WebSpider Wotbox
ZoomBot internetVista monitor

306
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 404enemy
360Spider
80legs
ADmantX
AIBOT
ASPSeek ASPSeek
Aboundexbot Abonti
AfD-Verbotsverfahren
Aipbot Aipbot
AllSubmitter
Alligator
AlphaBot AlphaBot
Anarchie Anarchie
Anarchy99
Ankit Ankit
Aspiegel Anthill
Attach Atomseobot
BackDoorBot BDCbot
BDFetch
BLEXBot
BackStreet BackStreet
BackWeb
BacklinkCrawler BacklinkCrawler
Badass Bandit
Barkrowler BatchFTP
Bigfoot Bigfoot
Bitacle Bitacle
BlackWidow
Blow Blow
BlowFish Bolt
BotALot Buck
Brandprotect
Buddy Buddy
CATExplorador CODE87
CCBot Cegbfeieh
CheTeam
CheeseBot CheeseBot
Claritybot CherryPicker
Cloud mapping ChinaClaw
Cosmos Cliqzbot
Collector
Craftbot Craftbot
Crescent Crescent
Curious CrunchBot
Custo Custo
DTS Agent DSearch
DatabaseDriverMysqli DatabaseDriverMysqli
Demon
Deusu Deusu
Discobot DigitalPebble
Dispatch Dirbuster
DomCopBot Disco
DomainCrawler
DomainSigmaCrawler
DomainStatsBot
Domains Project
Download Wonder
Dragonfly Dragonfly
Drip ECCP/1.0
Ecxi Evil
EirGrabber
EroCrawler
ExtLinksBot ExtLinksBot
ExtractorPro Extractor
Fimap EyeNetIE
FemtosearchBot
FlashGet
Flunky
FrontPage
Fuzz Fuzz
Fyrebot Fyrebot
Genieo
Getintent Getintent
Gigabot
Go-Ahead-Got-It
Gotit Gotit
GrapeshotCrawler GrapeshotCrawler
HTTP::Lite GridBot
HaosouSpider HEADMasterSEO
Havij Haansoft
Heritrix
Hloader Hloader
HybridBot IDBot
IDBTE4M Iblog
Id-search Id-search
IlseBot IlseBot
IndeedBot Image Fetch
Image Sucker
Indy Library Indy Library
InfoNaviRobot
InfoTekies InfoTekies
Intelliseek
Internet Ninja Internet Ninja
Iria
Iskanie
JOC Web Spider
JamesBOT JamesBOT
JennyBot Jbrofuzz
JetCar JetCar
Jyxobot JikeSpider
Jorgee
JustView
Keyword Density
Kozmosbot
LNSpiderguy LNSpiderguy
Larbin LeechGet
LexiBot LexiBot
Lftp LibWeb
Linkbot Libwhisker
LinkextractorPro Likse
LinqiaRSSBot LinkScan
LinqiaScrapeBot LinkpadBot
LinqiaMetadataDownloaderBot
Lipperhey Spider Lipperhey Spider
Litemage_walker Litemage_walker
Ltx71 Lmspider
MIDown tool
MIIxpc MIIxpc
MJ12bot MJ12bot
MTRobot
Mail.RU_Bot Mail.RU_Bot
Majestic SEO Majestic SEO
Mb2345Browser MarkWatch
Meanpathbot Mass Downloader
Mediatoolkitbot Masscan
Metauri Mata Hari
MauiBot
Minefield
Mister PiX Mister PiX
Moblie Safari Moblie Safari
Mojolicious Mr.4x3
Musobot Musobot
NICErsPRO NICErsPRO
NPbot Name Intelligence
Nameprotect
Navroad Navroad
Needle Needle
Nessus Nessus
Net Vampire Net Vampire
NetMechanic NetAnts
Netcraft NetLyzer
Nibbler Nettrack
Nmap Niki-bot
Not Nikto
Nutch Nutch
Octopus Octopus
OnCrawl Offline Explorer
Openfind Offline Navigator
Openvas
OrangeSpider OrangeSpider
OutclicksBot OutclicksBot
OutfoxBot
PECL::HTTP PECL::HTTP
POE-Component-Client-HTTP PHPCrawl
Page Analyzer
PageAnalyzer
PageScorer PageScorer
PageThing.com
Pandalytics
Panscient Panscient
Papa Foto
Pavuk
Petalbot
Pi-Monster Pi-Monster
Picsearch Picscout
PictureFinder Pimonster
Pixray PleaseCrawl
Pockey Pockey
Probethenet ProPowerBot
Psbot Pu_iN
QueryN Metasearch PyCurl
RSSingBot Quick-Crawler
RankActive RankActive
RankFlex RankActiveLinkBot
Rankivabot RankingBot
ReGet RankurBot
RedesScrapy
Ripper Ripper
RocketCrawler
Rogerbot Rogerbot
SEOkicks-Robot SBIder
SEOlyticsCrawler SISTRIX
SEOprofiler ScanAlert
SMTBot Scanbot
Scrapy Scrapy
Screaming Seekport
ScreenerBot
ScrepyBot
SearchmetricsBot
Semrush Semrush
SemrushBot Seomoz
SeoSiteCheckup Shodan
SeobilityBot
Siphon
Site Sucker Site Sucker
Sitebeam SiteSnagger
Snake SmartDownload
Snoopy Snoopy
SocialRankIOBot SocialRankIOBot
Sociscraper Sosospider
SpaceBison Sottopop
Spammen SpankBot
Spanner Spanner
Spinn3r Spbot
SputnikBot SputnikBot
Sucuri Sucker
SuperHTTP SurveyBot
Suzuran
Swiftbot
T0PHackTeam T0PHackTeam
Teleport
TeleportPro TeleportPro
Telesphorep The Intraformant
TheNomad TightTwatBot
Thumbor Toata
Titan Toweyabot
Tracemyfile Tracemyfile
Trendiction Trendictionbot
Turingos Turnitin
TurnitinBot
TwengaBot
Twice
Typhoeus
URLy Warning URLy Warning
URLy.Warning URLy.Warning
UnisterBot Vacuum
Upflow VelenPublicWebCrawler
Vagabondo
VeriCiteCrawler VeriCiteCrawler
VidibleScraper VidibleScraper
VoidEYE Voil
WBSearchBot
WEBDAV
WISENutbot WISENutbot
WWW-Collector-E WPScan
Web Auto Web Auto
Web Fuck Web Fuck
Web Sauger Web Sucker
WebAuto WebCopier
WebCollage WebEnhancer
WebFetch
WebFuck
WebImageCollector
WebStripper WebStripper
WebSucker WebWhacker
WebZIP
Webalta Webalta
Website Quester WebmasterWorldForumBot
WebsiteExtractor WebsiteQuester
Webster Who.is Bot
WiseGuys Robot WiseGuys Robot
Wonderbot
Wotbox Wotbox
Wprecon Wprecon
Xaldon WebSpider Xaldon WebSpider
Xaldon_WebSpider Xenu
Zade YoudaoBot
Zeus Zermelo
ZmEu ZoominfoBot
adscanner ZyBorg
arquivo.pt arquivo-web-crawler
autoemailspider
clark-crawler clark-crawler
cognitiveseo
crawl.sogou.com
crawler4j crawler4j
dataforseo.com
demandbase-bot
evc-batch evc-batch
facebookscraper gopher
heritrix
internetVista monitor
isitwp.com isitwp.com
lwp-trivial lwp-request
magpie-crawler mediawords
meanpathbot muhstik-scan
netEstate NE Crawler
oBot oBot
page scorer s1z.ru
pcBrowser
scalaj-http scalaj-http
scan.lol scan.lol
seobility seobility
seoscanners
seostar seostar
sexsearcher sp_auditbot
siteripz
sogouspider
spyfu spyfu
trendiction.com sysscan
trendiction.de
ubermetrics-technologies.com ubermetrics-technologies.com
webgains-bot webgains-bot
webmeup-crawler webmeup-crawler
webprosbot webprosbot
zgrab

298
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
ADmantX 404checker
ASPSeek 404enemy
Abonti 80legs
Aboundex ALittle Client
Aboundexbot
AfD-Verbotsverfahren AfD-Verbotsverfahren
AhrefsBot Aipbot
Alexibot
Anarchie
Anarchy99 Anarchy99
Anthill Anthill
Apexoo
AwarioRssBot
AwarioSmartBot AwarioSmartBot
BBBike
BDCbot BDCbot
BLEXBot BLEXBot
Bandit BackDoorBot
BatchFTP BacklinkCrawler
Battleztar Bazinga
BetaBot
Bigfoot Bigfoot
Bitacle BlowFish
BlackWidow
Blackboard
Bolt Bolt
Brandwatch Brandprotect
CATExplorador Buddy
CCBot BunnySlippers
Calculon CazoodleBot
Cegbfeieh Cegbfeieh
CheTeam
CheeseBot CheeseBot
CherryPicker CherryPicker
ChinaClaw
Chlooe
Claritybot
Cloud mapping Cloud mapping
Cocolyzebot
Collector Collector
Copyscape
Craftbot
CrazyWebCrawler CrazyWebCrawler
Curious
CyotekWebCopy CyotekWebCopy
DSearch DSearch
DataCha0s DataCha0s
DatabaseDriverMysqli
Demon Demon
Deusu Deusu
Devil
Digincore Digincore
Disco Disco
Discobot Discobot
DittoSpyder Dispatch
DomCopBot DnyzBot
DomainCrawler DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
DomainStatsBot Domains Project
Dotbot
Drip
ECCP/1.0 ECCP/1.0
Ecxi Ebingbong
Exabot EroCrawler
ExtLinksBot
ExtractorPro ExtractorPro
Extreme Picture Finder Extreme Picture Finder
EyeNetIE EyeNetIE
Ezooms
FDM FDM
FHscan
FemtosearchBot FemtosearchBot
Firefox/7.0 FrontPage
Foobot
Freeuploader
Fyrebot Fyrebot
G-i-g-a-b-o-t
GT::WWW GT::WWW
Genieo GalaxyBot
GetWeb GetRight
Getintent Getintent
Gigabot Go-Ahead-Got-It
GrabNet
Grabber
GrapeFX
GridBot
HEADMasterSEO HEADMasterSEO
HMView HTTrack
Haansoft
HaosouSpider HaosouSpider
Havij
Heritrix Heritrix
Hloader Humanlinks
HonoluluBot
HybridBot HybridBot
IDBTE4M IDBTE4M
IDBot
Iblog Iblog
Id-search
Image Sucker
IndeedBot IndeedBot
Intelliseek Intelliseek
InterGET
Internet Ninja
InternetSeer
Iskanie
JOC Web Spider JOC Web Spider
Jbrofuzz
JennyBot
JetCar JetCar
Jetty
Joomla
Jorgee Jorgee
JustView JustView
Kozmosbot Keybot Translation-Search-Machine
Lanshanbot Keyword Density
LeechGet Kinza
LWP::Simple
LexiBot
Lftp Lftp
Libwhisker Libwhisker
LinksManager LieBaoFast
Lightspeedsystems
LinkScan
LinkWalker
LinkextractorPro
LinqiaMetadataDownloaderBot
Lipperhey Lipperhey
Lipperhey Spider Litemage_walker
MIDown tool MIDown tool
MIIxpc
MQQBrowser MQQBrowser
MSFrontPage MSIECrawler
Mag-Net Magnet
Mail.RU_Bot
Majestic SEO Majestic SEO
Majestic-SEO Mass Downloader
Majestic12
MarkWatch
Mata Hari
MauiBot MauiBot
Mb2345Browser Mb2345Browser
MicroMessenger Mediatoolkitbot
Microsoft Data Access Mister PiX
Mojeek Moblie Safari
Mozlila Mojolicious
MolokaiBot
Morfeus Fucking Scanner
Msrabot Msrabot
NPbot Musobot
Needle Nameprotect
Nessus Navroad
NetMechanic NetSpider
Netcraft NetZIP
Nettrack
Netvibes Netvibes
Nikto
NimbleCrawler NimbleCrawler
Nimbostratus Nimbostratus
Ninja Ninja
Nutch Nutch
Offline Explorer Octopus
OpenVAS
Openvas Openvas
OrangeBot OrangeSpider
PECL::HTTP PECL::HTTP
POE-Component-Client-HTTP POE-Component-Client-HTTP
Panscient Page Analyzer
PageAnalyzer
PageThing.com
Papa Foto Papa Foto
PeoplePal Petalbot
Picscout Pi-Monster
Picsearch
Piepmatz
Pixray Pixray
Pockey PleaseCrawl
Pump ProPowerBot
PxBroker ProWebWalker
QueryN Metasearch Psbot
PyCurl
Quick-Crawler Quick-Crawler
RankActive RankingBot2
Re-re Rankivabot
RankurBot
ReGet ReGet
Recorder SEOkicks-Robot
RedesScrapy
Ripper
RocketCrawler
Rogerbot
SEOlyticsCrawler
SEOprofiler SEOprofiler
SEOstats SEOstats
ScanAlert SalesIntelligent
ScoutJet
Scrapy Scrapy
Screaming Screaming
SearchmetricsBot ScrepyBot
SemanticJuice Searchestate
SemrushBot SentiBot
SeoSiteCheckup
Seomoz Seomoz
Shodan
Siphon Siphon
Sitevigil SiteCheckerBotCrawler
Snoopy SiteLockSpider
SocialRankIOBot SiteSucker
Sogou web spider Snake
Snapbot
Sottopop Sottopop
SpaceBison Spammen
SpankBot SpankBot
Spanner Spanner
SputnikBot Spinn3r
Sqlworm Sqlmap
Sqworm
Steeler Steeler
Stripper
Sucuri Sucuri
SuperBot Swiftbot
SuperHTTP Szukacz
Surfbot T0PHackTeam
T8Abot Telesphorep
Teleport Thumbor
TightTwatBot
Toweyabot Toweyabot
Tracemyfile Trendictionbot
Turingos True_Robot
Typhoeus TurnitinBot
URLy Warning URLy.Warning
Vacuum Upflow
V-BOT
VB Project
Vagabondo Vagabondo
VoidEYE Voltron
WASALive-Bot
WBSearchBot WBSearchBot
WISENutbot WISENutbot
WPScan
WWW-Collector-E
WWW::Mechanize
WWWOFFLE
Wallpapers/3.0
WallpapersHD
Web Auto
Web Collage Web Collage
Web Enhancer Web Pix
WebAuto
WebCollage
WebCopier
WebFetch WebFetch
WebFuck
WebGo IS WebGo IS
WebImageCollector
WebLeacher WebLeacher
WebSauger WebReaper
WebSucker WebStripper
WebWhacker WebWhacker
WebZIP
Webshag
WebsiteQuester WebsiteQuester
Webster Webster
Whacker
Whatweb
Widow Widow
Woobot Woobot
Wprecon Wprecon
Xaldon WebSpider Xaldon_WebSpider
Xenu Xenu
YoudaoBot Zeus
Zauba
ZoominfoBot ZoominfoBot
ZumBot ZumBot
ZyBorg archive.org_bot
adscanner arquivo.pt
backlink-check
cah.io.community cah.io.community
evc-batch coccocbot
gopher com.plumanalytics
instabid crawler.feedback
internetVista monitor crawler4j
lwp-request eCatch
ips-agent
iubenda-radar
magpie-crawler
meanpathbot
mediawords
muhstik-scan muhstik-scan
oBot plumanalytics
page scorer
polaris version
probe-image-size
s1z.ru s1z.ru
scan.lol scan.lol
seobility seobility
seostar seocompany.store
sexsearcher
sitechecker.pro sitechecker.pro
sogouspider spyfu
webpros.com sysscan
webprosbot trendiction.de
ubermetrics-technologies.com
webgains-bot
x09Mozilla
xpymep1.exe
zauba.io zauba.io
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
buchananshardware.com beauby.ru
cacheimages.com enge-fotzen.info
cometorussia.net hdimagegallery.net
hundejo.com izoll.ru
iflycapetown.co.za marketingvici.com
kino2018.club mixed-wrestling.ru
layola.biz.tc pateaswing.com
tizanidinewithdrawal.blogspot.com peugeot-club.org
vancleefreplica.pw simplepooltips.com
wpthemedetector.co.uk uptimebot.net

922
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
1-free-share-buttons.com 123cha.com
2itech.ru 1314dh.com
300richmond.co.nz 1stat.ru
40cg.com 2345.com
4k-player.pl 2drittel.de
51unlim.ru 3th.co.in
57883.net 3weekdiet.com
5kstore.com 5i2.net
70casino.online 5ws.dating-app.ru
aa8780bb28a1de4eb5bff33c28a218a930.com 8xv8.com
adanih.com 91abcw.com
a.pr-cy.ru
above.com
absolutelyanalog.com
abtasty.com
addray.pro addray.pro
adloads.com adman.gr
adpremium.org adrenalin-stk.ru
ads.tremorhub.com adtech.de
adtech.us advancedsoftwaresupport.com
adultfullhd.com afmuseum.com
adventureparkcostarica.com ahmedabadwebs.com
advokateg.ru
agahinameh.com
alfapro.ru
all4invest.info
all4wap.ru all4wap.ru
allnews.md
alltheviews.com alltheviews.com
amateurlivechat.org
amazingninja.com amazingninja.com
anabolics.shop ampower.me
analytics-ads.xyz angigreene.com
animaltoplist.com apidiamondatanet-a.akamaihd.net
animebox.com.ua apimegabrowsebiz-a.akamaihd.net
antons-transporte.de apisecretsaucebi-a.akamaihd.net
ap.senai.br appmsr.org
apiqualitinknet-a.akamaihd.net approved.su
architecturebest.com aquarium-pro.ru
as5000.com arabgirls.adultgalls.com
asacopaco.tk arkkivoltti.net
asrvrep-a.akamaihd.net asophoto.com
atofilms.com atlasvkusov.ru
ausergrubhof.info avtointeres.ru
automobile-spec.com bablonow.ru
aviav.org baoxaydung.com.vn
avtochehli.by barbourjackets.online
awency.com
azazu.ru
azbukafree.com
babyguns.ru
baditri.com
balkanfarma.org
banan.tv
bannerconnect.net
bard-real.com.ua
bavsac.com
beachfix.co
beatifullandscape.co
beget.tech beget.tech
berdasovivan.ru bestcoin.cc
berlininsl.com besthoro.ru
best-businessman.ru bestplacetobuyeriacta.jw.lt
best-way.men bestwebsiteawards.com
bestmarriages.com beta.hotkeys.com
bestofferswalkmydogouteveryday.gq betshuckclean.com
bezlimitko.xyz betterscooter.com
bigcareer.info betwinservice.com
binomo.com beztuberkuleza.ru
bitcoinremote.com bidbuy.co.kr
bioscorp.ru
bitcoinpile.com
bitnote.co
bizcheapjerseyswholesalechina.com
bizru.info bizru.info
blackbabesporn.com bizzliving.com
blagovest-med.ru blagovest-med.ru
bmw-ark.ru blog.f00kclan.de
bmwhighperformers.com blogig.org
bobinoz.com bolegapakistan.com
bochemit.com.ua bongiornos.info
bonkers.name
bonzibuddi.com bonzibuddi.com
brainboosting.club book-bets.com
brothers-smaller.ru bookmark4you.com
buehne-fuer-menschenrechte.de bosman.pluto.ro
burkesales.com brainboostingsupplements.org
bus-offer.com brainsandeggs.blogspot.com
business-suggested.tk braintobe.win
buzz-porno.info bravegirlsclub.com
buzzsumo.com brimstonehillfortress.org
bvlgariring.xyz btnativenav.com
byme.se btvn.ru
cacheimages.com buyclomidonlaine.com
callawaygolfstore.online buyhoverboard.com
candytiner.com buytizanidineonline.blogspot.com
buzzonclick.com
candelluxsklep.pl
cannibalcheerleader.com cannibalcheerleader.com
carrauterie.be cartierring.xyz
cashonads.com casinopinup-wins.com
cheap-pills-norx.com cbb1smartlist12.click
cheapbarbour.online cdncash.com
cheapestjerseysonwholesale.com ceotrk.com
chatroulette.life
chatroulette.online
cheapjerseysbizwholesale.us cheapjerseysbizwholesale.us
chocolateslim-en-france.com cheatcode-lita12.rhcloud.com
chorus.terakeet.com chelyabinsk.xrus.org
cialisndbrx.com chinaelitecheapjerseys.com
cindymatches.com christianlouboutinshoes.xyz
cineacademy.ru ci.ua
cinemaflix.website cialis-samples.com
circlesl.com cialischmrx.com
cleanersoft.com cialisovercounteratwalmartusa.com
clickso.com cinemaenergy-hd.ru
clayvasedesigns.tk
click2pawn.com
clubfashionista.com clubfashionista.com
com-cleaner.systems cmd.kz
compliance-alex.xyz cntravelre.com
compliance-jessica.top cognitiveseo.com
comsysnet.com colehaanoutlet.store
contentsexpress.com compliance-checker-7.info
coolgamechannel.com compliance-george.xyz
corta.co cookie-law-enforcement-hh.xyz
counter.yadro.ru cookielawblog.wordpress.com
cowlmash.com cookingmeat.ru
crackzplanet.com culmimg.pw
cupday.com culturevie.info
customchocolate.business-for-home.com customerguru.in
czat.wp.pl cv.wallhade.co
dalavia.ru cyprusbuyproperties.com
darodar.com damedingel.ya.ru
datract.com dev-seo.blog
deanmoore.ie dirtpics.pw
delitime.info dividendo.ru
deshevo-nedorogo.ru
detki-opt.ru
digilander.libero.it
dimfour.com
dlya-android.org dlya-android.org
dogclothing.store doctissimo.fr
dokumentalkino.net dogoimage.com
domainanalyzing.xyz dogsrun.net
domaincdn.xyz domaincdn.xyz
domaineaneblanc.com domini.cat
dominterior.org dostavka-v-ukrainu.ru
domznaniy.ru dosug-lux.ru
dosugrostov.site downloadeer.net
downtuptv.gq downloadmefiranaratb1972.xpg.com.br
drunkmoms.net doxysexy.com
earncash.com.ua dprtb.com
earthmagic.info dumuelave.xyz
e-c.al
eas-seo.com eas-seo.com
easyshoppermac.com
ebooknovel.club
educontest.net
eets.net eets.net
efkt.jp ekn-art.se
ekspresihati.info electronicadirect.com
elektrozigarette2.wordpressy.pl eleimgo.pw
elektrozigarettekaufen1.devhub.com elektrozigarettekaufen1.devhub.com
elektrozigarettekaufen2.blogse.nl eloconcream.blogspot.com
emergencyneeds.org en.home-task.com
enge-fotzen.info energy-ua.com
enhand.se erotik-kostenlos.net
enternet.ee escortslet.net
eredijovon.com euromasterclass.ru
espaceinventoristes.com europages.com.ru
eu-cookie-law.blogspot.com excaliburfilms.com
euroskat.ru exchangeit.gq
evaashop.ru exportshop.us
ez8motelseaworldsandiego.com ez8motelseaworldsandiego.com
ezigarettenkaufen1.pagina.gr ezaz.info
ezrvrentals.com ezigaretteshop.webs.com
f07.de f-loaded.de
f1nder.org
facebook-mobile.xyz
factorynightclub.com factorynightclub.com
familienzahnaerzte.com fast-wordpress-start.com
farmingworm.com filesclub.net
fetishinside.com financepoints.eu
fetlifeblog.com findthe.pet
fidelityfunding.com finstroy.net
filesvine.com firma-legion.ru
finejewelryshop.ru fitfloponline.store
flashbannernow.com fitness-video.net
flex4launch.ru flauntyoursite.com
floating-share-buttons.com
floridahuntingfishingadventures.com
floridamhca.org floridamhca.org
forex-procto.ru food.dtu.dk
forum-engineering.ru foxjuegos.com
foto-telok.net fr-bearings.ru
fototravel.eu free-deals.faith
foxydeal.com freenode.info
franecki.net froggytube.com
free-share-buttons.com fullgirl.ru
freelinkbuilding.website.tk fungirlsgames.net
freeseedsonline.com galaxys6manual.info
freewhatsappload.com gamerextra.com
freewlan.info games.kolossale.ru
fres-news.com garciniaxt.us
freshwallpapers.info gearsadspromo.club
front.ru genericlowlatencyasiodriverhq.aircus.com
funnymama.com gepezz.info
galleryawesome.com get-free-traffic-now.com
game-mmorpg.net get-your-social-buttons.info
gameplexcity.com ghazel.ru
gate5.co.za ghernnqr.skyrock.com
gavuer.ru gosarhivrt.ru
gaygalls.net grand-chlen.ru
ge0ip.org greatdealshop.com
getaclueamerica.com greatidea.marketing
getlamborghini.ga gstatey.net
gettpromos.com guarrasdelporno.xxx
girlgamerdaily.com gunsvicceadadebt.tk
girlporn.ru hawaiisurf.com
glasof.es hkladys.com
gnuetella.com homedecoguide.info
googst2.ru hostingclub.lk
gov.yanao.ru hotblognetwork.com
graphicwe.org hoverboard360.at
gta-top.ru hoverboard360.de
gymi.name
hackers-crackers.tk
haikuware.com
hausfrauensex18.com
hccoder.info
hd-filmy.net
healmytrauma.info
hitmuzik.ru
hobbyhuren24.net
home.myplaycity.com
homedecorpicture.us
hostcritique.com
hoste.octopis.com
hotkeys.com
howlongdoestizanidinestayinyoursystem.blogspot.com howlongdoestizanidinestayinyoursystem.blogspot.com
husky-shop.cz https-legalrc.biz
hustoon.over-blog.com huimin764128.com
i-hobot.ru iccornacircri.cf
iboss.com ido3.com
indiasourcemart.in imagui.eu
infazavr.ru indiakino.net
infogame.name infogame.name
intellego.info investingclub.ru
investyb.com investmac.com
iqoption.com itbc.kiev.ua
istizanidineanopiate.blogspot.com
iusstf.org iusstf.org
ix20.ru ixora.pro
jerseychinabizwholesale.us janavibekken.no
jpcycles.com javcoast.com
karpun-iris.ru jbl-charge.info
kazka.ru jerseywholesalechinabiz.com
kemerovo.zrus.org jolic2.com
keywordblocks.com justdating.online
kinomaniatv.pl kakadu-interior.com.ua
kinosed.net kazan.zrus.org
kiwe-analytics.com kchaxton.com
konica.kz keenoutlet.online
konteiner24.com keywordbasket.com
koopilka.com kino-ecran.ru
koral.se kinozapas.org
koronirealestate.gr kmd-pto.ru
krasnodar-avtolombards.ru knowyournextmove.com
kriokomora.info kol-energo.ru
la-fa.ru kopihijau.info
laexotic.com kung-fu-ru.com
kunstaktien.info
kutikomi.net
labelwater.se
labvis.host.sk
lacasamorett.com
ladsup.com
lakomka.com.ua
languagecode.com languagecode.com
larose.jb2c.me laptoper.net
lashstudia.ru
layola.biz.tc layola.biz.tc
lenpipet.ru lechenie-gemorroya.com
lescinq.com libertybilisim.com
letmacwork.world liky.co.ua
lexaprogeneric.link lion.snapmap.us
linkpulse.com lirunet.ru
liran-locks.com lomb.co
livingcanarias.com lomza.info
localflirtbuddies.com lonely-mature.com
locationdesaison.com lordzfilmz.me
lol-smurfs.com
lonfon.xyz
luchshie-topcasino.ru luchshie-topcasino.ru
mac-shield.com luxmagazine.cf
mafcards.ru mafcards.ru
manyresultshub-a.akamaihd.net mall.uk
map028.com maridan.com.ua
mb140.ru marketingvici.com
mbiologi.ru matrixalchemy.com
megakino.net max-p.men
message-warning.net maximilitary.ru
microstatic.pl mebel-vstroika.ru
middlerush-a.akamaihd.net meddesk.ru
mmog-play.ru mediafresh.online
moncleroutletonline.pw mediaoffers.click
more-letom.ru medicine-4u.org
moyakuhnia.ru medkritika.ru
muschisexbilder.com mega-bony-2017.pl
musicvidz.ru menetie.ru
muycerdas.xxx metallosajding.ru
mwtpludn.review michaelkorsoutletstore.net
mydownloadengine.com mirtorrent.net
myxdate.info mixx.com
narosty.com mobifunapp.weebly.com
naturalshair.site mobioffertrck.com
moetomnenie.com
monarhs.info
monetizationking.net
moneymaster.ru
monitorwebsitespeed.com
monsterdivx.tv
moscow.xrus.org
mossmesi.com
mostantikor.ru
moviezbonkerssk.cf
mrbojikobi4.biz
mrsdalloways.com
my-big-family.com
myfreemp3.eu
mygameplus.com
mylida.org
naj-filmy24.pl
napalm51.nut.cc
nastydollars.com
navalwiki.info
negral.pluto.ro
netallergy.ru netallergy.ru
nextconseil.com netanalytics.xyz
nfljerseyscheapchinabiz.com nikkiewart.ru
nikitabuch.com nisuturnetdgu.tk
noumeda.com northfacestore.online
nucia.biz.ly notebook-pro.ru
nuker.com novgorod.xrus.org
oddamzadarmo.eu nvformula.ru
old-rock.com nw-servis.ru
onlineserialy.ru o-o-8-o-o.com
openlibrary.org obiavo.kz
oren-cats.ru obiavo.net
osnova3.ru obsessionphrases.com
oynat.info obuv-kupit.ru
paccohichetoti.ml okel.co
partner-high.men okmusic.jp
partnersafe.men olvanto.ru
paydayloanslocal.com one.net.in
paydayonlinecom.com online-hd.pl
pc-test.net online-podarki.com
pcimforum.com ooomeru.ru
pendelprognos.se openmediasoft.com
percin.biz.ly ops.picscout.com
perfectplanned.com ownshop.win
perm-profnastil.ru pammik.ru
perm.xrus.org paramountmarble.co.uk
picswe.com partner-stop.men
picturesfrom.com partners-ship.pro
pigrafix.at partnerworkroom.men
placid-rounded-coast.glitch.me pc4download.co
ploenjitmedia.azurewebsites.net pensplan.com
pogruztehnik.ru photo-clip.ru
polybuild.ru phpdevops.com
popup-jdh.xyz pk-services.ru
poquoson.org pl.aasoldes.fr
porno-dojki.net platezhka.net
porno-home365.com pod-muzyku.club
pornobrazzers.biz poem-paying.gq
pornolook.net pokemongooo.ml
pornoreino.com popads.net
pornosmola.info popunder.ru
pornosok.ru porndroids.com
portal-eu.ru portadd.men
portside.cc
pregnant.guru
pricheskaonline.ru pricheskaonline.ru
privat-girl.net privacyassistant.net
privatefx.all4invest.info privatbank46.ru
prizefestival.mobi prizeestates.cricket
produkto.net prlog.ru
projectforte.ru projefrio.com.br
prostitutki-kiev.org prostitutki-tyumeni.org
prostitutki-tolyatti.org puteshestvennik.com
pshare.biz
pskcijdc.bloger.index.hr
psoriasis-file.trade
pufip.com
pyramidlitho.webs.com
qexyfu.bugs3.com
quality-traffic.com quality-traffic.com
ranksonic.org quickcashlimited.com
rankexperience.com
ranking2017.ga
rapidsites.pro rapidsites.pro
realnye-otzyvy.info raschtextil.com.ua
redirectme.net razamicroelectronics.com
renewablewealth.com realmonte.net
retreatia.com rebuildermedical.com
rezeptiblud.ru reelheroes.net
rfd-split.hr registrysweeper.com
rialp.getenjoyment.net reklamuss.ru
riralmolamsaca.tk renecaovilla.online
robertefuller.com rentaremotecomputer.com
rossanasaavedra.net resultshub-a.akamaihd.net
royal-betting.net rockingclicks.com
rukino.org rome2rio.com
rusexy.xyz royalvegascasino.com
russintv.fr rozalli.com
saddiechoua.com ruf777.com
saratov.xrus.org rumamba.com
sardinie.us salut-camp.ru
sbt-aqua.ru sandhillsonline.com
sbtdesign.co.uk santechnik.jimdo.com
scanner-ivan.top sanyuprojects.com
scanner-margo.top sarahmilne.top
scanner-walter.top scanner-don.top
scanspyware.net
scopich.com
sdelatmebel.ru sdelatmebel.ru
se-welding.ru sdi-pme.com
search-goo.com
security60-e.com security60-e.com
seoholding.com securityallianceservices.com
sexad.net seo-smm.kz
sexreliz.com seofirmreviewsus.info
sextracker.com seorank.info
sexyteens.hol.es sexflirtbook.com
sheerseo.com sexkontakte-seite.com
shlyahten.ru sexphoto.site
shohanb.com sexs-foto.com
shop-electron.ru sexs-foto.top
shoppingjequiti.com.br sexsearch.com
shoppytoolmac.com sfj-ror.no
sildenafil-tadalafil.info shemale-sex.net
simul.co shinikiev.com.ua
site-speed-checker.site sicfor.bcu.cc
sivs.ru sigmund-freud.co.uk
sixcooler.de silktide.com
skachat-besplatno-obrazcy.ru simple-share-buttons.com
siteripz.net
skytraf.xyz skytraf.xyz
skyway24.ru sledstvie-veli.net
slim.sellany.ru smichovbike.cz
slomm.ru
slujbauborki.ru
smartpet.ru
snowplanes.com
soblaznu.net
social-buttons.xyz social-buttons.xyz
social-search.me
socialbookmarksubmission.org
soft-terminal.ru soft-terminal.ru
soochi.co soft1.ru
sortthemesitesby.com softonicads.com
sovinsteel.ru spamnuker.com
srdrvp.com speeddream.xyz
ssconstruction.co spinnerco.ca
sticken.co stairliftstrue.com
students-cheapskate.ml stal-rulon.ru
sukirgenk.dvrlists.com stanthonyscatholicchurch.org
superiends.org sundrugstore.com
sunflowerdrawingpaintings.blogspot.com
superlist.biz
supervesti.ru supervesti.ru
svtrd.com suralink.com
swagbucks.com surflinksmedical.com
swsociety.se surgut.zrus.org
takethatad.com tabletkinaodchudzanie.com.pl
talant-factory.ru tackletarts.co
taqywu51.soup.io tattooha.com
technopellet.gr telefonsexi.com
tedy.su telefonsexsofort.tk
teencastingporn.com
teenfuck.tv
telemetryverification.net telemetryverification.net
telsis.com tfxiq.com
textads.men the-usa-games.blogspot.com
the-torrent-tracker.blogspot.com thepokertimer.com
theallgirlarcade.com theporndude.com
themeforest.net titelhelden.eu
themestotal.com titslove.yopoint.in
thewomenlife.com tizanidine6mg.blogspot.com
titan-ads.life tizanidinenarcotic.blogspot.com
tizanidine4mgstreetvalue.blogspot.com tizanidineusedfor.blogspot.com
tizanidinehcl2mgtablet.blogspot.com tongkatmadura.info
tizanidinepain.blogspot.com tonivedu.it
tizanidinesideeffects.blogspot.com topappspro.com
tizanidinesleepaid.blogspot.com topdownloads.ru
tmtrck.com toproadrunner5.info
tonerbox.kz torrent-to-magnet.com
topads.men totu.info
tourismvictoria.com tourismvictoria.com
ttrraacckkrr.com trustaffs.com
twincitiescarservice.com tsan.net
tyumen.xrus.org tube8.com
ucsol.ru tupper-shop.ru
uptime-delta.net turist-strani.ru
urldelivery.com uk-zheu20.ru
usbggettwku.ga ukrobstep.com
vasileostrovsky-rayon.ru ultimatesetnewfreeallsoftupgradesystems.pw
vchulkah.net unitygame3d.com
veles.shop unrealcommander.biz
vielporno.net valoresito.com
vrotike.ru vgoloveboli.net
vsesubwaysurfers.com viagraneggrx.com
vvpg.ru viberdownload10.com
vzlomfb.com villakohlanta.nu
wallpapersist.com vkontaktemusic.ru
watchmyfb.pl vladimir.zrus.org
wdfdocando.com voluumtrk.com
webix.biz vseuznaem.com
website-analytics.online walkme.com
website-analyzer.info wareseeker.com
welck.octopis.com website-audit.com.ua
wemedinc.com wedding-salon.net
westsextube.com weightatraining.com
white-truck.life wejdz-tu.pl
win-spy.com weprik.ru
wovis.site wikes.20fr.com
wowas31.ucoz.ru workle.website
wpthemedetector.co.uk www.solartek.ru
www.888.com www.souvenirua.com
www.ehscloud.cn
www.pinnacle-bets.com
x-porno.video
xaylapdiendanang.com
xchangetrak.com
xdoza.com xdoza.com
xgames-04.com xedserver.com
xn--80aaajbdbddwj2alwjieei2afr3v.xn--p1ai xfire.com
xn--80adaggc5bdhlfamsfdij4p7b.xn--p1ai xn--24-glceagatoq7c2a6ioc.xn--p1ai
xn--80ajbshivpvn2i.xn--p1ai xn--80aaaks3bbhabgbigamdr2h.xn--p1ai
xxxrus.org xn--c1acygb.xn--p1ai
xxxsiterips.xyz xn--h1aakne2ba.xn--p1ai
xxxtubesafari.com xn--oogle-wmc.com
yaoguangdj.com xtrafficplus.com
yellowads.men xvideosbay.com
yoopsie.com xxxtube69.com
yourporngay.com xz618.com
yubikk.info xzlive.com
za-fun-offer.com yellowstonesafaritours.com
zakazvzloma.com yourads.website
zarabiaj-dzis.pl youresponsive.com
zdesoboi.com yourtemplatefinder.com
zeroredirect7.com za-music.mymobiplanet.com
znakomstva-moskva77.ru zagadki.in.ua
zoolubimets.ru zakonobosago.ru
zrizvtrnpale.tk zatjmuzu.info
zscaler.net zeroredirect11.com
zverokruh-shop.cz zeroredirect12.com
zetmaster.ru
zhuravlev.info
zona-aqua.ru
zoomovies.org
zscalerone.net

478
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
101raccoon.ru
12masterov.com
13tabs.com
1j7740kd.website
1millionusd.xyz
3rbseyes.com
4pp13.com 4pp13.com
72-news.com aadroid.net
DomainStatsBot aanapa.ru
adamoads.com abiente.ru
adminshop.com absolutelyanalog.com
adservme.com advertisingtag.net
adultactioncam.com aero2.ru
allboard.xobor.de alta-realestate.com
allkrim.com apifasterlightin-a.akamaihd.net
allwidewallpapers.com apiqualitinknet-a.akamaihd.net
anabolics.shop appearance-cool.com
androids-store.com arcteryxstore.online
apelsinnik.website artavenuegardenstudios.com
arkkivoltti.net
artdeko.info
atlasvkusov.ru atlasvkusov.ru
atleticpharm.org autoseo-traffic.com
auspolice.com
aviav.co
avirasecureserver.com avirasecureserver.com
bear.gotcher.us azartmix.com
bidr.trellian.com azarttoto.com
azbaseimages.net
backuperwebcam.weebly.com
bamo.xsl.pt
bankcrediti.ru
beauby.ru
bedandbreakfast.com
besthomemadepornsites.com
bestpornuha.com
bitcoinremote.com
bizzliving.com bizzliving.com
bkns.vn blue-square.biz
blogs.rediff.com brandehk.dk
bpro1.top bus-offer.com
buqyxa.rincian.info buyhoverboard.com
cacheimages.com buypanicdisorderpill.com
cartierring.xyz buytizanidineonlinenoprescription.blogspot.com
cartiertrinity.xyz bvlgariring.xyz
cellfun.mobi cars-modification.net
cheap-trusted-backlinks.com cascadelink.org
cindymatches.com cdn.walkme.com
citizenclsdriveri7.pen.io cheapsergiorossi.online
clayvasedesigns.tk cinemaflix.website
compliance-donald.xyz classiquebijoux.ru
compliance-elena.xyz cleanallspyware.com
compucelunlock.net compiko.info
cookie-law-enforcement-aa.xyz compliance-alexa.top
cool-wedding.net compliance-irvin.xyz
covadhosting.biz compliance-josh.top
craftburg.ru compliance-margo.top
cs-passion.pl connectionstrenth.com
customsua.com.ua contextualyield.com
dailyfinancefix.com continent-e.tv
danschawbel.com cupday.com
dbmkfhqk.bloger.index.hr danceuniverse.ru
dellalimov.com dawlenie.com
discountbarbour.online delta-line.men
div.as deshevo-nedorogo.ru
djihispano.com deutschland123.de
domcran.net dfwu1013.info
dosug-lux.ru distver.ru
drpornogratisx.xxx diy-handmade-ideas.com
drugspowerstore.com dtm-spain.com
dtnlyss.com ebonyporn.site
dvd-famille.com ecommerce-seo.com
ecxtracking.com elektroniksigaraankara.info
ege-russian.ru elidelcream.weebly.com
elektrozigaretten1.tumblr.com emediate.eu
elmacho.xyz eng-lyrics.com
elmifarhangi.com epngo.bz
eorogo.top esopini.com
eralph.tk essaypro.com
evogarage.com exchangeit.gq
eyelike.com.ua extlabs.io
ezigarettenkaufen1.deviantart.com ezigarettekaufen2.blox.pl
face.hostingx.eu ezigarettenshop1.yolasite.com
favorcosmetics.com florida-tourism.net
favoritemoney.ru focalink.com
filesdatabase.net formulaantiuban.com
foto-telok.net franklinfire.co
foto-weinberger.at free-laptop-reward.com
fotopop.club
foxinsocks.ru
free411games.com free411games.com
frustrated-favorable.gq freemaintenancesysforpcandmac.top
fuckbuddybestgilf.info fugarif.ga
fym.com.ru fullgirl.ru
game-top.su gamevalue7.weebly.com
girlsatgames.ru gamingspark.com
gogps.me gaz-voshod.ru
goodhousekeeping.com gktt.ru
googglet.com goforexvps.com
google-liar.ru
graphics8.info graphics8.info
gsbs.com.ua gsasearchengineranker.site
heartofpayne.xyz hesteel.pl
helicalpile.us hifidesign.ru
hermesbracelets.xyz hmmm.cz
historichometeam.com hobild.net
homedo.fabpage.com hol.es
hopeonthestreet.co.uk home.myplaycity.com
hothot.ru hotblognetwork.com
humanorightswatch.org houseofgaga.ru
hvd-store.com hunterboots.online
iklysha.ml iplogger.org
iminent.com itbc.kiev.ua
impotentik.com jamiembrown.com
interfucks.net jerseywholesalebizchina.com
intermesh.net jikoman.info
investsuccess.org jimmychoosale.online
investyb.com juliadiets.com
ipornox.xxx jumptap.com
iqupdatetmz.win junketjuice.blogspot.com
javitas.info jyvopys.com
johnrobertsoninc.com keki.info
journalhome.com klitimg.pw
kakablog.net kochanelli.com
kambasoft.com kongruan.com
karaganda.xkaz.org konkursov.net
kmgamex.cf labplus.ru
kometars.xyz lacave.ntic.fr
kredit-pod-zalog-krasnodar.ru lescinq.com
la-fa.ru lion.bolegapakistan.com
leadwayau.com lirunet.ru
lineshops.biz littlesexdolls.com
linkdebrideur.xyz locationdesaison.com
lollypopgaming.com lonfon.xyz
lostfilm-online.ru lostfilm-online.ru
lowephotos.info marsgatan.com
lunchrock.co masterseek.com
m3gadownload.pl mecash.ru
m4ever.net medanestesia.ru
magento-crew.net mediaoffers.click
magnetpress.sk moetomnenie.com
manyresultshub-a.akamaihd.net moinozhki.com
marketingvici.com monclercheap.xyz
masturbate.co.uk moz.com
maximilitary.ru mp3s.club
medkletki.ru muztops.ru
memberty.com mvpicton.co.uk
mesto-x.com mxgetcode.com
micasainvest.com mydirtystuff.com
michaelkorssaleoutletonline.net mywallpaper.top
microsoftportal.net mzdish.site
minet.club netpics.org
mitrasound.ru nfljerseyscheapbiz.us
modabutik.ru njkmznnb.ru
mojpregled.com nodding-passion.tk
money-every-day.com nootrino.com
moneymaster.ru
mosdverka.ru
mowser.com
mrvideospornogratis.xxx
mural.co
music.utrolive.ru
musicspire.online
narkologiya-voronezh.ru
nextlnk12.com
notebook-pro.ru notebook-pro.ru
obsessionphrases.com okayimage.com
offside2.5v.pl olvanto.ru
onlineslotmaschine.com online.ktc45.ru
ons-add.men onlinetvseries.me
optibuymac.com painting-planet.com
oxotl.com parfusale.se
par-fallen.ga partnerworkroom.men
pattersonsweb.com pattersonsweb.com
phormchina.com peugeot-club.org
picsearch.com pipki.r.acdnpro.com
picturesboss.com pizdeishn.com
pinsdaddy.com play-movie.pl
pixell.club playmsn.com
playfortuna-play.ru polybuild.ru
popander.mobi pony-business.com
pornogig.com pornoblood.com
pornosemki.info postclass.com
pornotubexxx.name praisong.net
pourvous.info princeadvantagesales.com
pregnant.guru
promodj.com promodj.com
psa48.ru
pukaporn.com
pussyfleet.com
putevka24.ru putevka24.ru
puzzleweb.ru pxhdwsm.com
ranking2017.ga qitt.ru
rapidgator-porn.ga quickloanbank.com
realnye-otzyvy.info r-e-f-e-r-e-r.com
relatodelpresente.com.ar raschtextil.com.ua
remont-ustanovka-tehniki.ru ratemodels.net
replica-watch.ru redirect.trafficreceiver.club
roofers.org.uk remote-dba.de
russia-today-video.ru ru-dety.ru
saddiechoua.com russkoe-zdorovie.ru
sankt-peterburg.nodup.ru saugatuck.com
sarahmilne.top scopich.com
scanner-jack.top scripted.com
screentoolkit.com search.pch.com
segol.tv searchtooknow-a.akamaihd.net
seinterface.com semrush.com
seksotur.ru seofirmreviewsus.info
servisural.ru seorankinglinks.com
sexs-foto.com sexkontakteao.info
sexs-foto.top sexsaoy.com
sharpchallenge.com shitmovs.com
sherlock.se shoppingjequiti.com.br
shop-electron.ru sildenafilcitratemed.com
snjatie-geroinovoy-lomki.ru silktide.com
spaceshipad.com sirpornogratis.xxx
spin2016.cf sisiynas.ru
spywareit.com sitiz.club
stathat.com sledstvie-veli.net
steebook.com smailik.org
stickers-market.ru splendorsearch-a.akamaihd.net
storehouse.ua stairliftsarea.com
streetfooduncovered.com stop-gepatit.te.ua
stroilka.info svbur.ru
stroimajor.ru tabakur77.com
supermama.top tattooha.com
swsociety.se thaisamkok.com
techart24.com tizanidinecost.blogspot.com
thegameriders.com token-lab.org
titan-cloud.life toondinsey.com
tizanidinevsvalium.blogspot.com top-study.work
tkanorganizma.ru top10-way.com
topshef.ru torrentgamer.net
torontoplumbinggroup.com trichizobswiv.agddns.net
torrentdownloadhub.com turvgori.ru
tracklead.net twincitiescarservice.com
trafficmonetize.org txxx.com
traffixer.com ultramart.biz
traxdom.ru unrealcommander.com
trkur.com url-img.link
tula.mdverey.ru valkiria-tk.ru
tvory.predmety.in.ua veopornogratis.xxx
ucsol.ru vesnatehno.com
ukkala.xyz videos-for-your-business.com
usadacha.net videotuber.ru
varbergsvind.se virtuagirl.com
vavilone.com vtc.pw
vip2ch.com vzglyadriv.kg
vut.com.ru w3javascript.com
website-audit.com.ua warningzscaler.heraeus.com
www.solartek.ru webmasterhome.cn
wyniki-lista.pl weburlopener.com
xclicks.net weprik.ru
xerox-douglas.cf wineitudes.wordpress.com
xn--80atua3d.xn--p1ai winwotgold.pl
xz618.com womama.ru
yatrk.xyz word-vorlagen.xyz
zaimite.ru worldis.me
zarepta.com xblognetwork.com
zigarettenonlinekaufen2.drupalgardens.com xn----7sbho2agebbhlivy.xn--p1ai
zootravel.com xn--1-8sbcpb0bdm8k6a.xn--p1ai
zreloeporno.tv xn--80aaaks3bbhabgbigamdr2h.xn--p1ai
xn--l1aengat.xn--p1ai
xn--oogle-wmc.com
zakon-ob-obrazovanii.ru
zaloro.com
zarabotat-na-sajte.ru
zmoda.hostreo.com
znakomstvaonlain.ru
zog.link
zojirushi-products.ru
zoogdisany.com
zvuker.net

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.126 0; 10.1.0.129 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.03.3014 ### Version: V4.2022.03.3015
### Updated: Thu Mar 31 10:14:56 UTC 2022 ### Updated: Thu Mar 31 22:04:39 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.03.3014 ### Version: V4.2022.03.3015
### Updated: Thu Mar 31 10:14:56 UTC 2022 ### Updated: Thu Mar 31 22:04:39 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##