vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-09 22:05:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.12.2755 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-12-04 08:03:36 +00:00
parent 7162f1d776
commit 570bcea278
51 changed files with 72821 additions and 72798 deletions
Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.12.2754 #### Version: V4.2021.12.2755
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [630](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.12.2754 #### Version: V4.2021.12.2755
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [630](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -11,9 +11,9 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2021.12.2754 #### Version: V4.2021.12.2755
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [630](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

8960
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_generator_lists/bad-user-agents.list
View file

@ -1,4 +1,3 @@
Citoid
01h4x.com 01h4x.com
360Spider 360Spider
404checker 404checker
@ -78,6 +77,7 @@ CheeseBot
CherryPicker CherryPicker
ChinaClaw ChinaClaw
Chlooe Chlooe
Citoid
Claritybot Claritybot
Cliqzbot Cliqzbot
Cloud\ mapping Cloud\ mapping

4
_google_analytics_ghost_spam/README.md
View file

@ -3,9 +3,9 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2021.12.2754 #### Version: V4.2021.12.2755
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [630](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

8967
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

8967
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

8967
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

1
dev-tools/_robots_input/robots-input.txt
View file

@ -77,6 +77,7 @@ CheeseBot
CherryPicker CherryPicker
ChinaClaw ChinaClaw
Chlooe Chlooe
Citoid
Claritybot Claritybot
Cliqzbot Cliqzbot
Cloud mapping Cloud mapping

8967
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

8967
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

8967
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -138,6 +138,7 @@
"~*(?:\b)CherryPicker(?:\b|)" 0; "~*(?:\b)CherryPicker(?:\b|)" 0;
"~*(?:\b)ChinaClaw(?:\b|)" 0; "~*(?:\b)ChinaClaw(?:\b|)" 0;
"~*(?:\b)Chlooe(?:\b|)" 0; "~*(?:\b)Chlooe(?:\b|)" 0;
"~*(?:\b)Citoid(?:\b|)" 0;
"~*(?:\b)Claritybot(?:\b|)" 0; "~*(?:\b)Claritybot(?:\b|)" 0;
"~*(?:\b)Cliqzbot(?:\b|)" 0; "~*(?:\b)Cliqzbot(?:\b|)" 0;
"~*(?:\b)Cloud\ mapping(?:\b|)" 0; "~*(?:\b)Cloud\ mapping(?:\b|)" 0;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

8967
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2754 2755

8967
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

8967
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.183.70.238 0; 20.124.134.31 0;

8967
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

8967
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

8967
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -138,6 +138,7 @@
"~*(?:\b)CherryPicker(?:\b|)" 0; "~*(?:\b)CherryPicker(?:\b|)" 0;
"~*(?:\b)ChinaClaw(?:\b|)" 0; "~*(?:\b)ChinaClaw(?:\b|)" 0;
"~*(?:\b)Chlooe(?:\b|)" 0; "~*(?:\b)Chlooe(?:\b|)" 0;
"~*(?:\b)Citoid(?:\b|)" 0;
"~*(?:\b)Claritybot(?:\b|)" 0; "~*(?:\b)Claritybot(?:\b|)" 0;
"~*(?:\b)Cliqzbot(?:\b|)" 0; "~*(?:\b)Cliqzbot(?:\b|)" 0;
"~*(?:\b)Cloud\ mapping(?:\b|)" 0; "~*(?:\b)Cloud\ mapping(?:\b|)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

8967
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.183.70.238 0; 20.124.134.31 0;

8967
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.183.70.238 0; 20.124.134.31 0;

8967
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -138,6 +138,7 @@
"~*(?:\b)CherryPicker(?:\b|)" 0; "~*(?:\b)CherryPicker(?:\b|)" 0;
"~*(?:\b)ChinaClaw(?:\b|)" 0; "~*(?:\b)ChinaClaw(?:\b|)" 0;
"~*(?:\b)Chlooe(?:\b|)" 0; "~*(?:\b)Chlooe(?:\b|)" 0;
"~*(?:\b)Citoid(?:\b|)" 0;
"~*(?:\b)Claritybot(?:\b|)" 0; "~*(?:\b)Claritybot(?:\b|)" 0;
"~*(?:\b)Cliqzbot(?:\b|)" 0; "~*(?:\b)Cliqzbot(?:\b|)" 0;
"~*(?:\b)Cloud\ mapping(?:\b|)" 0; "~*(?:\b)Cloud\ mapping(?:\b|)" 0;

18
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Anthill Anarchy99
Dragonfly BlowFish
HybridBot
Indy Library
Masscan
PeoplePal PeoplePal
Thumbor Probethenet
WebSucker Seomoz
ZmEu YoudaoBot
ZoomBot gopher
arquivo.pt
clark-crawler
evc-batch

304
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
AIBOT 360Spider
80legs
ALittle Client ALittle Client
Abonti Abonti
Acunetix Aboundexbot
AfD-Verbotsverfahren
AhrefsBot AhrefsBot
AiHitBot
Aipbot Aipbot
Alexibot Alexibot
Apexoo AllSubmitter
Asterias Alligator
Attach Anarchie
Anarchy
Ankit
Anthill
Aspiegel
AwarioSmartBot
BBBike
BackStreet BackStreet
Badass BackWeb
BetaBot
Bigfoot Bigfoot
Black Hole Black Hole
BlackWidow Blackboard
Blow Brandprotect
Brandwatch Buddy
BunnySlippers BuiltBotTough
CATExplorador BuzzSumo
CCBot CCBot
Calculon
CazoodleBot
CensysInspect CensysInspect
CheeseBot
ChinaClaw
Chlooe Chlooe
Citoid
Cliqzbot Cliqzbot
Craftbot Cocolyzebot
Crawling at Home Project Cogentbot
Cosmos
CrazyWebCrawler CrazyWebCrawler
Curious
Custo
DBLBot DBLBot
DIIbot DIIbot
DTS Agent DSearch
DataCha0s DatabaseDriverMysqli
Demon Demon
Deusu Disco
Devil Discobot
DigitalPebble DnBCrawler-Analytics
Discoverybot DomainAppender
DnyzBot DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
Domains Project Domains Project
Dotbot
Dragonfly Dragonfly
ECCP/1.0
EMail Siphon
EMail Wolf EMail Wolf
EasyDL ExtractorPro
Evil
EyeNetIE EyeNetIE
FDM FDM
FlashGet
Flunky
Foobot Foobot
Freeuploader FyberSpider
Fuzz Gigabot
Fyrebot
G-i-g-a-b-o-t
GermCrawler
Go-Ahead-Got-It Go-Ahead-Got-It
GoZilla
GrabNet
Grabber
Grafula Grafula
GrapeFX GrapeFX
HMView HTMLparser
HTTP::Lite
Haansoft
HaosouSpider HaosouSpider
Harvest
Havij Havij
Heritrix
Humanlinks Humanlinks
HybridBot
IDBot
IRLbot IRLbot
Iblog Iblog
Id-search IlseBot
Image Fetch Indy Library
Image Sucker
InfoNaviRobot
InfoTekies InfoTekies
Intelliseek Iria
InterGET
Internet Ninja
Iskanie Iskanie
JamesBOT
Jbrofuzz Jbrofuzz
JetCar JetCar
Jorgee Jetty
Jyxobot Joomla
Kenjin Spider Kenjin Spider
Keyword Density
Kinza
Kozmosbot
LNSpiderguy
LWP::Simple LWP::Simple
Lanshanbot
Larbin Larbin
Leap
LeechFTP
LeechGet LeechGet
Lftp LexiBot
LibWeb
LieBaoFast LieBaoFast
LinkWalker Likse
Linkbot LinksManager
Linkdexbot LinqiaMetadataDownloaderBot
LinkextractorPro LinqiaRSSBot
LinqiaScrapeBot
Lipperhey Lipperhey
Lipperhey Spider Litemage_walker
Lmspider MTRobot
Ltx71 Magnet
MIDown tool Majestic SEO
MJ12bot
MQQBrowser
MSFrontPage
Majestic12 Majestic12
MarkWatch
Masscan Masscan
Mata Hari Mata Hari
MauiBot Minefield
MeanPath Bot Morfeus Fucking Scanner
Meanpathbot NICErsPRO
MegaIndex.ru
Microsoft Data Access
Mojeek
Mr.4x3
Musobot
Nameprotect Nameprotect
Navroad
Nessus Nessus
Net Vampire
NetLyzer
NetMechanic NetMechanic
NetZIP
Netcraft Netcraft
Netvibes Nettrack
Nikto Nikto
Nutch Nimbostratus
Ninja
Nuclei
OnCrawl OnCrawl
Openfind OutfoxBot
OrangeBot PHPCrawl
POE-Component-Client-HTTP
Page Analyzer
PageGrabber
PageScorer PageScorer
Panscient Pavuk
Petalbot
Pi-Monster
PictureFinder
Piepmatz
Pimonster Pimonster
PleaseCrawl PleaseCrawl
Pockey Pockey
ProPowerBot
Pump
PxBroker
PyCurl PyCurl
Quick-Crawler RankingBot2
RankActive RankurBot
RankingBot
Re-re
ReGet
RebelMouse
Recorder Recorder
RedesScrapy
RepoMonkey RepoMonkey
Ripper RocketCrawler
Rogerbot
SBIder SBIder
SEOstats SEOkicks
Scanbot SISTRIX
SalesIntelligent
ScoutJet ScoutJet
Scrapy ScreenerBot
ScrepyBot SearchmetricsBot
Searchestate
SemanticJuice
SemrushBot SemrushBot
SentiBot SeoSiteCheckup
Site Sucker SeobilityBot
Siteimprove Seomoz
Sitevigil Siphon
SlySearch SiteCheckerBotCrawler
Snake SiteSucker
Snapbot Sitebeam
Sottopop SmartDownload
Snoopy
Sogou web spider
SpaceBison SpaceBison
Spammen Spammen
Spanner Spbot
Spinn3r Spinn3r
Stripper SputnikBot
Sqlmap
Steeler
Sucker Sucker
SuperBot SuperBot
Surfbot Swiftbot
T0PHackTeam
T8Abot
Teleport Teleport
TeleportPro TeleportPro
Telesoft
Telesphoreo
The Intraformant The Intraformant
TheNomad Titan
Toweyabot Toata
Trendiction True_Robot
Trendictionbot Turnitin
TurnitinBot
Twice Twice
Typhoeus Upflow
URLy.Warning Vagabondo
VB Project
Vacuum
VelenPublicWebCrawler VelenPublicWebCrawler
Voltron
WBSearchBot
WEBDAV WEBDAV
WWW-Collector-E WWW-Collector-E
WWW::Mechanize WWW-Mechanize
WWWOFFLE
Wallpapers
WallpapersHD WallpapersHD
WeSEE
Web Collage Web Collage
Web Enhancer
Web Fetch
Web Fuck
Web Pix
Web Sauger
WebAuto
WebBandit WebBandit
WebCollage
WebCopier WebCopier
WebFetch
WebFuck
WebGo IS WebGo IS
WebLeacher WebLeacher
WebPix WebPix
WebSucker WebReaper
WebSauger
WebWhacker WebWhacker
Webalta Webalta
WebmasterWorldForumBot WebmasterWorldForumBot
Website Quester Website Quester
Whacker WebsiteExtractor
Webster
Whack
Who.is Bot Who.is Bot
Widow
WinHTTrack WinHTTrack
WiseGuys Robot Woobot
Wotbox Wotbox
Zauba
ZoomBot
ZoominfoBot
ZyBorg ZyBorg
adscanner clark-crawler
arquivo.pt
autoemailspider
backlink-check
cah.io.community
check1.exe
coccocbot-web coccocbot-web
cognitiveseo cognitiveseo
crawler.feedback crawler.feedback
crawler4j
demandbase-bot demandbase-bot
domainsproject.org eCatch
evc-batch
gopher gopher
heritrix instabid
magpie-crawler internetVista monitor
isitwp.com
lwp-trivial
mediawords
muhstik-scan muhstik-scan
plumanalytics
polaris version polaris version
probe-image-size
ripz
satoristudio.net satoristudio.net
scalaj-http
scan.lol scan.lol
seocompany.store
seostar seostar
serpstatbot sogouspider
sexsearcher spyfu
siteripz voyagerx.com
sp_auditbot webmeup-crawler
sysscan
trendiction.com
ubermetrics-technologies.com
webpros.com webpros.com
webprosbot x22Mozilla
x09Mozilla zauba.io
xpymep1.exe
zgrab

306
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com
404checker
404enemy 404enemy
80legs AIBOT
ADmantX
ALittle Client ALittle Client
AiHitBot ASPSeek
AllSubmitter Aboundexbot
Alligator Acunetix
AlphaBot AfD-Verbotsverfahren
Anarchie AhrefsBot
Anarchy Anarchy99
Ankit Aspiegel
Asterias AwarioRssBot
BDCbot AwarioSmartBot
BLEXBot BBBike
BackStreet BackDoorBot
BackWeb Badass
Bandit Barkrowler
BatchFTP BetaBot
BlackWidow Bigfoot
Blow Bitacle
Bolt BlowFish
Boardreader
BotALot BotALot
Brandprotect Brandprotect
Brandwatch Brandwatch
Buddy
BuiltBotTough
BuiltWith BuiltWith
Bullseye Bullseye
BuzzSumo BunnySlippers
CATExplorador
CODE87 CODE87
CazoodleBot CSHttp
Cegbfeieh CheTeam
CheeseBot
ChinaClaw ChinaClaw
Cocolyzebot Cloud mapping
Copyscape Cogentbot
Craftbot Collector
CrazyWebCrawler Crawling at Home Project
Crescent
Curious
Custo Custo
DBLBot DBLBot
DIIbot DataCha0s
DatabaseDriverMysqli
Deusu Deusu
Digincore Devil
Dirbuster
Discobot Discobot
Discoverybot
Dispatch Dispatch
DittoSpyder DittoSpyder
DomCopBot DnBCrawler-Analytics
DomainCrawler DnyzBot
DomainSigmaCrawler Download Wonder
Domains Project Dragonfly
Drip Drip
EMail Siphon
EMail Wolf EMail Wolf
EirGrabber
ExtLinksBot ExtLinksBot
Extractor
Extreme Picture Finder Extreme Picture Finder
Ezooms Ezooms
FemtosearchBot FHscan
Firefox/7.0 Firefox/7.0
FlashGet Flunky
Foobot Freeuploader
Fuzz Fuzz
FyberSpider
Fyrebot Fyrebot
G-i-g-a-b-o-t
GT::WWW GT::WWW
Genieo Getintent
GermCrawler Go-Ahead-Got-It
GetWeb
GoZilla GoZilla
Gotit GrabNet
GrapeFX Grafula
HEADMasterSEO HTTP::Lite
HTTrack HTTrack
Havij
Heritrix
Hloader Hloader
IDBTE4M Id-search
IRLbot
Image Fetch Image Fetch
Image Sucker Indy Library
InfoNaviRobot
InfoTekies InfoTekies
Intelliseek Intelliseek
InternetSeer InterGET
Iria
Iskanie Iskanie
IstellaBot
JOC Web Spider JOC Web Spider
Jbrofuzz Jbrofuzz
JikeSpider JikeSpider
JustView Jorgee
Jyxobot
Kenjin Spider Kenjin Spider
Keyword Density Kozmosbot
LNSpiderguy LNSpiderguy
Leap
LeechFTP LeechFTP
LibWeb
Libwhisker
LieBaoFast LieBaoFast
Likse Likse
LinkScan
LinkWalker LinkWalker
LinkpadBot LinkpadBot
Lipperhey LinqiaRSSBot
Lipperhey Spider
Lmspider
MIDown tool
MJ12bot MJ12bot
MSIECrawler Mail.RU_Bot
MTRobot
Magnet
Majestic-SEO Majestic-SEO
MarkMonitor
MarkWatch
Mass Downloader Mass Downloader
Masscan Masscan
Mb2345Browser MauiBot
MeanPath Bot MeanPath Bot
Meanpathbot Metauri
Mediatoolkitbot Microsoft URL Control
MegaIndex.ru Mister PiX
Microsoft Data Access Moblie Safari
Minefield
Morfeus Fucking Scanner Morfeus Fucking Scanner
Musobot Name Intelligence
Nameprotect NetAnts
Navroad NetLyzer
NearSite NetMechanic
Needle NetSpider
Netvibes Netcraft
Niki-bot
Nikto Nikto
Nimbostratus Nimbostratus
Nmap Ninja
Nuclei
Nutch
OnCrawl OnCrawl
OpenLinkProfiler
Openfind Openfind
OrangeBot OrangeBot
OutfoxBot
Page Analyzer
PageAnalyzer PageAnalyzer
Pandalytics PageGrabber
Pavuk PageScorer
Panscient
PeoplePal
Petalbot Petalbot
Picscout Pi-Monster
Picsearch Picsearch
Pimonster Pimonster
ProPowerBot
Probethenet Probethenet
Pump Pump
QueryN Metasearch
Quick-Crawler Quick-Crawler
RankFlex RankFlex
Re-re RankingBot2
Rankivabot
ReGet ReGet
RealDownload RealDownload
Reaper RebelMouse
RocketCrawler RepoMonkey
SEOkicks-Robot Rogerbot
SEOkicks
SEOlyticsCrawler SEOlyticsCrawler
SEOprofiler SISTRIX
Scanbot
ScoutJet ScoutJet
ScreenerBot Screaming
ScrepyBot ScrepyBot
Searchestate Searchestate
Semrush SearchmetricsBot
SentiBot
SeoSiteCheckup
Seomoz Seomoz
Shodan
Siphon Siphon
SiteCheckerBotCrawler
SiteExplorer SiteExplorer
Sitebeam SiteLockSpider
SiteSucker
Siteimprove
SlySearch SlySearch
Snoopy
SocialRankIOBot
Sogou web spider
Sottopop Sottopop
Spammen Spbot
Spinn3r
SputnikBot
Sqlmap
Sqlworm Sqlworm
Steeler
Sucker Sucker
Surfbot SuperBot
Suzuran SuperHTTP
Swiftbot Swiftbot
T8Abot Szukacz
T0PHackTeam
TeleportPro TeleportPro
TheNomad Telesphorep
TightTwatBot Thumbor
Titan
Tracemyfile
Trendiction
True_Robot
Turnitin Turnitin
TurnitinBot
Typhoeus Typhoeus
URLy.Warning URLy Warning
V-BOT V-BOT
VCI VB Project
Vacuum Vacuum
Vagabondo VidibleScraper
VeriCiteCrawler VoidEYE
WBSearchBot Voltron
WASALive-Bot
WPScan WPScan
WWW-Collector-E WWW::Mechanize
WWW-Mechanize
WWWOFFLE
Wallpapers
WeSEE WeSEE
Web Collage Web Collage
Web Fetch
Web Fuck
Web Sauger
Web Sucker Web Sucker
WebBandit WebAuto
WebCollage WebCollage
WebFetch
WebFuck WebFuck
WebSauger WebWhacker
WebStripper WebZIP
WebSucker WebsiteExtractor
Webalta
Webshag
Website Quester
WebsiteQuester WebsiteQuester
Whack Whatweb
WiseGuys Robot Widow
Wonderbot Wprecon
Xenu Xaldon WebSpider
Zade Xaldon_WebSpider
Zermelo adscanner
Zitebot
ZmEu
ZoomBot
ZoominfoBot
ZyBorg
archive.org_bot archive.org_bot
arquivo.pt arquivo-web-crawler
autoemailspider autoemailspider
backlink-check
cah.io.community
check1.exe check1.exe
coccocbot-web clark-crawler
com.plumanalytics cognitiveseo
crawler.feedback dataforseo.com
crawler4j evc-batch
demandbase-bot
domainsproject.org
facebookscraper
gopher gopher
heritrix
instabid instabid
internetVista monitor
isitwp.com isitwp.com
lwp-trivial lwp-trivial
magpie-crawler mediawords
polaris version muhstik-scan
ripz page scorer
scalaj-http satoristudio.net
scan.lol scan.lol
seobility seocompany.store
seoscanners seoscanners
serpstatbot serpstatbot
sitechecker.pro sitechecker.pro
sogouspider spyfu
trendiction.de sysscan
voyagerx.com tAkeOut
webgains-bot trendiction.com
webprosbot x22Mozilla
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
appartement-stumm.at bidbuy.co.kr
appmsr.org com-onlinesupport.host
buqyxa.rincian.info daisye.top
energydiet-info.ru downvids.net
kollesa.ru extremez.net
localmatchbook.com kirov.zrus.org
mnogolok.info krasula.pp.ua
pateaswing.com lrac.info
renhacklids.tk ragecash.com
rfserial.net reining.lovasszovetseg.hu

944
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
0c47f8422d3f.com 0n-line.tv
1008.su 101lesbian.xyz
123cha.com 10bet.com
1pamm.ru 1adult.com
1xbet4you.com
2015god.org 2015god.org
202ch.com 40cg.com
28n2gl3wfyb0.ru 5-steps-to-start-business.com
2itech.ru 72-news.com
2nt.ru 8jn.dating-app.ru
34.gs a96527gi.beget.tech
3weekdiet.com abtasty.com
4webmasters.org acc.eu.org
7zap.com activecampaign.dreamhosters.com
8gold.com adf.ly
aa8b68101d388c446389283820863176e7.com adman.gr
aanapa.ru adminshop.com
actices.com agahinameh.com
actionnooz.com alert-fjg.xyz
admeasures.com alienwheel.es
adultgalls.com all-streaming-media.com
advancedsoftwaresupport.com allesohnegirls.net
airlandsea.info
aktivator-windows10.blogspot.com
all4invest.ru
allboard.xobor.de
alloysteel.ru
alpharma.net
alt-servis.ru alt-servis.ru
amehdaily.com amanda-porn.ga
angel1777.kz ameblo.jp
animalia-life.club anaseracresar.tk
anniemation.com anmysite.com
aosexkontakte.net anonymousfox.co
apisecretsaucebi-a.akamaihd.net apartmentbay.ru
appearance-cool.com apelsinnik.website
apple.com-webbrowsing-security.review apiappenableinfo-a.akamaihd.net
artdestshop.eu apidiamondatanet-a.akamaihd.net
artparquet.ru arabgirls.adultgalls.com
arate.ru
arenda-yeisk.ru
artdeko.info
asmxsatadriverin.aircus.com
asophoto.com asophoto.com
atelielembrancaqueencanta.com.br avtochehli.by
auto.rusvile.lt avtolombard-krasnodar.ru
autogrep.ru ayeartoforget.com
avtocredit-legko.ru babosas.com
azbuka-mo.ru baditri.com
azlex.uz balans.shahterworld.org
babespcs.com beachfix.co
bagcionderlab.com belinka.com.ua
baixar-musicas-gratis.com bestdraws.com
balook.com bestfortraders.com
beachdriveblog.com bestwaystogetpaid.us
belsetirehafi.tk
beremenyashka.com
berlininsl.com
best-businessman.ru
bestprofits-there.com
betshuckclean.com betshuckclean.com
bezcmexa.ru bezlimitko.xyz
bibys.com
bimmerpost.com bimmerpost.com
bioscorp.ru bingo8888.com
blobar.org bitcoin-ua.top
bmsco.co bizlist.com.de
bolitgorlo.net bizru.info
bonzybuddy.com blog.koorg.ru
bosman.pluto.ro blpmovies.com
braindaily.xyz bluerobot.info
brendbutik.ru bnt-team.com
brewdom.ru bongiornos.info
bukleteg.xyz bonvillan.com
burger-imperia.com boobsimge.com
businesn.men bosefux.esy.es
business-suggested.tk brains2.biz
c-english.ru brothers-smaller.ru
candlewooddentalcentre.com.au browserprotecter.com
candy-glam-hp.com bryansk.zrus.org
canrioloadm.gq builtwith.com
canrole.com buyessayonline19.blogspot.ru
capsderfudd.tk buynorxx.com
carina-sy.de buypanicdisorderpill.com
carsplate.com bvlgariring.xyz
cartierreplica.top bytimedance.ru
celejihad.info calc-for-credit.ru
chatroulette.video carloans.com
checktext.org carstrends2015.com
chimiver.info cartierring.xyz
chinese-amezon.com cartierwatch.xyz
chuckguilford.com cayado.snn.gr
civilwartheater.com certifywebsite.win
cheapsergiorossi.online
cheapwebsitehoster.com
checkpagerank.net
chocolateslim-original.com
choice-credit.ru
cialgenisrx.com
cleanallspyware.com
cleanmypc.com cleanmypc.com
clickbanksites.info codysbbq.com
clickintext.net com-onlinesupport.host
clickprotects.com
clubfashionista.com
cocolyze.com
com-cleaner.systems
com-supportcenter.website
cometorussia.net
compiko.info compiko.info
compliance-alex.top
compliance-alex.xyz
compliance-alexa.xyz
compliance-brian.top
compliance-brian.xyz compliance-brian.xyz
compliance-jack.top compliance-don.top
comsysnet.com compliance-willy.top
copenergo.ru connectingsingles.com
copyrightclaims.org conocer-sanabria.com
cordstrap.cc cookingmeat.ru
creams.makeforum.eu cool-wedding.net
counciltally.com
counter.yadro.ru
crazyboost.pro
credit-respect.ru credit-respect.ru
credit.co.ua customboxes4less.com
creditmoney.com.ua customcatchcan.com
cubook.supernew.org d2jsp.org
customerpromos-a.akamaihd.net danashop.ru
d-black.bz day-news.info
danceuniverse.ru deluxedumps.com
dandiyabeats.in detki-opt.ru
daooda.com deutschehobbyhuren.net
datadepths.com dfiles.me
dawlenie.com digest-project.ru
dayibiao.com dikqlyremy.info
dealighted.com dimfour.com
dekoration.us dimkino.ru
den-noch24.ru documentbase.net
depositfiles-porn.ga dogbreedspicture.net
destinationrealestate.com dolohen.com
detmebel.su doreenblog.online
divku.ru dprtb.com
dojki-hd.com druzhininevgeniy63.blogspot.com
domik-derevne.ru dumpsmania24.com
dorratex.tn eandsgallery.com
downloaddy.net ekto.ee
drivotracker.com
druzhbany.ru
e-avon.ru
edge.sharethis.com
education-cz.ru
egovaleo.it
ekosmetyki.net.pl
eleimgo.pw
elektronischezigarette2.mex.tl
elektronischezigarettekaufen2.tumblr.com
elidelcream.weebly.com elidelcream.weebly.com
elitedollars.com embedle.com
elstal.com.pl en.home-task.com
espaceinventoristes.com epngo.bz
essaypro.com ero-advertising.com
essayservicewriting.org esnm.ru
etm-consult.de etotupo.ru
evaashop.ru evening-dating-club.info
evogarage.com exchanges-bet.com
exhibitionplus.eu excitacao.com
exlarseva.webblog.es extractorandburner.com
ezigarettekaufen2.mpbloggar.se extremal-blog.com
ezigarettenshop2.myblog.de eyes-on-you.ga
ezigaretteshop2.vefblog.net ez8motelseaworldsandiego.com
falcon-images.blogspot.com ezaz.info
familienzahnaerzte.com ezofest.sk
feorina.ru f07.de
fickenprivat.info f1nder.org
filesmonster.porn facecup.top
finansov.info farm26.ru
fishingwholesale.us fashion-mk.net
flashbannernow.com fast-wordpress-start.com
flashlarevista.com feel-planet.com
flavors.me fellowshipoftheminds.com
foodgid.net filesdatabase.net
footbalive.org finejewelryshop.ru
forex-indextop20.ru finemanteam.com
forexunion.net firesub.pl
formseo.com floridamhca.org
flowwwers.com
footballfarrago.com
forminecrafters.ru
forms-mtm.ru
formulaantiuban.com
forodvd.com forodvd.com
forum.tvmir.org foxydeal.com
free-floating-buttons.com freetangodownload.com
freenom.link fuck-paid-share-buttons.xyz
froggytube.com funcrushgames.com
fullfileaccess.com funnypica.com
fun-mobi.pl gameonasia.com
futbolkisales.ru gamerscorps.com
fyxabomiw.ru gate5.co.za
galeon.com gayxperience.com
galleryawesome.com ge0ip.net
gaming-journal.com geoads.com
generic-pills-online.com gerhardhealer.com
getmiro.com getlaid-xxxhookupdirect.com
getridofstretchmarks.org getmyads24.com
gigixo.com gifspics.com
girlgamerdaily.com glass-msk.ru
gk-atlant.info global-ics.co.za
glicol.kz globatur.ru
globalsurfari.com gnuetella.com
gloverid.site go2mike.ru
go2jump.org goforexvps.com
golmau.host.sk goldandcard.ru
graphicwe.org golden-praga.ru
growshop.es golfresa.lucania.se
grupografico-pilar.com.ar greendream.com.ua
gsou.cf gribokstop.com
gungamesz.com gsasearchengineranker.space
gwebtools.com gymi.name
hacktougroup.ru h2monline.com
handicapbathtubarea.com hasshe.com
hanwei.us hccoder.info
hao123.com hdfreeporno.net
hdimagegallery.net hdseriale.pl
healthcarestore.info healmytrauma.info
help.tpu.ru hentai-manga.porn
hjaoopoa.top herehloadibs.cf
howtowhitenteethfast.xyz highstairs-a.akamaihd.net
hydropump.su hit-kino.ru
hyhj.info hkdiiohi.skyrock.com
hyiphunter.org hollywoodactress.info
i4track.net hoterika.com
hoverboard360.es
hoverboardforsaledirect.com
hscsscotland.com
husky-shop.cz
ideawheel.com
iflycapetown.co.za iflycapetown.co.za
igadgetsworld.com igtools.club
igithab.com imagerydatabase.com
igrovyeavtomaty777.ru images-graphics-pics.com
ihc2015.info imagui.eu
images-free.net imediadesk.com
imitex-plus.ru img.wallpaperstock.net
incest-ru.com in-tandem.co
incomekey.net incomekey.net
indiakino.net infektsii.com
intellektmedia.at infoupdate.org
itservicesthatworkforyou.com inome.com.ua
ixora.pro intellego.info
japfm.com interior-stickers.ru
jerseywholesalechinabiz.com investpamm.ru
jetsli.de iopeninghours.co.uk
jo24news.com iphantom.com
jobgirl24.ru iqoption.com
kchaxton.com istock-mebel.ru
kichenaid.ru ivanstroi.ru
kino-filmi.com izhevsk.xrus.org
kinomaniatv.pl jjbabskoe.ru
kinoplen.ru johnrobertsoninc.com
konoplisemena.com kazka.ru
kountrylife.com keywordbasket.com
krasivoe-hd.com kidskunst.info
kskjco.club kino2018.club
ku6.com klikbonus.com
kupit-adenu.ru klumba55.ru
kuzinsp.ru kollekcioner.ru
ladsblue.com kongruan.com
law-check-nine.xyz kostenlos-sexvideos.com
lazymanyoga.com krasivoe-hd.net
letslowbefast.site ktotut.net
lidoradio.com kupiproday.com.ua
lincolntheatre.com kurwa.win
liver-chintai.org labelwater.se
locatellicorretor.com.br lacapilla.info
locationdesaison.com lafriore.ru
locksmith.jp lanadelreyfans.us
locustdesign.co lancheck.net
lomza.info laptoper.net
longgreen.info link.ac
lovasszovetseg.hu linkrr.com
lovi-moment.com.ua lion.bolegapakistan.com
lowephotos.info livefixer.com
ltvperf.com livesex-amateure.info
m.b00kmarks.com localflirtbuddies.com
makedo.ru lonfon.xyz
manifestation.betteroffers.review longlifelomilomi.info
marblestyle.ru lotto6888.com
marwer.info lsp-awak-perikanan.com
medkletki.ru lulea-auktionsverk.se
medtherapy.ru m1media.net
meduza-consult.ru magazin-pics.ru
meet-flirt-dating.com magento-crew.net
megagrabber.ru mapquestz.us
megahdporno.net marcowebonyodziezowe.pl
meget.co.za masserect.com
millionare.com matrixalchemy.com
minet.club mcnamaratech.com
minharevisao.com meccadumps.net
miniads.ca mediaoffers.click
mobilierland.com medicinacom.ru
mobot.site meetingrainstorm.bid
moinozhki.com megavolt.net.ua
mojpregled.com mesto-x.com
money-for-placing-articles.com mexicosleevegastrectomy.com
moscow.online-podarki.com micasainvest.com
moviezbonkerssk.cf microsoftportal.net
moxo.com minneapoliscopiers.com
mp3films.ru mirobuvi.com.ua
mirtorrent.net
mistr-x.org
mixx.com
mnogabukaff.net
mojowhois.com
monclercheap.xyz
monclercoats.xyz
moncleroutletonline.xyz
morepoweronmac.trade
mosdverka.ru
mowser.com
mp3s.club mp3s.club
muz-shoes.ru mrwhite.biz
na15.ru musflashtv.com
naperehresti.info music.utrolive.ru
narutonaruto.ru music7s.me
naughtyconnect.com mycouponizemac.com
navalwiki.info myseoconsultant.com
netanalytics.xyz nagdak.ru
nikitabuch.com nastydollars.com
naval.jislaaik.com
net-profits.xyz
newfilmsonline.ru
newstudio.tv
ningessaybe.me
nofreezingmac.click
nordvpn.com nordvpn.com
noscrapleftbehind.co normalegal.ru
npoet.ru o-o-6-o-o.com
o-o-8-o-o.ru obesidadealgarve.com
oakleyglassesonline.us okel.co
obiavo.com okonich.com.ua
oddamzadarmo.eu onlinewritingjobs17.blogspot.ru
offer.camp onstrapon.purplesphere.in
offtopic.biz openlibrary.org
okmedia.sk ops.picscout.com
onclkads.com
onlinadverts.com
onlinemegax.com
onload.pw
openstat.com
osagonline.ru osagonline.ru
ostroike.org ostrovtaxi.ru
ourtherapy.ru ozas.net
palvira.com.ua p-business.ru
paperwritingservice17.blogspot.ru pacificair.com
par-fallen.ga pammik.ru
partner-high.men parlament.biz
partnerads.men partner-print.men
pastaleads.com perfectpracticeweb.com
pattersonsweb.com pic2fly.com
pdns.download pijoto.net
piluli.info pinstake.com
plastjulgranar.se pinup-casino1.ru
playtap.us piter.xrus.org
pobeiranie.pl piulatte.cz
popander.mobi pizdeishn.com
popugaychiki.com pizdopletka.club
pornoelita.info pk-pomosch.ru
pornosexrolik.com ploenjitmedia.azurewebsites.net
pornoslive.net podshipniki-nsk.ru
pogodnyyeavarii.gq
polcin.de
pornhive.org
pornoblood.com
pornofeuer.com
pornofiljmi.com
pornoinn.com
pornosee.info
pornosok.ru pornosok.ru
portside.cc privacyassistant.net
pospr.waw.pl privatefx-in.ru
postclass.com prokotov.com
pozdravleniya-c.ru prombudpostach.com.ua
pozvonim.com pron.pro
preventheadacheguide.info prostitutki-tyumeni.org
pricheski-video.com pufip.com
printdirectforless.com pwwysydh.com
privhosting.com quality-traffic.com
proekt-gaz.ru rankingchart.de
prom23.ru raschtextil.com.ua
rangjued.com razamicroelectronics.com
rapidokbrain.com reckonstat.info
rentaremotecomputer.com reklama-i-rabota.ru
responsinator.com remontgruzovik.ru
revistaindustria.com rennlist.com
risparmiocasa.bz.it replicaclub.ru
robotixix.com reversing.cc
rogervivierforsale.com roma-kukareku.livejournal.com
romhacking.ru rvtv.ru
rospromtest.ru sabaapress.com
rubanners.com sakhboard.ru
s.lollypopgaming.com sale-japan.com
safe-app.net salut-camp.ru
samo-soznanie.ru saneyes.com
savingsslider-a.akamaihd.net santasgift.ml
scopich.com sardinie.us
se.bnt-team.com satoristudio.net
seadragonherbery.com scanner-irvin.top
search-error.com scanner-julia.top
searchengineranker.email scansafe.net
sdelatmebel.ru
sdrescher.net
searchimpression.com searchimpression.com
secret.xn--oogle-wmc.com semprofile.com
seksotur.ru semrush.com
seocdvig.ru seokicks.de
seomarketings.online sergiorossistore.online
seozoom.it sexpornotales.com
servingnotice.com sextracker.de
sex-watch.com sfj-ror.no
sexphoto.site shemale-sex.net
sexpornotales.net sherlock.se
shakhtar-doneck.ru shiksabd.com
shisha-swag.de shopfishing.com.ua
signoredom.com sibdevice.ru
simple-share-buttons.com simon3.ru
simplesite.com sims-sims.ru
sinel.info siteripz.net
site-auditor.online skyway24.ru
siteaero.com slavia.info
skanninge.se small-game.com
slimcdn.com snts.shell-pmr.ru
smsactivator.ru social-buttons.xyz
snegozaderzhatel.ru social-s-hhh.xyz
social-vestnik.ru sodexo.com
socialmonkee.com softtor.com
socialsignals24.com sonnikforme.ru
soda.media souvenirua.com
songplanet.ru spammen.de
soviet-portal.do.am staff.prairiesouth.ca
space4update.pw startufa.ru
spbchampionat.ru styro.ru
spravka-medosmotr.ru supernew.org
squidoo.com
sr-rekneskap.no
srecorder.com
startwp.org
steame.ru
streetfire.net
studentguide.ru
sukarame.net
superlist.biz
superstats.com
supervesti.ru
t3chtonic.com
tanieaukcje.com.pl
taqplayer.info
tasteidea.com tasteidea.com
tedxrj.com
telefonsexi.com telefonsexi.com
thewebsitetemplate.info tengohydar.tk
tizanidineonline.blogspot.com theallgirlarcade.com
tizanidinepain.blogspot.com thebluffs.com
tizanidinerecreationaluse.blogspot.com thedownloadfreeonlinegames.blogspot.com
tizanidinewithdrawalsymptoms.blogspot.com thepokertimer.com
tomatis.gospartner.com thewomenlife.com
tonerbox.kz time-japan.ru
top-karkas.ru tizanidine4mgtablets.blogspot.com
torontoplumbinggroup.com tizanidine4mguses.blogspot.com
torrents.life tizanidinedosageforsleep.blogspot.com
touchmods.fr tizanidineforopiatewithdrawal.blogspot.com
track-rankings.online topbestgames.com
tracksz.co topcasinoratings.ru
tracxn.com tracfone.com
trackmedia101.com
traffic-club.info
traffic2cash.org
trafficreceiver.club trafficreceiver.club
truck-rece.life tri-slona.org
tryrating.com tripper.de
tt-ipd.info twelvevisionspartyofcolorado.com
tube8.com u-cheats.ru
tubeline.biz udayavani.com
tvand.ru udsgame.online
typimga.pw ukrtextbook.com
uac.net.au
umityangin.net umityangin.net
unacittaconte.org unece.org
urccvfmc.bloger.index.hr univerfiles.com
upstore.me
urlopener.com
usacasino.com
userequip.com userequip.com
usswrite.com varikozdok.ru
uvozdeckych.info vepad.com
v-doc.co
vchulkah.net
versaut.xxx-cam.webcam versaut.xxx-cam.webcam
vgoloveboli.net vezuviy.su
via-gra.webstarts.com viagra.pp.ua
viktoria-center.ru viagroid.ru
videooko.weebly.com
vinylvault.co.uk vinylvault.co.uk
vip-dom.in vipromoffers.com
vizag.kharkov.ua
vkonche.com
vkontarkte.com
volgograd.xrus.org
voloomoney.com
vriel.batcave.net
vsesubwaysurfers.com
vzlomfb.com vzlomfb.com
vzlomsn.org watracker.net
w-journal.ru
w3javascript.com
warezkeeper.com
wcb.su wcb.su
we-are-gamers.com
webads.co.nz
webadvance.club webadvance.club
webscouter.net webpromotion.ae
website-analytics.online website-speed-checker.site
website-speed-check.site webtherapy.ru
wejdz-tu.pl whatistizanidine2mg.blogspot.com
westsextube.com whatsappdownload10.com
wetgames.ru williamrobsonproperty.com
wheelchairliftsarea.com womama.ru
white-truck.life ww1943.ru
win-spy.com www.pinnacle-bets.com
workle.website www.souvenirua.com
wowas31.ucoz.ru x-diesel.biz
wrz0iuebwhp5fg.freeddns.com x-stars.ru
wsgames.ru xmronta.com
wstroika.ru xn-----6kcaabbafhu7cskl7akvongwpo7hvjj.xn--p1ai
wygraj-teraz.com xn-----6kcaacnblni5c5bicdpcmficy.xn--p1ai
x-lime.net xn----8sbdbjgb1ap7a9c4czbh.xn--p1acf
x-true.info xn----btbdvdh4aafrfciljm6k.xn--p1ai
xblog.in xn--80aaagvmjabrs1aoc9luc.xn--p1ai
xcc24.pl xn--80atua3d.xn--p1ai
xgames-04.com xn--b1ag5cfn.xn--p1ai
xlovecam.com xpresscare.ru
xmlinde.com xsfetish.org
xn----8sbarihbihxpxqgaf0g1e.xn--80adxhks xxxdatinglocal.us
xn--80aeahghtf8ac5i.xn--p1ai xxxkaz.org
xn--80ak6aa92e.com xxxtubesafari.com
xn--b1adccaf1bzj.xn--p1ai yadro.ru
xzlive.com youbrainboost.asia
yellowads.men youjizz.com
yellowproxy.net yourads.website
yginekologa.com zaim-pod-zalog-krasnodar.ru
youdao.com zawyna.ua
yougame.biz zeroredirect7.com
yourmovies.pl zigarettenonlinekaufen2.bloog.pl
youtubologia.it zkjovpdgxivg.ga
z-master.ru znaturaloriginal.com
za-fun-offer.com zooggames.com
zaidia.xhost.ro
zaimhelp.ru
zeleznobeton.ru
zeroredirect.com
zhacker.net
zionstar.net
zixizop.net.ru
znakomstva-moskva77.ru
znaniyapolza.ru
zoogdisany.com
zrus.org
zx6.ru

488
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
1000n1.ru 00webcams.com
4istoshop.com 03p.info
777-club.ru 2drittel.de
a.frcls.fr 34.gs
addtoadd.men 4kplayer.pl
adf.ly 5i2.net
adultfullhd.com a.pr-cy.ru
advancedcleaner.com aaa38852e886ac4af1a3cff9b47cab6272.com
agapovdg.ru abiente.ru
alessandraleone.com above.com
alif-ba-ta.com adexprts.com
all4invest.info adinterax.com
amateurlivechat.org adnotbad.com
apparel-offer.com adultwebhosting.info
architecturebest.com adzerg.com
arenanews.com.ua afora.ru
arquapetrarca.info aktivator-windows10.blogspot.com
asophoto.com amehdaily.com
auto-zapchasti.org android4fun.org
azartniy-bonus.com animenime.ru
b2b-lounge.com anonymous-redirect.com
becuo.com ap.senai.br
bedandbreakfast.com apartmentbay.ru
begalka.xyz artefakct.com
big-boards.info audiofree.ru
bimatoprost-careprost.com.ua autodo.info
bjetjt.com autoseo-traffic.com
blogqpot.com autoua.top
bpro1.top avon-novosib.ru
canrioloadm.gq azarttoto.com
cardul.ru babs.com.ua
carfax.com.ua bbtec.net
casinorewards.info beauby.ru
chocolateslim-en-espana.com berdasovivan.ru
clicksor.net bestofupload.info
clksite.com bestplacetobuyeriacta.jw.lt
club-lukojl.ru bobinoz.com
club-samodelkin.ru brendbutik.ru
coldfilm.ru buqyxa.rincian.info
compliance-andrew.top camdolls.com
compliance-brian.xyz candy-glam-hp.com
compliance-checker-7.info carapics.com
connectionstrenth.com cartierlovebraceletreplica.xyz
crirussian.ru casite-513049.cloudaccess.net
cubook.supernew.org cfacarrosserie74.com
dear-diary.ru cheapwebsitehoster.com
decoratinghomes.ga chinese-amezon.com
decorationspcs.com chlooe.com
dekorkeramik.ru choice-credit.ru
delayreferat.ru christianlouboutinsaleoutletonline.us
deutschehobbyhuren.net claytransformations.info
diffbot.com clips.ua.ac.be
digilander.libero.it clknsee.com
dikx.gdn coaching-netz.info
diy-handmade-ideas.com compliance-andrew.xyz
dneprsvet.com.ua compliance-elena.top
domashneeruporno.com compliance-ivan.xyz
doublepimp.com concealthyself.com
dynainbox.com data.vtc.pw
easyukraine.com ddospanel.com
elektrischezigarette1.onsugar.com dirtpics.pw
elvel.com.ua docsportal.net
enginebay.ru doktoronline.no
enternet.ee downloadeer.net
expolicenciaslatam.co doxysexy.com
ezigarettenkaufen2.dreamwidth.org doyouknowtheword-flummox.ml
falopicm.pw drunkmoms.net
fapgon.com e-commerce-seo.com
favorcosmetics.com edwinkonijn.com.au
fickenbumsen.net erotikchat-24.com
fikasound.tk etm-consult.de
filmfanatic.com europages.com.ru
financehint.eu evening-dating-club.info
finejewelryshop.ru excitacao.com
firstdrugmall.ru exdocsfiles.com
forum-engineering.ru eywords-monitoring-your-success.com
forum.tvmir.org farmprofi.net
foto-basa.com fast-wordpress-start.com
franch.info figuringmoneyout.com
frankofficial.ru filefilter.weebly.com
frivgame250.com film-one.ru
gamblingnerd.com filmgo.ru
garciniaxt.us filmidivx.com
gazobeton-p.com.ua financepoints.eu
goatse.ru floridahuntingfishingadventures.com
goodnightjournal.com forsex.info
googlepositions.com freesitetest.com
gopixdatabase.com funponsel.com
green-tea.tv gambarxkata.co
grizzlysgrill.com girlsfuckdick.com
hackers-crackers.tk gotcher.us
healbio.ru gratuitxblcodes.com
hetmanship.xyz growmyfunds.ca
homegardenlova.com helpmymacfaster.trade
hoverboard360.se histock.info
hoverboardforsaledirect.com hkladys.com
hunthillfarmtrust.org housediz.com
iamsport.org
idegenvezeto.eu
ifmo.ru
ihc2015.info ihc2015.info
iklysha.ml incolors.club
impisr.ru infazavr.ru
indiasourcemart.in infogame.name
iron-age.info informatiecentro.be
kamalsinha.com inspiring-desperate.tk
keywordsdoctor.com interesnie-faktu.ru
komp-pomosch.ru iqoption.pro
kredit-pod-zalog-krasnodar.ru izhevsk.xrus.org
ktotut.net jackpotchances.com
lesbian.xyz javrip.net
letslowbefast.today javxxx18.com
limads.men joessmogtestonly.com
liumimgx.pw kamen-e.ru
localflirtbuddies.com kerei.ru
loginduepunti.it ketrzyn.pl
losangeles-ads.com kmgamex.cf
m0r0zk0-krava.ru kometars.xyz
mabdoola.blogspot.com krasivoe-hd.net
macdamaged.space ladsblue.com
magento-crew.net lenvred.org
maju.bluesalt.co lifebyleese.com
meetingrainstorm.bid lignofix.ua
megakino.net lineavabit.it
member-quiz.com longlifelomilomi.info
memberty.com luchshie-topcasino.ru
merryhouse.co.uk luckyshop.net.ua
miniads.ca luxup.ru
moscow.xrus.org maranbrinfo.com.br
moskva.nodup.ru marcoislandvacations.net
moviemail-online.co.uk marmitaco.cat
movies-in-theaters.net masturbate.co.uk
mrmoneymustache.com matrixalchemy.com
musflashtv.com meble-bogart.info
mycouponizemac.com medicine-4u.org
myindospace.com meds-online24.com
myseoconsultant.com medtherapy.ru
neobux-bg.info metalonly.info
netcheckcdn.xyz mielec.pl
newstraveller.ru minegam.com
nextbackgroundcheck.gq monclerjacketsoutlet.win
nsatc.net montredemarque.nl
nudo.ca moscow-clining.ru
obiavo.by mosrif.ru
mp3films.ru
mrpornogratis.xxx
musezone.su
my-floor.in.ua
mypets.by
myyour.eu
nbsproject.ru
net-profits.xyz
netpics.org
nfljerseyscheapchinabiz.com
niroo.info
no-fuel.org
noscrapleftbehind.co
o-o-8-o-o.com
o00.in
offtopic.biz offtopic.biz
okuos.com oktube.ru
omgtnc.com online-sbank.ru
onblastblog.online online-x.ru
onlineserialy.ru onlinefilmz.net
onlineslotmaschine.com onlineporno.site
origin-my.ru origin-my.ru
outshop.ru ostroike.org
ownshop.cf ownshop.cf
paidonlinesites.com pacificair.com
panamaforbeginners.com perm.xrus.org
paphoselectricianandplumber.com pinapchik.com
parajumpersstore.online pix-hd.com
partnersafe.men platinumdeals.gr
phormchina.com play-movie.pl
php-market.ru popunder.ru
pinup-casino1.ru potoideas.us
piulatte.cz promoheads.com
pixgood.com
pl.aasoldes.fr
porndairy.in
pornoinn.com
pravoholding.ru
priora-2.com
professionalwritingservices15.blogspot.ru
prombudpostach.com.ua
propranolol40mg.blogspot.com
prostitutki-kharkova.org prostitutki-kharkova.org
purplesphere.in prostitutki-rostova.org
purplestats.com pskcijdc.bloger.index.hr
qualitymarketzone.com quelle.ru
quickbuck.com quick-offer.com
raisedseo.com rangjued.com
rapidokbrain.com rarbg.to
registrationdomainsite.com redirectingat.com
relax.ru rock-cafe.info
remorcicomerciale.ro rockma.se
retreatia.com
russintv.fr
rutor.vip rutor.vip
s-iwantyou.com sanatorrii.ru
saveindex.xyz sashagreyblog.ga
searchinquire.com savetubevideo.com
semp.net scanmarine.info
seorankinglinks.us scanner-walt.top
serialsway.ucoz.ru searchlock.com
shymkent.xkaz.org seasonvar.ru
si-unique.com seo-smm.kz
sideeffectsoftizanidine.blogspot.com sevenstars7.com
siteimprove.com shell-pmr.ru
siteripz.net sibvitr.ru
sledstvie-veli.net sientalyric.co
sport7777.net silvermature.net
stairliftstrue.com sim-service.net
statoutlook.info sms2x2.ru
stuff-about-money.com socialbookmarksubmission.org
sunflowerdrawingpaintings.blogspot.com sophang8.com
superstarfloraluk.com sosdepotdebilan.com
svolze.com sprttrack.com
symbaloo.com stockspmb.info
t-bygg.com strana-krasoty.ru
telsis.com studentguide.ru
the-usa-games.blogspot.com suchenindeutschland.com
tracksz.co telefonsexkostenlos.tk
twu.com.ua thegamerznetwork.com
u555u.info thewebsitetemplate.info
ublaze.ru time-japan.ru
uchebavchehii.ru timeallnews.ru
ultimateclassicrock.com tizanidineonline.blogspot.com
um-razum.ru tizanidineusedfor.blogspot.com
uytmaster.ru tomatis.gospartner.com
valkiria-tk.ru torrentred.games
valuado.com totu.info
videochat.tv.br trichizobswiv.agddns.net
visionwell.com.cn uhod-za-sobakoj.ru
vita.com.hr unlimitdocs.net
viteonlusarezzo.it v720hd.ru
vladimir.xrus.org vchulkax.com
wapsite.me venta-prom.ru
website-speed-checker.site via-energy-order.com
website-speed-up.site videochat.world
westen-z.life videokrik.net
westvilletowingservices.co.za vkonche.com
x-diesel.com voditeltrezviy.ru
xaylapdiendanang.com web.cvut.cz
xblog.in webtherapy.ru
xingzi-vision.com wowas31.ucoz.ru
xn----ctbigni3aj4h.xn--p1ai wsgames.ru
xn--d1aifoe0a9a.top wygraj-teraz.com
xxxnatelefon.ru x-stars.ru
xz618.com xmronta.com
yachts-cruise.info xn----7sbaphztdjeboffeiof6c.xn--p1ai
yougame.biz xn----7sbbagbq7bd5aheftfllo4m.xn--p1ai
yourporngay.com xn--80aagddcgkbcqbad7amllnejg6dya.xn--p1ai
yousense.info xn--80ahvj9e.xn--p1ai
yugk.net xportvusbdriver8i.snack.ws
yun56.co xvideospornoru.com
yunque.pluto.ro ynymnwbm.bloger.index.hr
zahvat.ru yourporn.com
zakazvzloma.com yoursearch.me
zarabotok--doma.ru youtube-downloader.savetubevideo.com
zeroredirect5.com zigarettenonl.canalblog.com
zixizop.net.ru znakomstva-moskva77.ru
zootoplist.com zojirushi-products.ru
zs2vm.top zytpirwai.net
zzlgxh.com

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.130 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

8
dev-tools/tmp/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.12.2754 ### Version: V4.2021.12.2755
### Updated: Fri Dec 3 22:01:37 UTC 2021 ### Updated: Sat Dec 4 07:57:54 UTC 2021
### Bad Bot Count: 630 ### Bad Bot Count: 631
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -173,6 +173,8 @@ User-agent: ChinaClaw
Disallow:/ Disallow:/
User-agent: Chlooe User-agent: Chlooe
Disallow:/ Disallow:/
User-agent: Citoid
Disallow:/
User-agent: Claritybot User-agent: Claritybot
Disallow:/ Disallow:/
User-agent: Cliqzbot User-agent: Cliqzbot

8
robots.txt/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.12.2754 ### Version: V4.2021.12.2755
### Updated: Fri Dec 3 22:01:37 UTC 2021 ### Updated: Sat Dec 4 07:57:54 UTC 2021
### Bad Bot Count: 630 ### Bad Bot Count: 631
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -173,6 +173,8 @@ User-agent: ChinaClaw
Disallow:/ Disallow:/
User-agent: Chlooe User-agent: Chlooe
Disallow:/ Disallow:/
User-agent: Citoid
Disallow:/
User-agent: Claritybot User-agent: Claritybot
Disallow:/ Disallow:/
User-agent: Cliqzbot User-agent: Cliqzbot