vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-04 19:50:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.03.2955 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-03-03 09:39:51 +00:00
parent 76cae899ed
commit bb4111b122
51 changed files with 95690 additions and 95667 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.03.2954 #### Version: V4.2022.03.2955
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [636](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.03.2954 #### Version: V4.2022.03.2955
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [636](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -13,9 +13,9 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.03.2954 #### Version: V4.2022.03.2955
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [636](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

11820
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_generator_lists/bad-user-agents.list
View file

@ -1,4 +1,3 @@
Keybot\ Translation-Search-Machine
01h4x.com 01h4x.com
360Spider 360Spider
404checker 404checker
@ -221,6 +220,7 @@ Jorgee
JustView JustView
Jyxobot Jyxobot
Kenjin\ Spider Kenjin\ Spider
Keybot\ Translation-Search-Machine
Keyword\ Density Keyword\ Density
Kinza Kinza
Kozmosbot Kozmosbot

4
_google_analytics_ghost_spam/README.md
View file

@ -3,9 +3,9 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.03.2954 #### Version: V4.2022.03.2955
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [636](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

11827
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11827
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

11827
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

1
dev-tools/_robots_input/robots-input.txt
View file

@ -220,6 +220,7 @@ Jorgee
JustView JustView
Jyxobot Jyxobot
Kenjin Spider Kenjin Spider
Keybot Translation-Search-Machine
Keyword Density Keyword Density
Kinza Kinza
Kozmosbot Kozmosbot

11827
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 1; 10.1.0.14 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 0; 10.1.0.14 0;
127.0.0.1 0; 127.0.0.1 0;

11827
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 1; 10.1.0.14 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 0; 10.1.0.14 0;
127.0.0.1 0; 127.0.0.1 0;

11827
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 1; 10.1.0.14 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -281,6 +281,7 @@
"~*(?:\b)JustView(?:\b|)" 0; "~*(?:\b)JustView(?:\b|)" 0;
"~*(?:\b)Jyxobot(?:\b|)" 0; "~*(?:\b)Jyxobot(?:\b|)" 0;
"~*(?:\b)Kenjin\ Spider(?:\b|)" 0; "~*(?:\b)Kenjin\ Spider(?:\b|)" 0;
"~*(?:\b)Keybot\ Translation-Search-Machine(?:\b|)" 0;
"~*(?:\b)Keyword\ Density(?:\b|)" 0; "~*(?:\b)Keyword\ Density(?:\b|)" 0;
"~*(?:\b)Kinza(?:\b|)" 0; "~*(?:\b)Kinza(?:\b|)" 0;
"~*(?:\b)Kozmosbot(?:\b|)" 0; "~*(?:\b)Kozmosbot(?:\b|)" 0;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 0; 10.1.0.14 0;
127.0.0.1 0; 127.0.0.1 0;

11827
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2954 2955

11827
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

11827
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
40.77.69.97 0; 20.43.231.35 0;

11827
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 1; 10.1.0.14 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 0; 10.1.0.14 0;
127.0.0.1 0; 127.0.0.1 0;

11827
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 1; 10.1.0.14 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 0; 10.1.0.14 0;
127.0.0.1 0; 127.0.0.1 0;

11827
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 1; 10.1.0.14 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -281,6 +281,7 @@
"~*(?:\b)JustView(?:\b|)" 0; "~*(?:\b)JustView(?:\b|)" 0;
"~*(?:\b)Jyxobot(?:\b|)" 0; "~*(?:\b)Jyxobot(?:\b|)" 0;
"~*(?:\b)Kenjin\ Spider(?:\b|)" 0; "~*(?:\b)Kenjin\ Spider(?:\b|)" 0;
"~*(?:\b)Keybot\ Translation-Search-Machine(?:\b|)" 0;
"~*(?:\b)Keyword\ Density(?:\b|)" 0; "~*(?:\b)Keyword\ Density(?:\b|)" 0;
"~*(?:\b)Kinza(?:\b|)" 0; "~*(?:\b)Kinza(?:\b|)" 0;
"~*(?:\b)Kozmosbot(?:\b|)" 0; "~*(?:\b)Kozmosbot(?:\b|)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 0; 10.1.0.14 0;
127.0.0.1 0; 127.0.0.1 0;

11827
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
40.77.69.97 0; 20.43.231.35 0;

11827
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
40.77.69.97 0; 20.43.231.35 0;

11827
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 1; 10.1.0.14 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -281,6 +281,7 @@
"~*(?:\b)JustView(?:\b|)" 0; "~*(?:\b)JustView(?:\b|)" 0;
"~*(?:\b)Jyxobot(?:\b|)" 0; "~*(?:\b)Jyxobot(?:\b|)" 0;
"~*(?:\b)Kenjin\ Spider(?:\b|)" 0; "~*(?:\b)Kenjin\ Spider(?:\b|)" 0;
"~*(?:\b)Keybot\ Translation-Search-Machine(?:\b|)" 0;
"~*(?:\b)Keyword\ Density(?:\b|)" 0; "~*(?:\b)Keyword\ Density(?:\b|)" 0;
"~*(?:\b)Kinza(?:\b|)" 0; "~*(?:\b)Kinza(?:\b|)" 0;
"~*(?:\b)Kozmosbot(?:\b|)" 0; "~*(?:\b)Kozmosbot(?:\b|)" 0;

18
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
ADmantX ADmantX
GermCrawler HEADMasterSEO
LinkScan Iblog
MicroMessenger NearSite
SiteExplorer Panscient
T8Abot Psbot
WebImageCollector Turnitin
YoudaoBot Web Sucker
oBot pcBrowser
sogouspider siteripz

292
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com
404enemy
ASPSeek ASPSeek
Aboundexbot Abonti
AfD-Verbotsverfahren Aboundex
AhrefsBot AhrefsBot
AiHitBot
Aipbot
Alexibot Alexibot
AllSubmitter AllSubmitter
AlphaBot
Anarchy Anarchy
Ankit Ankit
Apexoo Atomseobot
Aspiegel BLEXBot
AwarioSmartBot BackDoorBot
BDFetch
Backlink-Ceck Backlink-Ceck
Bandit Badass
Barkrowler
BatchFTP BatchFTP
Battleztar Bazinga
Bigfoot Bigfoot
Bitacle Bitacle
Black Hole
BlackWidow BlackWidow
BlowFish Blackboard
BotALot Bolt
Buck Buck
Buddy BunnySlippers
CazoodleBot BuzzSumo
Cegbfeieh CODE87
CheTeam
CheeseBot
Citoid
Cloud mapping
Cogentbot
Collector
Copyscape Copyscape
Craftbot Cosmos
CrazyWebCrawler Crescent
CrunchBot Custo
DBLBot
DIIbot DIIbot
DataCha0s DSearch
DatabaseDriverMysqli DatabaseDriverMysqli
Devil Demon
Digincore
DigitalPebble
Dirbuster
Disco Disco
Discobot
Discoverybot Discoverybot
DnBCrawler-Analytics
DnyzBot DnyzBot
DomainStatsBot DomCopBot
DomainAppender
DomainSigmaCrawler
Domains Project
Dotbot Dotbot
Dragonfly ECCP/1.0
EMail Wolf
EasyDL
EroCrawler EroCrawler
Extractor Evil
Fimap
FlashGet
Flunky Flunky
Freeuploader Foobot
FrontPage
G-i-g-a-b-o-t
GT::WWW
GalaxyBot GalaxyBot
Genieo Genieo
GermCrawler GetRight
Getintent
Gigabot
Go!Zilla
Gotit Gotit
GrabNet
Grabber Grabber
GrapeFX Grafula
GrapeshotCrawler GrapeshotCrawler
GridBot GridBot
HMView HTTrack
HTTP::Lite Havij
Haansoft Hloader
Heritrix IDBTE4M
Humanlinks
IDBot IDBot
IRLbot
Iblog
Image Fetch Image Fetch
Image Sucker Image Sucker
InterGET Indy Library
Internet Ninja Iria
InternetSeer
IstellaBot IstellaBot
JOC Web Spider JennyBot
Jbrofuzz
JetCar JetCar
Jorgee Jorgee
Jyxobot JustView
Kenjin Spider Kenjin Spider
LWP::Simple Keyword Density
Larbin Kinza
LieBaoFast Kozmosbot
LNSpiderguy
Lanshanbot
LeechGet
Likse
Linkbot Linkbot
Linkdexbot
LinkextractorPro LinkextractorPro
LinqiaScrapeBot LinqiaMetadataDownloaderBot
Lipperhey Lmspider
Litemage_walker
Ltx71 Ltx71
MTRobot MIIxpc
Mail.RU_Bot Magnet
MarkMonitor MarkMonitor
MarkWatch MarkWatch
Mass Downloader
Masscan Masscan
Mata Hari Mata Hari
Mb2345Browser Mediatoolkitbot
MegaIndex.ru MegaIndex.ru
Metauri Metauri
MicroMessenger MicroMessenger
Mojolicious
MolokaiBot MolokaiBot
Morfeus Fucking Scanner
Musobot Musobot
Navroad NICErsPRO
NPbot
Name Intelligence
Nameprotect
NearSite NearSite
Nessus
Net Vampire
NetAnts NetAnts
NetLyzer NetMechanic
NetSpider Nettrack
NetZIP
Netvibes
NextGenSearchBot
Niki-bot Niki-bot
Nikto
NimbleCrawler NimbleCrawler
Ninja
Not Not
Nuclei
Offline Navigator Offline Navigator
OnCrawl OutclicksBot
Openfind PHPCrawl
Openvas
OrangeBot
PECL::HTTP
POE-Component-Client-HTTP
Page Analyzer Page Analyzer
PageAnalyzer PageThing.com
Petalbot Pavuk
Pi-Monster PeoplePal
Picscout Picscout
Picsearch PictureFinder
Pimonster
Pixray Pixray
PleaseCrawl
Pockey Pockey
ProPowerBot ProPowerBot
Probethenet
Psbot Psbot
Pump Pu_iN
PyCurl PxBroker
QueryN Metasearch Quick-Crawler
RankingBot RankingBot
RankingBot2 RankingBot2
Rankivabot
Re-re Re-re
ReGet
Reaper Reaper
RedesScrapy RebelMouse
Recorder
RocketCrawler RocketCrawler
Rogerbot SEOkicks
SBIder SEOstats
SMTBot SMTBot
SalesIntelligent
ScanAlert
ScoutJet
Screaming
ScreenerBot ScreenerBot
ScrepyBot
Searchestate
SemanticJuice SemanticJuice
SentiBot SemrushBot
SeobilityBot Seomoz
Siphon Siphon
SiteExplorer Site Sucker
SiteCheckerBotCrawler
SiteSnagger
Sitebeam Sitebeam
Siteimprove Siteimprove
SlySearch Snapbot
SmartDownload SocialRankIOBot
Sociscraper Sociscraper
Sogou web spider Sosospider
Sottopop SpankBot
Spanner
Spinn3r
Sqlmap Sqlmap
Sqlworm Sqworm
Stripper Sucuri
Sucker SuperBot
Surfbot Surfbot
Swiftbot Swiftbot
T8Abot Szukacz
TeleportPro Teleport
The Intraformant TightTwatBot
Titan
Tracemyfile
Trendiction Trendiction
True_Robot True_Robot
Turingos
Turnitin Turnitin
TwengaBot
UnisterBot UnisterBot
V-BOT
VB Project
Vacuum Vacuum
Vagabondo Vagabondo
Virusdie VeriCiteCrawler
VoidEYE VoidEYE
Voil
Voltron
WBSearchBot WBSearchBot
WEBDAV WPScan
WWW-Mechanize WWW-Mechanize
WWWOFFLE
Wallpapers Wallpapers
Wallpapers/3.0 Wallpapers/3.0
WallpapersHD Web Auto
WeSEE
Web Enhancer Web Enhancer
Web Fetch Web Fuck
Web Pix Web Pix
Web Sauger Web Sucker
WebAuto WebBandit
WebCollage WebCollage
WebEnhancer WebEnhancer
WebFetch
WebFuck
WebPix
WebSauger
WebStripper WebStripper
WebSucker WebZIP
WebWhacker Webalta
WebsiteQuester WebsiteQuester
Whack
Whacker
Widow Widow
WinHTTrack
WiseGuys Robot WiseGuys Robot
Wonderbot Woobot
Wotbox Wotbox
Wprecon
Xaldon WebSpider
Zade
Zauba
Zermelo
Zeus Zeus
Zitebot ZmEu
ZoominfoBot ZumBot
ZyBorg
adscanner adscanner
arquivo-web-crawler arquivo-web-crawler
backlink-check clark-crawler
check1.exe coccocbot
domainsproject.org cognitiveseo
facebookscraper crawl.sogou.com
crawler.feedback
evc-batch
internetVista monitor
isitwp.com isitwp.com
magpie-crawler lwp-trivial
muhstik-scan muhstik-scan
netEstate NE Crawler oBot
pcBrowser pcBrowser
plumanalytics
polaris version
probe-image-size probe-image-size
ripz satoristudio.net
s1z.ru seoscanners
seocompany.store serpstatbot
sexsearcher
siteripz siteripz
sogouspider
spyfu
sysscan sysscan
trendiction.com tAkeOut
webmeup-crawler ubermetrics-technologies.com
webpros.com webprosbot
x09Mozilla
x22Mozilla x22Mozilla
zauba.io xpymep1.exe
zgrab zgrab

310
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -2,249 +2,249 @@
360Spider 360Spider
404checker 404checker
80legs 80legs
ADmantX ASPSeek
Aboundex
Acunetix
AfD-Verbotsverfahren AfD-Verbotsverfahren
Alexibot Aipbot
Anarchy AllSubmitter
Ankit AlphaBot
Apexoo Apexoo
Asterias
Atomseobot
BDCbot BDCbot
BDFetch Bandit
BackWeb BatchFTP
Badass Battleztar Bazinga
Bigfoot Bigfoot
Bitacle Bitacle
Blackboard BlackWidow
Blow BlowFish
Boardreader
Brandprotect
Buddy
BuiltWith
Bullseye Bullseye
BunnySlippers
BuzzSumo BuzzSumo
Calculon CCBot
Cegbfeieh CODE87
CazoodleBot
CensysInspect CensysInspect
CheeseBot Collector
ChinaClaw Copier
Citoid
Claritybot
Cliqzbot
Cloud mapping
CopyRightCheck CopyRightCheck
Copyscape Curious
Craftbot
Crescent
Custo
DBLBot DBLBot
DSearch DSearch
DTS Agent Digincore
Demon DigitalPebble
Devil Dirbuster
Discobot Discobot
Discoverybot
DittoSpyder
DnBCrawler-Analytics
DnyzBot DnyzBot
DomCopBot DomCopBot
DomainSigmaCrawler DomainCrawler
DomainStatsBot
Dragonfly
Drip
ECCP/1.0 ECCP/1.0
EasyDL EasyDL
Ebingbong Evil
Ecxi Exabot
ExtLinksBot
ExtractorPro ExtractorPro
FHscan
FemtosearchBot FemtosearchBot
Fimap Fimap
Flunky Fuzz
G-i-g-a-b-o-t FyberSpider
GT::WWW Fyrebot
GalaxyBot GalaxyBot
Genieo Genieo
Getintent GermCrawler
Go-Ahead-Got-It GetWeb
GrabNet
Grabber Grabber
Grafula Grafula
GrapeFX
HEADMasterSEO
HTMLparser HTMLparser
HTTP::Lite HTTP::Lite
HaosouSpider HaosouSpider
Harvest
Havij Havij
Heritrix Heritrix
Hloader HybridBot
IDBTE4M
Id-search Id-search
IlseBot
Image Sucker Image Sucker
Indy Library IndeedBot
InfoNaviRobot
InfoTekies InfoTekies
InterGET
Internet Ninja Internet Ninja
Iria Iria
JOC Web Spider IstellaBot
Jbrofuzz JamesBOT
JennyBot
Jetty Jetty
JikeSpider JikeSpider
Jorgee Joomla
JustView
Kenjin Spider
Keybot Translation-Search-Machine
Keyword Density Keyword Density
Kinza LNSpiderguy
Lanshanbot Larbin
LieBaoFast LeechFTP
LibWeb
LinkWalker
LinkextractorPro
LinksManager
LinqiaRSSBot
LinqiaScrapeBot LinqiaScrapeBot
Lipperhey
Ltx71 Ltx71
MIDown tool MIIxpc
MSIECrawler MJ12bot
Mag-Net MQQBrowser
Mail.RU_Bot
Majestic-SEO
MarkMonitor MarkMonitor
Mass Downloader
Masscan
Mata Hari
MauiBot
Mb2345Browser Mb2345Browser
Microsoft URL Control MeanPath Bot
Mister PiX Metauri
Morfeus Fucking Scanner MicroMessenger
Microsoft Data Access
Mojeek
Mojolicious
MolokaiBot
Mozlila Mozlila
Msrabot Musobot
Nameprotect NICErsPRO
Navroad NPbot
NetLyzer Name Intelligence
NetSpider NearSite
Needle
Nessus
Net Vampire
NetAnts
NetMechanic
NetZIP NetZIP
NextGenSearchBot Nettrack
Nibbler Netvibes
Nimbostratus Niki-bot
NimbleCrawler
Nmap Nmap
Not Octopus
OnCrawl
OpenVAS OpenVAS
Openfind
Openvas Openvas
OrangeBot OrangeBot
OutclicksBot OrangeSpider
PECL::HTTP Page Analyzer
PHPCrawl
PageAnalyzer
PageScorer
Pandalytics Pandalytics
Panscient PeoplePal
Papa Foto Pi-Monster
Pavuk
Petalbot
Picsearch
PictureFinder PictureFinder
Piepmatz Piepmatz
Pimonster ProWebWalker
Probethenet
Psbot Psbot
Quick-Crawler PyCurl
RSSingBot QueryN Metasearch
RankActiveLinkBot
RankingBot RankingBot
RankingBot2 RankingBot2
Rankivabot RankurBot
Re-re RealDownload
RebelMouse Reaper
Recorder RedesScrapy
RepoMonkey Ripper
SBIder RocketCrawler
Rogerbot
SEOkicks
SEOkicks-Robot
SEOlyticsCrawler SEOlyticsCrawler
SEOstats SEOprofiler
ScoutJet SMTBot
Scrapy
Screaming Screaming
ScrepyBot
Searchestate
Seekport Seekport
SemrushBot SemrushBot
SentiBot SeoSiteCheckup
Seomoz
Siphon
Site Sucker Site Sucker
SiteExplorer SiteCheckerBotCrawler
SiteLockSpider
SiteSucker
Sitebeam
Siteimprove Siteimprove
SlySearch SmartDownload
Snake
Sociscraper Sociscraper
Sogou web spider Sogou web spider
Sosospider
Sottopop Sottopop
SpaceBison SpaceBison
Spammen Spammen
Spinn3r Spanner
Steeler SputnikBot
Surfbot Sqlworm
SuperBot
SuperHTTP
SurveyBot SurveyBot
Suzuran T0PHackTeam
Swiftbot
T8Abot T8Abot
Teleport
TeleportPro TeleportPro
Telesoft
Telesphoreo Telesphoreo
Telesphorep
Thumbor Thumbor
Toweyabot Toweyabot
Tracemyfile Tracemyfile
Trendiction Trendictionbot
Turingos
TurnitinBot
TwengaBot TwengaBot
Typhoeus Twice
URLy.Warning URLy Warning
UnisterBot Upflow
V-BOT VB Project
VelenPublicWebCrawler VCI
VidibleScraper
VoidEYE VoidEYE
Voil
Voltron Voltron
Wallpapers WASALive-Bot
WWW-Collector-E
WWWOFFLE
Wallpapers/3.0 Wallpapers/3.0
WeSEE WeSEE
Web Auto
Web Collage Web Collage
Web Fuck Web Fetch
Web Sucker Web Sucker
WebAuto WebAuto
WebBandit WebReaper
WebStripper WebStripper
Webalta
WebmasterWorldForumBot WebmasterWorldForumBot
Webshag
Website Quester Website Quester
WebsiteExtractor Whack
Webster Whacker
Whatweb
Who.is Bot
WinHTTrack
WiseGuys Robot WiseGuys Robot
Wonderbot
Woobot
Wprecon Wprecon
Xaldon_WebSpider Xenu
YoudaoBot
Zade
Zauba
ZoominfoBot ZoominfoBot
ZumBot ZyBorg
adscanner
archive.org_bot archive.org_bot
backlink-check backlink-check
cah.io.community
check1.exe check1.exe
crawler.feedback clark-crawler
cognitiveseo
dataforseo.com
domainsproject.org
evc-batch evc-batch
heritrix facebookscraper
instabid
internetVista monitor internetVista monitor
meanpathbot ips-agent
muhstik-scan isitwp.com
lwp-trivial
magpie-crawler
netEstate NE Crawler netEstate NE Crawler
pcBrowser ripz
probe-image-size s1z.ru
satoristudio.net scan.lol
seobility seoscanners
serpstatbot sitechecker.pro
sexsearcher siteripz
sysscan trendiction.com
tAkeOut
trendiction.de
ubermetrics-technologies.com
voyagerx.com voyagerx.com
webgains-bot webprosbot
webpros.com zgrab
xpymep1.exe
zauba.io

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
dignitasdata.se amateurlivechat.org
hannasolution.ru anal-acrobats.hol.es
haveinc.xyz baixar-musicas-gratis.com
jjbabskoe.ru czat.wp.pl
laexotic.com eorogo.top
lolnex.us esfchat.tk
sexspornotub.com kinosed.net
wholesalejerseyscheapjerseys.us.com topappspro.com
www.arenda-yeisk.ru wygraj-teraz.com
xmlinde.com youandcredit.ru

930
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
007angels.com 1001watch.com.ua
0lovespells0.blogspot.com 101lesbian.xyz
1008.su 10bet.com
21h2o.com 123locker.com
3th.co.in 1kdailyprofit.me
4istoshop.com 24-ak.ru
5elementov.ru 3rbseyes.com
a.pr-cy.ru 4kpics.rocks
aanapa.ru 5000-cotydzien.com
abcdefh.xyz 7a2rnuey1tw9ar.ru
abclauncher.com 7wind.ru
absolute-s.ru 7xc4n.com
academia-nsk.org 7zap.com
adanih.com 8gold.com
add-add.men 9icmzvn6.website
adnotbad.com DomainStatsBot
aaa38852e886ac4af1a3cff9b47cab6272.com
abcdeg.xyz
above.com
activecampaign.dreamhosters.com
adexprts.com
adloads.com
adrunnr.com adrunnr.com
adtiger.tk adsloads.com
adult3dgames.com
advancedsoftwaresupport.com
advertisingtag.net
agapovdg.ru
agreda.pluto.ro agreda.pluto.ro
allboard.xobor.de ahhjf.com
allmarketsnewdayli.gdn ahmedabadwebs.com
alpharma.net alf-img.com
alpinism.ru aludecor.info
amanda-porn.ga amazingninja.com
amazon-adsystem.com amigobulls.com
amehdaily.com ananumous.ru
ample-awards-today.us android-vsem.org
anaksma.info api.stathat.com
apidigidocketnet-a.akamaihd.net apifasterlightin-a.akamaihd.net
apishops.ru appartement-stumm.at
apiwebwebgetcom-a.akamaihd.net apple.com-webbrowsing-security.science
arcadepages.com appsaurus.com
aruplighting.com apptester.tk
ausergrubhof.info arate.ru
automobile-spec.com arendakvartir.kz
avon-novosib.ru asseenontv.ru
avramstroy.ru atmovs.com
avtochehli.by autoblogger24.info
avtolombard-krasnodar.ru autolombard-krasnodar.ru
ayodhya.co autowebmarket.com.ua
azartniy-bonus.com availit.weebly.com
azbuka-mo.ru avek.ru
aviav.ru
avirasecureserver.com
axbocz.net
azarttoto.com
azbaseimages.net
babosas.com
backuperwebcam.weebly.com backuperwebcam.weebly.com
baksman.com
balla-boo.se
banan.tv banan.tv
barnfurnituremart.com bank.uz
basisches-wasser.net batanga.net
battlecarnival.su belsetirehafi.tk
bavsac.com best-seo-solution.com
bdsmgalls.net bestbuy.ca
beachpics.com bestfortraders.com
bedandbreakfast.com bestwaystogetpaid.us
belinka.com.ua betshuckclean.com
benchmarkcommunications.co.uk bimmerpost.com
berlininsl.com bio.trade-jp.net
bestempresas.es bkgr.se
beta.hotkeys.com blackhatworth.com
beyan.host.sk blogporn.in
bgviagrachrx.com bouda.kvalitne.cz
bingoporno.com brainboostingsupplements.org
bitcoin-ua.top braindaily.xyz
bitporno.sx
biznesschat.net
bjorkbacken.nu
blog.koreadaily.com
bmw.afora.ru
brainboosting.club
brandov.ru brandov.ru
brendbutik.ru brianhenry.co
brokergid.ru bristolhotel.com.ua
bumskontakte.org bumskontakte.org
burkesales.com burger-imperia.com
buscarfoto.com
buttons-for-website.com buttons-for-website.com
buybest1.biz buy-cheap-online.info
buynorxx.com c-english.ru
buytizanidineonline.blogspot.com californianews.cf
cakesplus.com.au callcafe.info
callmd5map.com candelluxsklep.pl
calstaterealty.us capacitacionyaprendizaje.com
cam-kontakte.org capsderfudd.tk
cand.jp cartechnic.ru
carabela.com.do casite-513049.cloudaccess.net
carstrends2015.com caveavins.fr
cartierbracelet.xyz cdncash.com
cartierjusteunclou.xyz cdnnetwok.xyz
cayado.snn.gr cenoval.ru
cdn.walkme.com cherrypointplace.ca
centre-indigo.org.ua chidporn.com
cerev.info chocolateslim-original.com
cheapjerseysap.com christianlouboutinreplica.pw
cheapmoncler.pw cialisndbrx.com
check-this-out-now.online ciceros.co
chiblackhawks-jerseys.com cindymatches.com
chocoslim.pro clarithromycin500mg.com
ci.ua cleanersoft.com
cialgenisrx.com clipartnew.com
clicksotrk.com cocyq.inwtrade.com
consorzioilmosaico.org compliance-alexa.xyz
converse.ddsoldes.fr compliance-don.xyz
coolwallpapers-hd.com compliance-fred.xyz
countercrazy.com compliance-ivan.xyz
covetnica.com compliance-jessica.top
cryptoswap.biz cool-wedding.net
csgo4.win coprofam.org
damasarenai.info crazyboost.pro
dame-ns.kz cyclobenzaprinevstizanidine.blogspot.com
danashop.ru dailystrength.org
data1.scopich.com daretodonate.co
dataloading.net darkbooks.org
davidovic.info dbmkfhqk.bloger.index.hr
dellalimov.com dealighted.com
dev-seo.blog decorazilla.com
devochki.top detoxmed24.ru
diegolopezcastan.com deutschehobbyhuren.net
dinkolove.ya.ru dfiles.me
directrev.com digital-sale.su
discovertreasure-a.akamaihd.net digitalfaq.com
disruptingdinnerparties.com dikqlyremy.info
distver.ru dimaka.info
djonwatch.ru dipstar.org
dmmspy.com discountliv.com
dnepropetrovsk.mistr-x.org diy-handmade-ideas.com
dokfilms.net dojki365.online
domain-submit.info domain-submit.info
domain.webkeyit.com
domainanalyzing.xyz domainanalyzing.xyz
domainsphoto.com domaincdn.xyz
dosugrostov.site domainsatcost.com
draniki.org downloadeer.net
druzhininevgeniy63.blogspot.com dyt.net
dumpsmania24.com
dumuelave.xyz
easycommerce.cf
ebonyporn.site ebonyporn.site
ebooknovel.club electro-prom.com
edwinkonijn.com.au elektrozigarette1.dreamwidth.org
egovaleo.it
elektrischezigaretten2.beeplog.com
elektronischezi.livejournal.com
elektronischezigarettekaufen2.tumblr.com
elektrozigaretten1.webs.com
elitesportsadvisor.com elitesportsadvisor.com
en.home-task.com eloconcream.blogspot.com
eonpal.com elvenar.com
eu-cookie-law.blogspot.com erot.co
eyessurgery.ru european-torches.ru
ezigaretteshop2.vefblog.net europeanwatches.ru
ezrvrentals.com exchanges-bet.com
faracontrol.ir expertnaya-ocenka.ru
fatfasts-4tmz.com eyemagination.com
fba-mexico.com ezigaretteshop.webs.com
fbdownloader.com ezigaretteshop2.mywapblog.com
felizporno.com f0815.de
fetroshok.ru facecup.top
fickluder69.com falllow.gq
filerockstar298.weebly.com fanpagerobot.com
firma-legion.ru fanrto.com
fishtauto.ru fealq.com
flipper.top fearcrow.com
flooringinstallation-edmonton.com fellowshipoftheminds.com
floridahuntingfishingadventures.com filesclub.net
flprog.com filkhbr.com
fortevidyoze.net finansov.info
forum20.smailik.org findpics.pw
foxinsocks.ru findsexguide.com
frankofficial.ru finejewelryshop.ru
free-fb-traffic.com firesub.pl
free-laptop-reward.com florida-tourism.net
free-traffic.xyz formularz-konkurs.tk
freevpn.space forum.tvmir.org
freewebs.com foto-telok.net
freza-sverlo.ru foxweber.com
frameimage.org
freegamesplay.online
freeseedsonline.com
frighteningremain.cf frighteningremain.cf
frivgame250.com frustrated-favorable.gq
futbolkisales.ru gambarkataku.co
fxtips.ru gameprimary.com
fyxabomiw.ru gdcentre.ru
gameonasia.com ge0ip.net
gaypornmovie.net geile-lelly.eu
gazporno.com get-free-social-traffic.com
gearcraft.us girlsatgames.ru
gepezz.info go2jump.org
gettpromos.com gogps.me
gigixo.com googlsucks.com
globatur.ru gorabagrata.ru
gotcher.us guidefs.ru
gotomontenegro.net guruofcasino.com
gov.yanao.ru gwebtools.com
gowreckdiving.com gwhwpxbw.bloger.index.hr
gratuitbaise.com hamptonoaks.ca
gratuitxblcodes.com
gstatey.net
guiadeserraazul.com
handicapvantoday.com
hasfun.com hasfun.com
hchha.com hdimagelib.com
hdhc.ru hdwallpapers-free.com
heroz.fr hebr.myddns-flir.com
hledejvshopech.cz help.tpu.ru
homegardenlova.com herokuapp.com
homik.pw hetmanship.xyz
hoporno.com heygidday.biz
hotloans.ru high-speed1.net
housekuba.org hotcore.info
hoverboard360.de houston-vikings.com
htmlcorner.com hoverboard360.nl
hubbble.com hvd-store.com
iboss.com hyhj.info
hyiphunter.org
iccornacircri.cf
ico.re
ideibiznesa2015.ru ideibiznesa2015.ru
imagerydatabase.com imadedinner.net
imgarit.pw images-graphics-pics.com
imgata.com img.wallpaperstock.net
immigrational.info imitex-plus.ru
incep.imagine-ex.co impisr.ru
indetiske.ya.ru import-sales.com
incest-ru.com
indiasourcemart.in indiasourcemart.in
infazavr.ru infokonkurs.ru
int.search.mywebsearch.com insider.pro
internetproviderstucson.com iopeninghours.co.uk
iskalko.ru ipchicken.com
iphantom.com
ipostroika.ru
irkutsk.online-podarki.com
isistaylorporn.info
istizanidineanarcoticdrug.blogspot.com
istmira.ru istmira.ru
iwantmyfreecash.com ivanovo.zrus.org
izoll.ru ivoiretechnocom.ci
jacago.com jackwolfskinoutlet.online
javstock.com james13prix.info
je7.us jamiembrown.com
johnrobertsoninc.com javcoast.com
jwss.cc jbl-charge.info
katushka.net kabbalah-red-bracelets.com
kerei.ru kandidos.com
kerwinandcariza.com kazrent.com
keywordglobal.co.za khadastoafarde.tk
keywordhouse.com kndxbkdx.bloger.index.hr
kiinomaniak.pl kovrenok.ru
kino-rf.ru ks1234.com
kliksaya.com ladsup.com
klitimg.pw lawyersinfo.org
konturkrasoty.ru lazymanyoga.com
kostenlos-sexvideos.com leadn.pl
kovesszucs.atw.hu lechenie-gemorroya.com
kozhniebolezni.com likesuccess.com
krasivoe-hd.com linkbux.ru
ktotut.net linksharingt.com
kung-fu-ru.com
kurwa.win
kuzinsp.ru
laudit.ru
laxdrills.com
ldrtrack.com
le-clos-des-alouettes.com
leboard.ru
lexixxx.com
lifeinsurancekb.com
light.ifmo.ru
linkwithin.com
live-sexcam.tk
liver-chintai.org liver-chintai.org
localmatchbook.com ljusihus.se
lombardfinder.ru llastbuy.ru
lonfon.xyz lockerz.com
lottospring.com longgreen.info
lsex.xyz louboutinbooties.xyz
lutherstable.org lsp-awak-perikanan.com
mabdoola.blogspot.com luxmagazine.cf
machicon-ueno.info m0r0zk0-krava.ru
macrotek.ru macnewtech.com
maglid.ru macresource.co.uk
mantramusic.ru magiadinamibia.blogspot.com
marketingtechniques.info mainhp.net
masserect.com manifestation.betteroffers.review
meccadumps.net marblestyle.ru
marwer.info
master-muznachas.ru
mastershef.club
masturbate.co.uk
mebeldekor.com.ua
mediafresh.online mediafresh.online
menhealed.net medicine-4u.org
merryhouse.co.uk meds-online24.com
mesto-x.com memberty.com
metascephe.com michellblog.online
michaelkorsoutletstore.net mikozstop.com
mirmedinfo.ru militarysale.pro
mixx.com miniature.io
mirobuvi.com.ua
mixtapetorrent.com
mjchamonix.org
mlf.hordo.win
mmstat.com
mobilemedia.md mobilemedia.md
mobioffertrck.com moesexy.com
mobot.site mojowhois.com
modenamebel.ru
moinozhki.com
mompussy.net
monarhs.info
moncleroutletonline.xyz moncleroutletonline.xyz
moneymaster.ru moskva.nodup.ru
moroccosurfadventures.com mountainstream.ms
moscow.xrus.org music.utrolive.ru
moviemail-online.co.uk musicpro.monster
musicvidz.ru muznachas-service.ru
musirc.com mypornfree.ru
mwtpludn.review myyour.eu
mycouponizemac.com na-telefon.biz
mysex21.com narutonaruto.ru
nalogovyykodeks.ru naturtreenspicerx.pw
namenectar.com neks.info
needtosellmyhousefast.com nicovideo.jp
newstaffadsshop.club nikitsyringedrivelg.pen.io
nikhilbahl.com notaria-desalas.com
niroo.info novgorod.xrus.org
no-fuel.org nrjmobile.fr
nodup.ru nudejapan.net
obiavo.in o-o-8-o-o.com
oecnhs.info obsessionphrases.com
office-windows.ru offer.camp
okout.ru officedocuments.net
onlainbesplatno.ru okel.co
onlywoman.org oneminutesite.it
openfrost.net onko-24.com
optibuymac.com onlinadverts.com
oz-offers.com onlineslotmaschine.com
paccohichetoti.ml onlythegames.com
palocco.it ontargetseo.us
paydayloanslocal.com oren-cats.ru
pdns.download ownshop.win
petrovka-online.com owohho.com
oxford-book.com.ua
paighambot.com
pammik.ru
paparazzistudios.com.au
partner-trustworthy.men
partnersafe.men
pcgroup.com.uy
pdfprof.com
pensplan.com
photochki.com
phuketscreen.com phuketscreen.com
piccshare.com pivka.xyz
picturesify.com pladform.ru
picurams.pw platesauto.com
pijoto.net playfortuna-play.ru
piuminiita.com poems.com.ua
pl.aasoldes.fr pokemongooo.ml
pl.id-forex.com pontiacsolstice.info
pogodnyyeavarii.gq
popads.net popads.net
popcash.net popcash.net
popup-jdh.xyz porno-dojki.net
porno-home365.com pornodojd.ru
positive2b.ru pornohd1080.online
pourvous.info pornohub.me
pozdravleniya-c.ru pornolook.net
portnoff.od.ua
potolokelekor.ru
predmety.in.ua
printdirectforless.com
printie.com printie.com
privatbank46.ru pro-tec.kz
prizestohandle.club profit-opportunity.com
prlog.ru profitfx.online
prodess.ru putevka24.ru
produkto.net quickchange.cc
prombudpostach.com.ua
promgirldresses.xyz
prospekt-st.ru
pskcijdc.bloger.index.hr
psvita.ru
purchasepillsnorx.com
quangcaons.com
quelle.ru
quick-offer.com
quickloanbank.com quickloanbank.com
rabotaetvse.ru qwesa.ru
ratemodels.net r-e-f-e-r-e-r.com
realmonte.net radiodigital.co
realting-moscow.ru randki-sex.com
recinziireale.com ranksonic.org
rednise.com realnye-otzyvy.info
reimageplus.com redhotfreebies.co.uk
reklamuss.ru registratciya-v-moskve.ru
remontbiz.ru registrationdomainsite.com
repeatlogo.co.uk registry-cleaner.net
retailwith.com reklama-i-rabota.ru
revistaindustria.com renecaovilla.online
reyel1985.webnode.fr renecaovillasale.online
rogervivierforsale.com rethinkwasteni.info
ruclicks.com ribieiendom.no
rumamba.com robot-forex.biz
rus-pornuha.com rossanasaavedra.net
rusexy.xyz rucrypt.com
russintv.fr ruscoininvest.company
s8-nowy-wygraj.comli.com ruspdd.com
sack.net sakhboard.ru
safe-app.net saletool.ru
samara.rosfirm.ru samara.rosfirm.ru
samolet.fr saratov.xrus.org
saveindex.xyz sarf3omlat.com
sax-sex.com scanner-walter.top
sbt-aqua.ru schalke04fc.info
scanspyware.net score-ads.men
schoolfiles.net
scrnet.biz.ua
sdsjweb.com
se-welding.ru
searchinquire.com
securityallianceservices.com
semalt.com
semprofile.com semprofile.com
seorankinglinks.xyz seoanalyses.com
sexobzor.info seolab.top
sexpornotales.com serpstat.com
sexsearch.com sexsaoy.com
sexspornotub.com shohanb.com
share-buttons-for-free.com shoppingmiracles.co.uk
sherlock.se
shitmovs.com
shoesonlinebuy.xyz
shymkent.xkaz.org
si-unique.com si-unique.com
sierraapps.com signal03.ru
sildenafilcitratemed.com simple-share-buttons.com
sisiynas.ru simplepooltips.com
site-speed-check.site site-auditor.online
siteonomy.com site-speed-checker.site
skachat-besplatno-obrazcy.ru slayerlife.com
smartadserver.com slujbauborki.ru
smichovbike.cz smallseotools.com
solnplast.ru snsdeainavi.info
sonnikforme.ru soft-program.com
sortthemesitesby.com sogimlecal.tk
spy-app.info sotechco.co
statistici.ro spbchampionat.ru
stockspmb.info specialfinanceoffers.com
streha-metalko.si sports-supplements.us
stretchmate.net ssconstruction.co
stuff-about-money.com star61.de
svbur.ru stockquotes.wooeb.com
syvertsen-da.no streetfooduncovered.com
tacbelarus.ru stroiminsk.com
tacbibirfa.tk stroyhelp-dv.ru
tampabaywatch.org studiokamyk.com.pl
tandvardshuset.net supermama.top
tds-advert005.info t-bygg.com
tecnoteakviareggio.it tastyfoodideas.com
teguh.info tech4master.com
templates.franklinfire.co tedxrj.com
the-trader.net tehngr.ru
thepantonpractice.co.uk thaisamkok.com
thefotosgratis.eu
theporndude.com
theprofitsmaker.net theprofitsmaker.net
thiegs.reco.ws tizanidinedruginteractions.blogspot.com
threecolumnblogger.com tizanidinevssoma.blogspot.com
titelhelden.eu top1-seo-service.com
tizanidineandgabapentin.blogspot.com top10-way.com
tizanidinedosageforsleep.blogspot.com top250movies.ru
tizanidinemusclerelaxant.blogspot.com topsy.com
toloka.hurtom.com
tootoo.to
top-karkas.ru
topbestgames.com
topmira.com
topvidos.ru
torrent-to-magnet.com torrent-to-magnet.com
torrentgamer.net toursmaps.com
traxdom.ru trkdf.com
tri-slona.org truemfilelj.gq
trk-4.net tubeoffline.com
tt-ipd.info turist-strani.ru
turvgori.ru turvgori.ru
tx41tclega.ru u17795.netangels.ru
udsgame.online uac.net.au
uk-zheu20.ru ublaze.ru
unionmarkt.de ukrup.com
upupa.net unlimitdocs.net
url-img.link uroki.net
urlcut.ru uzungil.com
userequip.com v720hd.ru
versaut.xxx-cam.webcam valoresito.com
via-gra.webstarts.com velobikestock.com
videochat.cafe viagengrarx.com
vielporno.net video-chat.in
vintontech.info videositename.com
voltrknc1.com vip-parfumeria.ru
vpnmouse.com virus-respirators.com
vriel.batcave.net vksaver-all.ru
vysigy.su vladhistory.com
vltai.com
vmnmvzsmn.over-blog.com
vod.com.ua
vonradio.com
vroze.com
vvon.co.uk
vzlomtw.com
w3data.co
wallpapers-best.com
wallpapersdesk.info
wallpapersist.com wallpapersist.com
wapsite.me webstatsdomain.org
webcamtalk.net wemarketing.se
webenlace.com.ar wholesalecheapjerseysfree.com
welcomeauto.ru
wheelchairliftsarea.com
whereverdesperate.gq
wholesalejerseyscheapjerseys.us.com
wholinkstome.com wholinkstome.com
wnoz.de wpthemedetector.co.uk
wordpress-crew.net ww1943.ru
worldoffiles.ru www.pinnacle-bets.com
wrz0iuebwhp5fg.freeddns.com
ws.ampower.me
wwwadultcheck.com wwwadultcheck.com
wygraj-skiny.win wygraj-teraz.com
xep.info wzgyyq.com
xjlottery.com x-lime.com
xn--1-8sbcpb0bdm8k6a.xn--p1ai x-porno.video
xn--e1aggki3c.xn--80adxhks xblog.in
xportvusbdriver8i.snack.ws xedserver.com
xtubeporno.net xerox-douglas.cf
xvideosbay.com xkaz.org
xvideospornoru.com xn------7cdbapdecfd4ak1bn0amjffj7afu3y.xn--p1ai
xxxkaz.org xn--l1aengat.xn--p1ai
xrus.org
yachts-cruise.info
yatrk.xyz yatrk.xyz
youbloodyripper.com yginekologa.com
youresponsive.com yoshkarola.zrus.org
yourporngay.com yourmovies.pl
zagadki.in.ua yoursearch.me
zarenica.net yourtemplatefinder.com
zeroredirect12.com yunque.pluto.ro
zeroredirect8.com yurgorod.ru
zhacker.net za-music.mymobiplanet.com
zirondelli.it zeikopay.com
zolotoy-lis.ru zhuravlev.info
zrus.org zoogdiesney.com
zynax.ua zoompegs.com

488
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
1-free-share-buttons.com 101flag.ru
108shot.com 28n2gl3wfyb0.ru
1hwy.com 3rbseyes.com
9med.net 4kplayer.pl
aanapa.ru 4pp13.com
abcdeg.xyz 8558l.jobs.net
adamoads.com aac52006ec82a24e08b665f4db2b5013f7.com
addblueoff.com.ua allbizne.co.ua
adsref.men anapa-inns.ru
adultwebhosting.info
albuteroli.com
allmarketsnewdayli.gdn
allsilver925.co.il
alphacarolinas.com
androids-store.com androids-store.com
apifasterlightin-a.akamaihd.net api.stathat.com
apimossnetinfo-a.akamaihd.net apiportalmorecom-a.akamaihd.net
b-styles.xyz appfixing.space
backuperwebcam.weebly.com arewater.com
bang-hotties.com atleticpharm.org
benchmarkcommunications.co.uk autolombard-krasnodar.ru
berdasovivan.ru aviva-limoux.com
best-mam.ru avon-severozapad.ru
best-seo-offer.com avramstroy.ru
biketank.ga azartmsl.com
bioca.org balla-boo.se
bitcoinpile.com bangla-chat-uk-paralud.ga
bjanshee.ru bashtime.ru
blog-hits.com bestcoin.cc
blog.axant.it bestcurs.org
bocoarchives.org bestdooz.com
brus.city besthomemadepornsites.com
btc4u.ru bird1.ru
buchananshardware.com blockety.co
buyk.host.sk brendbutik.ru
bwlx.prepedu.cn bristolhotel.com.ua
canzoni.ru brk-rti.ru
castingbank.ru builtwith.com
centrdebut.ru buttons-for-your-website.com
cheapsergiorossi.online callmd5map.com
chinese-amezon.com cartierlove2u.com
click2pawn.com cartierlovebraceletreplica.xyz
clothesforcash.com cartierloveringreplica.xyz
clubfashionista.com cashkitten-a.akamaihd.net
coaching-netz.info cerev.info
collegeessay19.blogspot.ru cezartabac.ro
compliance-george.top chatseo.com
compliance-john.top cheapestjerseysonwholesale.com
compliance-julianna.top cheapmoncler.pw
copblock.org chlooe.com
crackguru.tk cialis-samples.com
culturevie.info cialischmrx.com
cyprusbuyproperties.com cialisovercounteratwalmartusa.com
dataloading.net cocolyze.com
deniven.1bb.ru com-cleaner.systems
deshevo-nedorogo.ru compliance-jack.top
detective01.ru confib.ifmo.ru
domainsphoto.com covadhosting.biz
donvito.unas.cz cubs-jerseys.us
e2click.com customerguru.in
ee77ee.com cyse.tk
elektrozigarettekaufen1.devhub.com daretodonate.co
englishdictionaryfree.com ddsoldes.fr
espaceinventoristes.com delitime.info
ezofest.sk demon-tweeks.com
factorynightclub.com descargar-musica-gratis.net
familyholiday.ml dfwu1013.info
fanoboi.com dgfitness.co
farmingworm.com diamond-necklace.info
dikx.gdn
diminishedvalueoforegon.com
dojki365.online
donna7753191.ru
doublepimp.com
downloaddy.net
downvids.net
efnor-ac.com
egovaleo.it
ekosmetyki.net.pl
eleimgo.pw
exhibitionplus.eu
ezb.elvenmachine.com
f-loaded.de
failingmarriege.blogspot.com
family1st.ca
fast-torrent.ru fast-torrent.ru
fermersovet.ru flex4launch.ru
fickenprivat.info freshdz.com
fil.ru g.starmoe.xyz
filmetricsasia.com gallery.rennlist.com
flowwwers.com gate5.co.za
fm-upgrade.ru gazporno.com
foodcrafts.website generalporn.org
freelinkbuilding.website.tk gifspics.com
freevpn.space greatdealshop.com
frivgame250.com greendream.com.ua
front.ru harmonyglen.us
funnypica.com helvetia.com.ua
gamingspark.com hledejvshopech.cz
gdebestkupit.ru hyhj.info
get-free-social-traffic.com ikearugs.xyz
given2.com ilbe.club
glass-msk.ru imabase.com
global-ics.co.za ip-guide.com
gnuetella.com irunfar.com
gogps.me iyasimasennka.com
goosefishpost.bid japfm.com
graphicwe.org javcoast.com
greatgrace.ru karaganda.xkaz.org
gsasearchengineranker.space
hamilton.ca
hanink.biz.ly
hawaiisurf.com
hdfreeporno.net
homedo.fabpage.com
iflycapetown.co.za
ihtec2019.org
images-graphics-pics.com
imitex-plus.ru
impresagaia.it
incest-ru.com
jillepille.com
journalhome.com
journeydownthescale.info
kashubadesign.ru kashubadesign.ru
kenaba.su kerch.site
krassh.ru keywords-monitoring-your-success.com
ks1234.com kinomaniatv.pl
kupit-adenu.ru krasnodar.xrus.org
leftofcentrist.com la-fa.ru
legionalpha.com lacapilla.info
lennatin.info lacasamorett.com
letsart.ru lampokrat.ws
lexixxx.com landmania.ru
linkbuddies.com laulini.soclog.se
locimge.pw lawyersinfo.org
losangeles-ads.com letmacwork.world
mafa-free.com letslowbefast.today
magiadinamibia.blogspot.com loftdigital.eu
malwareremovalcenter.com longlifelomilomi.info
marketingtechniques.info lyngdalhudterapi.no
marktforschung-stuttgart.com makis.nu
matpre.top mantramusic.ru
metarip.ru megahdporno.net
middlerush-a.akamaihd.net meteocast.net
milblueprint.com
mlf.hordo.win mlf.hordo.win
mlvc4zzw.space mlvc4zzw.space
monclercheap.xyz modenamebel.ru
mtmtv.info moesen-ficken.com
mug-na-chas-moscow.ru moncleroutletonline.win
muz-shoes.ru moncleroutletonline.xyz
myxdate.info monetizationking.net
newyorkhotelsmotels.info musicspire.online
nicefloor.co.uk nfljerseys.online
novosibirsk.xrus.org nfljerseyscheapbiz.us
nrv.co.za nullrefer.com
ntic.fr okuos.com
nufaq.com onclkads.com
oballergiya.ru online-hd.pl
obesidadealgarve.com onlinetvseries.me
obsessionphrases.com oralsexfilme.net
office2web.com paparazzistudios.com.au
okout.ru paramountmarble.co.uk
omoikiri-japan.ru partnerworkroom.men
online-templatestore.com pawli.eu
onlinedomains.ru pixshark.com
papasdelivery.ru pojdelo.weebly.com
photo.houseofgaga.ru porndroids.com
photosaga.info pornogratisdiario.com
piccshare.com pornophoto.xyz
platesauto.com pozdrawleniya.com
poptool.net profitwithalex.info
pornhubforum.tk pufip.com
pornozhara.com qexyfu.bugs3.com
pornstartits.xblog.in rainbowice.ru
portal-eu.ru rapidsites.pro
preconnubial.usuby.site rasteniya-vs-zombi.ru
preg.marketingvici.com redirect.trafficreceiver.club
productarium.com registry-cleaner.net
py100.ru remont-ustanovka-tehniki.ru
ranksonic.info renhacklids.tk
razamicroelectronics.com ringtonepartner.com
registratciya-v-moskve.ru riralmolamsaca.tk
renecaovilla.online ruscoininvest.company
renecaovillasale.online saugatuck.com
rep-am.com scanner-julia.top
rique.host.sk searchmywindow-a.akamaihd.net
rockprogblog.com seksotur.ru
rospromtest.ru seminarykansas904.ml
sabizonline.com semrush.com
scanmarine.info servicecenter.co.ua
seasaltwithfood.com sexiporno.net
sergiorossistore.online sexkontakteao.info
sextracker.com sexsaoy.com
sharpchallenge.com sherlock.se
shoppytoolmac.com shitting.pro
shopvilleroyboch.com.ua shop.xz618.com
sibtest.ru sibecoprom.ru
simple-image.com.ua sideeffectsoftizanidine.blogspot.com
smartadserver.com site.ru
snip.to smart-scripts.com
snts.shell-pmr.ru smartpet.ru
soc-econom-problems.ru social-button.xyz
soc-proof.su social-buttons.com
socialtrade.biz social-vestnik.ru
spyfu.com softtor.com
strag-invest.ru soochi.co
stroydetali.ru souvenir.cc
superfish.com sovinsteel.ru
svensk-poesi.com sportbetfair.com
svnuppsalaorebro.se standardchartered-forex.com
tagil.zrus.org studiofaca.com
teenfuck.tv superinterstitial.com
telegramdownload10.com svetodiodoff.ru
thecoolimages.net tam-gde-more.ru
thelottosecrets.com thebestphotos.eu
thetardistimes.ovh timetorelax.biz
topbestgames.com tizanidinehcl2mgsideeffects.blogspot.com
topdownloads.ru tizanidinehcl4mgtablet.blogspot.com
trafficmp.com tizanidinerecreationaluse.blogspot.com
traffmonster.info topcasinoratings.ru
trion.od.ua topquality.cf
tryrating.com trackzapper.com
tt-ipd.info traxdom.ru
tvand.ru tuckpointingmasonrysystems.com
uggbootsoutletsale.us twinderbella.com
uptime.com tytoona.com
urengoy.pro um-razum.ru
valmetrundan.se umg-stroy.ru
valsalud.com unrealcommander.org
vancleefreplica.pw uroki.net
vclicks.net uzungil.com
vellings.info varikoz24.com
vgoloveboli.net vchulkah.net
videonsk.com veopornogratis.xxx
videosbox.ru vesnatehno.com
vietimgy.pw via-energy-order.com
villacoloniale.com viagra.pp.ua
visionwell.com.cn viagroid.ru
voloslove.ru video-chat.cn
vsexkontakte.net videojam.tv
vzlomsn.org videospornogratisx.net
watsonrealtycorp.com vipromoffers.com
waysbetter.cn visa-pasport.ru
viveresaniesnelli.it
vsdshnik.com
watracker.net
web-analytics.date
webenlace.com.ar
website-audit.com.ua
websiteaccountant.de websiteaccountant.de
welck.octopis.com wholesalecheapjerseysfree.com
whatsappbot.flyland.ru wholesalenfljerseys.us.com
while.cheapwebsitehoster.com wordkeyhelper.com
wladimirpayen.com worldtraveler.world
womensplay.net wtsindia.in
word-vorlagen.net ww1943.ru
worst-sites.online xlovecam.com
xblognetwork.com xmnb.net
xn----itbeirbjbi7bc6bh2d.xn--p1ai xn----7sbbagbq7bd5aheftfllo4m.xn--p1ai
xn--80aagddcgkbcqbad7amllnejg6dya.xn--p1ai xn----7sbifcamovvfggw9d.xn--p1ai
xn--80adaggc5bdhlfamsfdij4p7b.xn--p1ai xn----9sbebi2bvzr7h.xn--p1ai
xn--80ahvj9e.xn--p1ai xn--80abgj3a5acid6ghs.top
xrus.org xvideospornoru.com
xvideosbay.com yoshkarola.zrus.org
yoluxuryevents.com yoursite.com
youresponsive.com ytmnd.com
yubikk.info zahvat.ru
zeroredirect12.com zebradudka.com
zootoplist.com zootravel.com
zrelaya.pw

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.184 0; 10.1.0.14 0;
127.0.0.1 0; 127.0.0.1 0;

8
dev-tools/tmp/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.03.2954 ### Version: V4.2022.03.2955
### Updated: Wed Mar 2 22:01:33 UTC 2022 ### Updated: Thu Mar 3 09:34:02 UTC 2022
### Bad Bot Count: 636 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -459,6 +459,8 @@ User-agent: Jyxobot
Disallow:/ Disallow:/
User-agent: Kenjin Spider User-agent: Kenjin Spider
Disallow:/ Disallow:/
User-agent: Keybot Translation-Search-Machine
Disallow:/
User-agent: Keyword Density User-agent: Keyword Density
Disallow:/ Disallow:/
User-agent: Kinza User-agent: Kinza

8
robots.txt/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.03.2954 ### Version: V4.2022.03.2955
### Updated: Wed Mar 2 22:01:33 UTC 2022 ### Updated: Thu Mar 3 09:34:02 UTC 2022
### Bad Bot Count: 636 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -459,6 +459,8 @@ User-agent: Jyxobot
Disallow:/ Disallow:/
User-agent: Kenjin Spider User-agent: Kenjin Spider
Disallow:/ Disallow:/
User-agent: Keybot Translation-Search-Machine
Disallow:/
User-agent: Keyword Density User-agent: Keyword Density
Disallow:/ Disallow:/
User-agent: Kinza User-agent: Kinza