vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-10 15:34:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.07.3252 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-07-26 22:07:05 +00:00
parent 35b46bf669
commit 5204ac1304
47 changed files with 101651 additions and 101661 deletions
Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.07.3251 #### Version: V4.2022.07.3252
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.07.3251 #### Version: V4.2022.07.3252
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.07.3251 #### Version: V4.2022.07.3252
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12572
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.07.3251 #### Version: V4.2022.07.3252
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12576
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12576
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

12576
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12576
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 1; 10.1.1.152 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 0; 10.1.1.152 0;
127.0.0.1 0; 127.0.0.1 0;

12576
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 1; 10.1.1.152 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 0; 10.1.1.152 0;
127.0.0.1 0; 127.0.0.1 0;

12576
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 1; 10.1.1.152 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 0; 10.1.1.152 0;
127.0.0.1 0; 127.0.0.1 0;

12576
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3251 3252

12576
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

12576
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.1.185.27 0; 20.115.107.197 0;

12576
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 1; 10.1.1.152 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 0; 10.1.1.152 0;
127.0.0.1 0; 127.0.0.1 0;

12576
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 1; 10.1.1.152 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 0; 10.1.1.152 0;
127.0.0.1 0; 127.0.0.1 0;

12576
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 1; 10.1.1.152 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 0; 10.1.1.152 0;
127.0.0.1 0; 127.0.0.1 0;

12576
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.1.185.27 0; 20.115.107.197 0;

12576
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.1.185.27 0; 20.115.107.197 0;

12576
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 1; 10.1.1.152 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
EMail Siphon Demon
EroCrawler MauiBot
Fimap Mr.4x3
HEADMasterSEO Picsearch
LibWeb SeobilityBot
MFC_Tear_Sample Upflow
OpenLinkProfiler VB Project
ProPowerBot WebPix
Web Collage Zitebot
page scorer sitechecker.pro

298
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
360Spider
404checker 404checker
404enemy Abonti
80legs Aboundex
ALittle Client
Acunetix
AhrefsBot AhrefsBot
AiHitBot AiHitBot
AllSubmitter
Alligator
AlphaBot AlphaBot
Anarchie Anarchy
Anarchy99 Ankit
Apexoo Apexoo
Asterias
Atomseobot Atomseobot
Attach
AwarioRssBot AwarioRssBot
AwarioSmartBot BBBike
BDCbot BDFetch
BackStreet
BackWeb BackWeb
Backlink-Ceck Backlink-Ceck
Badass
Barkrowler
Battleztar Bazinga
Black Hole Black Hole
BlackWidow BotALot
Blackboard BuiltWith
Boardreader
Brandwatch
Buddy
BuiltBotTough
Bullseye
BunnySlippers BunnySlippers
BuzzSumo
CATExplorador CATExplorador
CCBot
CODE87 CODE87
CSHttp CSHttp
Calculon Calculon
Cegbfeieh CazoodleBot
CherryPicker CensysInspect
ChinaClaw
Citoid
Claritybot
Cliqzbot
Cloud mapping Cloud mapping
Cogentbot Copier
Collector
CopyRightCheck CopyRightCheck
Cosmos
Craftbot
Crawling at Home Project
CrazyWebCrawler
Crescent Crescent
Curious Curious
CyotekWebCopy
DBLBot
DTS Agent
DatabaseDriverMysqli
Demon Demon
Digincore Deusu
Discobot Discobot
Dispatch Domains Project
DittoSpyder Download Wonder
DnBCrawler-Analytics Dragonfly
DomainSigmaCrawler
Dotbot
EMail Siphon
EMail Wolf EMail Wolf
EasyDL Ebingbong
Evil Exabot
ExtLinksBot Express WebPictures
Extreme Picture Finder ExtractorPro
Ezooms
FHscan
FlashGet
Flunky
Foobot Foobot
Freeuploader
Fuzz Fuzz
FyberSpider
Fyrebot
G-i-g-a-b-o-t G-i-g-a-b-o-t
Genieo
GermCrawler
GetRight
Gigabot
GrapeFX GrapeFX
HTTrack GrapeshotCrawler
HMView
Haansoft Haansoft
Humanlinks HaosouSpider
HybridBot Harvest
IDBot Havij
Heritrix
Hloader
IDBTE4M
Id-search Id-search
Image Fetch Image Fetch
Image Sucker
IndeedBot IndeedBot
InfoNaviRobot
Iria Iria
IstellaBot Iskanie
JamesBOT JamesBOT
Jbrofuzz Jbrofuzz
Jetty Jetty
JikeSpider JustView
Jorgee Keyword Density
Jyxobot
Kenjin Spider
Kinza Kinza
Kozmosbot Kozmosbot
Larbin LNSpiderguy
Leap
LexiBot
Lftp Lftp
LinkScan LinkWalker
Linkbot LinqiaRSSBot
LinkpadBot
LinqiaMetadataDownloaderBot
LinqiaScrapeBot LinqiaScrapeBot
Lipperhey Spider Lmspider
Litemage_walker
Ltx71
MFC_Tear_Sample MFC_Tear_Sample
MIDown tool
MIIxpc MIIxpc
MJ12bot MJ12bot
MQQBrowser MQQBrowser
MSIECrawler
Mail.RU_Bot Mail.RU_Bot
Majestic12 Majestic SEO
Majestic-SEO
MarkMonitor MarkMonitor
Masscan Masscan
Mata Hari Mata Hari
Mb2345Browser Mb2345Browser
Meanpathbot MeanPath Bot
Mediatoolkitbot
Metauri Metauri
MicroMessenger MicroMessenger
Microsoft URL Control Morfeus Fucking Scanner
Minefield Mozlila
Mojolicious
Mr.4x3
NICErsPRO NICErsPRO
NPbot NPbot
Nameprotect Nameprotect
Nessus Needle
NetSpider NetLyzer
NextGenSearchBot Netcraft
Netvibes
Nibbler
Niki-bot Niki-bot
Nikto Nikto
NimbleCrawler Nmap
Nutch Not
Offline Explorer Nuclei
OrangeBot OpenLinkProfiler
OrangeSpider OrangeSpider
OutclicksBot OutclicksBot
OutfoxBot
PECL::HTTP PECL::HTTP
POE-Component-Client-HTTP Pandalytics
Page Analyzer
PageAnalyzer
PageGrabber
Panscient Panscient
Pavuk Pimonster
Petalbot PleaseCrawl
Piepmatz ProPowerBot
Probethenet
Psbot
Pump
PyCurl PyCurl
RankActive RankActive
RankFlex RankFlex
Rankivabot RankingBot
RankurBot
Re-re Re-re
ReGet RealDownload
Reaper Reaper
Recorder RedesScrapy
Rogerbot RocketCrawler
SBIder SBIder
SEOkicks
SEOkicks-Robot SEOkicks-Robot
SEOstats SEOlyticsCrawler
SISTRIX SISTRIX
SalesIntelligent SMTBot
ScoutJet Scrapy
ScreenerBot
ScrepyBot
Searchestate
SearchmetricsBot
Seekport Seekport
Semrush SeoSiteCheckup
SeobilityBot
Seomoz Seomoz
SiteSucker Siphon
Sitebeam SiteCheckerBotCrawler
SiteLockSpider
SiteSnagger
SlySearch
SmartDownload SmartDownload
Snoopy Snapbot
SocialRankIOBot SocialRankIOBot
Sociscraper SpaceBison
SpankBot Spanner
Spbot Spbot
Spinn3r Sqlworm
SputnikBot
Sqlmap
Sqworm Sqworm
Stripper
Sucker Sucker
Sucuri SuperHTTP
SuperBot
SurveyBot
Suzuran Suzuran
Szukacz
T0PHackTeam
T8Abot
Teleport
TeleportPro TeleportPro
Telesoft Telesphoreo
The Intraformant The Intraformant
Thumbor Thumbor
Toweyabot Toweyabot
Tracemyfile Tracemyfile
TurnitinBot TurnitinBot
VB Project TwengaBot
UnisterBot
Upflow
V-BOT
Vagabondo Vagabondo
VidibleScraper VelenPublicWebCrawler
Voltron VoidEYE
WBSearchBot
WEBDAV
WPScan
WWW-Collector-E WWW-Collector-E
WWW::Mechanize
Wallpapers
WallpapersHD
WeSEE WeSEE
Web Auto Web Auto
Web Fetch
Web Pix
Web Sucker
Web Collage Web Collage
WebEnhancer Web Fetch
Web Fuck
Web Pix
Web Sauger
Web Sucker
WebBandit
WebCopier
WebEnhancer
WebGo IS
WebImageCollector
WebSauger
WebStripper
WebSucker WebSucker
Webalta
WebmasterWorldForumBot
WebsiteQuester WebsiteQuester
WebsiteQuester Whack
WinHTTrack Whacker
Wonderbot Widow
Wprecon
Xaldon_WebSpider Xaldon_WebSpider
Xenu Zade
YoudaoBot
Zauba
ZmEu
ZoomBot
ZoominfoBot ZoominfoBot
ZumBot
ZyBorg ZyBorg
adscanner arquivo-web-crawler
arquivo.pt autoemailspider
backlink-check
cah.io.community cah.io.community
check1.exe
clark-crawler clark-crawler
coccocbot coccocbot
cognitiveseo
com.plumanalytics
crawl.sogou.com crawl.sogou.com
crawler4j crawler4j
dataforseo.com domainsproject.org
evc-batch eCatch
facebookscraper
gopher
internetVista monitor internetVista monitor
iubenda-radar ips-agent
linkdexbot isitwp.com
lwp-request meanpathbot
magpie-crawler mediawords
muhstik-scan
oBot oBot
probe-image-size plumanalytics
serpstatbot scalaj-http
sitechecker.pro scan.lol
seostar
siteripz
sogouspider
sysscan
trendiction.de trendiction.de
webprosbot ubermetrics-technologies.com
voyagerx.com
x09Mozilla x09Mozilla
xpymep1.exe xpymep1.exe

282
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
404checker 404checker
ALittle Client ADmantX
Aboundexbot AIBOT
Acunetix ASPSeek
Abonti
AhrefsBot AhrefsBot
Anarchie Aipbot
Anarchy Alexibot
Apexoo AllSubmitter
AlphaBot
Asterias Asterias
Atomseobot Atomseobot
Attach
AwarioRssBot AwarioRssBot
BBBike
BDCbot
BDFetch BDFetch
BLEXBot BackDoorBot
BackStreet
BackWeb BackWeb
Backlink-Ceck Bandit
BacklinkCrawler
Barkrowler
BatchFTP
Battleztar Bazinga
BetaBot
Bigfoot Bigfoot
BlackWidow
Blackboard
Blow
BlowFish BlowFish
Boardreader
BuiltBotTough BuiltBotTough
BuiltWith
Bullseye Bullseye
BuzzSumo CATExplorador
CazoodleBot CCBot
Cegbfeieh CensysInspect
CheTeam CheTeam
CheeseBot
ChinaClaw
Chlooe Chlooe
Citoid Citoid
Cliqzbot Claritybot
CopyRightCheck Cloud mapping
Cocolyzebot
Collector
Copyscape
Craftbot Craftbot
CrunchBot Crawling at Home Project
Custo CrazyWebCrawler
CyotekWebCopy Crescent
DSearch DSearch
DTS Agent Digincore
DigitalPebble Dirbuster
Disco
DittoSpyder DittoSpyder
DnBCrawler-Analytics ECCP/1.0
DnyzBot
DomainAppender
DomainSigmaCrawler
Dotbot
EMail Siphon
EMail Wolf EMail Wolf
EasyDL ExtLinksBot
Exabot
Extreme Picture Finder
Ezooms Ezooms
FDM
FemtosearchBot FemtosearchBot
Fimap Fimap
FlashGet FlashGet
Foobot Flunky
Freeuploader Freeuploader
Fuzz
Fyrebot Fyrebot
GT::WWW
Genieo
GetWeb
Getintent Getintent
Gigabot Gigabot
Go!Zilla
Go-Ahead-Got-It Go-Ahead-Got-It
GoZilla
GrabNet GrabNet
Grabber GrapeshotCrawler
Grafula
HEADMasterSEO
HMView
HTTrack HTTrack
HaosouSpider Haansoft
Havij
Humanlinks Humanlinks
IDBTE4M HybridBot
IDBot IRLbot
Id-search Iblog
Image Fetch Image Fetch
Image Sucker Image Sucker
IndeedBot InfoNaviRobot
Internet Ninja
IstellaBot IstellaBot
JOC Web Spider Jbrofuzz
JennyBot
JetCar JetCar
JikeSpider Jetty
Joomla
Kinza Kinza
LNSpiderguy
LWP::Simple
Lanshanbot
Larbin Larbin
LeechFTP LeechFTP
LeechGet LeechGet
Lftp Lftp
LibWeb LieBaoFast
Lightspeedsystems Lightspeedsystems
Likse
LinkScan LinkScan
LinkWalker LinkextractorPro
Linkbot LinksManager
LinkpadBot
LinqiaMetadataDownloaderBot
LinqiaRSSBot LinqiaRSSBot
Lipperhey
Lipperhey Spider Lipperhey Spider
Litemage_walker
Lmspider Lmspider
Ltx71 Ltx71
MJ12bot MIIxpc
MTRobot MTRobot
Mail.RU_Bot Mail.RU_Bot
Majestic SEO
MarkMonitor MarkMonitor
MarkWatch
Mass Downloader Mass Downloader
Mb2345Browser MauiBot
MeanPath Bot MeanPath Bot
Meanpathbot MegaIndex.ru
Mojeek Microsoft Data Access
Mojolicious Mojolicious
NPbot Musobot
NICErsPRO
Navroad Navroad
NearSite NetSpider
NetLyzer
Netcraft Netcraft
Nettrack Nettrack
Netvibes
NextGenSearchBot NextGenSearchBot
Niki-bot Nibbler
Nikto Nikto
NimbleCrawler
Nmap
Not Not
Nuclei Octopus
Nutch OnCrawl
OpenLinkProfiler OpenLinkProfiler
Openvas OrangeSpider
OutclicksBot OutclicksBot
OutfoxBot PECL::HTTP
Page Analyzer Page Analyzer
PageGrabber PageThing.com
PageScorer
Panscient
Papa Foto Papa Foto
Pavuk Picscout
Pi-Monster Picsearch
PictureFinder Piepmatz
Pimonster PleaseCrawl
Pockey
Probethenet Probethenet
PxBroker PxBroker
PyCurl PyCurl
QueryN Metasearch
Quick-Crawler
RankActive
RankActiveLinkBot
RankingBot2 RankingBot2
Rankivabot
RankurBot RankurBot
Re-re
ReGet
RealDownload
Reaper
Recorder Recorder
RepoMonkey RocketCrawler
SEOkicks-Robot Rogerbot
SEOlyticsCrawler SalesIntelligent
SEOstats ScanAlert
Scanbot Scanbot
ScreenerBot ScoutJet
Scrapy
Screaming
ScrepyBot ScrepyBot
Semrush Searchestate
SemanticJuice
SeoSiteCheckup SeoSiteCheckup
Seomoz
Shodan
Site Sucker Site Sucker
SiteCheckerBotCrawler
SiteExplorer
SiteLockSpider SiteLockSpider
SiteSnagger
Siteimprove
Sitevigil Sitevigil
SmartDownload
Snake Snake
Sociscraper
Sosospider Sosospider
SpaceBison
Spanner
Spbot Spbot
Spinn3r
SputnikBot SputnikBot
Sqlworm Sqlworm
Steeler Sucker
Stripper Sucuri
SuperHTTP SuperHTTP
SurveyBot Surfbot
Szukacz Szukacz
T0PHackTeam T8Abot
TeleportPro Teleport
The Intraformant Telesphoreo
TheNomad
Thumbor Thumbor
Titan Tracemyfile
Trendiction
Trendictionbot
Turingos Turingos
Turnitin Turnitin
URLy Warning Typhoeus
URLy.Warning UnisterBot
VCI
Vacuum Vacuum
Vagabondo Vagabondo
VelenPublicWebCrawler
Virusdie Virusdie
VoidEYE VoidEYE
Voltron Voltron
WASALive-Bot WBSearchBot
WEBDAV
WPScan
WWW-Collector-E WWW-Collector-E
WWW::Mechanize WWW::Mechanize
Wallpapers/3.0 WWWOFFLE
WallpapersHD WeSEE
Web Auto Web Enhancer
Web Collage Web Fetch
Web Fuck
Web Sauger
Web Sucker Web Sucker
WebBandit WebBandit
WebFetch WebImageCollector
WebFuck
WebGo IS
WebLeacher WebLeacher
WebReaper
WebSauger
WebSucker
Webalta
WebmasterWorldForumBot WebmasterWorldForumBot
Webshag Webshag
Webster
Whack Whack
Whacker
Whatweb Whatweb
Woobot Widow
Wotbox WinHTTrack
Xaldon WebSpider Wonderbot
Wprecon
Xenu
Zade Zade
Zeus
Zitebot Zitebot
ZmEu
ZumBot
ZyBorg ZyBorg
archive.org_bot archive.org_bot
arquivo-web-crawler arquivo-web-crawler
check1.exe arquivo.pt
clark-crawler autoemailspider
cognitiveseo
com.plumanalytics com.plumanalytics
crawl.sogou.com dataforseo.com
crawler.feedback
demandbase-bot
evc-batch evc-batch
facebookscraper facebookscraper
instabid
linkdexbot linkdexbot
lwp-request
lwp-trivial lwp-trivial
meanpathbot plumanalytics
oBot polaris version
page scorer probe-image-size
s1z.ru ripz
seoscanners satoristudio.net
tAkeOut scan.lol
seocompany.store
sexsearcher
sysscan
trendiction.de trendiction.de
ubermetrics-technologies.com webgains-bot
webprosbot xpymep1.exe
x09Mozilla zauba.io

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
biographiya.com 7search.com
doktoronline.no bibys.com
food.dtu.dk cinemaenergy-hd.ru
jav-idol.com diusyawiga.tk
mountainstream.ms eropho.net
pdn-4.com inboundlinks.win
pooleroadmedicalcentre.co.uk lion.playtap.us
realmonte.net nibbler.silktide.com
rightenergysolutions.com.au seo-smm.kz
univerfiles.com swinon.site

938
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
03p.info 01apple.com
0912701309f8ce.com 1-free-share-buttons.com
1000n1.ru 101flag.ru
1008.su 123kuma.com
1flag.co.za 123locker.com
1pamm.ru 12bet.com
202ch.com 12masterov.com
2345.com 1bet.com
4inn.ru 2pxg8bcf.top
4webmasters.org 34.gs
5forex.ru 3w24.com
6hopping.com 4kpics.rocks
7fon.ru 5kstore.com
aac52006ec82a24e08b665f4db2b5013f7.com 5tey7463.weebly.com
aasoldes.fr 5ws.dating-app.ru
absugars.com 98oi.ru
actionnooz.com a14download.com
adconscious.com a1security.com.ua
adminshop.com aa9d046aab36af4ff182f097f840430d51.com
aihelen.net aaronabel.com
allcredits.su abbanreddy.com
alpharma.net abovetherivernc.com
alpinism.ru activecampaign.dreamhosters.com
antons-transporte.de adclickthru.net
apartment.ru admanaerofoil.com
apioutoboxnet-a.akamaihd.net adrenalin-stk.ru
adsref.men
adtology.com
adultnet.in
aktobe.xkaz.org
alert.scansafe.net
alf-img.com
all-number.com
all4invest.info
allboard.xobor.de
alldezservice.kz
alloysteel.ru
analnoeporno.tv
analytics-ads.xyz
andpolice.com
animali.deagostinipassion.it
animenime.ru
apiwoodensealcom-a.akamaihd.net
appearance-cool.com
appfastplay.com
appsecurityr.com appsecurityr.com
arenanews.com.ua arabgirls.adultgalls.com
artel116.ru arkkivoltti.net
ascat.porn atdedinotuho.tk
asiengirls.net auspolice.com
audiobangout.com autobudpostach.club
automobile-spec.com autochoixspinelli.com
aviav.co automatic-seo.com
azartniy-bonus.com avon-novosib.ru
azte.ch azadnegar.com
babs.com.ua azartmix.com
baikaleminer.com back.dog
balois.worldbreak.com bannerads.de
balook.com basisches-wasser.net
bamo.xsl.pt beachpics.com
basedecor.ru beauty-things.com
beachfix.co bedandbreakfast.com
beetpics.pw beetpics.pw
bestmobilityscooterstoday.com bestcoin.cc
bestofferswalkmydogouteveryday.gq bestempresas.es
bestprofits-there.com bestserials.com
betune.onlinewebshop.net betterscooter.com
bezcmexa.ru bigames.online
bibys.com bigpictures.club
bingo8888.com
biocn.dx.am
biographiya.com
bioscorp.ru bioscorp.ru
bitcoin-ua.top bjanshee.ru
bizru.info blackplanet.com
bnt-team.com blackwitchcraft.ru
bombla.org blobar.org
bongiornos.info bloke.com
bookhome.info bochemit.com.ua
bpro1.top bonzybuddy.com
brainsandeggs.blogspot.com boole.onlinewebshop.net
bravegirlsclub.com brianhenry.co
bryansk.zrus.org brokergid.ru
bscodecs.com budilneg.xyz
btc4u.ru burkesales.com
burger-imperia.com buyessayonline19.blogspot.ru
businesxxl.com bwlx.prepedu.cn
buttons-for-website.com callejondelpozo.es
buyessay3.blogspot.ru camsex.xxx-cam.webcam
buypillsorderonline.com candypeople.se
bycontext.com capture-room.com
cacheimages.com cartechnic.ru
cactussoft.biz
callawaygolfoutlet.online
cam-kontakte.org
canzoni.ru
cardul.ru
carloans.com
cartierbracelet.xyz cartierbracelet.xyz
cartierbraceletsreplica.pw cartiertrinity.xyz
casite-513049.cloudaccess.net casinorewards.info
caveavins.fr cdnanalytics.xyz
cayado.snn.gr cdnnetwok.xyz
cete.ru certifywebsite.win
cheapsergiorossi.online cfcl.co.uk
chinaelitecheapjerseys.com cheapestjerseys-wholesale.com
chocolateslim-en-espana.com cheapjerseysbizwholesale.us
chocolateslim-original.com checkpagerank.net
ciekawinki.pl chocolateslim-u-srbiji.com
cineacademy.ru chuckguilford.com
cityadspix.com cialisndbrx.com
clean-start.net cindymatches.com
cleanersoft.com ciproandtizanidine.blogspot.com
comissionka.net classiquebijoux.ru
community.allhiphop.com com-onlinesupport.site
compliance-brian.xyz compiko.info
compliance-fred.top compliance-andrew.top
compliance-george.xyz compliance-checker.info
compliance-viktor.top compliance-ivan.xyz
confib.ifmo.ru cracksplanet.com
cool-wedding.net craftburg.ru
coolgramgoods.com cupday.com
copblock.org dailyrank.net
cosmetic.donna7753191.ru damedingel.ya.ru
cougfan.info ddospanel.com
counter.bloke.com
covid-schutzmasken.de
crirussian.ru
culmimg.pw
customboxes4less.com
cwetochki.ru
danceuniverse.ru
dataloading.net
db.speedup.gdn
ddpills.com
dealighted.com
dealitright.click dealitright.click
delayreferat.ru decoratinghomes.ga
deniven.1bb.ru demenageur.com
digilander.libero.it detki-opt.ru
digital-sale.su devochki.top
dikx.gdn
dimkino.ru
dinkolove.ya.ru dinkolove.ya.ru
dolohen.com dogbreedspicture.net
downloadeer.net dojki-hd.com
downloader12.ru dollartree.info
downloadmefiranaratb1972.xpg.com.br domain-tracker.com
dverimegapolis.ru domination.ml
dvervmoskvu.ru doreenblog.online
dyshagi.ru drillsaw.ru
dyt.net drugspowerstore.com
easync.io drugstoreforyou.com
ee77ee.com dumpsmania24.com
elektrischezigarette2.onsugar.com eachdayisagift.review
elektronischezigarettekaufen2.tumblr.com easyshoppermac.com
eloconcream.blogspot.com education-cz.ru
eugenevaultstorage.com edwinkonijn.com.au
eupornstar.info ekaterinburg.xrus.org
euronis-free.com ekspertmed.com
evening-dating-club.info ekto.ee
election.interferencer.ru
elektrischezigaretten1.blogse.nl
elektrozigarettekaufen2.blogse.nl
elektrozigaretten1.tumblr.com
energydiet24.ru
eorogo.top
erectile.bid
erotik0049.com
errorfixing.space
ertelecom.ru
espaceinventoristes.com
evehemming.blogspot.com.au
everytalk.tv everytalk.tv
exonline.info eyelike.com.ua
ez8motelseaworldsandiego.com ezigarettekaufen1.hpage.com
ezigarettenkaufen1.pagina.gr ezrvrentals.com
f012.de face.hostingx.eu
f07.de facecup.top
f0815.de falcon-images.blogspot.com
falcoware.com fanoboi.com
farm26.ru fashionindeed.ml
fearcrow.com fast-wordpress-start.com
ferretsoft.com fealq.com
fetishinside.com filmania-x.ru
film-one.ru finteks.ru
financehint.eu firma-legion.ru
findclan.org fitnesspiks.com
firstsiteguide.com flowwwers.com
flavors.me fodelsedagspresenter.nu
flytourisme.org foto-weinberger.at
forex.osobye.ru freelotto.com
fortwosmartcar.pw fres-news.com
foxjuegos.com
fr.netlog.com
franch.info
franecki.net
franklinfire.co
frcls.fr
free-social-buttons.xyz
free-video-tool.com
freelinkbuilding.website.tk
freemaintenancesysforpcandmac.top
freeseedsonline.com
frivgame250.com
front.to front.to
ftns.ru
fuckmill.com
fullfileaccess.com
funnypica.com
funponsel.com
fx-brokers-review.com
fxund.us fxund.us
g33.org galeria-zdjec.com
galleryawesome.com gamedayassist.com
gambarkataku.co gamerextra.com
gameplexcity.com gamerscorps.com
gameprimary.com garciniaxt.us
gaypornmovie.net gcup.ru
gembird.com generic-pills-online.com
get-free-social-traffic.com glall.ru
getlaid-xxxhookupdirect.com glass-msk.ru
gidonline.one glcomputers.ru
gktt.ru gobongo.info
glavprofit.ru goforexvps.com
global-ics.co.za gogalleryawesome.com
goldenggames.com googlsucks.com
golfresa.lucania.se goosefishpost.bid
googglet.com grand-chlen.ru
gpms.org.my
gratuitxblcodes.com gratuitxblcodes.com
greenshop.su greatdealshop.com
gsasearchenginerankerhelp.com gsasearchengineranker.space
gwagka.com guiadeserraazul.com
hamilton.ca hao123.com
handsandlegs.ru haveinc.xyz
heygidday.biz hdfreeporno.net
hop.clickbank.net heartofbeijing.blogspot.com
hoverboard360.se histock.info
homelygarden.com
hoterika.com
house.sieraddns.com
hoverboardforsaledirect.com hoverboardforsaledirect.com
howtowhitenteethfast.xyz howtoclean.club
hunterboots.online hunterboots.online
iblogpress.xyz hydropump.su
icaseclub.ru hyipmanager.in
ideibiznesa2015.ru ifirestarter.ru
iflycapetown.co.za images-graphics-pics.com
iframe-toloka.com imguramx.pw
ilovevitaly.xyz imperiafilm.ru
imgarcade.com importchinacoach-teach.com
imgarit.pw inboxdollars.com
imgata.com incest-ru.com
immigrational.info incolors.club
intelhdgraphicsgtdrive6w.metroblog.com informatiecentro.be
ipostroika.ru interesnie-faktu.ru
iwanttodeliver.com interior-stickers.ru
ixora.pro internetproviderstucson.com
izismile.com inzn.ru
jackwolfskinoutlet.online istizanidineacontrolledsubstance.blogspot.com
japan-bearings.ru iweblist.info
jasonpartington.com javidol.site
jav-idol.com jumptap.com
jillepille.com kazka.ru
jolic2.com ketrzyn.pl
jovencitas.gratis keywordsuggest.org
jurajskie.info kicknights.gq
kadashihotel.com kimcurlrvsms.com
karachev-city.ru
keenoutlet.online
kevblog.top
keywordsdoctor.com
keywordspay.com
kiinomaniak.pl
kino-key.info
kino-rating.ru kino-rating.ru
kino2018.club kinohall.ru
kinobest.pl kiwi237au.tk
kinogolos.ru klejonka.info
kinogonew.ru klosetkitten.com
kinopolet.net konkursowo-24.pl
klin3952.ru koversite.info
kollekcioner.ru krasivye-devushki.net
konoplisemena.com krasnodar-avtolombards.ru
koral.se kunstaktien.info
kountrylife.com lacloop.info
krasnodar.xrus.org landmania.ru
kriokomora.info lashstudia.ru
labelwater.se le-clos-des-alouettes.com
lancheck.net ledis.top
laptoper.net lego4x4.ru
law-check-nine.xyz lemon-ade.site
lesbian.xyz
lescinq.com
lexiangwan.com
lflash.ru
lida-ru.com lida-ru.com
limads.men
linkbolic.com
linkredirect.biz linkredirect.biz
liumimgx.pw linksharingt.com
liverpool.gsofootball.com lion.conboy.us
luciddiagnostics.in live-cam6.info
luckyday.world liver-chintai.org
manimpotence.com locatellicorretor.com.br
mapquestz.us lockerz.com
mararoom.ru lolnex.us
markjaybeefractal.com lombia.com
mathgym.com.au mac-shield.com
mazda-roadsters.com magda-gadalka.ru
message-warning.net makis.nu
mexicosleevegastrectomy.com malls.com
mezaruk.info malwareremovalcenter.com
minharevisao.com manuscript.su
miniads.ca marmotstore.online
mistr-x.org mature.free-websites.com
mmog-play.ru mediawhirl.net
mobplayer.ru medkletki.ru
moesonce.com mere.host.sk
monclerboots.xyz mex-annushka.ru
moonci.ru middlerush-a.akamaihd.net
moscow.online-podarki.com mini.7zap.com
muizre.ru mirzonru.net
musirc.com mnogolok.info
myftpupload.com modnie-futbolki.net
myhealthcare.com moivestiy.biz
nakrutka.cc moncleroutletonline.pw
namenectar.com money-every-day.com
naughtyconnect.com money-for-placing-articles.com
moomi-daeri.com
more-letom.ru
most.gov.iq
mturkcontent.com
musicktab.com
musicstock.me
musicvidz.ru
muz-tracker.net
myhitmp3.club
mylovelibrabry.com
myonlinepayday.co
narkologiya-voronezh.ru
nash-krym.info
nastroyke.net
natali-forex.com
naturalbreakthroughsresearch.com
navalwiki.info navalwiki.info
negral.pluto.ro nikitabuch.com
neks.info nina.az
newsperuse.com notfastfood.ru
nippon-bearings.ru nw-servis.ru
nmrk.ru oballergiya.ru
nonameread45.live offer.camp
noumeda.com oivcvx.website
o-o-6-o-o.ru okonich.com.ua
omsk.xrus.org okroshki.ru
onlainbesplatno.ru openfrost.com
onlinadverts.com openlibrary.org
onlinemegax.com oren-cats.ru
ons-add.men owathemes.com
osb.se11.ru ownshop.cf
pacificair.com oxotl.com
panchro.xyz papasdelivery.ru
parajumpersjakkesalgnorge.info parajumpersoutlet.online
partner-trustworthy.men partnerline.men
pcimforum.com pearlisland.ru
pdfprof.com perfection-pleasure.ru
pdn-4.com petitions.whitehouse.gov
perfectplanned.com photo-clip.ru
photo.houseofgaga.ru picsforkeywordsuggestion.com
photokitchendesign.com picturesfrom.com
picscout.com pills24h.com
picturesmania.com pisanieprac.info
pigrafix.at piter.xrus.org
piratecams.com play-mp3.com
pirateday.ru pony-business.com
pix24x7.com
pixgood.com
pk-pomosch.ru
pl-vouchers.com
pochtovyi-index.ru
poiskzakona.ru
pojdelo.weebly.com
popup-fdm.xyz popup-fdm.xyz
popup-hgd.xyz pornoreino.com
porno-home365.com pornosee.info
porno-transsexuals.ru pornozhara.com
pornoelita.info portadd.men
pornofeuer.com pravoholding.ru
pornotubs.com prchecker.info
powc.r.ca.d.sendibm2.com preparevideosafesystem4unow.space
powitania.pl pro-okis.ru
pozdrawleniya.ru producm.ru
preparevideosafesystem4unow.site profitfx.online
princeadvantagesales.com
privatefx.all4invest.info
prodess.ru
productarium.com
profitkode.com profitkode.com
proftests.net profitsport.club
proposal-engine.com projefrio.com.br
prostitutki-odessa.org promgirldresses.xyz
prostitutki-tyumeni.org propranolol40mg.blogspot.com
rapidgator-porn.ga prostitutki-astana.org
qexyfu.bugs3.com
quelle.ru
quick-seeker.com
quickcashlimited.com
rankexperience.com
ranksonic.org
rapidsites.pro rapidsites.pro
ratemodels.net recinziireale.com
razorweb-a.akamaihd.net redirectingat.com
realitykings.com registry-clean-up.net
rebuildermedical.com registrydomainservices.com
reckonstat.info renecaovilla.online
remmling.de responsive-test.net
rentehno.ru rocketchange.ru
rostov.xrus.org roleforum.ru
rozalli.com rome2rio.com
s1z.ru ros-ctm.ru
safe-app.net royal-betting.net
sakhboard.ru royalads.net
salmonfishingsacramentoriver.com rukino.org
satoristudio.net ruspdd.com
sax-sex.com rusprostitute.com
scanner-margo.top rvzr-a.akamaihd.net
scopich.com samoiedo.it
sanyuprojects.com
scanner-barak.top
scanner-jane.top
scanner-jess.top
scanner-julianna.top
schalke04fc.info
screentoolkit.com
se-welding.ru
search.pch.com
searchinquire.com searchinquire.com
secret.xn--oogle-wmc.com searchkut.com
securesmrt-dt.com segol.tv
sel-hoz.com semaltmedia.com
sell-fb-group-here.com
semxiu.com
seocheckupx.com
servethis.com servethis.com
sexad.net service.adtech.us
sharpchallenge.com sex-dating.co
sex-pr.net
sex.hotblog.top
sexiporno.net
sexkontakte-seite.com
sexkontakteao.info
shisha-swag.de shisha-swag.de
shivafurnishings.com shivafurnishings.com
shoesonlinebuy.cn shoppytoolmac.com
shohanb.com sibvitr.ru
shoplvlv.us siha.de
shoppingjequiti.com.br
sierraapps.com
sildenafil-tadalafil.info
simon3.ru
simplesite.com simplesite.com
sitechecker.pro simpoed.ufop.br
siteripz.net sklad-24.ru
slim.sellany.ru smichovbike.cz
smartadserver.com snsdeainavi.info
sobecjvuwa.com.ru social-s-hhh.xyz
softomix.com soft1.ru
sonata-arctica.wz.cz softwaretrend.net
space2019.top spasswelt.net
spammen.de staff.prairiesouth.ca
standardchartered-forex.com statistici.ro
stardevine.com
stariy-baku.com
stats-public.grammarly.io stats-public.grammarly.io
steebook.com sticken.co
store-rx.com stop-zavisimost.com
stpicks.com students-cheapskate.ml
strag-invest.ru sumo.com
stroiminsk.com supermodni.com.ua
studiokamyk.com.pl sweepstakes.rewardit.com
studworks.org szucs.ru
suchenindeutschland.com tacbelarus.ru
supermesta.ru takeflyte.com
symbaloo.com takeprofitsystem.com
taaaak.com tds-advert005.info
tabakur77.com techart24.com
tdsing.ru themestotal.com
tecnoteakviareggio.it thepokertimer.com
teenporn18.net thesmartsearch.net
texbaza.by tilido.com
thaisamkok.com tizanidine4mgstreetvalue.blogspot.com
themeforest.net tizanidinehydrochloride4mgstreetvalue.blogspot.com
thexart.club tizanidinesideeffectsweightloss.blogspot.com
tizanidineandhydrocodone.blogspot.com tizanidinezanaflex.blogspot.com
tizanidinemusclerelaxant.blogspot.com tjkckpytpnje.com
tizanidinerestlesslegsyndrome.blogspot.com topclickguru.com
tomck.com torrent-newgames.com
tonivedu.it torrentdownloadhub.com
topquality.cf tovaroboom.vast.ru
toproadrunner5.info tradgardspartner.se
torrents-tracker.com traffic2cash.org
touchmods.fr trafficcentr.xyz
tozup.com
track.deriv.com
trafficmania.com trafficmania.com
traffmonster.info trumpetedextremes.com
trustl.life tsstcorpcddvdwshbbdriverfb.aircus.com
tula.mdverey.ru
turist-strani.ru turist-strani.ru
tvory.predmety.in.ua twitlinks.com
twinderbella.com tzritel.tk
typer.one ugguk.online
u-cheats.ru umg-stroy.ru
ua-company.ru underthesite.com
uhdtv.website unisexjewelry.org
undergroundcityphoto.com unrealcommander.com
uni.me uptime-alpha.net
upstore.me userequip.com
urlopener.com vacuumcleanerguru.com
vasileostrovsky-rayon.ru versaut.xxx-cam.webcam
vchulkax.com vesnatehno.ru
vinsit.ru vgoloveboli.net
visa-pasport.ru via-energy-order.com
vitanail.ru
vk-mus.ru
vkonche.com
vltai.com vltai.com
vsesubwaysurfers.com voloo.ru
warningwar.ru vsdshnik.com
watchmygf.net wait3sec.org
webkeyit.com wallpaperaccess.com
webshoppermac.com watch-movies.ru
website-analyzer.info watchmyfb.pl
website-speed-up.site waterpurifier.club
websiteexplorer.info wdrake.com
websitevaluebot.com weboptimizes.com
wejdz-tu.pl webpromotion.ae
wesharepics.com website-speed-up.top
wetgames.ru weightatraining.com
whatsappbot.flyland.ru welcomeauto.ru
weprik.ru
westen-v.life
whatzmyip.net
while.cheapwebsitehoster.com while.cheapwebsitehoster.com
wildcattube.com wholesalejerseychinaoutlet.com
wiosenny-bon-1500.pl winner7777.net
wma-x.com worldhistory.biz
woman-h.ru wosik-dach.service-for-web.de
world-mmo.com wygraj-teraz.com
ws.ampower.me x-lime.com
www.arenda-yeisk.ru xerox-douglas.cf
x-musics.com xn-----6kcaacnblni5c5bicdpcmficy.xn--p1ai
xblog.in xn----8sbarihbihxpxqgaf0g1e.xn--80adxhks
xlolitka.com xn----8sblgmbj1a1bk8l.xn----161-4vemb6cjl7anbaea3afninj.xn--p1ai
xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai xn--80ab4aa2g.xn--p1ai
xn----7sbbagbq7bd5aheftfllo4m.xn--p1ai xn--80aeahghtf8ac5i.xn--p1ai
xn----8sbdbjgb1ap7a9c4czbh.xn--p1acf xsfetish.org
xn----ctbigni3aj4h.xn--p1ai yellowstonevisitortours.com
xn--80ajbshivpvn2i.xn--p1ai your-bearings.com
xn--h1ahbi.com.ua yourtemplatefinder.com
y8games-free.com yur-p.ru
yadro.ru zambini.ru
yebocasino.co.za zarabotok--doma.ru
yebocasino.com
yellowproxy.net
yourads.website
youresponsive.com
yourothersite.com
yourporngay.com
yurgorod.ru
zarabiaj-dzis.pl
zastenchivosti.net zastenchivosti.net
zdesoboi.com zdesoboi.com
zeleznobeton.ru zhuravlev.info
zeroredirect12.com zigarettenonlinekaufen2.drupalgardens.com
zeroredirect6.com zigzog.ru
zetgie.com.pl zona-aqua.ru
zigarettenonl.canalblog.com zscaler.net
zoosexart.com zvetki.ru
zytpirwai.net

484
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
1001watch.com.ua 0912701309f8ce.com
1flag.co.za 1-99seo.com
2nt.ru 101lesbian.xyz
4rent.ru 108shot.com
51unlim.ru 202ch.com
5ws.dating-app.ru 2rich4bitches.com
abclauncher.com 4k-player.pl
above.com aa625d84f1587749c1ab011d6f269f7d64.com
absolutelyanalog.com aaa38852e886ac4af1a3cff9b47cab6272.com
adultgalls.com abiente.ru
albamargroup.com actulite.com
allwidewallpapers.com adamoads.com
allwomen.info adcash.com
alveris.ru adf.ly
ampower.me admitad.com
apiallgeniusinfo-a.akamaihd.net advancedmassagebysara.com
apidigidocketnet-a.akamaihd.net aero2.ru
appiq.mobi agapovdg.ru
as5000.com amateurgalls.com
atleticpharm.org anime.dougasouko.com
atmovs.com anti-virus-removal.info
auto-zapchasti.org aosheng-tech.com
avtoarenda.by arcteryxsale.online
avtochehli.by atlasvkusov.ru
baoxaydung.com.vn auto-moto-elektronika.cz
best-games.today bagsonsale.online
bestfortraders.com beclean-nn.ru
betshuckclean.com belstaffstore.online
big-cash.net bensbargains.net
bimatoprost-careprost.com best-seo-software.xyz
bio-optomarket.ru bestssaker.com
bizru.info besttorrentknifta.weebly.com
bjetjt.com biketank.ga
bjgugu.net.ua bmwhighperformers.com
blog.koreadaily.com bongiornos.info
blpmovies.com brainboosting.club
bradleylive.xyz braindaily.xyz
break-the-chains.com businescoop.men
builtwith.com cam-kontakte.org
bus-offer.com cand.jp
callcafe.info candycrushshop.com
camel-beach.com cbozhe.com
carsdined.org cheapjerseysbizwholesale.us
cartierlovebracelet.xyz clickaider.com
cezartabac.ro cntravelre.com
chatroulette.world co.lumb.co
cl.s7.exct.net compliance-jane.top
compliance-nelson.top counter.yadro.ru
concordexoticrentals.com craftburg.ru
construmac.com.mx ddsoldes.fr
copblock.org demenageur.com
cutalltheshit.com detskie-zabavi.ru
cv.wallhade.co dms-sw.ru
czat.wp.pl donvito.unas.cz
daneshetabiat.com downtuptv.gq
darkbooks.org dpihatinh.gov.vn
day-news.info drugspowerstore.com
deutschland123.de drugstoreforyou.com
dignitasdata.se eloconcream.blogspot.com
diminishedvalueoforegon.com es5.com
dinkolove.ya.ru escortplius.com
div.as espaceinventoristes.com
doeco.ru etm-consult.de
domaineaneblanc.com evaashop.ru
donna7753191.ru ewebarticle.info
doxyporno.com expdom.com
dverimegapolis.ru eyes-on-you.ga
ec-file.info fast-torrent.ru
education-cz.ru faz99.com
ekspertmed.com fettefrauen.net
eldiariodeguadalajara.com fil.ru
elitedollars.com findthe.pet
eloxal.ru fingerprintjs.com
eshop.md flash4fun.com
euronis-free.com flashlarevista.com
everypony.ru floridamhca.org
everytalk.tv forex-procto.ru
exci.ru fortwosmartcar.pw
extremal-blog.com forums.d2jsp.org
fapgon.com fullgirl.ru
fast-wordpress-start.com gael-s.ru
fireads.men game-mmorpg.net
fishtauto.ru gameprimary.com
fiuxy.com gaypornmovie.net
flashbannernow.com gdcentre.ru
foodcrafts.website gepezz.info
forex-indextop20.ru
forexgb.ru
fortunejack.com
fotoxxxru.com
freakycheats.com
freeseedsonline.com
freshberry.com.ua
fungamelands.com
gerhardhealer.com
girlsfuckdick.com girlsfuckdick.com
groupmoney.ru gk-atlant.info
gwagka.com golden-praga.ru
gz2.bbsoldes.fr googglet.com
hawaiielectriclight.com googlefeud.com
help.tpu.ru goosefishpost.bid
heroz.fr green-tea.tv
guge.io
gwebtools.com
headpharmacy.com
himazin.info
histats.com
hitcpm.com
host-protection.com
hosting-tracker.com
hostingclub.lk
hotchatdate.com
hyipmanager.in
i-midias.net.br
iblogpress.xyz
ibmdatamanagement.co
iccornacircri.cf iccornacircri.cf
ifirestarter.ru igtools.club
igrovyeavtomaty777.ru ihc2015.info
insurple.com iminent.com
intim-uslugi.info indetiske.ya.ru
investzalog.ru infobanks.ru
iqupdatetmz.win irkutsk.zrus.org
ispac.org istripper.com
itservicesthatworkforyou.com it-max.com.ua
iusstf.org ivearchenceinflu.cf
jasonpartington.com kaliningrad.zrus.org
jikoman.info kemerovo.zrus.org
junketjuice.blogspot.com keywordbasket.com
justbcause.com kinomaniatv.pl
jwss.cc kvartiry-remont.ucoz.ru
kaidalibor.de ledpolice.ru
kandidos.com letslowbefast.today
kiwi237au.tk linkpulse.com
kleine-titten.biz listiseltemournlan.gq
knigonosha.net loginduepunti.it
kredit-blog.ru lombardfinder.ru
kursy-ege.ru lovi-moment.com.ua
landoftracking.com lulea-auktionsverk.se
laxob.com mabdoola.blogspot.com
lazymanyoga.com marketingtechniques.info
lignofix.ua masthopehomes.com
likesuccess.com mediaoffers.click
linkrr.com megahdporno.net
lionking-1994.blogspot.com mejoresfotos.eu
m3gadownload.pl mikrobiologies.ru
makis.nu minyetki.ru
manualterap.roleforum.ru moncleroutletonline.win
maranbrinfo.com.br musicktab.com
matsdale.com my-floor.in.ua
maxthon.com mydownlodablefiles.com
medanestesia.ru newstraveller.ru
medkritika.ru nordstar.pro
megaindex.ru noumeda.com
mitrasound.ru o-o-8-o-o.com
mlvc4zzw.space openfrost.com
moneyzzz.ru outshop.ru
mrwhite.biz owathemes.com
musezone.ru ozas.net
muzaporn.com pagesense.com
mydirtystuff.com partner-stop.men
myphotopipe.com pastaleads.com
nastroyke.net pdns.cz
negociosdasha.com perfectpracticeweb.com
newyorkhotelsmotels.info perosan.com
nfljerseys.online pigrafix.at
norththeface.store pornokajf.com
ntic.fr poshiv-chehol.ru
oecnhs.info princevc.com
ofanda.com
onemactrckr.com
online-hd.pl
ons-add.men
onthemarch.co
ostroike.org
panicatack.com
pcimforum.com
penisvergrotendepillennl.ovh
pertlocogasilk.tk
photokitchendesign.com
pochtovyi-index.ru
podshipniki-nsk.ru
porno-play.net
preconnubial.usuby.site
privatamateure.com
privhosting.com
pro-okis.ru pro-okis.ru
prostitutki-astana.org putitin.me
rednise.com qpypcx.com
registry-clean-up.net rating-bestcasino.com
relatodelpresente.com.ar reckonstat.info
relax.ru replica-watch.ru
retreatia.com rezeptiblud.ru
ring4rhino.com rollercoin.com
roofers.org.uk royalvegascasino.com
sbornik-zakonov.ru rucrypt.com
scanner-jane.top rvzr-a.akamaihd.net
seminarykansas904.ml saletool.ru
serpstat.com samara.rosfirm.ru
sex-dating.co samoiedo.it
sexflirtbook.com sandhillsonline.com
shopfishing.com.ua scanner-fred.top
sibecoprom.ru scanner-jack.top
sk.golden-praga.ru search.alot.com
skincrate.net seeingmeerkat.com
slomm.ru seoanalyses.com
slonechka.ru seocheki.net
sloopyjoes.com serialsway.ucoz.ru
sluganarodu.ru sexfreepornoxxx.com
snip.to sexpartygirls.net
social-button.xyz sexreliz.com
social-fun.ru sexuria.net
sotkal.lark.ru shariki-zuma-lines.ru
sovinsteel.ru sideeffectsoftizanidine.blogspot.com
space-worry.ml sitechecker.pro
spravka-medosmotr.ru sizeplus.work
start.myplaycity.com social-search.me
staticfs.host softwaretrend.net
solinf.co
solution4u.com
songplanet.ru
spabali.org
stockquotes.wooeb.com
strana-krasoty.ru strana-krasoty.ru
studiofmp.com stroimajor.ru
sumo.com studworks.org
superlist.biz
supermodni.com.ua
superoboi.com.ua superoboi.com.ua
supervesti.ru surfbuyermac.com
teencastingporn.com takeprofitsystem.com
tizanidinefibromyalgia.blogspot.com telegraf.by
tizanidinestreetvalue.blogspot.com tizanidineformigraines.blogspot.com
tizanidinewithdrawal.blogspot.com tizanidinehcl2mgsideeffects.blogspot.com
track112.site tizanidineshowupondrugtest.blogspot.com
traffic2cash.org tizanidinesideeffectsweightloss.blogspot.com
tri-slona.org top-karkas.ru
ttrraacckkrr.com track-rankings.online
uchetunet.su trafaret74.ru
udsgame.online traffic100.com
urll.eu trahic.ru
ussearche.cf trainoffend.ml
usswrite.com truck-rece.life
tuckpointingmasonrysystems.com
twu.com.ua
uggsale.online
ukkala.xyz
ultramart.biz
um-razum.ru
unitygame3d.com
urengoy.pro
ustion.ru
utiblog.fr
vancleefreplica.pw
vapomnoncri.tk
vavilone.com vavilone.com
velpanex.ru visa-pasport.ru
venerologiya.com vonradio.com
vezuviy.su wareseeker.com
via-gra.webstarts.com watchinf.com
videochat.mx wdfdocando.com
vielporno.net webalan.ru
visionwell.com.cn weblibrary.win
walkme.com whereverdesperate.gq
warning.or.kr wholesalenfljerseys.us.com
website-analyzer.info wjgony.com
westsextube.com
wiosenny-bon-1500.pl
wnoz.de
worldhistory.biz
worldtraveler.world worldtraveler.world
x-lime.com writersgroup580.web.fc2.com
x-porno.video xn-----7kceclhb4abre1b4a0ccl2fxch1a.xn--p1ai
xblognetwork.com
xfluro.com
xn----7sbaaabaei0cc8aj5bj0bncejx.xn--p1ai
xn----9sbebi2bvzr7h.xn--p1ai xn----9sbebi2bvzr7h.xn--p1ai
xn----ctbbcjd3dbsehgi.xn--p1ai xn----btbdvdh4aafrfciljm6k.xn--p1ai
xn--80ak6aa92e.com youdao.com
xn--d1abj0abs9d.in.ua zajm-zalog-krasnodar.ru
xxxdatinglocal.us zigarettenonlinekaufen2.drupalgardens.com
zaloro.com zoompegs.com
zhuravlev.info zvezdagedon.ru

10
dev-tools/test_units/ratelimittest-beta.txt
View file

@ -581,14 +581,4 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html> </html>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.86 0; 10.1.1.152 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.07.3251 ### Version: V4.2022.07.3252
### Updated: Tue Jul 26 10:03:55 UTC 2022 ### Updated: Tue Jul 26 22:01:20 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.07.3251 ### Version: V4.2022.07.3252
### Updated: Tue Jul 26 10:03:55 UTC 2022 ### Updated: Tue Jul 26 22:01:20 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##