vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-03 19:20:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.07.3240 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-07-21 10:09:17 +00:00
parent 4cbec1e107
commit e68af16ce1
48 changed files with 98489 additions and 98509 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.07.3239 #### Version: V4.2022.07.3240
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.07.3239 #### Version: V4.2022.07.3240
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.07.3239 #### Version: V4.2022.07.3240
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12174
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.07.3239 #### Version: V4.2022.07.3240
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12178
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12178
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

12178
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12178
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 1; 10.1.0.237 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 0; 10.1.0.237 0;
127.0.0.1 0; 127.0.0.1 0;

12178
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 1; 10.1.0.237 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 0; 10.1.0.237 0;
127.0.0.1 0; 127.0.0.1 0;

12178
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 1; 10.1.0.237 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 0; 10.1.0.237 0;
127.0.0.1 0; 127.0.0.1 0;

12178
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3239 3240

12178
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

12178
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.38.8.115 0; 52.186.140.140 0;

12178
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 1; 10.1.0.237 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 0; 10.1.0.237 0;
127.0.0.1 0; 127.0.0.1 0;

12178
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 1; 10.1.0.237 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 0; 10.1.0.237 0;
127.0.0.1 0; 127.0.0.1 0;

12178
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 1; 10.1.0.237 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 0; 10.1.0.237 0;
127.0.0.1 0; 127.0.0.1 0;

12178
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.38.8.115 0; 52.186.140.140 0;

12178
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.38.8.115 0; 52.186.140.140 0;

12178
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 1; 10.1.0.237 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
DittoSpyder Aboundex
Fyrebot BBBike
Lmspider Craftbot
ProWebWalker CrunchBot
RankingBot2 Havij
SeoSiteCheckup LeechFTP
Web Collage Nameprotect
WebImageCollector OpenVAS
WebReaper PleaseCrawl
Website Quester Web Enhancer

310
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
360Spider 404checker
404enemy 404enemy
80legs ADmantX
ASPSeek ALittle Client
Aboundexbot Abonti
AhrefsBot AllSubmitter
AiHitBot Alligator
Anarchie Anarchy99
Ankit
Aspiegel Aspiegel
Atomseobot Atomseobot
AwarioRssBot Attach
AwarioSmartBot BDCbot
BDFetch BLEXBot
BackDoorBot BackWeb
Backlink-Ceck
Barkrowler Barkrowler
Battleztar Bazinga Bigfoot
Black Hole Blow
Boardreader Boardreader
Bolt
BotALot
Brandprotect Brandprotect
Buck Brandwatch
CazoodleBot BuiltBotTough
CensysInspect BunnySlippers
CCBot
Calculon
Cegbfeieh
CheeseBot
Claritybot Claritybot
Cliqzbot Cliqzbot
Copier Cloud mapping
CopyRightCheck Cogentbot
Collector
Copyscape Copyscape
Craftbot Crawling at Home Project
Crescent Crescent
Curious CyotekWebCopy
DBLBot DBLBot
DIIbot DIIbot
DTS Agent
DatabaseDriverMysqli DatabaseDriverMysqli
Digincore Devil
Discobot Discobot
Discoverybot Dispatch
DittoSpyder DittoSpyder
DnBCrawler-Analytics DnBCrawler-Analytics
DnyzBot DnyzBot
DomainAppender
DomainSigmaCrawler
DomainStatsBot DomainStatsBot
Domains Project Domains Project
Dotbot Download Wonder
EMail Siphon Dragonfly
EirGrabber Drip
Express WebPictures ECCP/1.0
ExtLinksBot EMail Wolf
EasyDL
Ecxi
Evil
Extractor Extractor
Extreme Picture Finder ExtractorPro
FDM EyeNetIE
Fimap Ezooms
Firefox/7.0 FHscan
FlashGet FemtosearchBot
Foobot Foobot
G-i-g-a-b-o-t FyberSpider
GetRight Fyrebot
GT::WWW
GalaxyBot
GermCrawler
GetWeb GetWeb
Getintent Gigabot
Go-Ahead-Got-It Go-Ahead-Got-It
GoZilla GoZilla
Grabber Gotit
GrapeFX GrapeshotCrawler
GridBot
HEADMasterSEO HEADMasterSEO
HMView
HTMLparser HTMLparser
HTTP::Lite HTTP::Lite
Haansoft
HaosouSpider HaosouSpider
Harvest
Heritrix Heritrix
HybridBot HybridBot
Iblog IDBot
IRLbot
Id-search Id-search
Image Fetch
Image Sucker Image Sucker
Indy Library Indy Library
InfoTekies InfoTekies
Intelliseek InterGET
InternetSeer Internet Ninja
JetCar JamesBOT
JennyBot
Jetty Jetty
JikeSpider JikeSpider
Joomla Kenjin Spider
Keybot Translation-Search-Machine LNSpiderguy
Kozmosbot
LWP::Simple LWP::Simple
Lftp Lanshanbot
Libwhisker Leap
LexiBot
LibWeb
LieBaoFast LieBaoFast
LinkWalker Likse
LinkScan
Linkbot
LinkextractorPro LinkextractorPro
LinksManager
LinqiaMetadataDownloaderBot LinqiaMetadataDownloaderBot
LinqiaRSSBot
LinqiaScrapeBot LinqiaScrapeBot
Lipperhey
Litemage_walker Litemage_walker
Lmspider Lmspider
Ltx71 MFC_Tear_Sample
MIDown tool
MIIxpc
MJ12bot
MQQBrowser MQQBrowser
Magnet MSFrontPage
Majestic-SEO
Mass Downloader
MauiBot
MeanPath Bot
Meanpathbot Meanpathbot
Mediatoolkitbot
MegaIndex.ru MegaIndex.ru
Metauri Metauri
MicroMessenger Microsoft URL Control
Microsoft Data Access
Minefield
Mister PiX
Morfeus Fucking Scanner
Mozlila Mozlila
Musobot
NICErsPRO NICErsPRO
Nameprotect NPbot
Navroad Name Intelligence
Nessus Nessus
NetSpider Netcraft
Nettrack
Netvibes Netvibes
NextGenSearchBot Nikto
Nibbler NimbleCrawler
Nimbostratus
Ninja Ninja
Nmap Nmap
OrangeSpider Not
Nuclei
Offline Explorer
Offline Navigator
OrangeBot
OutfoxBot OutfoxBot
PECL::HTTP POE-Component-Client-HTTP
PHPCrawl
PageScorer PageScorer
PageThing.com
Panscient
Pi-Monster Pi-Monster
Picscout Picscout
Picsearch PictureFinder
Piepmatz Pixray
Pockey
ProPowerBot ProPowerBot
RSSingBot ProWebWalker
RankActiveLinkBot Psbot
RankFlex Pump
RankingBot RankActive
Re-re RankingBot2
ReGet
RealDownload
RebelMouse
Recorder Recorder
RepoMonkey RepoMonkey
Ripper RocketCrawler
Rogerbot SEOkicks
SEOlyticsCrawler SMTBot
SEOprofiler
SISTRIX
SalesIntelligent
ScoutJet ScoutJet
Screaming
ScrepyBot
Searchestate Searchestate
Seekport Semrush
SiteCheckerBotCrawler SemrushBot
SiteSucker SentiBot
Siteimprove Siphon
Site Sucker
Sitebeam
Sitevigil Sitevigil
SlySearch SlySearch
SmartDownload Snoopy
Sociscraper SocialRankIOBot
Sogou web spider SpaceBison
SpankBot Spammen
Spbot
Spinn3r
SputnikBot SputnikBot
Sqlmap
Sqlworm
Sqworm Sqworm
Stripper Sucker
SuperBot SuperBot
Surfbot Suzuran
SurveyBot Swiftbot
T0PHackTeam
T8Abot
Teleport Teleport
TeleportPro TeleportPro
Telesoft
Telesphoreo
Telesphorep
The Intraformant
TheNomad
TightTwatBot
Titan Titan
Tracemyfile TurnitinBot
True_Robot
Turnitin
TwengaBot TwengaBot
Twice
Typhoeus
URLy Warning
UnisterBot UnisterBot
Upflow
V-BOT
VCI VCI
Vacuum Vacuum
Vagabondo
VelenPublicWebCrawler
VeriCiteCrawler VeriCiteCrawler
Virusdie WWWOFFLE
VoidEYE WallpapersHD
Voltron Web Fuck
WBSearchBot
WWW-Collector-E
WWW::Mechanize
Wallpapers
WeSEE
Web Sauger
Web Sucker Web Sucker
WebBandit WebAuto
WebCopier WebCopier
WebFetch
WebFuck WebFuck
WebImageCollector WebGo IS
WebReaper WebSucker
WebStripper WebWhacker
WebmasterWorldForumBot Webalta
Webster
Whacker Whacker
Whatweb
WinHTTrack
WiseGuys Robot
Wonderbot Wonderbot
Woobot
Wotbox
Wprecon
Zade Zade
Zauba
Zermelo Zermelo
Zeus
Zitebot Zitebot
ZmEu ZoomBot
ZoominfoBot ZoominfoBot
ZyBorg
adscanner adscanner
archive.org_bot archive.org_bot
arquivo-web-crawler arquivo-web-crawler
arquivo.pt arquivo.pt
autoemailspider
cah.io.community cah.io.community
check1.exe coccocbot
clark-crawler
crawler.feedback
crawler4j crawler4j
dataforseo.com dataforseo.com
demandbase-bot eCatch
facebookscraper instabid
heritrix isitwp.com
internetVista monitor
ips-agent
lwp-request lwp-request
magpie-crawler lwp-trivial
meanpathbot meanpathbot
plumanalytics page scorer
scalaj-http polaris version
ripz
seobility
seoscanners seoscanners
seostar sysscan
sitechecker.pro tAkeOut
spyfu
trendiction.com
trendiction.de trendiction.de
webgains-bot ubermetrics-technologies.com
webprosbot voyagerx.com
zgrab

316
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
360Spider
404checker 404checker
404enemy 80legs
ADmantX Abonti
ASPSeek Acunetix
Aboundex
Aboundexbot
AfD-Verbotsverfahren AfD-Verbotsverfahren
AhrefsBot Alexibot
Aipbot Anarchie
AllSubmitter Anarchy
AlphaBot
Anarchy99 Anarchy99
Ankit Anthill
Apexoo
Aspiegel Aspiegel
Asterias Atomseobot
BackDoorBot Attach
BackWeb AwarioSmartBot
Backlink-Ceck BDCbot
Badass BLEXBot
BackStreet
Bandit
BatchFTP BatchFTP
Battleztar Bazinga
BetaBot BetaBot
Bitacle
BlackWidow
Blow
BlowFish BlowFish
Boardreader Boardreader
Brandprotect Bolt
Brandwatch
Buck
Buddy Buddy
BuiltWith BuiltBotTough
BunnySlippers
BuzzSumo BuzzSumo
CCBot
Calculon
Cegbfeieh Cegbfeieh
CensysInspect CheeseBot
CherryPicker
ChinaClaw ChinaClaw
Claritybot Claritybot
Cogentbot Cocolyzebot
Copier Collector
CopyRightCheck
Copyscape
Cosmos Cosmos
CrazyWebCrawler CrazyWebCrawler
Crescent Crescent
CrunchBot
Curious Curious
Demon Custo
Deusu DBLBot
DigitalPebble DIIbot
DataCha0s
Digincore
Dirbuster
Disco Disco
Discobot Discobot
DnBCrawler-Analytics Discoverybot
Download Wonder Dispatch
DomCopBot
DomainAppender
Dotbot
Dragonfly Dragonfly
EMail Siphon ECCP/1.0
EasyDL EMail Wolf
Ecxi Ebingbong
ExtLinksBot EroCrawler
ExtractorPro Evil
Ezooms Express WebPictures
Extractor
Extreme Picture Finder
FDM FDM
FlashGet FemtosearchBot
FyberSpider Fimap
Fyrebot Firefox/7.0
GermCrawler Foobot
Getintent GT::WWW
Gigabot Genieo
Go-Ahead-Got-It
GoZilla
Gotit Gotit
Grafula GrabNet
GrapeshotCrawler GridBot
HEADMasterSEO HEADMasterSEO
HMView HMView
HTMLparser HTMLparser
HTTP::Lite HTTrack
Haansoft
HaosouSpider
Harvest
Heritrix
Hloader Hloader
HonoluluBot IDBTE4M
Humanlinks IRLbot
HybridBot Iblog
Id-search IlseBot
Image Sucker
IndeedBot IndeedBot
Indy Library InterGET
InfoNaviRobot InternetSeer
Internet Ninja Iskanie
Iria
IstellaBot
Jbrofuzz
Jetty Jetty
JikeSpider JikeSpider
JustView JustView
Jyxobot Kenjin Spider
Keybot Translation-Search-Machine
Keyword Density Keyword Density
LWP::Simple Kinza
Lanshanbot LNSpiderguy
Larbin LieBaoFast
Leap Likse
LeechFTP
LeechGet
Lightspeedsystems
Linkbot Linkbot
LinqiaMetadataDownloaderBot LinqiaMetadataDownloaderBot
LinqiaRSSBot LinqiaRSSBot
LinqiaScrapeBot Litemage_walker
Lipperhey MFC_Tear_Sample
MIIxpc
MJ12bot
MQQBrowser
Magnet
Mail.RU_Bot Mail.RU_Bot
Majestic SEO
Majestic-SEO
Majestic12
MarkWatch
Mass Downloader
Masscan Masscan
Mediatoolkitbot Mata Hari
MicroMessenger MauiBot
Meanpathbot
Microsoft Data Access Microsoft Data Access
Microsoft URL Control
Moblie Safari Moblie Safari
Name Intelligence Morfeus Fucking Scanner
Mozlila
Mr.4x3
Msrabot
Musobot
NPbot
Nameprotect
NearSite NearSite
Needle
NetAnts NetAnts
NetSpider NetLyzer
NetZIP Nibbler
Nettrack
NimbleCrawler
Nimbostratus
Ninja
Not Not
Nuclei Nuclei
Octopus
Offline Explorer
Offline Navigator Offline Navigator
OnCrawl Openvas
OpenLinkProfiler OrangeBot
OrangeSpider OutfoxBot
PECL::HTTP PECL::HTTP
PageAnalyzer PageAnalyzer
PageGrabber
PageThing.com PageThing.com
Pandalytics Pandalytics
Pavuk Petalbot
Pi-Monster
Picsearch Picsearch
ProPowerBot PictureFinder
Pimonster
Pockey
Psbot Psbot
RankActive Pump
RankFlex Quick-Crawler
RSSingBot
RankActiveLinkBot
RankingBot2 RankingBot2
RankurBot RankurBot
ReGet ReGet
RedesScrapy RealDownload
SEOkicks RepoMonkey
SEOlyticsCrawler RocketCrawler
SBIder
SEOkicks-Robot
SEOprofiler SEOprofiler
SMTBot Scanbot
SalesIntelligent ScreenerBot
ScoutJet ScrepyBot
Scrapy
Searchestate Searchestate
SearchmetricsBot SearchmetricsBot
Seekport
SemanticJuice SemanticJuice
Semrush
SemrushBot SemrushBot
SentiBot SentiBot
SeoSiteCheckup
Siphon
Site Sucker
SiteCheckerBotCrawler
Sitebeam Sitebeam
Siteimprove
SlySearch SlySearch
SmartDownload Snapbot
Snoopy Snoopy
SocialRankIOBot SocialRankIOBot
Sosospider Sogou web spider
Sottopop Sottopop
SpaceBison
Spammen Spammen
SpankBot
SputnikBot
Sqworm Sqworm
Steeler Steeler
Stripper Sucker
Surfbot SurveyBot
Suzuran Szukacz
Swiftbot
T8Abot T8Abot
Teleport Telesoft
TeleportPro
Telesphorep Telesphorep
The Intraformant The Intraformant
TheNomad
Titan Titan
Toata Toata
Toweyabot Toweyabot
True_Robot
Turingos Turingos
UnisterBot Typhoeus
V-BOT Upflow
VB Project Vacuum
VelenPublicWebCrawler
VeriCiteCrawler VeriCiteCrawler
VidibleScraper WEBDAV
Virusdie
WBSearchBot
WPScan WPScan
WWWOFFLE WWW-Mechanize
Wallpapers/3.0 WWW::Mechanize
WallpapersHD WeSEE
Web Auto Web Enhancer
Web Sucker Web Fuck
WebBandit WebEnhancer
WebCollage WebSauger
WebCopier
WebFuck
WebGo IS
WebImageCollector
WebPix
WebStripper WebStripper
WebWhacker WebWhacker
WebZIP WebZIP
Webalta
WebmasterWorldForumBot
Webshag
Website Quester Website Quester
Whack WebsiteQuester
Whacker
Who.is Bot Who.is Bot
WinHTTrack
Woobot
Wotbox Wotbox
Wprecon Wprecon
YoudaoBot Zermelo
Zauba ZoomBot
Zitebot
ZumBot ZumBot
archive.org_bot arquivo.pt
autoemailspider backlink-check
clark-crawler check1.exe
com.plumanalytics coccocbot
crawl.sogou.com cognitiveseo
crawler4j crawler4j
demandbase-bot
eCatch
evc-batch evc-batch
facebookscraper
instabid
internetVista monitor internetVista monitor
meanpathbot ips-agent
linkdexbot
lwp-request
magpie-crawler
muhstik-scan muhstik-scan
page scorer
pcBrowser
plumanalytics plumanalytics
polaris version ripz
satoristudio.net
scalaj-http scalaj-http
scan.lol
seocompany.store
seoscanners
seostar seostar
serpstatbot sitechecker.pro
sogouspider
trendiction.com trendiction.com
ubermetrics-technologies.com
webgains-bot webgains-bot
webmeup-crawler x09Mozilla
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
amehdaily.com adtology.com
cmd.kz classiquebijoux.ru
dollartree.info eropornosex.ru
filmci.pro freemags.cc
get-seo-domain.com iphantom.com
sportbetfair.com job.icivil.ir
vielporno.net mobifunapp.weebly.com
webadvance.club vod.com.ua
womama.ru wurr.voila.net
xxxtubesafari.com xblog.in

932
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
0912701309f8ce.com 12u.info
101lesbian.xyz 23kw.ru
101raccoon.ru 3w24.com
123any.com 40cg.com
1314dh.com 4istoshop.com
202ch.com 4rent.ru
2345.com 4replicawatch.net
4webmasters.com 5tey7463.weebly.com
6128786.com 7a2rnuey1tw9ar.ru
7ooo.ru 7fon.ru
83net.jp 991mostfm.co.id
9med.net a1security.com.ua
a96527gi.beget.tech aa14ab57a3339c4064bd9ae6fad7495b5f.com
aa9d046aab36af4ff182f097f840430d51.com
aarbur.com
abcdeg.xyz abcdeg.xyz
actionnooz.com adsref.men
actulite.com adviceforum.info
adclickthru.net air-edem.ru
adsloads.com airmaxshoes-2016.com
adult3dgames.com alborzan.com
advancedmassagebysara.com allblogroll.com
advancedsoftwaresupport.com allmarketsnewdayli.gdn
afftrack001.com allpdfmags.net
agecheckadult.com aloofly.com
ahrefs.com alpharma.net
albamargroup.com aludecor.info
alekseevec.ru anniemation.com
alfabot.xyz apartamentwroclaw.eu
allbizne.co.ua apimossnetinfo-a.akamaihd.net
allesohnegirls.net appearance-cool.com
alyeskaresort.com apptester.tk
ameblo.jp aquarium-pro.ru
an-donut.com arcteryxstore.online
android-style.com atdedinotuho.tk
animenime.ru aviav.co
apiadanaknet-a.akamaihd.net aviva-limoux.com
approvedlocal.co.za avon-ukraine.com
aramaicmedia.org ayanaspa.com
asdfz.pro azazaporn.com
asrv-a.akamaihd.net azazu.ru
atout-energie-69.com azbukadiets.ru
autonew.biz babieca.com
aviapanda.ru babs.com.ua
awaybird.ru bankiem.pl
bad-stars.net bankofthewext.com
baersaratov.ru bausparen.bz.it
baladur.ru best-coupon-offer.com
banksrf.ru bestdooz.com
battle.net bestmarriages.com
beauty-things.com bestwaystogetpaid.us
begalka.xyz bigcareer.info
best-games.today bigcities.org
bestkfiledxd.cf bin-brokers.com
bestwebsitesawards.com binomo.kz
betune.onlinewebshop.net bio.trade-jp.net
bget.ru biographiya.com
bidr.trellian.com bizlist.com.de
bigpictures.club blockety.co
billiard-classic.com.ua blog.f00kclan.de
bioca.org bnt-team.com
bioscorp.ru boltalko.xyz
birzha-truda.eu bombla.org
blackbabesporn.com boole.onlinewebshop.net
blackcurranthumidifiers.site brains2.biz
blackplanet.com brandedleadgeneration.com
blacktwhite.com brandov.ru
bloggerads.net buigas.00it.com
bloke.com buy-cheap-online.info
bonzibuddi.com buzzzg.men
bouda.kvalitne.cz bvlgariring.xyz
brothers-smaller.ru cactussoft.biz
burkesales.com canacopegdl.com
bus-offer.com carfax.com.ua
businesxxl.com carsdined.org
buy-forum.ru cartierlovebracelet.xyz
calcularpagerank.com caulderwoodkennels.com
candycrushshop.com cdncash.org
cannibalcheerleader.com centre-indigo.org.ua
carloans.com cezartabac.ro
carrauterie.be chatseo.com
cartierlovebraceletreplica.xyz chiblackhawks-jerseys.com
cashkitten-a.akamaihd.net citizenclsdriveri7.pen.io
cayado.snn.gr
celebsopics.com
cementaresearch.se
centrdebut.ru
cheapjerseysfootballshop.com
check-host.net
chorus.terakeet.com
cialiswithoutadoctor.net
clarithromycin500mg.com
cleansearch.net
clevernt.com clevernt.com
clickaider.com clickzzs.nl
cmhomestayagency.com clipartnew.com
commoncrawl.org club-lukojl.ru
communisave.co.za coindirect.io
compliance-brian.xyz coinsspb.com
com-onlinesupport.site
compliance-donald.xyz
compliance-george.top
computernetworksonline.com
connectingsingles.com
cookie-law-enforcement-ee.xyz cookie-law-enforcement-ee.xyz
coolingoods.com
coolwallpapers-hd.com coolwallpapers-hd.com
cottageofgrace.com copblock.org
counterbot.com copypaste.traderzplanet.in
covetnica.com cornomase.win
counter.bloke.com
crystalslot.com crystalslot.com
cscwtalkto.site cubook.supernew.org
cupday.com czeshop.info
custom-product-labels.com dalavia.ru
customerpromos-a.akamaihd.net darkages.info
d0t.ru day-news.info
d2jsp.org daymusam.com
dame-ns.kz delta-line.men
daptravel.com demon-tweeks.com
db.speedup.gdn detective01.ru
dealitright.click docstoc.com
defenderxtactical.com doctormakarova.ru
den-noch24.ru dojki-hd.com
design-ideas.info dominterior.org
detalizaciya-tut.biz downloader12.ru
dev78.dioniqlabb.se drunkenstepfather.com
diarioaconcagua.com e-biznes.info
doctissimo.fr e-collantes.com
domaincrawler.com e-poker-2005.com
domainsatcost.com eachdayisagift.review
donvito.unas.cz ee77ee.com
doreenblog.online elektrischezigarette2.onsugar.com
drugs-no-rx.info elidelcream.weebly.com
drunkmoms.net
drupa.com
dtnlyss.com
e2click.com
e90post.com
easycommerce.cf
edudocs.net
elektrischezigarette2.devhub.com
elektronischezigarettekaufen1.beeplog.com
elektronischezigarettekaufen1.myblog.de
elektrozigarette2.wordpressy.pl
elektrozigarettekaufen1.devhub.com
elektrozigaretten2.yn.lt
elexies.info
elitedollars.com
elmifarhangi.com elmifarhangi.com
enginebay.ru emailaccountlogin.co
englate.com envaseslotusama.com
etm-consult.de epngo.bz
exoneration-project.us europages.com.ru
eyemagination.com evogarage.com
expdom.com
exportshop.us
extlinks.com
extrabot.com
ezigarettekaufen1.hpage.com
ezigarettekaufen2.yolasite.com ezigarettekaufen2.yolasite.com
f5mtrack.com facialporntube.com
falopicm.pw fakehandbags.xyz
familyholiday.ml familyphysician.ru
feargames.ru fickkontaktehobbyhuren.com
femmesdenudees.com fikasound.tk
feorina.ru filmfanatic.com
ferretsoft.com filmi-onlain.info
fettefrauen.net
finansov.info finansov.info
findpik.com findthe.pet
finstroy.net flashlarevista.com
fitness-video.net floridahuntingfishingadventures.com
florida-tourism.net forum69.info
folowsite.com free-share-buttons.com
forminecrafters.ru free411games.com
formulaantiuban.com fuckbuddybestgilf.info
foto-sisek.porngalleries.top fugarif.ga
franklinfire.co fyxabomiw.ru
frankofficial.ru gabeshop.ru
free-gluten.ru galaxy-family.ru
free-share-buttons.top game-top.su
free-social-buttons.xyz gamebackyard.com
free-stock-illustration.com
fuckmill.com
fungirlsgames.net
fxund.us
galeria-zdjec.com
gamerextra.com
gamevalue7.weebly.com
gamewrath.com
gaygalls.net
gazoblok.net.ua
gdcentre.ru
ge0ip.com
ge0ip.net
gearcraft.us gearcraft.us
genericlowlatencyasiodriverhq.aircus.com genericlowlatencyasiodriverhq.aircus.com
gettpromos.com genericviagrasildenafiled.net
girls-ufa.ru gidonline.one
go2album.com girlgamerdaily.com
goodhumor24.com girlspicsa.com
googlsucks.com given2.com
gopro-online.info gkvector.ru
gotomontenegro.net glogow.pl
gpms.org.my golaya.pw
green-tea.tv goldpanningtools.com
gwagka.com golfresa.lucania.se
gwebtools.com.br gombita.info
haikuware.com googlefeud.com
haveinc.xyz gq-catalog.gq
health-medical-portal.info greamimgo.pw
hentaiheroes.com grtyi.com
hermesbelts.xyz gsou.cf
heygidday.biz hamilton.ca
hifidesign.ru handicapvantoday.com
hkladys.com hasfun.com
home-task.com hausfrauensex18.com
homedecorpicture.us hccoder.info
homemature.net hidefiles.org
hoporno.com historichometeam.com
hotblog.top hollywoodactress.info
hotelcrocenzi.sm hotgirlhdwallpaper.com
howmuchdoestizanidinecost.blogspot.com hotshoppymac.com
https-legalrc.biz housekuba.org
hyhj.info houseofgaga.ru
idegenvezeto.eu hoverboard360.at
igadgetsworld.com hoverboard360.se
igtools.club hreade.com
huhn.altervista.org
humanelydrew.com
hyipmanager.in
iamsport.org
ibb.com.ua
iccornacircri.cf
iklysha.ml iklysha.ml
iloveitaly.ru ilmen.net
images.gyffu.com ilovevitaly.xyz
imk.com.ua imgarcade.com
impisr.edunsk.ru immigrational.info
infowarcraft.ru intelhdgraphicsgtdrive6w.metroblog.com
intellektmedia.at intervsem.ru
itag.pw intimshop-fantasy.ru
ivoiretechnocom.ci itis4you.com
iweblist.info ivanovo.zrus.org
izismile.com jamiembrown.com
jav-fetish.site janavibekken.no
japfm.com
javlibrary.cc javlibrary.cc
jbl-charge.info
jerseysbizwholesalecheap.com
jerseywholesalebizchina.com
jetsli.de
justbcause.com justbcause.com
kakadu-interior.com.ua justucalling32211123456789.tk
kamorel.com kam-dom.ru
kareliatobacco.ru katushka.net
katjimej.blog.fc2.com kemerovo.zrus.org
kerch.site keywordhouse.com
kinopolet.net kino-rf.ru
kmgamex.cf kinostorm.net
knigonosha.net klitimg.pw
kollesa.ru kollekcioner.ru
komputernaya-pomosh-moscow.ru kostenloser-sex.com
kongruan.com kuzinsp.ru
konkursowo-24.pl lacapilla.info
koversite.info landscaping.center
lawyers.cafe larutti.ru
layola.biz.tc layola.biz.tc
letolove.ru lazymanyoga.com
light.ifmo.ru lego4x4.ru
linkrr.com letto.by
listiseltemournlan.gq lexixxx.com
liver-chintai.org lion.misba.us
livetsomudvekslingstudent.bloggersdelight.dk lombardfinder.ru
loadopia.com lombia.com
loginduepunti.it lonerangergames.com
lottospring.com lookingglassemb.com
lovi-moment.com.ua lordzfilmz.me
m1media.net lubetube.com
m4ever.net lyrics.home-task.com
mackeeper-center.club mainhp.net
mapquestz.us mansparskats.com
martlinker.com manualterap.roleforum.ru
medanestesia.ru marketland.ml
medicine-4u.org matchpal-a.akamaihd.net
medkritika.ru mature.free-websites.com
megaapteka.ru mcar.in.ua
mere.host.sk mebel-vstroika.ru
mesbuta.info mebelcomplekt.ru
med-zdorovie.com.ua
medmajor.ru
meendo-free-traffic.ga
meltwater.com
memberty.com
menstennisforums.com
merryhouse.co.uk
metascephe.com metascephe.com
mhtr.be mezaruk.info
middlerush-a.akamaihd.net mikozstop.com
mindbox.co.za miniature.io
mini-modus.ru mirtorrent.net
minneapoliscopiers.com misswell.net
moncleronline.xyz mixx.com
monetizationking.net mmgq.ru
mp3downloadhq.com mmofreegames.online
mmostrike.ru
mnogabukaff.net
mobifunapp.weebly.com
modnie-futbolki.net
mojpregled.com
mojpreskumanie.com
mompussy.net
monarhs.info
money-every-day.com
moneymaster.ru
moneyzzz.ru
more-letom.ru
most.gov.iq
motherboard.vice.com
moyakuhnia.ru
mrbitsandbytes.com mrbitsandbytes.com
my-aladin.com mtmtv.info
mydirtystuff.com musicdaddy.net
musicvidz.ru
myblogregistercm.tk
mydearest.co
mymercy.info
myplaycity.com myplaycity.com
nastroyke.net nailsimg.com
natali-forex.com nalogovyy-kodeks.ru
narkologiya-orel.ru
nationalbreakdown.com
naturalshair.site
naturtreenspicerx.pw naturtreenspicerx.pw
news-readers.ru negociosdasha.com
newstudio.tv nero-us.com
njkmznnb.ru nicefloor.co.uk
notasprensa.info nicovideo.jp
novosibirsk.xrus.org niki-mlt.ru
ntic.fr nlfjjunb5.ru
nudo.ca no-rx.info
nzfilecloud.weebly.com o-o-8-o-o.ru
obiavo.kz oakleyglassesonline.us
ofanda.com obiavo.by
ohmyrings.com offer.camp
onemantrip.com old-rock.com
online-templatestore.com onko-24.com
online-zaymy.ru oops-cinema.ru
onlineporno.site outpersonals.com
ontargetseo.us p-business.ru
osagonline.ru parfusale.se
ossmalta.com partners-ship.pro
paintingplanet.ru
partner-trustworthy.men
partnersafe.men
partnerworkroom.men
pcboa.se
pdns.cz
pdns.download pdns.download
pendelprognos.se pekori.to
pestomou.info pewit.pw
phelissota.xyz phormchina.com
photostudiolightings.com picswe.com
php-market.ru pictures-and-images.net
pinsdaddy.com piulatte.cz
pinwallpaper.xyz plastweb.ru
pk-pomosch.ru pokemongooo.ml
plastgranar.nu popup-hgd.xyz
play.leadzupc.com pornoklad.net
polytopesexempt.com pornosmola.info
pooleroadmedicalcentre.co.uk pospr.waw.pl
poppen-nw.net
popunder.ru
popup-jdh.xyz
pornhubforum.tk
poshiv-chehol.ru
potoideas.us potoideas.us
preg.marketingvici.com pozdrawleniya.com
pregnant.guru praisong.net
privacylocationforloc.com privatov-zapisi.ru
prizeestates.cricket prokotov.com
profitfx.online pussyspace.net
prointer.net.ua qexyfu.bugs3.com
promoheads.com qld10000.net
prostitutki-almata.org quebec-bin.com
pulseonclick.com
purplestats.com
qor360.com
qualitymarketzone.com
queerspace.com queerspace.com
questionmarque.ch ragecash.com
radiogambling.com realizmobi.com
randalljhoward.com rebuildermedical.com
rank-checker.online
ratemodels.net
recipedays.ru recipedays.ru
regdefense.com regionshop.biz
registrysweeper.com registry-clean-up.net
remmling.de remontvsamare.su
remote-dba.de remote-dba.de
rfid-locker.co replica-watch.ru
rockma.se ring4rhino.com
rome2rio.com ringtonepartner.com
runofilms.ru rogervivierforsale.com
runstocks.com ros-ctm.ru
sale-japan.com rrutw.com
samoiedo.it rucrypt.com
sarafangel.ru rusvideos.su
sarm.tk rutor.vip
sashagreyblog.ga sbtdesign.co.uk
savefrom.com scanner-irvin.top
saveriopiazza.it scanner-olga.top
sbprabooks.com scrapinghub.com
scanner-george.top sdelatmebel.ru
scanner-margo.top sdi-pme.com
scottbywater.com seasaltwithfood.com
seoheap.com seo-smm.kz
seozoom.it seocheckupx.com
serpstat.com sexs-foto.top
service-core.ru sexyteens.hol.es
sevenstars7.com shoppytoolmac.com
sexvideo-sex.com sildenafilcitratemed.com
sexytrend.ru simpoed.ufop.br
ship-marvel.co.ua sisi-go.ru
shop-electron.ru site-analyzer.com
shymkent.xkaz.org
signoredom.com
silvermature.net
sitebeam.net
sitevaluation.org sitevaluation.org
sitevalued.com sizeplus.work
skuteczna-dieta.co.pl skuteczna-dieta.co.pl
skytraf.xyz slavia.info
slonechka.ru slim.sellany.ru
slujbauborki.ru small-games.biz
soc-proof.su smart-scripts.com
socas.pluto.ro smartadserver.com
sms2x2.ru
snjack.info
sobecjvuwa.com.ru
social-fun.ru
socialmediasuggest.com
socialseet.ru
socialtrade.biz
softonicads.com
softwaretrend.net softwaretrend.net
sovetogorod.ru soietvousmaime.fr
speeddream.xyz solmarket.by
spl63.fr soundfrost.org
sovinsteel.ru
space4updating.win
splendorsearch-a.akamaihd.net
sprttrack.com sprttrack.com
spylog.com startwp.org
sterva.cc statistici.ro
stillmiracle.com stmassage.ru
stockquotes.wooeb.com strfls.com
streetfire.net
stroilka.info
stroiminsk.com stroiminsk.com
stromerrealty.com stroy-portal22.ru
strongsignal-a.akamaihd.net stroyhelp-dv.ru
stroymonolit.su strv.se
sundrugstore.com sugarkun.com
surcentro.com suggest-keywords.com
szamponrevita.pl sunflowerdrawingpaintings.blogspot.com
taihouse.ru svarkagid.com
tastyfoodideas.com svetodiodoff.ru
tds-advert002.info svnuppsalaorebro.se
swinger-mobil.net
tabakur77.com
tacbibirfa.tk
techart24.com
technopellet.gr
teencastingporn.com teencastingporn.com
tehngr.ru teenforporn.com
telefonsexi.com
telsis.com telsis.com
tengohydar.tk theguardlan.com
themestotal.com thelottosecrets.com
thepokertimer.com ticketsys.inetwd.com
threecolumnblogger.com timecrimea.ru
tiens2010.ru titan-ads.life
tizanidineandhydrocodone.blogspot.com tizanidine4mgtablets.blogspot.com
tizanidinehcl4mgisitanarcotic.blogspot.com
tizanidinehclsideeffects.blogspot.com tizanidinehclsideeffects.blogspot.com
tizanidineoral.blogspot.com tizanidineorflexeril.blogspot.com
tizanidinesideeffectsweightloss.blogspot.com tjkckpytpnje.com
tizanidineusedfor.blogspot.com tomck.com
tizanidinewithdrawal.blogspot.com tonivedu.it
tkanorganizma.ru topshef.ru
tnctrx.com tour-line.net
top250movies.ru tourismvictoria.com
tovaroboom.vast.ru track112.site
trackzapper.com tracksz.co
triumf-realty.ru tryrating.com
truck-land.life tsyndicate.com
truemfilelj.gq tuberkuleznik.ru
tuckpointingmasonrysystems.com tvnewsclips.info
turbo-suslik.org tvoystartup.ru
tzritel.tk twincitiescarservice.com
u17795.netangels.ru uchebavchehii.ru
uac.net.au uhdtv.website
ublaze.ru ultimatesetnewfreeallsoftupgradesystems.pw
uggsale.online unblocksit.es
uginekologa.com universals.com.ua
unlimitdocs.net url-img.link
unrealcommander.com urzedowski.eu
uprour.com
uptimechecker.com
upupa.net
urengoy.pro
usacasino.com
userequip.com userequip.com
vacances-voyages.info utrolive.ru
varikoz24.com uzporno.mobi
vellings.info v-doc.co
venerologiya.com valsalud.com
vepad.com vancleefreplica.pw
vandrie-ict.nl
vbtracker.net
viagraneggrx.com
video-chat.in
videokrik.net
vietimgy.pw vietimgy.pw
vita.com.hr vip-file.com
vitoriacabos.com vipsexfinders.com
vladhistory.com vitanail.ru
voltrknc1.com vkmusics.ru
vonradio.com volgograd.xrus.org
voprosotvet24.ru voloomoney.com
vzlom-na-zakaz.com vote-up.ru
warezaccess.com vroze.com
watchmyfb.pl vut.com.ru
waysbetter.cn wallpaperaccess.com
webmasters.stackexchange.com wallpapersdesk.info
webradiology.ru webalan.ru
weclipart.com webcamtalk.net
welcomeauto.ru webextract.profound.net
wellcome2slovenia.ru webpromotion.ae
westen-v.life website-audit.com.ua
white-truck.life wedding0venues.tk
wholinkstome.com weightatraining.com
wildworld.site wejdz-tu.pl
win-spy.com wingsofrefuge.net
wsgames.ru wovis.site
wufak.com writingservices17.blogspot.ru
www.bookmaker-bets.com
www.pinnacle-bets.com
wzgyyq.com wzgyyq.com
xclicks.net x69ty.ru
xev.ru xgames-04.com
xfluro.com xn--80aaafbn2bc2ahdfrfkln6l.xn--p1ai
xjlottery.com xn--80ab4aa2g.xn--p1ai
xmladserver.com xn--80ahvj9e.xn--p1ai
xn-----6kcaacnblni5c5bicdpcmficy.xn--p1ai xn--80atua3d.xn--p1ai
xn--b1addnj3cah.xn--p1ai xolodremont.ru
xxxmania.top xsfetish.org
yaderenergy.ru xtremeeagles.net
yadro.ru xxxhdvideo.site
yoluxuryevents.com xz618.com
youdesigner.kz youporn-forum.ga
yougotanewdomain.com youraticles.pl
youradulthosting.com yugk.net
youresponsive.com zaobao.com.sg
yourmovies.pl zhongwenlink.com
yurgorod.ru zkjovpdgxivg.ga
z-master.ru zolotoy-lis.ru
zajm-pod-zalog-nedvizhimosti.ru zona-aqua.ru
zaloro.com
zdesoboi.com
zebramart.ru
zetgie.com.pl
zigarettenonl.canalblog.com
zryydi.com

484
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
1bet.com 01apple.com
5kstore.com 03p.info
7minuteworkout.com 0912701309f8ce.com
883zy.com 123cha.com
activecampaign.dreamhosters.com 2rich4bitches.com
ad-words.ru aa9d046aab36af4ff182f097f840430d51.com
ads.tremorhub.com abcdeg.xyz
adverster.com adinterax.com
all4wap.ru ahmedabadwebs.com
allsilver925.co.il alevinefesleri.com
aosheng-tech.com algerianembassy.co.in
apiallgeniusinfo-a.akamaihd.net am-se.com
app5.letmacworkfaster.world aosexkontakte.net
arclk.net apiitsthirteende-a.akamaihd.net
aviav.ru.com arenanews.com.ua
backuperwebcam.weebly.com astrochicks.com
balkanfarma.org atelielembrancaqueencanta.com.br
banan.tv avkzarabotok.com
battle.net azartmsl.com
belstaffstore.online b-styles.xyz
bet-prognoz.com babosas.com
bigames.online bard-real.com.ua
biglistofwebsites.com beauty-b0x.pl
bitnote.co beauty-bracelet.com
best-mam.ru
bestofpicture.com
bigcities.org
binaryoptionscops.info
bio-japan.net
bizcheapjerseyswholesalechina.com
biznesschat.net
bkgr.se
blavia.00author.com blavia.00author.com
bongacams.com blockworld.ru
bookmaker-bet.com bravegirlsclub.com
brainboostingsupplements.org bristolhotel.com.ua
bristolhotel.com bronzeaid-a.akamaihd.net
buchananshardware.com browsepulse-a.akamaihd.net
bumble.cheapwebsitehoster.com bumble.cheapwebsitehoster.com
buscarfoto.com buybest1.biz
buy-loft.ru candelluxsklep.pl
bwlx.prepedu.cn capture-room.com
californianews.cf
cartierreplica.pw
cartierreplica.win
cejewelry.xyz
chatroulette.world
chinacheapelitejerseys.com chinacheapelitejerseys.com
click2pawn.com chinese-amezon.com
clickaider.com citetick.com
clickpapa.com collegerentals.com
crackzplanet.com comissionka.net
crd.clan.su compliance-irvin.xyz
daretodonate.co compliance-mark.top
darkbooks.org copro.pw
dev78.dioniqlabb.se d2jsp.org
diminishedvalueoforegon.com dandiyabeats.in
dinkolove.ya.ru data-ox.com
dnepr-avtospar.com.ua data1.scopich.com
docsportal.net deadlinkchecker.com
domaincdn.xyz destinationrealestate.com
domaincheck.io dispo.de
domainsphoto.com dnsrsearch.com
domini.cat dojki-hd.com
domoysshop.ru eachdayisagift.review
downvids.net ee77ee.com
dvr.biz.ua elektronischezigarette2.mex.tl
el-nation.com elektrozigaretten1.webs.com
elektrischezigaretten2.beeplog.com energydiet24.ru
elitedollars.com errorfixing.space
elvenmachine.com eu-cookie-law.info
eropho.com everypony.ru
erotikchat-24.com excitacao.com
extlinks.com f012.de
extstat.com faptitans.com
eyemagination.com fatfasts-4tmz.com
fapgon.com fetishinside.com
feeriaclub.ru filmci.pro
fickkontakte.org folowsite.com
fickkontaktehobbyhuren.com foto-sisek.porngalleries.top
filesvine.com foxjuegos.com
findpik.com free-share-buttons.com
fitnesspiks.com freza-sverlo.ru
free-social-buttons.com fz139.ttk.ru
free-stock-illustration.com gamesprite.me
galaxys6manual.info gamewrath.com
gallerily.com gazobeton-p.com.ua
gameprimary.com genetworx.com
gaytube.com gentamicineyedrops.blogspot.com
geckoandfly.com getresponse.com
get-free-traffic-now.com gfaq.ru
gomusix.com gigapeta.com
greenidesign.co glicol.kz
greenshop.su globalsurfari.com
growmyfunds.ca gonextmedia.com
grupografico-pilar.com.ar gorodservis.ru
gsasearchengineranker.space gstatey.net
guruofcasino.com hackers-crackers.tk
gwebtools.com hanink.biz.ly
homelygarden.com hdhc.ru
hopeonthestreet.co.uk hdpixion.com
hotxnights.info hentaiheroes.com
iboss.com high-speed1.net
ifirestarter.ru hoporno.com
ilo134uloh.com hostnow.men
instabid.tech hully.altervista.org
internet-apteka.ru ido3.com
ip-guide.com ie.57883.net
iptool.xyz indo-export.ru
iyasimasennka.com interior-stickers.ru
jerseyschinabizwholesale.us investpamm.ru
jetsli.de javatex.co.id
jobgirl24.ru javcoast.com
joyceblog.top jyrxd.com
kashubadesign.ru kanimage.com
kazinogames.lv karaganda.xkaz.org
klejonka.info katadhin.co
kongruan.com ketoanhanoi.info
krynica.info keywordblocks.com
kustanay.kz keywordsking.com
lakomka.com.ua kinozapas.com
lapolis.it kostenloser-sex.com
ledis.top kovrenok.ru
li-er.ru kretpicf.pw
libertybilisim.com ktotut.net
limads.men lankarns.com
linkwithin.com law-check-nine.xyz
livefixer.com leadwayau.com
lsex.xyz letslowbefast.today
machicon-ueno.info littleberry.ru
makis.nu lkbennettstore.online
malwareremovalcenter.com mactechinfo.info
masserect.com manualterap.roleforum.ru
masterseek.com marcowebonyodziezowe.pl
mastershef.club matpre.top
medmajor.ru mebel-alait.ru
medosmotr-ufa.ru mebel-ekb.com
meduza-consult.ru mediawhirl.net
menstennisforums.com mesto-x.com
minharevisao.com metabar.ru
mobile.ok.ru microstatic.pl
mobplayer.net mmstat.com
moesen-ficken.com monetizer.com-01.site
mp3ringtone.info money-every-day.com
mttwtrack.com moscow-clining.ru
muizre.ru msfsaar.de
mulberryoutletonlineeu.com mulberryoutletonlineeu.com
myanyone.net mundoaberrante.com
mycouponizemac.com mygameplus.ru
myghillie.info narco24.me
na15.ru narkologiya-penza.ru
nagdak.ru navalwiki.info
neodownload.webcam nextrent-crimea.ru
netallergy.ru nisuturnetdgu.tk
ngps1.ru nofreezingmac.click
nikitabuch.com nofreezingmac.work
ningessaybe.me nudepatch.net
niroo.info nuit-artisanale.com
notaria-desalas.com obiavo.uz
obiavo.net oksrv.com
odoratus.net onlineslotmaschine.com
okout.ru orenburg-gsm.ru
online-zaymy.ru partners-ship.pro
online.ktc45.ru pcgroup.com.uy
ororodnik.goodbb.ru pensplan.com
ourtherapy.ru piccdata.com
ozoz.it piski.top
page2rss.com pizdopletka.club
parvezmia.xyz primedice.com
pawli.eu privatamateure.com
picquery.com privatov-zapisi.ru
plastgran.com prointer.net.ua
pokerniydom.ru promalp-universal.ru
popcash.net prostitutki-tyumeni.org
poptool.net prostitutki-yaroslavlya.org
popunder.net pussysaga.com
popunder.ru
pornhub-ru.com
pornoklad.ru
prizestohandle.club
projectforte.ru
prospekt-st.ru
randalljhoward.com randalljhoward.com
rating-bestcasino.com rangoman.date
reimageplus.com refads.pro
remont-komputerov-notebook.ru rossmark.ru
remybutler.fr royalads.net
rennlist.com runofilms.ru
robot-forex.biz rvtv.ru
rockma.se sabizonline.com
ros-ctm.ru sanyuprojects.com
rumamba.com scanner-barak.top
scripted.com scanner-julia.top
sell-fb-group-here.com scanner-viktor.top
seojokes.net scmor.ilxc.cc
servisural.ru searchaddis.com
sexs-foto.com sexad.net
sharebutton.org sexkrasivo.net
si-unique.com share-buttons-for-free.com
skytraf.xyz siha.de
slimcdn.com sisi-go.ru
slowmac.tech sloopyjoes.com
smart-scripts.com snsdeainavi.info
sobecjvuwa.com.ru socas.pluto.ro
space4update.pw social-buttons.com
spasswelt.net socialbutton.xyz
spasswelt.xyz socialmediasuggest.com
stairliftstrue.com softxaker.ru
stariy-baku.com spidtest.org
stroymonolit.su sports-supplements.us
sugvant.ru stackthatbucks.com
t-bygg.com stanthonyscatholicchurch.org
td-l-market.ru stard.shop
teencastingporn.com stevemonsen.com
testbotprocessor44.com strangeduckfilms.com
thenews-today.info stroy-portal22.ru
tilido.com suhanpacktech.com
timhost.ru support.nopeas.sk
tizanidinehcl2mgtablet.blogspot.com tacbibirfa.tk
touchmods.fr tasteidea.com
tourcroatia.co.uk thecoral.com.br
traffictrade.life tizanidinehclsideeffects.blogspot.com
tramadolandtizanidine.blogspot.com torontoplumbinggroup.com
tsan.net torrentgamer.net
twodollarshows.com track112.site
uggsale.online traffic2cash.org
unrealcommander.com trafficgenius.xyz
uprour.com trafficstars.com
uptimebot.net trk-4.net
url-img.link tubeline.biz
vancleefreplica.pw turbo-suslik.org
vavilone.com twittruth.com
venta-prom.ru twojebook.pl
vinsit.ru validdomain.xyz
vip-file.com vchulkax.com
vedomstvo.net
via-energy-order.com
viagengrarx.com
visionwell.com.cn
viveresaniesnelli.it
vltai.com
voltrknc1.com voltrknc1.com
wait3sec.org
wallpaperaccess.com
wallpaperstock.net
waterpurifier.club waterpurifier.club
wdrake.com weboptimizes.com
webtherapy.ru website-analytics.online
wesharepics.site wleuaprpxuvr.ga
worldinternetauthority.com ws.ampower.me
worldis.me wttavern.com
www.solartek.ru xn--e1aggki3c.xn--80adxhks
wygraj-skiny.win xxxkaz.org
xgames-04.com xxxmania.top
xitjw.info yellowstonevisitortours.com
xn-----6kcaacnblni5c5bicdpcmficy.xn--p1ai
xn----9sbebi2bvzr7h.xn--p1ai
xn----ctbbcjd3dbsehgi.xn--p1ai
xn--80aaafbn2bc2ahdfrfkln6l.xn--p1ai
yopoint.in yopoint.in
youtube-downloader.savetubevideo.com youandcredit.ru
z-master.ru zawyna.ua
zacreditom.ru zerocash.msk.ru
zaimhelp.ru zocaparj.kz
zastenchivosti.net zrizvtrnpale.tk
zazagames.org
zolotoy-lis.ru

10
dev-tools/test_units/ratelimittest-beta.txt
View file

@ -581,14 +581,4 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html> </html>

10
dev-tools/test_units/ratelimittest-master.txt
View file

@ -581,14 +581,4 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html> </html>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.203 0; 10.1.0.237 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.07.3239 ### Version: V4.2022.07.3240
### Updated: Wed Jul 20 22:01:23 UTC 2022 ### Updated: Thu Jul 21 10:03:15 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.07.3239 ### Version: V4.2022.07.3240
### Updated: Wed Jul 20 22:01:23 UTC 2022 ### Updated: Thu Jul 21 10:03:15 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##