vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-04 19:50:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2023.01.3544 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2023-01-12 22:10:57 +00:00
parent f3dbaa726c
commit 5a8fe84169
47 changed files with 98491 additions and 98484 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2023.01.3543 #### Version: V4.2023.01.3544
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2023.01.3543 #### Version: V4.2023.01.3544
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2023.01.3543 #### Version: V4.2023.01.3544
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12166
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2023.01.3543 #### Version: V4.2023.01.3544
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12170
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12170
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

12170
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12170
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 1; 10.1.0.107 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 0; 10.1.0.107 0;
127.0.0.1 0; 127.0.0.1 0;

12170
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 1; 10.1.0.107 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 0; 10.1.0.107 0;
127.0.0.1 0; 127.0.0.1 0;

12170
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 1; 10.1.0.107 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 0; 10.1.0.107 0;
127.0.0.1 0; 127.0.0.1 0;

12170
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3543 3544

12170
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

12170
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
23.98.136.32 0; 13.91.166.1 0;

12170
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 1; 10.1.0.107 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 0; 10.1.0.107 0;
127.0.0.1 0; 127.0.0.1 0;

12170
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 1; 10.1.0.107 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 0; 10.1.0.107 0;
127.0.0.1 0; 127.0.0.1 0;

12170
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 1; 10.1.0.107 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 0; 10.1.0.107 0;
127.0.0.1 0; 127.0.0.1 0;

12170
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
23.98.136.32 0; 13.91.166.1 0;

12170
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
23.98.136.32 0; 13.91.166.1 0;

12170
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 1; 10.1.0.107 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
BackDoorBot Aboundexbot
BotALot AllSubmitter
Craftbot Aspiegel
Image Sucker Blackboard
Lanshanbot Domains Project
LinkWalker NextGenSearchBot
Psbot Offline Explorer
WASALive-Bot Sqlmap
WiseGuys Robot Web Enhancer
archive.org_bot crawler.feedback

308
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 360Spider
80legs 80legs
ADmantX ALittle Client
ASPSeek ASPSeek
Aboundex Abonti
Aboundexbot AhrefsBot
Aipbot Aipbot
Alexibot AllSubmitter
Anarchy Alligator
Anthill Ankit
Apexoo Apexoo
Aspiegel
Atomseobot
Attach Attach
BDCbot AwarioRssBot
BDFetch BDFetch
BLEXBot
BackDoorBot BackDoorBot
BackStreet BackStreet
BackWeb BackWeb
Backlink-Ceck Badass
Bandit Barkrowler
Battleztar Bazinga Bigfoot
Black Hole
BlackWidow BlackWidow
Blackboard Blackboard
BlowFish
Brandwatch
Buck
Buddy
BuiltWith BuiltWith
BunnySlippers Bullseye
CATExplorador
CODE87 CODE87
CSHttp CSHttp
Calculon CensysInspect
Cegbfeieh
CheeseBot
CherryPicker CherryPicker
ChinaClaw
Chlooe Chlooe
Cliqzbot
Cloud mapping Cloud mapping
Cocolyzebot Cocolyzebot
CopyRightCheck Cosmos
Craftbot Crescent
CrazyWebCrawler CrunchBot
Curious
Custo Custo
CyotekWebCopy DIIbot
DSearch DataCha0s
DTS Agent DigitalPebble
Demon Dirbuster
Deusu Discoverybot
Discobot
Dispatch Dispatch
DnBCrawler-Analytics
DnyzBot
DomCopBot
DomainCrawler DomainCrawler
DomainStatsBot DomainSigmaCrawler
Domains Project EMail Siphon
Dotbot
Download Wonder
EasyDL EasyDL
EirGrabber Extractor
Evil Extreme Picture Finder
Exabot Ezooms
Express WebPictures FDM
EyeNetIE Firefox/7.0
FHscan
FlashGet FlashGet
Flunky Flunky
Foobot Fuzz
G-i-g-a-b-o-t GT::WWW
GalaxyBot Gigabot
Getintent Go-Ahead-Got-It
GoZilla GrabNet
Gotit Grabber
Harvest Grafula
IDBot GrapeFX
IRLbot GrapeshotCrawler
HTMLparser
HaosouSpider
IDBTE4M
Id-search
IlseBot IlseBot
IndeedBot
Indy Library Indy Library
InfoNaviRobot
InfoTekies InfoTekies
Intelliseek
InterGET InterGET
InternetSeer InternetSeer
Iria Iria
Iskanie
JOC Web Spider
JamesBOT JamesBOT
Jbrofuzz
Jetty Jetty
Joomla JikeSpider
Jorgee
Keybot Translation-Search-Machine Keybot Translation-Search-Machine
Kozmosbot
LNSpiderguy
LWP::Simple LWP::Simple
Larbin
Leap Leap
LeechFTP LeechGet
LexiBot
Lftp Lftp
Lightspeedsystems Lightspeedsystems
Likse LinkWalker
LinkScan
Linkbot
LinkextractorPro LinkextractorPro
LinkpadBot LinkpadBot
LinksManager
LinqiaMetadataDownloaderBot LinqiaMetadataDownloaderBot
LinqiaRSSBot
Lipperhey Spider
Litemage_walker Litemage_walker
Lmspider
MSFrontPage
MSIECrawler MSIECrawler
MTRobot MTRobot
Magnet Magnet
Majestic SEO Mail.RU_Bot
Majestic-SEO Mass Downloader
Majestic12 Masscan
MarkWatch MauiBot
Mata Hari
Mediatoolkitbot
MicroMessenger MicroMessenger
Microsoft URL Control
Moblie Safari
Mojolicious Mojolicious
MolokaiBot MolokaiBot
Mozlila Musobot
Mr.4x3 Nameprotect
NPbot Needle
Navroad Nessus
NearSite Net Vampire
NetAnts
NetLyzer NetLyzer
NetMechanic NetMechanic
Nettrack NetZIP
Niki-bot Nibbler
Nikto Nimbostratus
NimbleCrawler
Ninja Ninja
Nmap
Not Not
Nutch Nutch
OpenLinkProfiler OpenLinkProfiler
OpenVAS
Openvas Openvas
OrangeSpider OutclicksBot
PHPCrawl Page Analyzer
PageAnalyzer PageAnalyzer
PageGrabber
PageScorer PageScorer
PageThing.com
Pandalytics
Panscient
Papa Foto Papa Foto
Picscout Pavuk
ProPowerBot PeoplePal
Psbot Pimonster
Pu_iN PleaseCrawl
ProWebWalker
Pump Pump
Quick-Crawler RankActiveLinkBot
RankActive RankFlex
RankingBot
Rankivabot Rankivabot
RealDownload Re-re
Reaper ReGet
RedesScrapy
RocketCrawler
Rogerbot Rogerbot
SEOprofiler
SEOstats SEOstats
SISTRIX
SMTBot SMTBot
Screaming SalesIntelligent
SentiBot ScreenerBot
Site Sucker ScrepyBot
SiteCheckerBotCrawler Searchestate
SiteSucker SemrushBot
SiteExplorer
Sitebeam Sitebeam
Sitevigil Siteimprove
SmartDownload SlySearch
Snapbot Sociscraper
Snoopy Sosospider
Sogou web spider
Sottopop Sottopop
Spammen
SpankBot SpankBot
Spbot Spinn3r
Sqlmap
Sqlworm
Sqworm
Steeler
Stripper Stripper
Sucuri Sucuri
Suzuran T8Abot
T0PHackTeam Telesoft
TeleportPro
Telesphorep Telesphorep
The Intraformant
TheNomad TheNomad
Thumbor
TightTwatBot TightTwatBot
Titan Tracemyfile
Trendictionbot Trendiction
True_Robot True_Robot
Turingos URLy Warning
Turnitin
URLy.Warning URLy.Warning
UnisterBot
V-BOT V-BOT
VCI Vacuum
Vagabondo VelenPublicWebCrawler
Virusdie
Voil Voil
Voltron
WASALive-Bot WASALive-Bot
WBSearchBot
WEBDAV WEBDAV
WISENutbot Wallpapers/3.0
WPScan WallpapersHD
WWW-Collector-E
WWWOFFLE
Wallpapers
WeSEE
Web Auto Web Auto
Web Fetch Web Enhancer
Web Fuck
Web Sauger Web Sauger
WebAuto WebFuck
WebCollage WebImageCollector
WebFetch
WebReaper
WebSauger
WebWhacker
WebZIP
Webalta
Webshag Webshag
WebsiteExtractor Whack
Whacker Whatweb
Widow Wonderbot
WiseGuys Robot
Wotbox Wotbox
Wprecon Zitebot
Xenu
YoudaoBot
Zermelo
ZoomBot ZoomBot
ZoominfoBot ZoominfoBot
ZyBorg
adscanner adscanner
arquivo.pt
autoemailspider
backlink-check
cah.io.community
coccocbot coccocbot
cognitiveseo
crawl.sogou.com crawl.sogou.com
crawler.feedback
crawler4j crawler4j
dataforseo.com dataforseo.com
evc-batch demandbase-bot
domainsproject.org
facebookscraper
gopher
heritrix heritrix
instabid
internetVista monitor
linkdexbot
lwp-request
magpie-crawler magpie-crawler
oBot meanpathbot
pcBrowser mediawords
probe-image-size plumanalytics
s1z.ru polaris version
ripz
scan.lol scan.lol
seobility
seocompany.store
seoscanners seoscanners
seostar sitechecker.pro
sp_auditbot siteripz
sysscan
tAkeOut tAkeOut
trendiction.com trendiction.com
trendiction.de ubermetrics-technologies.com
voyagerx.com
webgains-bot webgains-bot
webmeup-crawler
webpros.com
webprosbot
x09Mozilla
x22Mozilla
xpymep1.exe xpymep1.exe
zauba.io
zgrab zgrab

298
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
ALittle Client 404enemy
ASPSeek 80legs
Abonti
Aboundex
Acunetix Acunetix
AhrefsBot AhrefsBot
AllSubmitter Alligator
Atomseobot Anarchie
BDCbot Anarchy
BackStreet Apexoo
Aspiegel
AwarioRssBot
AwarioSmartBot
BBBike
BDFetch
BackWeb BackWeb
Backlink-Ceck Backlink-Ceck
BacklinkCrawler
Bandit
Barkrowler Barkrowler
BetaBot BatchFTP
Bitacle Battleztar Bazinga
Bigfoot
Black Hole Black Hole
Blow BlowFish
Boardreader
Bolt Bolt
Brandprotect Buck
Brandwatch
BuiltBotTough
Bullseye Bullseye
BuzzSumo BuzzSumo
CATExplorador
CODE87 CODE87
CSHttp
Calculon Calculon
CazoodleBot CazoodleBot
CensysInspect Cegbfeieh
CherryPicker CheeseBot
Citoid Chlooe
Claritybot
Cliqzbot Cliqzbot
Cocolyzebot Cocolyzebot
Cogentbot
Collector Collector
Copier Copier
CopyRightCheck Cosmos
Copyscape
Craftbot Craftbot
CrazyWebCrawler CrazyWebCrawler
Curious Crescent
CyotekWebCopy Custo
DSearch DSearch
DataCha0s
DatabaseDriverMysqli DatabaseDriverMysqli
DittoSpyder Deusu
DigitalPebble
Disco
Discoverybot
DnyzBot
DomCopBot
DomainAppender DomainAppender
DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
Domains Project Dotbot
Download Wonder
Drip Drip
EMail Siphon
EMail Wolf EMail Wolf
Exabot Ebingbong
Express WebPictures EroCrawler
EyeNetIE ExtLinksBot
ExtractorPro
FDM FDM
FHscan
Firefox/7.0 Firefox/7.0
Flunky
Foobot
Fuzz Fuzz
FyberSpider
G-i-g-a-b-o-t
GT::WWW GT::WWW
Genieo GoZilla
GermCrawler GrabNet
GetWeb
Getintent
Gigabot
Go!Zilla
Grabber Grabber
GrapeFX
GrapeshotCrawler GrapeshotCrawler
GridBot GridBot
HEADMasterSEO
HMView HMView
HTTP::Lite
HaosouSpider
Harvest
Hloader Hloader
Humanlinks HonoluluBot
HybridBot
IRLbot IRLbot
IndeedBot
Indy Library
InfoNaviRobot InfoNaviRobot
InfoTekies
Intelliseek
InterGET
InternetSeer InternetSeer
Iria
Iskanie Iskanie
IstellaBot Jbrofuzz
Jetty JennyBot
JikeSpider JikeSpider
Joomla
Jyxobot Jyxobot
Kinza Kinza
Kozmosbot Kozmosbot
Larbin LNSpiderguy
LeechGet
LexiBot
Lftp
LibWeb
LieBaoFast LieBaoFast
Lightspeedsystems Lightspeedsystems
Linkbot
LinkpadBot
LinqiaMetadataDownloaderBot LinqiaMetadataDownloaderBot
Lipperhey MIIxpc
Lipperhey Spider
Ltx71
MJ12bot MJ12bot
MTRobot MSFrontPage
Magnet Magnet
Majestic SEO Majestic-SEO
Majestic12 MarkWatch
MarkMonitor Mass Downloader
Mb2345Browser Mata Hari
MeanPath Bot MeanPath Bot
Mediatoolkitbot
MegaIndex.ru MegaIndex.ru
Metauri Metauri
MicroMessenger MicroMessenger
Minefield
Moblie Safari
Mojolicious
MolokaiBot MolokaiBot
Morfeus Fucking Scanner
Mozlila
Mr.4x3
Msrabot
Musobot
NICErsPRO
NPbot NPbot
Nameprotect NearSite
Needle Needle
Nessus
NetAnts NetAnts
NetMechanic Nettrack
NetZIP NextGenSearchBot
Netcraft Niki-bot
NimbleCrawler NimbleCrawler
Ninja
Nmap
Not
Nutch Nutch
Octopus Octopus
Offline Explorer
Offline Navigator Offline Navigator
OnCrawl OnCrawl
OpenLinkProfiler OpenVAS
Openfind Openfind
PECL::HTTP OutclicksBot
OutfoxBot
PHPCrawl PHPCrawl
PageGrabber PageGrabber
PageThing.com PageThing.com
Pandalytics Papa Foto
Panscient PictureFinder
Pavuk
Pi-Monster
Picsearch
Piepmatz Piepmatz
PleaseCrawl Pixray
ProWebWalker Probethenet
Psbot Psbot
Pump PxBroker
Quick-Crawler PyCurl
RankingBot RSSingBot
RankActive
Rankivabot
RankurBot RankurBot
Re-re ReGet
Reaper Reaper
Recorder Recorder
Ripper RedesScrapy
SBIder RepoMonkey
SEOlyticsCrawler RocketCrawler
SEOkicks-Robot
SEOstats SEOstats
SISTRIX SalesIntelligent
ScanAlert
ScoutJet
Scrapy Scrapy
ScreenerBot
ScrepyBot ScrepyBot
SemanticJuice SearchmetricsBot
SemrushBot SeoSiteCheckup
Siphon Seomoz
SiteCheckerBotCrawler SiteLockSpider
SiteExplorer
SiteSnagger SiteSnagger
Sitebeam Sitebeam
SmartDownload Siteimprove
Sociscraper Snapbot
Sosospider SocialRankIOBot
SpaceBison Sogou web spider
Spammen Spammen
Spanner SpankBot
Spbot
SputnikBot SputnikBot
Sqlmap Sqlmap
Sqworm Sqlworm
Sucuri Steeler
SuperBot SuperBot
SuperHTTP SuperHTTP
SurveyBot Surfbot
Swiftbot Swiftbot
Szukacz T0PHackTeam
T8Abot
Teleport Teleport
Telesphoreo The Intraformant
Thumbor Thumbor
TightTwatBot Toata
Turnitin Toweyabot
TurnitinBot TwengaBot
Typhoeus Twice
URLy Warning
UnisterBot UnisterBot
VCI Upflow
Vacuum VB Project
Virusdie
VoidEYE VoidEYE
Voil WBSearchBot
WEBDAV
WWW-Collector-E WWW-Collector-E
WWW::Mechanize WWW::Mechanize
Wallpapers
Wallpapers/3.0
Web Fetch
Web Pix Web Pix
Web Sucker
WebAuto
WebCollage
WebEnhancer
WebFetch
WebImageCollector
WebReaper WebReaper
WebSauger
WebStripper WebStripper
WebWhacker WebSucker
WebmasterWorldForumBot Webalta
WebsiteExtractor
WebsiteQuester
Webster Webster
Whack
Whatweb Whatweb
Who.is Bot Who.is Bot
Wonderbot WinHTTrack
Woobot ZumBot
Wotbox
Xaldon_WebSpider
Xenu
YoudaoBot
Zade
Zitebot
ZoominfoBot
ZyBorg ZyBorg
arquivo-web-crawler adscanner
arquivo.pt arquivo.pt
autoemailspider backlink-check
cah.io.community cognitiveseo
clark-crawler com.plumanalytics
crawl.sogou.com demandbase-bot
domainsproject.org
eCatch
facebookscraper facebookscraper
ips-agent
iubenda-radar
linkdexbot linkdexbot
lwp-request
lwp-trivial
magpie-crawler magpie-crawler
page scorer oBot
plumanalytics
probe-image-size probe-image-size
scan.lol ripz
seobility
seocompany.store seocompany.store
seoscanners
sexsearcher
sitechecker.pro
siteripz
sp_auditbot sp_auditbot
sysscan spyfu
trendiction.de trendiction.com
ubermetrics-technologies.com webpros.com
webgains-bot webprosbot
x22Mozilla x09Mozilla
xpymep1.exe xpymep1.exe
zauba.io
zgrab zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
bannerconnect.net bjorkbacken.nu
bitcoin-ua.top cottageofgrace.com
gymi.name forum.poker4life.ru
lexixxx.com havepussy.com
normalegal.ru kinoduh.ru
sexad.net massage-info.nl
stats-public.grammarly.io remont-comp-pomosh.ru
superiends.org sunflowerdrawingpaintings.blogspot.com
visa-china.ru vegan-foods.us
zapiszto.pl xn--d1aifoe0a9a.top

944
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
000free.us 101flag.ru
1stat.ru 10bet.com
23kw.ru 11pikachu.ru
2nt.ru 2015god.org
300richmond.co.nz 2daytrendingnews.com
404.mn 2girls1cup-free.com
70casino.online 8si.ru
72-news.com abcdeg.xyz
8558l.jobs.net abwa.tk
8gold.com activepr.ru
8lufu.com
abiente.ru
acgs.tk
addbin.men
adelly.bg adelly.bg
adhome.biz adminshop.com
adloads.net adservme.com
adtech.de adult3dgames.com
afftrack001.com advancedcleaner.com
akusherok.ru advancedmassagebysara.com
alfapro.ru agapovdg.ru
alibestsale.com ahrntal.verymes.xyz
alive-ua.com albamargroup.com
amigobulls.com alert-fdm.xyz
anal-acrobats.com alert-fjg.xyz
anonymizeme.pro alfa9.com
apartamentwroclaw.eu alienwheel.es
api.stathat.com allstatesugarbowl.org
apimossnetinfo-a.akamaihd.net alpinism.ru
amanda-porn.ga
ameblo.jp
amt-k.ru
analnoeporno.tv
anatomiy.com
antons-transporte.de
apartmentbay.ru
apelsinnik.website
apiitsthirteende-a.akamaihd.net
applyneedy.xyz
approvedlocal.co.za
arabsexxxtube.com
arcteryxsale.online
ardimobileinfo.ml ardimobileinfo.ml
as5000.com arenda-yeisk.ru
ascat.porn arkartex.ru
asociatia-tipografilor-transilvania.ro artdeko.info
automatic-seo.com astrochicks.com
autonew.biz atleticpharm.org
autoseo-traffic.com autoseo-traffic.com
babespcs.com avon-ukraine.com
bablonow.ru avtointeres.ru
babs.com.ua avtovolop.ru
baksman.com awency.com
ayanaspa.com
azazaporn.com
azbukafree.com
azte.ch
bag77.ru
baixar-musicas-gratis.com
bamo.xsl.pt bamo.xsl.pt
beatifullandscape.co bankmib.ru
belinka.com.ua bear.gotcher.us
belinvestdom.by beauty-things.com
best-coupon-offer.com beget.tech
bestwebsiteawards.com bestcalovebracelet.cn
bet-prognoz.com bestofpicture.com
betwinservice.com bestplacetobuyeriacta.jw.lt
bfz.biz beyan.host.sk
bio-optomarket.ru bezsporno.ru
biznesschat.net bidbuy.co.kr
bkgr.se bikini-image.com
bkns.vn bio-japan.net
blagovest-med.ru blog-hits.com
bluerobot.info blog.axant.it
bonzybuddy.com blog.koorg.ru
boobsimge.com bloggers.nl
bookhome.info blogos.kz
bracketsmackdown.com boltalko.xyz
bonky.biz
bradleylive.xyz bradleylive.xyz
brandimensions.com
bravegirlsclub.com bravegirlsclub.com
brendbutik.ru brg8.com
brk-rti.ru bristolhotel.com
brus.city
bryansk.zrus.org bryansk.zrus.org
buchananshardware.com bumble.cheapwebsitehoster.com
budilneg.xyz buy-meds24.com
bum.com.ru buybest1.biz
burger-imperia.com buypanicdisorderpill.com
butstrap.space buyviagraa.blogspot.com
callstevens.com bzero1jewelry.net
cacheimages.com
callcafe.info
cand.jp cand.jp
candcstuccoandstone.com candiceloves.us
cannibalcheerleader.com capture-room.com
carfax.com.ua carabela.com.do
ccbill.com carapics.com
centrdebut.ru cardsharp1.ru
cfcl.co.uk carloans.com
changde.58.com cartierloveringreplica.xyz
cheapjerseysap.com cartierwatch.xyz
cheapmoncler.win caulderwoodkennels.com
checkhit.com cayado.snn.gr
cialis-samples.com centre-indigo.org.ua
ceotrk.com
cheapbarbour.online
cheapjerseysa.com
cheapjerseysfootballshop.com
chocolateslim-u-srbiji.com
chorus.terakeet.com
christianlouboutinreplica.pw
chuckguilford.com
cialischstgerts.com cialischstgerts.com
cindymatches.com classiquebijoux.ru
clickbank.net clayvasedesigns.tk
clickhype.com clickbanksites.info
clickpapa.com clmforexeu.com
collegerentals.com club-samodelkin.ru
compliance-walter.top coin-hive.com
colobit.biz
com-cleaner.systems
comsysnet.com comsysnet.com
cookie-law-enforcement-ee.xyz concealthyself.com
cookingmeat.ru converse.ddsoldes.fr
coolwallpapers-hd.com counter.bloke.com
coop-gamers.ru cpabegins.ru
copblock.org crazyboost.pro
copro.pw dairyindia.in
counter.yadro.ru damedingel.ya.ru
crazyprotocol.com data.vtc.pw
crystalslot.com deniven.1bb.ru
customchocolate.business-for-home.com deplim.com
cypernhuset.se dfiles.me
daisye.top dikqlyremy.info
danschawbel.com discovertreasure-a.akamaihd.net
darkbooks.org diusyawiga.tk
data-ox.com div.as
dating-time-now.com dktr.ru
day-news.info domain.webkeyit.com
ddpills.com domination.ml
dealighted.com dosug-lux.ru
demon-tweeks.com download-of-the-warez.blogspot.com
detoxmed24.ru drivotracker.com
dev78.dioniqlabb.se drugspowerstore.com
devochki.top e-kwiaciarz.pl
dgfitness.co easyshoppermac.com
diffbot.com ekn-art.se
djonwatch.ru ekspertmed.com
djstools.com ekspresihati.info
dlya-android.org elitedollars.com
docsportal.net eluxer.net
dojki365.online empathica.com
dokfilms.net energy-ua.com
domaincrawler.com epngo.bz
domashneeruporno.com eropornosex.ru
domoysshop.ru eupornstar.info
donvito.unas.cz extremepornos.net
doreenblog.online extstat.com
dprtb.com eyes-on-you.ga
droidlook.net ezigarettekaufen2.yolasite.com
dumpsmania24.com ezigarettenshop2.myblog.de
elektrischeziga.livejournal.com fakehandbags.xyz
elektrischezigarette1.blog.pl familienzahnaerzte.com
elitepcgames.com fantasticpixcool.com
elvenmachine.com fapgon.com
enge-fotzen.info ff30236ddef1465f88547e760973d70a.com
engines-usa.com filmfanatic.com
enternet.ee financehint.eu
eredijovon.com financeloan.us
eropho.com financepoints.eu
erot.co find1friend.com
erotische-geschichten-xxl.com fingerprintjs.com
eugenevaultstorage.com firstsiteguide.com
evehemming.blogspot.com.au floating-share-buttons.com
exchanges-bet.com florida-tourism.net
expertnaya-ocenka.ru folowsite.com
extremez.net forzeronly.com
ezigarettenkaufen1.pagina.gr foto-basa.com
fastcrawl.com foto-sisek.porngalleries.top
favorcosmetics.com franklinfire.co
fearcrow.com frcls.fr
femmesdenudees.com freewareseek.com
ferrotodo.com freza-sverlo.ru
filerockstar298.weebly.com fullfileaccess.com
findanysex.com fungamelands.com
finejewelryshop.ru gay-file.com
fireads.men gdcentre.ru
flagcounter.me getresponse.com
foodgid.net glall.ru
forensicpsychiatry.ru gloverid.site
forex.osobye.ru golfresa.lucania.se
forminecrafters.ru googlemare.com
formseo.com gotwebsite1.com
foxydeal.com gox.com.ua
freeuploader.ml gq-catalog.gq
freshdz.com gstatey.net
fuckmill.com guardlink.org
fxund.us guildebzh.info
gambarkatabaru.com hanink.biz.ly
gamblingnerd.com hdhc.ru
game300.ru hem.passagen.se
gamedayhouse.com hermesbirkinhandbagoutlets.com
gameplexcity.com hitmuzik.ru
gazobeton-p.com.ua homedecorpicture.us
gearsadspromo.club hotshoppymac.com
geileweiber.tk hyhj.info
gemara.com idealtits.net
girlsatgames.ru igithab.com
glcomputers.ru impisr.ru
globalscam.ga infobanks.ru
goo.ne.jp insomniagamingfestival.com
goosefishpost.bid inspiring-desperate.tk
greendream.com.ua io9.com
gsasearchenginerankerdiscount.com iqoption-bin.com
gsmtlf.ru isitpaleo.info
guardlink.com it-max.com.ua
hamilton.ca ix20.ru
handicapvansarea.com iyasimasennka.com
hatedriveapart.com jasonpartington.com
hayate.biz jav-way.site
hcate.com javspace.net
health-medical-portal.info jerseychinabizwholesale.us
hentaiheroes.com
hit-kino.ru
hmmm.cz
hmywwogw.bloger.index.hr
host-tracker.com
hothot.ru
hotkeys.com
housekuba.org
hoverboard360.de
hoztorg-opt.ru
icaseclub.ru
ikearugs.xyz
imagefinder.site
imagerydatabase.com
images.gyffu.com
imagez.co
instabid.tech
ipornox.xxx
ipostroika.ru
izoll.ru
janerikholst.se
joessmogtestonly.com joessmogtestonly.com
journeydownthescale.info jwss.cc
justprofit.xyz kaidalibor.de
kambasoft.com kamalsinha.com
kandidos.com kanimage.com
karachev-city.ru
kartiniresto.com kartiniresto.com
katushka.net kazan.xrus.org
keywordglobal.co.za kazan.zrus.org
kinostorm.net kazrent.com
kinozapas.com kinky-fetishes.com
kirov.zrus.org krasivoe-hd.com
kliksaya.com krasnodar.xrus.org
klin3952.ru kredit-blog.ru
knogg.net ktotut.net
komputers-best.ru kurbappeal.info
konkursov.net labafydjxa.su
kosmetyki.tm.pl lalalove.ru
kozhniebolezni.com layola.biz.tc
krynica.info leboard.ru
kupit-adenu.ru letolove.ru
lafriore.ru linkdebrideur.xyz
lenvred.org lion.bolegapakistan.com
lietaer.com
life.biz.ua
lineavabit.it
liquimondo.com
livingcanarias.com
lkbennettoutlet.online
lkbennettstore.online
llastbuy.ru llastbuy.ru
lmrauction.com loftdigital.eu
lolnex.us lombia.com
lonfon.xyz lomza.info
lordzfilmz.me low-format.ru
lostaruban.com lsp-awak-perikanan.com
louboutinbooties.xyz lubetube.com
louboutinshoes.xyz lyrster.com
macdamaged.space m.b00kmarks.com
madot.onlinewebshop.net machicon-akihabara.info
masthopehomes.com maglid.ru
mazda-roadsters.com magnetic-bracelets.ru
message-warning.net mail.allnews24.in
mesto-x.com mailemedicinals.com
metalonly.info marblestyle.ru
mexicosleevegastrectomy.com marktforschung-stuttgart.com
milblueprint.com maslenka.kz
millionare.com matsdale.com
minegam.com mebeldekor.com.ua
mini.7zap.com mediafresh.online
mlf.hordo.win meetingrainstorm.bid
modnie-futbolki.net metafilter.com
moneymaster.ru methodsmarketing.com
moneyzzz.ru mhtr.be
moscow.xrus.org minharevisao.com
mozello.ru misswell.net
mrvideospornogratis.xxx mobilierland.com
multgo.ru mobplayer.ru
muschisexbilder.com mojpreskumanie.com
muycerdas.xxx mokrayakiska.com
myiptest.com monclercheap.xyz
myshopmatemac.com moncleroutletonline.pw
nac-bearings.ru morpicert.pw
nacap.ru moz.com
narutonaruto.ru mp3ritm.top
naturalbreakthroughsresearch.com mrpornogratis.xxx
neobux-bg.info musezone.ru
newstraveller.ru musflashtv.com
newstudio.tv mydirtyhobby.com
nfljerseyscheapchinabiz.com mygameplus.com
nhl17coins.exblog.jp myliveblog.ru
nikitabuch.com mystats.xyz
nina.az namenectar.com
nofreezingmac.click narco24.me
novgorod.xrus.org netallergy.ru
nudo.ca networkad.net
nzfilecloud.weebly.com networkcheck.xyz
o-o-8-o-o.com nfljerseysforsalewholesaler.com
oakridgemo.com ningessaybe.me
obiavo.uz nippon-bearings.ru
obnal.org no-rx.info
nodup.ru
nootrino.com
northfacestore.online
o333o.com
obiavo.su
oecnhs.info oecnhs.info
of-ireland.info offergroup.info
ofanda.com office-windows.ru
okonich.com.ua oivcvx.website
onblastblog.online ok-ua.info
onlinadverts.com online-hd.pl
online7777.com online-sbank.ru
opinionreelle.com online.ktc45.ru
ovirus.ru onlyforemont.ru
ownshop.cf onstrapon.purplesphere.in
palvira.com.ua ooomeru.ru
panchro.co.uk open-odyssey.org
paramountmarble.co.uk optibuymac.com
partners-ship.pro parajumpersstore.online
paydayonlinecom.com partner-trustworthy.men
pcgroup.com.uy patterntrader-en.com
pelfind.me pelfind.me
penisvergrotendepillennl.ovh picsfair.com
pepperstyle.ru piluli.info
perfectplanned.com pio.polytopesexempt.com
picturesmania.com pistonclasico.com
pills24h.com pizza-tycoon.com
piski.top plastgranar.nu
pk-services.ru platesauto.com
playmsn.com platinumdeals.gr
playtap.us playboyfiles.xblog.in
porno-home365.com plugingeorgia.com
porno-transsexuals.ru podshipniki-nsk.ru
pornoklad.net pojdelo.weebly.com
pornoreino.com polcin.de
powc.r.ca.d.sendibm2.com pooleroadmedicalcentre.co.uk
preg.marketingvici.com popugauka.ru
primedice.com popup-jdh.xyz
printingpeach.com pornoelita.info
priscilarodrigues.com.br pornoforadult.com
privetsochi.ru pornokajf.com
promoheads.com preconnubial.usuby.site
prostitutki-novgoroda.org prizeestates.cricket
prostitutki-odessa.org pro-tec.kz
prostitutki-tyumeni.org pron.pro
pukaporn.com ptr.ruvds.com
pussysaga.com pyramidlitho.webs.com
qor360.com qualitymarketzone.com
ranksonic.info raisedseo.com
razorweb-a.akamaihd.net rankings-analytics.com
realmonte.net realizmobi.com
remont-fridge-tv.ru registry-cleaner.net
relax.ru
remontbiz.ru
renecaovilla.online renecaovilla.online
rentaremotecomputer.com rezeptiblud.ru
repeatlogo.co.uk roll123.com
respublica-otel.ru romhacking.ru
retreatia.com rostov.xrus.org
rff-cfal.info royalvegascasino.com
ric.info rukino.org
rniaeba.ga ruscoininvest.company
rocketchange.ru rvi.biz
rome2rio.com saitevpatorii.com
royal-betting.net sammlungfotos.online
runstocks.com sarf3omlat.com
rutor.group saugatuck.com
sa-live.com saveindex.xyz
sakhboard.ru saveriopiazza.it
sanyuprojects.com scanner-jack.top
sbtdesign.co.uk schalke04fc.info
scanner-irvin.top scottbywater.com
scanner-marwin.top sdelatmebel.ru
scenarii-1-sentyabrya.uroki.org.ua seasaltwithfood.com
scrapinghub.com seemoreresultshu-a.akamaihd.net
search-error.com seksvideoonlain.com
searchlock.com seo-platform.com
secretscook.ru seobility.net
securesmrt-dt.com seoexperimenty.ru
segol.tv seofirmreviewsus.info
semalt.com sevendays.com.ua
seoanalyses.com
seocheki.net
seolab.top
service-core.ru
sethrollins.net
sex-watch.com sex-watch.com
sex-znakomstva.online sexiporno.net
share-buttons-for-free.com sexkontakteao.info
sharebutton.net sexphoto.site
shitmovs.com sexvideo-sex.com
shtaketniki.kz sexystrippe.info
shikiso.info
shinikiev.com.ua
shoppingjequiti.com.br
shymkent.xkaz.org shymkent.xkaz.org
simpoed.ufop.br sibvitr.ru
site3.free-share-buttons.com sisi-go.ru
siteheart.net site.ru
slkrm.ru siteimprove.com
slutloadlive.com skyway24.ru
smokewithrabbits.com sloopyjoes.com
snsdeainavi.info smallseotools.com
sofit-dmd.ru social-fun.ru
soft1.ru social-search.me
soietvousmaime.fr sockshare.net
space-worry.ml softlinesolutions.me
space4update.pw spb.ru
special-porn.com speedup-my.site
sportbetfair.com
sps-shop.com
standardchartered-forex.com
stard.shop stard.shop
stickers-market.ru start.myplaycity.com
sterva.cc
stillmiracle.com
stockquotes.wooeb.com stockquotes.wooeb.com
strangeduckfilms.com success-seo.com
stretchingabuckblog.com
strigkaomsk.ru
studiofaca.com
sukarame.net sukarame.net
superinterstitial.com
superkanpo.com superkanpo.com
supermesta.ru superlist.biz
svarkagid.com supervesti.ru
svtrd.com sureone.pro
sweepstakes.rewardit.com svetodiodoff.ru
syndicate.fun swagbucks.com
tattooha.com sygraem.com
teastory.co takeflyte.com
tecspb.ru takeprofitsystem.com
teenforporn.com taqplayer.info
teenporn18.net taximytishi.ru
telefonsexi.com telefonsexsofort.tk
telefonsexkostenlos.tk the-trader.net
tetracsaudi.com thecoral.com.br
tgtclick.com
thegioixekhach.com
thejournal.ru
themestotal.com themestotal.com
thepantonpractice.co.uk theporndude.com
thexart.club timer4web.com
tiens2010.ru tizanidinehcl2mgsideeffects.blogspot.com
timhost.ru tizanidineshowupondrugtest.blogspot.com
tizanidinepills.blogspot.com tizanidinewithdrawalsymptoms.blogspot.com
tksn.ru trackmedia101.com
topdownloads.ru trafficshaper.com
torontoplumbinggroup.com
torrentgamer.net
toy-shop.top
tradedeals.biz
trafficstars.com trafficstars.com
trkdf.com traffictrade.life
trubywriting.com truck-addzilla.life
tuberkuleznik.ru ttsq.fr
turvgori.ru turist-strani.ru
tvorozhnaja-zapekanka-recept.ru twu.com.ua
twiclub.in ua.tc
twitlinks.com uamtrk.com
twojebook.pl uchil.net
umnovocaminho.com ucoz.ru
unitygame3d.com ucsol.ru
universals.com.ua ultimateclassicrock.com
unmaroll.ya.ru ultramart.biz
upproar.com unblocksit.es
uptime-alpha.net unimodemhalfduplefw.pen.io
usswrite.com unpredictable.ga
velpanex.ru vacances-voyages.info
videooko.weebly.com vapomnoncri.tk
virtuagirl.com vektorpress.ru
vodkoved.ru vesnatehno.com
volgograd.xrus.org via-energy-acquistare.com
voloo.ru videochat.bz
vykup-avto-krasnodar.ru videotuber.ru
vzubah.com videtubs.pl
wallpapersist.com viktoria-center.ru
web-betting.ru visa-pasport.ru
websiteexplorer.info vizitki.net
weightatraining.com vkmusics.ru
wemarketing.se vsdshnik.com
westermarkanjou.se vseigru.one
westum.se vzlom-na-zakaz.com
whatistizanidine2mg.blogspot.com wallpaperaccess.com
webkeyit.com
webpromotion.ae
website-datenbank.de
website-speed-check.site
wedding0venues.tk
wholinkstome.com
wildworld.site
williamrobsonproperty.com
wineration.com
wingsoffury2.com wingsoffury2.com
winx-play.ru wiosenny-bon-1500.pl
wma-x.com witherrom55.eklablog.fr
womens-journal.net wordpress-crew.net
womensterritory.ru www.solartek.ru
wpsecurity.website www.timer4web.com
www.ehscloud.cn wygraj-skiny.win
www.pinnacle-bets.com xn--80aanaardaperhcem4a6i.com
x-diesel.com xn--b1adccaf1bzj.xn--p1ai
xitjw.info xn--c1acygb.xn--p1ai
xkaz.org xn--e1afanlbnfckd7c3d.xn--p1ai
xlolitka.com xnxx699.com
xmronta.com xtremeeagles.net
xn----7sbaaabaei0cc8aj5bj0bncejx.xn--p1ai yes-com.com
xn--80abgj3a5acid6ghs.top yginekologa.com
xn--h1ahbi.com.ua yougame.biz
xvideosporn.biz yourmovies.pl
xwatt.ru yuweng.info
xxxasianporn.net zaimhelp.ru
xxxdatinglocal.us zastroyka.org
xxxmania.top zeroredirect10.com
yellowfootprints.com zeroredirect5.com
yoshkarola.zrus.org zeroredirect9.com
youbrainboost.asia zixizop.net.ru
yourads.website
youraticles.pl
yourporngay.com
yunque.pluto.ro
zaimite.ru
zajm-zalog-krasnodar.ru
zeroredirect7.com
zhorapankratov7.blogspot.com
zigarettenonlinekaufen2.bloog.pl
zmoda.hostreo.com
znakomstva-piter78.ru znakomstva-piter78.ru
zolotoy-lis.ru
zoodrawings.com
zoogdiesney.com zoogdiesney.com
zoompegs.com
zophim.me zophim.me
zscaler.net zverokruh-shop.cz

478
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
10bet.com 12bet.com
4kepics.com 178evakuator178.ru
4pp13.com 2000k.ru
4replicawatch.net 2drittel.de
4webmasters.com 4webmasters.org
57883.net 5kstore.com
8xv8.com 8jn.dating-app.ru
aa9d046aab36af4ff182f097f840430d51.com 999webdesign.com
abcguru.xyz adsloads.com
accessoires-mode.in
adf.ly
adtiger.tk adtiger.tk
adultfullhd.com advocatemsk.ru
albamargroup.com adzerg.com
alldezservice.kz alert.scansafe.net
amateurmatch.com alfa-img.com
anal-acrobats.com allfinweb.com
apartmentratings.com allwidewallpapers.com
appmsr.org alpharma.net
apptester.tk anaseracresar.tk
asdfz.pro android-systems.ru
asscenihotosidea.blogspot.com apiitsthirteende-a.akamaihd.net
atley.eu.pn apimossnetinfo-a.akamaihd.net
ausmepa.org.au apper.de
autoseo-traffic.com articlesdirectoryme.info
avkzarabotok.info arvut.org
azartmsl.com asiavirtualsolutions.com
bala.getenjoyment.net atleticpharm.org
betterhdporn.com awency.com
bezlimitko.xyz axbocz.net
billiard-classic.com.ua ayanaspa.com
bitcoinremote.com ayerbo.xhost.ro
bitporno.sx azbaseimages.net
bloggedporn.com beauby.ru
brains2.biz bestprofits-there.com
brus-vsem.ru bikini-image.com
bumascloset.com blogig.org
bobba.dzaba.com
bocoarchives.org
bonanza-fortune.men
bonvillan.com
browserprotecter.com
btnativenav.com
buzz-porno.info
c-english.ru
cactussoft.biz cactussoft.biz
callawaygolfoutlet.online canacopegdl.com
callawaygolfstore.online carina-sy.de
candiceloves.us carsplate.com
candycrushshop.com cdncash.org
cartierlove.xyz cellfun.mobi
casinopinup-wins.com checkhit.com
catalogs-parts.com chorus.terakeet.com
cfjump.com citysecurity.nu
chidporn.com clash-clans.ru
chuckguilford.com classiquebijoux.ru
cicaki.net
clubfashionista.com
cmhomestayagency.com
cocolyze.com
coderstate.com
collegerentals.com
compliance-fred.xyz compliance-fred.xyz
compliance-josh.top cookie-law-enforcement-hh.xyz
copblock.org coolwallpapers-hd.com
cornomase.win counciltally.com
country-chic.ru covadhosting.biz
craftburg.ru crackzplanet.com
creditmoney.com.ua crynet.cc
custodieva.ru danschawbel.com
dame-ns.kz data-mining.tk
decoratinghomes.ga data.vtc.pw
dev-seo.blog dating-time-now.com
devochki.top day-news.info
dyshagi.ru divatraffic.com
econom.co dktr.ru
edwinkonijn.com.au dolohen.com
efkt.jp download-of-the-warez.blogspot.com
eleimgo.pw doxysexy.com
elektronischezigarettekaufen1.beeplog.com educatemetv.com
erotikstories.ru ee77ee.com
excitacion.info en.home-task.com
ezigarettenkaufen1.deviantart.com enternet.ee
ezigaretteshop2.mywapblog.com erotische-geschichten-xxl.com
familyphysician.ru eshop.md
farmingworm.com feargames.ru
ferieboligkbh.dk fickkontakte.org
fickblock18.com filmfanatic.com
findclan.org flex4launch.ru
fingerprintjs.com footbalive.org
fishingwholesale.us forex21.ru
food.dtu.dk forpostlock.ru
free-floating-buttons.com forum69.info
frustrated-favorable.gq fotosfotos.eu
gateway.zscalerone.net free-share-buttons.com
getoutofdebtfree.org front.ru
growmyfunds.ca funponsel.com
hanwei.us fx-brokers-review.com
hchha.com fxtips.ru
hdimagegallery.net game300.ru
heliko.no gaz-voshod.ru
howlongdoestizanidinestayinyoursystem.blogspot.com goodhumor24.com
husky-shop.cz googlepositions.com
i-hobot.ru googst2.ru
ilmexico.com gratuitxblcodes.com
imagerydatabase.com growboxbl.ru
indianmedicaltourismshop.com handicapvansarea.com
handicapvantoday.com
hccoder.info
herehloadibs.cf
hermesbracelets.xyz
hidemyass.com
hotblognetwork.com
hoverboard360.es
ictizanidinehcl4mg.blogspot.com
igtools.club
imperiafilm.ru
impisr.edunsk.ru
importchinacoach-teach.com
in-tandem.co
inboxdollars.com
innodgfdriverhm.aircus.com innodgfdriverhm.aircus.com
iqs.biz.ua insurple.com
iron-age.info iqoption.pro
iscblog.info irunfar.com
iwanttodeliver.com ispac.org
jav-way.site iyasimasennka.com
jetsli.de javcoast.com
juliadiets.com junglenet-a.akamaihd.net
kakablog.net jyrxd.com
keywordblocks.com kalb.ru
keywordhut.com kareliatobacco.ru
keywords-monitoring-success.com kasino-money.pw
khadastoafarde.tk
kino-key.info kino-key.info
kino-rating.ru kleine-titten.biz
kinoflux.net klosetkitten.com
kinohit1.ru kovesszucs.atw.hu
kuzinsp.ru ladsup.com
lazymanyoga.com lawyers.cafe
le-clos-des-alouettes.com letolove.ru
linkwithin.com linkpulse.com
livefixer.com linkredirect.biz
livetsomudvekslingstudent.bloggersdelight.dk liveinternet.ru
livingroomdecor.info lonerangergames.com
localflirtbuddies.com long-beach-air-conditioning.com
machicon-ueno.info longlifelomilomi.info
macnewtech.com lumb.co
mafa-free.com m3gadownload.pl
magento-crew.net mackeeper-center.club
map028.com macotool.com
mathgym.com.au magazin-pics.ru
medicovi.com magda-gadalka.ru
mega-bony-2017.pl managerpak204.weebly.com
megavolt.net.ua masserect.com
mere.host.sk mmoguider.ru
mobilemedia.md mnogabukaff.net
mobot.site moncleronline.xyz
moi-glazki.ru moskva.nodup.ru
monclercheap.xyz musicstock.me
moneyzzz.ru myseoconsultant.com
morf.snn.gr nfljerseysforsalewholesaler.com
mp3s.club nudejapan.net
msk.afora.ru oakleyglassesonline.us
musicpro.monster office-windows.ru
muznachas-service.ru onko-24.com
nalogovyy-kodeks.ru open-odyssey.org
negral.pluto.ro openfrost.net
net-profits.xyz palvira.com.ua
nomuos.it
nullrefer.com
o-o-8-o-o.com
offside2.5v.pl
okonich.com.ua
old-rock.com
onlinadverts.com
ostroike.org
ourtherapy.ru
owohho.com
oyster-green.com
p-business.ru
page2rss.com
paighambot.com
panicatack.com
paperwritingservice17.blogspot.ru
par-fallen.ga
paramountmarble.co.uk paramountmarble.co.uk
partner-trustworthy.men partner-stop.men
pinsdaddy.com partnerline.men
pirateday.ru patterntrader-en.com
piulatte.cz phrcialiled.com
pix24x7.com pihl.se
pornoblood.com plusnetwork.com
pornosok.ru pobeiranie.pl
pornozhara.com porngalleries.top
positive2b.ru porno-raskazy.ru
privatamateure.com potoideas.us
prizestohandle.club preconnubial.usuby.site
privetsochi.ru
pro-poly.ru pro-poly.ru
progonrumarket.ru projefrio.com.br
prombudpostach.com.ua prosmibank.ru
psoriasis-file.trade py100.ru
qwarckoine.com rada.ru
rankexperience.com ranksays.com
rankingchart.de razvratnoe.org
rating-bestcasino.com recinziireale.com
reklama-i-rabota.ru replicalouboutin.xyz
replicaclub.ru reversing.cc
resant.ru rfid-locker.co
retailwith.com rfserial.net
reyel1985.webnode.fr riralmolamsaca.tk
ros-ctm.ru risparmiocasa.bz.it
rustic-quiver.win rvzr-a.akamaihd.net
s.lollypopgaming.com ryetaw.com
scanner-george.top sabaapress.com
sdsjweb.com santechnik.jimdo.com
semrush.com satellite.maps.ilovevitaly.com
seoheap.com scanner-alex.top
schoolfiles.net
seocheckupx.com
seoexperimenty.ru
seofirmreviewsus.info
serpstat.com serpstat.com
sex-tracker.de sex-dating.co
sfd-chess.ru sfd-chess.ru
shell-pmr.ru sitechecker.pro
shillyourcoins.com siteimprove.com
shtora66.ru skatestick.bid
sigmund-freud.co.uk slim.sellany.ru
signal03.ru snjack.info
sims-sims.ru social-s-ggg.xyz
siteexpress.co.il sogimlecal.tk
skypasss.com spin2016.cf
smadihome.com
socialbutton.xyz
solmarket.by
spl63.fr
stair.registrydomainservices.com stair.registrydomainservices.com
steebook.com staynplay.net
stroilka.info strangeduckfilms.com
stromerrealty.com strigkaomsk.ru
superlist.biz studiofaca.com
sweepstakes.rewardit.com
syndicate.fun syndicate.fun
tedy.su tabakur77.com
templates.radiodigital.co tehngr.ru
themeforest.net telefonsexsofort.tk
tizanidinewithdrawal.blogspot.com timecrimea.ru
tomck.com tizanidine4mg.blogspot.com
tonerbox.kz tizanidinehcl2mgsideeffects.blogspot.com
torontoplumbinggroup.com tizanidineonline.blogspot.com
traffic2money.com tjkckpytpnje.com
trafficnetzwerk.de trainoffend.ml
ttsq.fr tri-slona.org
tvand.ru trubywriting.com
ucoz.ru truck-addzilla.life
umnovocaminho.com tzritel.tk
uptime.com unacittaconte.org
usdx.us unece.org
via-gra.webstarts.com vavilone.com
wallpapers-best.com vereo.eu
wesharepics.com vesnatehno.ru
wildnatureimages.com videosbox.ru
winx-play.ru vpnmouse.com
workle.website wardreapptokone.tk
worst-sites.online webjam.com
www.888.com wemarketing.se
www.get-free-traffic-now.com wjgony.com
xn----7sbbahaq9bb5afgiqfliv4m.xn--p1ai wstroika.ru
xn--b1ag5cfn.xn--p1ai www.souvenirua.com
yopoint.in x-true.info
youbrainboost.asia xbaboon.com
zaim-pod-zalog-krasnodar.ru xn-----8kcatubaocd1bneepefojs1h2e.xn--p1ai
zaloadi.ru xn--80aodinpgi.xn--p1ai
zeroredirect9.com xrus.org
zigarettenonlinekaufen.tumblr.com xvideosporn.biz
yellowstonesafaritours.com
youporn-ru.com
ytmnd.com
zambini.ru

139
dev-tools/test_units/ratelimittest-master.txt
View file

@ -8,72 +8,6 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html>ml>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><html> </html><html>
<head><title>503 Service Temporarily Unavailable</title></head> <head><title>503 Service Temporarily Unavailable</title></head>
<body> <body>
@ -494,6 +428,69 @@
<hr><center>nginx/1.18.0 (Ubuntu)</center> <hr><center>nginx/1.18.0 (Ubuntu)</center>
</body> </body>
</html> </html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<!DOCTYPE html> <!DOCTYPE html>
<html> <html>
<head> <head>
@ -584,4 +581,14 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html> </html>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.20 0; 10.1.0.107 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2023.01.3543 ### Version: V4.2023.01.3544
### Updated: Tue Jan 10 22:04:29 UTC 2023 ### Updated: Thu Jan 12 22:04:48 UTC 2023
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2023.01.3543 ### Version: V4.2023.01.3544
### Updated: Tue Jan 10 22:04:29 UTC 2023 ### Updated: Thu Jan 12 22:04:48 UTC 2023
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##