vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-02 18:50:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.06.3148 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-06-05 10:09:15 +00:00
parent e3e1910a26
commit 4e07e464e6
46 changed files with 104424 additions and 104424 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.06.3147 #### Version: V4.2022.06.3148
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.06.3147 #### Version: V4.2022.06.3148
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.06.3147 #### Version: V4.2022.06.3148
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12916
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.06.3147 #### Version: V4.2022.06.3148
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

12920
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12920
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

12920
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

12920
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 1; 10.1.0.118 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 0; 10.1.0.118 0;
127.0.0.1 0; 127.0.0.1 0;

12920
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 1; 10.1.0.118 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 0; 10.1.0.118 0;
127.0.0.1 0; 127.0.0.1 0;

12920
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 1; 10.1.0.118 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 0; 10.1.0.118 0;
127.0.0.1 0; 127.0.0.1 0;

12920
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3147 3148

12920
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

12920
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.242.57.154 0; 20.225.239.106 0;

12920
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 1; 10.1.0.118 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 0; 10.1.0.118 0;
127.0.0.1 0; 127.0.0.1 0;

12920
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 1; 10.1.0.118 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 0; 10.1.0.118 0;
127.0.0.1 0; 127.0.0.1 0;

12920
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 1; 10.1.0.118 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 0; 10.1.0.118 0;
127.0.0.1 0; 127.0.0.1 0;

12920
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.242.57.154 0; 20.225.239.106 0;

12920
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.242.57.154 0; 20.225.239.106 0;

12920
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 1; 10.1.0.118 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Blow BlowFish
BunnySlippers CCBot
Copyscape Custo
Curious Dragonfly
Humanlinks ExtLinksBot
Jetty Litemage_walker
LWP::Simple Moblie Safari
RankActiveLinkBot SentiBot
WBSearchBot SmartDownload
cognitiveseo s1z.ru

318
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com ADmantX
AIBOT
ALittle Client
ASPSeek ASPSeek
Aboundexbot Aboundex
Acunetix AiHitBot
AfD-Verbotsverfahren Aipbot
AhrefsBot
AllSubmitter
Alligator Alligator
Anarchie
Anarchy Anarchy
Anarchy99 Anthill
Ankit Apexoo
Attach
BackStreet
BackWeb BackWeb
BacklinkCrawler BacklinkCrawler
BatchFTP Badass
BlackWidow Bitacle
Blackboard
Blow Blow
BlowFish Bolt
Buck Buck
Buddy Buddy
Bullseye BuiltBotTough
BuzzSumo
CATExplorador CATExplorador
CODE87 CCBot
CSHttp
Calculon
Cegbfeieh Cegbfeieh
CheTeam CheeseBot
CherryPicker Chlooe
Citoid Cliqzbot
Cloud mapping
Cocolyzebot Cocolyzebot
Cosmos Cogentbot
Craftbot Copyscape
Crawling at Home Project
Crescent
CrunchBot CrunchBot
CyotekWebCopy DBLBot
DSearch Deusu
DataCha0s DigitalPebble
Dirbuster
Disco Disco
DittoSpyder DnBCrawler-Analytics
DomainAppender DnyzBot
DomCopBot
DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
DomainStatsBot
Download Wonder Download Wonder
Dragonfly Dragonfly
Drip EMail Wolf
EMail Siphon
Ebingbong Ebingbong
EirGrabber EroCrawler
Exabot Extractor
ExtLinksBot FDM
ExtractorPro FemtosearchBot
Extreme Picture Finder
EyeNetIE
Firefox/7.0
Flunky Flunky
FrontPage Freeuploader
Fuzz GT::WWW
G-i-g-a-b-o-t
GalaxyBot GalaxyBot
Getintent Genieo
Gigabot GetRight
Go-Ahead-Got-It
GoZilla GoZilla
Grabber Grabber
GrapeshotCrawler Grafula
GridBot
HEADMasterSEO HEADMasterSEO
HMView HTMLparser
HTTrack
Haansoft Haansoft
Harvest IDBTE4M
HonoluluBot Id-search
IRLbot
IlseBot
Image Fetch
Image Sucker Image Sucker
IndeedBot IndeedBot
InfoTekies Indy Library
Internet Ninja Intelliseek
IstellaBot Iskanie
JamesBOT
JetCar JetCar
JikeSpider Joomla
Kenjin Spider Kenjin Spider
Keybot Translation-Search-Machine Kinza
Keyword Density
Kozmosbot Kozmosbot
LNSpiderguy Lanshanbot
LWP::Simple
Leap
LexiBot
Libwhisker Libwhisker
Lightspeedsystems
Likse
LinkScan
Linkbot Linkbot
LinkextractorPro LinkpadBot
LinksManager
LinqiaMetadataDownloaderBot LinqiaMetadataDownloaderBot
LinqiaRSSBot LinqiaRSSBot
LinqiaScrapeBot Lipperhey
Lipperhey Spider
Litemage_walker
Ltx71
MFC_Tear_Sample MFC_Tear_Sample
MIDown tool Mag-Net
MSFrontPage Magnet
MSIECrawler
Majestic SEO
Majestic-SEO
Majestic12 Majestic12
MarkMonitor Mata Hari
Mass Downloader
MauiBot
Mb2345Browser
MeanPath Bot MeanPath Bot
Meanpathbot
Mediatoolkitbot Mediatoolkitbot
Minefield Metauri
Mister PiX Mister PiX
Morfeus Fucking Scanner Mojolicious
Mozlila MolokaiBot
Msrabot NICErsPRO
Nameprotect
NearSite NearSite
Net Vampire
NetAnts
NetMechanic NetMechanic
NetSpider
NetZIP NetZIP
Netcraft Netcraft
Nettrack Nettrack
NextGenSearchBot Netvibes
Nimbostratus Nikto
Ninja Ninja
Nmap Nmap
Nutch Offline Navigator
Octopus
OnCrawl OnCrawl
OpenVAS Openfind
OutclicksBot Openvas
OutfoxBot OutfoxBot
PHPCrawl PECL::HTTP
PageAnalyzer PageGrabber
PageScorer
PageThing.com PageThing.com
Panscient Panscient
Pavuk Pavuk
PeoplePal
Petalbot Petalbot
Picscout Picscout
Picsearch PictureFinder
Piepmatz
Pimonster
Pockey Pockey
ProWebWalker ProPowerBot
Psbot Probethenet
PyCurl Quick-Crawler
QueryN Metasearch
RSSingBot RSSingBot
RankActiveLinkBot Rankivabot
RankurBot Re-re
Recorder ReGet
RealDownload
RebelMouse
RedesScrapy RedesScrapy
RepoMonkey
Ripper Ripper
RocketCrawler
SEOkicks-Robot SEOkicks-Robot
SEOprofiler SEOlyticsCrawler
SMTBot SEOstats
ScanAlert SalesIntelligent
Scanbot
ScoutJet
Scrapy
Screaming
ScreenerBot ScreenerBot
SearchmetricsBot ScrepyBot
Seekport Semrush
SeoSiteCheckup Shodan
Seomoz
Siphon
Site Sucker Site Sucker
SiteLockSpider SiteLockSpider
SiteSnagger SiteSnagger
SlySearch SiteSucker
SmartDownload Sitebeam
Snake Sitevigil
Snoopy
SocialRankIOBot SocialRankIOBot
Sogou web spider Sogou web spider
Sosospider
Sottopop
SpaceBison
Spammen Spammen
Spanner Spanner
Spinn3r Spinn3r
Sqlmap SputnikBot
Sqlworm Stripper
SuperBot Sucker
Surfbot SurveyBot
Swiftbot
Szukacz Szukacz
T8Abot
Teleport Teleport
Telesphoreo The Intraformant
Telesphorep
Titan Titan
Trendiction Toata
True_Robot Turnitin
TurnitinBot
TwengaBot TwengaBot
Twice URLy.Warning
Typhoeus UnisterBot
Upflow VelenPublicWebCrawler
V-BOT VidibleScraper
Vacuum VoidEYE
VeriCiteCrawler WASALive-Bot
Voltron WEBDAV
Wallpapers/3.0 WISENutbot
WWW-Collector-E
WWWOFFLE
Wallpapers
WallpapersHD
WeSEE WeSEE
Web Auto Web Auto
Web Collage
Web Enhancer Web Enhancer
Web Fetch
Web Fuck
Web Pix
Web Sauger Web Sauger
WebAuto WebBandit
WebCollage
WebCopier WebCopier
WebImageCollector WebEnhancer
WebFetch
WebGo IS
WebLeacher WebLeacher
Website Quester WebWhacker
Whack Webalta
Whacker WebmasterWorldForumBot
Webshag
Webster
Who.is Bot Who.is Bot
Widow Widow
WinHTTrack
Wonderbot Wonderbot
Xaldon_WebSpider Wprecon
Zauba YoudaoBot
Zermelo Zermelo
adscanner Zeus
ZoomBot
ZumBot
ZyBorg
arquivo-web-crawler arquivo-web-crawler
arquivo.pt
backlink-check backlink-check
check1.exe check1.exe
com.plumanalytics clark-crawler
domainsproject.org coccocbot
evc-batch cognitiveseo
gopher crawler4j
facebookscraper
heritrix
isitwp.com isitwp.com
netEstate NE Crawler lwp-trivial
page scorer meanpathbot
pcBrowser pcBrowser
plumanalytics plumanalytics
polaris version polaris version
probe-image-size ripz
scalaj-http s1z.ru
seobility scan.lol
seoscanners seocompany.store
seostar
serpstatbot serpstatbot
siteripz siteripz
sogouspider
sp_auditbot
spyfu
sysscan
tAkeOut
trendiction.de
ubermetrics-technologies.com ubermetrics-technologies.com
webmeup-crawler webmeup-crawler
webpros.com webprosbot
x22Mozilla
xpymep1.exe
zauba.io zauba.io
zgrab

304
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
360Spider 360Spider
80legs 404enemy
ADmantX ADmantX
AIBOT
Acunetix Acunetix
AfD-Verbotsverfahren AfD-Verbotsverfahren
AhrefsBot Aipbot
Alexibot
AllSubmitter
Alligator
Anarchie
Anarchy99 Anarchy99
Ankit Apexoo
Anthill Asterias
Aspiegel
Atomseobot Atomseobot
Attach Attach
BBBike BBBike
BDCbot BDFetch
BackDoorBot BackWeb
BackStreet Backlink-Ceck
Badass Badass
Bandit
Barkrowler Barkrowler
BatchFTP Bigfoot
Black Hole
BlackWidow BlackWidow
Blackboard Blackboard
Blow Blow
BlowFish Boardreader
BotALot
Brandprotect Brandprotect
Buck Buck
Buddy BuiltBotTough
BuiltWith BuiltWith
Bullseye Bullseye
BunnySlippers BunnySlippers
CATExplorador
CCBot CCBot
Calculon CODE87
CazoodleBot CSHttp
Cegbfeieh
CensysInspect CensysInspect
CheTeam CheTeam
CheeseBot
CherryPicker CherryPicker
Citoid
Claritybot
Cliqzbot
Cloud mapping
Cocolyzebot Cocolyzebot
Cogentbot Collector
Crawling at Home Project Crescent
CrazyWebCrawler
CrunchBot CrunchBot
Custo Curious
DIIbot DBLBot
DSearch DataCha0s
DatabaseDriverMysqli
Deusu Deusu
Devil DigitalPebble
DittoSpyder Dirbuster
DnBCrawler-Analytics Discobot
DomainAppender
DomainCrawler
DomainSigmaCrawler
DomainStatsBot DomainStatsBot
Express WebPictures Dotbot
Dragonfly
ECCP/1.0
EMail Siphon
EirGrabber
EroCrawler
ExtLinksBot ExtLinksBot
ExtractorPro Extreme Picture Finder
EyeNetIE EyeNetIE
Flunky Fimap
Freeuploader Freeuploader
FrontPage Fyrebot
G-i-g-a-b-o-t GalaxyBot
GermCrawler
GetRight GetRight
GetWeb GetWeb
GrabNet Getintent
Go!Zilla
GoZilla
Grabber Grabber
Grafula
GrapeFX
GrapeshotCrawler GrapeshotCrawler
GridBot GridBot
HEADMasterSEO
HMView
HTMLparser
HTTP::Lite HTTP::Lite
Harvest Hloader
Heritrix Humanlinks
HybridBot HybridBot
IDBTE4M
IRLbot IRLbot
Iblog
IlseBot IlseBot
Image Fetch Image Sucker
Indy Library IndeedBot
InfoNaviRobot InfoNaviRobot
InfoTekies
InterGET
Internet Ninja
InternetSeer
Iskanie Iskanie
IstellaBot IstellaBot
JOC Web Spider
JamesBOT JamesBOT
Jbrofuzz
JetCar JetCar
JustView Jetty
Keyword Density JikeSpider
Kenjin Spider
LNSpiderguy LNSpiderguy
Larbin LWP::Simple
Leap Lanshanbot
Lftp
LibWeb
Libwhisker
LieBaoFast LieBaoFast
Likse Likse
Linkbot
LinkextractorPro LinkextractorPro
LinksManager LinksManager
LinqiaRSSBot
LinqiaScrapeBot LinqiaScrapeBot
Lipperhey Lipperhey
Lipperhey Spider
Lmspider
MJ12bot
MQQBrowser MQQBrowser
MSFrontPage MSFrontPage
MSIECrawler
MTRobot MTRobot
Magnet Mail.RU_Bot
Majestic SEO MarkMonitor
Majestic-SEO
Majestic12
MarkWatch MarkWatch
MauiBot Mass Downloader
Mata Hari
Mb2345Browser Mb2345Browser
Mediatoolkitbot Meanpathbot
Metauri Metauri
Microsoft Data Access Moblie Safari
Microsoft URL Control
Mister PiX
Mojeek
Mojolicious Mojolicious
MolokaiBot MolokaiBot
Morfeus Fucking Scanner Mozlila
Msrabot
Nameprotect Nameprotect
Navroad
NearSite NearSite
Nessus Net Vampire
NetAnts
NetSpider NetSpider
NetZIP NetZIP
Nettrack
Netvibes Netvibes
Nmap Ninja
Offline Explorer Nuclei
OnCrawl Offline Navigator
OrangeSpider OpenLinkProfiler
OutclicksBot Openfind
OutfoxBot
PageGrabber PageGrabber
PeoplePal Pavuk
Petalbot Petalbot
Pi-Monster Pi-Monster
Pimonster Pimonster
Pixray
ProWebWalker ProWebWalker
Pump
PxBroker PxBroker
RankActiveLinkBot Quick-Crawler
RankActive
RankFlex
Rankivabot Rankivabot
RankurBot Reaper
Re-re
RebelMouse RebelMouse
RedesScrapy Recorder
RepoMonkey
Rogerbot Rogerbot
SEOkicks SBIder
SEOkicks-Robot Scanbot
SEOstats ScoutJet
SISTRIX Scrapy
SalesIntelligent ScreenerBot
ScanAlert SearchmetricsBot
ScrepyBot Seekport
SemanticJuice SeoSiteCheckup
Semrush Siphon
SeobilityBot SiteExplorer
Seomoz Sitevigil
Shodan
SiteCheckerBotCrawler
SiteSucker
SlySearch SlySearch
SmartDownload SmartDownload
Sogou web spider Snapbot
Spanner SocialRankIOBot
Sqlmap Sociscraper
Sqlworm SpaceBison
Sqworm Spammen
Sucuri SpankBot
SuperHTTP Spinn3r
Steeler
Stripper
SuperBot
Surfbot Surfbot
Suzuran SurveyBot
Swiftbot Telesphorep
T0PHackTeam TightTwatBot
TeleportPro Titan
Telesoft
The Intraformant
Thumbor
Toata
Toweyabot
Tracemyfile
Trendiction Trendiction
Typhoeus Trendictionbot
URLy Warning Twice
URLy.Warning UnisterBot
VB Project VB Project
Vagabondo VCI
Vacuum
VelenPublicWebCrawler VelenPublicWebCrawler
Voltron VoidEYE
WEBDAV WASALive-Bot
WISENutbot WISENutbot
WPScan WPScan
WWW-Collector-E WWW-Mechanize
Wallpapers WallpapersHD
Web Fetch Web Sucker
Web Pix
WebAuto
WebCopier WebCopier
WebEnhancer WebEnhancer
WebFuck WebImageCollector
WebGo IS
WebLeacher
WebPix WebPix
WebReaper WebmasterWorldForumBot
WebWhacker
Webalta
Webshag Webshag
Website Quester Website Quester
WebsiteExtractor Whatweb
Widow WiseGuys Robot
Wprecon Wonderbot
Xaldon WebSpider Woobot
Xenu Xenu
YoudaoBot YoudaoBot
Zade Zade
Zeus Zermelo
ZmEu ZmEu
ZoomBot ZumBot
arquivo-web-crawler ZyBorg
archive.org_bot
arquivo.pt arquivo.pt
autoemailspider
cah.io.community cah.io.community
check1.exe clark-crawler
coccocbot coccocbot
cognitiveseo cognitiveseo
com.plumanalytics crawler.feedback
crawl.sogou.com dataforseo.com
crawler4j evc-batch
ips-agent gopher
isitwp.com instabid
iubenda-radar linkdexbot
meanpathbot
muhstik-scan
plumanalytics
polaris version polaris version
ripz ripz
s1z.ru seobility
sitechecker.pro serpstatbot
sogouspider sp_auditbot
sysscan spyfu
tAkeOut tAkeOut
trendiction.com voyagerx.com
trendiction.de
webgains-bot webgains-bot
webpros.com
webprosbot webprosbot
x22Mozilla x09Mozilla
xpymep1.exe zauba.io
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
alphahoverboards.com budilneg.xyz
bambi.ck.ua extstat.com
ebooknovel.club fashionindeed.ml
elektir.ru fatfasts-4tmz.com
forum69.info hustoon.over-blog.com
fusoradio.info mini.7zap.com
gungamesz.com sneakyboy.com
osnova3.ru spasswelt.net
pixell.club timecrimea.ru
twojebook.pl x-mix.info

934
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
00webcams.com 11235813.webzdarma.cz
03e.info 13tabs.com
14b.info 1688.com
1qingdao.com 1flag.co.za
1stat.ru 1hwy.com
2ads.co.uk 21h2o.com
2nt.ru 2pxg8bcf.top
3dracergames.com 45en.ru
4webmasters.com 4kepics.com
55wmz.ru 4webmasters.org
5elementov.ru
70casino.online
91abcw.com
DomainStatsBot
a.frcls.fr
a14download.com
aab94f698f36684c5a852a2ef272e031bb.com
absugars.com
abtasty.com
actulite.com actulite.com
acunetix-referrer.com adanih.com
addbin.men adhome.biz
ads-cool.pro adminshop.com
adultwebhosting.info adnotbad.com
affrh2015.com adobereader-free.ru
adultactioncam.com
afora.ru
alert-fdm.xyz
alfa-medosmotr.ru
algarveglobal.com algarveglobal.com
alibestsale.com alkoravto.ru
all4invest.info allproblog.com
android-systems.ru amanda-porn.ga
angkortours.vn ananumous.ru
animali.deagostinipassion.it anime.dougasouko.com
animaltoplist.com ap.senai.br
animebox.com.ua apartamentwroclaw.eu
anlimebel.kiev.ua apibrowsesmartne-a.akamaihd.net
apibetweenlinesn-a.akamaihd.net apidigidocketnet-a.akamaihd.net
apilinkswiftco-a.akamaihd.net apiholdingmypage-a.akamaihd.net
apimountainbikei-a.akamaihd.net apispringsmartne-a.akamaihd.net
approvedlocal.co.za approvedlocal.co.za
appsecurityr.com arclk.net
architecturebest.com asrvvv-a.akamaihd.net
arendakvartir.kz asseenontv.ru
asrvrep-a.akamaihd.net atovh.local-finders.com
aussie-prizes.com australia-opening-times.com
auto4style.ru autochoixspinelli.com
autobudpostach.club aviav.ru.com
autorn.ru avramstroy.ru
autovideobroadcast.com
aviav.co
aviav.org
avtoarenda.by avtoarenda.by
avtochehli.by azbukafree.com
avtocredit-legko.ru babyfactory.fr
ayerbo.xhost.ro backlinkwatch.com
azartmix.com bannerbank.ru
babosas.com bannertesting.com
banki76.ru bestdooz.com
battleforupdating.site binomo.com
bcmp.org bioca.org
beauty-b0x.pl bitcoin-ua.top
beauty-bracelet.com bizlist.com.de
best-seo-software.xyz blog-hits.com
besttorrentknifta.weebly.com bloggerads.net
bestwebsitesawards.com bloggers.nl
bezcmexa.ru blogqpot.com
bimatoprost-careprost.com bluesalt.co
bingo8888.com bmw-ark.ru
blavia.00author.com bnt-team.com
blog.koreadaily.com bonzbuddy.com
bluejays-jerseys.us boobsimge.com
bolegapakistan.com boom-celebs.com
boleznikogi.com bpro1.top
bonvillan.com brainxs.us
bonzibuddi.com brillianty.info
brandbucket.com buckspar.gq
brewdom.ru budpost.com.ua
brg8.com bugshoot.cn
brimstonehillfortress.org buqayy0.livejournal.com
browsepulse-a.akamaihd.net candlewooddentalcentre.com.au
buigas.00it.com canrioloadm.gq
buttons-for-website.com
bvlgaribracelet.xyz
c-english.ru
candelluxsklep.pl
capsderfudd.tk capsderfudd.tk
cashonads.com capture-room.com
cercacamion.it cardsdumps.com
cfjump.com cartierreplica.pw
checkpagerank.net cdncash.org
chimiver.info celejihad.info
chuckguilford.com charmstroy.info
cienum.fr chidporn.com
cityadspix.com chinajerseyswholesalecoupons.com
chorus.terakeet.com
cialisovercounteratwalmartusa.com
cjs.com.ru
clickbanksites.info
clicksotrk.com clicksotrk.com
clothesforcash.com clippingphotoindia.com
companies-catalog.com club-samodelkin.ru
compliance-alexa.xyz communisave.co.za
concealthyself.com compliance-brian.top
continent-e.tv conboy.us
cookingmeat.ru conocer-sanabria.com
counciltally.com cordstrap.cc
courtshipgift.com countbertwistdisp26.soup.io
covadhosting.biz covetnica.com
creatives.adbetclickin.pink cowblog.fr
credit-online.ws cscwtalkto.site
creditwell.ru dafatiri.com
customsua.com.ua daisye.top
cypernhuset.se dame-ns.kz
dailyrank.net
daooda.com daooda.com
dating-time-now.com dating-time-now.com
datsun-do.com daydream-studio.ru
dengi-pod-zalog-nedvizhimosti.ru ddpills.com
destinationrealestate.com deadlinkchecker.com
devochki.top dear-diary.ru
dfwu1013.info degerlund.net
dfwu1019.info dekoration.us
discovertreasure-a.akamaihd.net delta-line.men
disruptingdinnerparties.com dentfidemountpreach.tk
diusyawiga.tk deshevo-nedorogo.ru
dnepropetrovsk.mistr-x.org digital-video-processing.com
docs4all.com diminishedvalueoforegon.com
domainanalyzing.xyz discountbarbour.online
domini.cat dms-sw.ru
draniki.org dogoimage.com
drpornogratisx.xxx domoysshop.ru
drupa.com dptaughtme.com
dtm-spain.com druzhininevgeniy63.blogspot.com
dverimegapolis.ru dscaas.website
dvervmoskvu.ru e2click.com
e-commerce-seo1.com earncash.com.ua
ekn-art.se eas-seo.com
ekosmetyki.net.pl ecommerce-seo.com
ekspertmed.com educontest.net
election.interferencer.ru elektrischezigaretten1.blogse.nl
elektrischezigarettekaufen2.cowblog.fr emediate.eu
elektronischezigarettekaufen1.beeplog.com englate.com
elitesportsadvisor.com enpolis.ru
elvenmachine.com enternet.ee
emerson-rus.ru erotikchat-24.com
eropornosex.ru euronis-free.com
euroskat.ru
evaashop.ru evaashop.ru
event-tracking.com ezb.elvenmachine.com
evrotekhservis.ru ezigarettekaufen2.yolasite.com
excitacao.com ezigarettenkaufen1.deviantart.com
extrabot.com ezigarettenkaufen2.dreamwidth.org
extremal-blog.com faithe.top
ezigarettekaufen2.mpbloggar.se fanoboi.com
f-online.de farm26.ru
factorynightclub.com fast-wordpress-start.com
fantasticpixcool.com favornews.com
faptitans.com fetlifeblog.com
farmingworm.com filmfanatic.com
fast-torrent.ru filmi-v.online
fenoyl.batcave.net find1friend.com
fickkontakte.org foodgid.net
fighrofacciufreesig.ga forodvd.com
filesclub.net forum.poker4life.ru
findthe.pet foxtechfpv.com
foto-weinberger.at franecki.net
franklinfire.co
frcls.fr
freakycheats.com freakycheats.com
free-fb-traffic.com freefoto.ca
fsalas.com gallerylisting.com
funponsel.com gambarkataku.co
fxund.us gameonasia.com
galaxy-family.ru games.kolossale.ru
gambarkatabaru.com gaypornmovie.net
gameplexcity.com generousdeal-a.akamaihd.net
gazporno.com gepezz.info
gembird.com
gemgrab-a.akamaihd.net
germetiki.com.ua germetiki.com.ua
getlaid-xxxhookupdirect.com girlsfuckdick.com
gk-atlant.info
goldadpremium.com goldadpremium.com
goldandcard.ru golmau.host.sk
gomusix.com goo.ne.jp
growmyfunds.ca goodwriterssales.com
gsasearchengineranker.space googlemare.com
guge.io googleseo.com.tr
gungamesz.com gov.yanao.ru
hamacapty.com grand-chlen.ru
happy.new.yeartwit.com grizzlysgrill.com
harmonyglen.us groupmoney.ru
headpress.ru guitar-master.org
himgaws.pw gwagka.com
hmmm.cz hdpixion.com
homeandhealth.ru hdxnxxtube.mobi
hopeonthestreet.co.uk healgastro.com
host-protection.com hexpilot.com
hostingclub.lk hifidesign.ru
hotblog.top hiwibyh.bugs3.com
hollywoodactress.info
homik.pw
horoshieokna.com
hotblognetwork.com hotblognetwork.com
hothor.se hoterika.com
housediz.com ico.re
howlongdoestizanidinestayinyoursystem.blogspot.com
howtowhitenteethfast.xyz
hubbble.com
humbmosquina.tk
hut1.ru
i-midias.net.br
igadgetsworld.com
ilikevitaly.com ilikevitaly.com
imicrovision.com images-free.net
incolors.club imguramx.pw
infobabki.ru impisr.ru
instabid.tech import-sales.com
intelhdgraphicsgtdrive6w.metroblog.com indianmedicaltourismshop.com
intim-uslugi.info indiasourcemart.in
investingclub.ru infogame.name
ivearchenceinflu.cf interesnie-faktu.ru
ivoiretechnocom.ci interior-stickers.ru
izoll.ru internetproviderstucson.com
jackwolfskinoutlet.online investmac.com
janettabridal.com invivo.hu
jav-fetish.site inzn.ru
javitas.info iplogger.org
jongose.ninja ipornox.xxx
jwss.cc isitpaleo.info
kaliningrad.zrus.org isotoner.com
keywordspay.com itag.pw
iwantmyfreecash.com
jbl-charge.info
jerseywholesalebizchina.com
justdating.online
kandidos.com
kashubadesign.ru
keywordhut.com
kfon.eu
kino-ecran.ru kino-ecran.ru
kinocccp.net kiprinform.com
konkursowo-24.pl kit-opt.ru
koversite.info kleine-titten.biz
kraljeva-sutjeska.com kochanelli.com
kriokomora.info koleso24.com.ua
laptoper.net kopihijau.info
lesbian.xyz kosova.de
letto.by kotaku.com
lietaer.com krafte.ru
lineavabit.it lanadelreyfans.us
linksharingt.com landmania.ru
lion.misba.us laxdrills.com
littlesunraiser.com lazy-z.com
live-cam6.info letolove.ru
live-sexcam.tk likesuccess.com
liveu.infoteka.hu liquimondo.com
lkbennettoutlet.online listiseltemournlan.gq
lonely-mature.com lol-smurfs.com
lost-alpha.ru lonfon.xyz
lovelycraftyhome.com louboutinshoes.xyz
luckyshop.net.ua lrac.info
m5home.ru luciddiagnostics.in
mahnwachen-helfen.info lunamedia.co
mantramusic.ru lyrics.home-task.com
mb140.ru m4ever.net
mega-polis.biz.ua macfix.life
mafa-free.com
magicalfind-a.akamaihd.net
make-money-online.com
makedo.ru
makemoneyonline.com
manimpotence.com
mansiondelrio.co
markjaybeefractal.com
mbiologi.ru
mearns-tractors.co.uk
meccadumps.net
meet-flirt-dating.com
megaindex.ru megaindex.ru
middlerush-a.akamaihd.net mere.host.sk
mikrobiologies.ru midst.eu
modnie-futbolki.net migente.com
moncleroutletonline.pw mini-modus.ru
moncleroutletonline.xyz mixed-wrestling.ru
monitorwebsitespeed.com mmostrike.ru
monthlywinners.com mnogabukaff.net
moscow-region.ru modabutik.ru
motherboard.vice.com moinozhki.com
mototsikl.org mole.pluto.ro
moviemail-online.co.uk moncleroutletonline.win
mrinsidesales.com moneymaster.ru
mrmoneymustache.com
msfsaar.de
mural.co mural.co
muzaporn.com musicpro.monster
my-big-family.com musicspire.online
myfreecams.com myfreetutorials.com
myhitmp3.club na-telefon.biz
myindospace.com najaden.no
myplaycity.com newhairstylesformen2014.com
mywallpaper.top nextconseil.com
nalogovyy-kodeks.ru nhl17coins.exblog.jp
nalogovyykodeks.ru nippon-bearings.ru
namecrumilchlet.tk nonews.co
narkologiya-penza.ru notasprensa.info
narkologiya-peterburg.ru nucia.biz.ly
narutonaruto.ru oballergiya.ru
net-profits.xyz obiavo.su
newstraveller.ru obnal.org
next-dentists.tk onlineku.com
nightvision746.weebly.com onlywoman.org
norththeface.store
novatech.vn
npoet.ru
ntic.fr
nudo.ca
o-o-11-o-o.com
obiavo.com
offer.camp
offers.bycontext.com
omsk.xrus.org
oneminutesite.it
oneshotdate.com
onlinedomains.ru
ooomeru.ru ooomeru.ru
orgasmus-virtual.com oops-cinema.ru
orhonit.com openlibrary.org
panamaforbeginners.com openstat.com
perm-profnastil.ru oralsexfilme.net
osb.se11.ru
ovirus.ru
painting-planet.com
panicatack.com
parajumpersstore.online
park.above.com
parlament.biz
pateaswing.com
paydayloanslocal.com
pcimforum.com
pekori.to
perfectpracticeweb.com
perm.xrus.org
photo-clip.ru
photorepair.ru photorepair.ru
phpdevops.com picsfair.com
picquery.com plastgranar.nu
picurams.pw playboyfiles.xblog.in
pigrafix.at
pihl.se
pl.aasoldes.fr
pod-muzyku.club pod-muzyku.club
popup.matchmaker.com poem-paying.gq
porn555.com polcin.de
popugaychiki.com
popunder.net
pornhive.org
porno-chaman.info porno-chaman.info
pornosee.info porno-play.net
pornosemki.info pornodojd.ru
pornoklad.ru
pornonik.com
pornophoto.xyz
pornotubs.com pornotubs.com
pornoxxx.com.mx
portal-eu.ru portal-eu.ru
powc.r.ca.d.sendibm2.com postclass.com
pravoholding.ru priora-2.com
priceg.com prodvigator.ua
pro-poly.ru progress-upakovka.ru
productarium.com prohoster.info
prombudpostach.com.ua prokotov.com
promover.org pulse33.ru
pronorm.fr
proposal-engine.com
prostitutki-odessa.org
pufip.com
purplestats.com
pushdata.sendpulse.com pushdata.sendpulse.com
putitin.me qld10000.net
quizzitch.net queerspace.com
qwertty.net rankexperience.com
r-e-f-e-r-e-r.com realizmobi.com
rapevideosmovies.com realtytimes.com
rapidsites.pro registrysweeper.com
rczhan.com reklama1.ru
recipedays.ru remorcicomerciale.ro
refads.pro renhacklids.tk
reimageplus.com restaurantlescampi.com
resant.ru rightenergysolutions.com.au
reversing.cc rniaeba.ga
ritlweb.com romhacking.ru
roleforum.ru ru-mediaget.ru
rozalli.com rubanners.com
rus-teh.narod.ru rusprostitute.com
rusvideos.su sadaholding.com
saddiechoua.com saecsa.co
santasgift.ml sajatvelemeny.com
satoristudio.net sanatorrii.ru
sax-sex.com sashagreyblog.ga
scanner-jane.top scanner-brian.top
scanner-jess.top scanner-marwin.top
schalke04fc.info scanner-walter.top
scripted.com scansafe.net
sdelai-prosto.ru scrapy.org
searchimpression.com se-welding.ru
searchinquire.com sei80.com
secret.xn--oogle-wmc.com seksvideoonlain.com
seinterface.com semprofile.com
semaltmedia.com semrush.com
seo18.su seo-traffic-ranking.info
seocheki.net seomarketings.online
seolab.top seorankinglinks.com
seorank.info seorankinglinks.us
servingnotice.com service-core.ru
servisural.ru sexphoto.site
seryeznie-znakomstva.ru sexspornotub.com
sexstream.pl sexstream.pl
sexy-pings.com sexuria.net
sexyteens.hol.es sharebutton.org
shikiso.info shareyards.com
shillyourcoins.com shopcheermakeup.info
shlyahten.ru shtaketniki.kz
shoesonlinebuy.cn
sicfor.bcu.cc sicfor.bcu.cc
similardeals.net sideeffectsoftizanidine.blogspot.com
sindragosa.comxa.com
sirpornogratis.xxx sirpornogratis.xxx
sisiynas.ru sisi-go.ru
site3.free-share-buttons.com site-speed-checker.site
siteripz.net sixcooler.de
sitevaluation.org sky-mine.ru
sitopreferito.it slavic-magic.ru
sizeplus.work smadihome.com
skladvaz.ru smartadserver.com
skutecznetabletkinaporostwlosow.pl snip.tw
snip.to socialbutton.xyz
snowplanes.com socialsignals24.com
snsdeainavi.info softwaretrend.net
soaksoak.ru sonnikforme.ru
socialmadesimple.com sosdepotdebilan.com
socialtrade.biz souvenir.cc
spbchampionat.ru spabali.org
sport7777.net sportbetfair.com
stevemonsen.com stair.registrydomainservices.com
stickers-market.ru startwp.org
streetfire.net staynplay.net
stroiminsk.com stillmiracle.com
stroyplus.ru stop-gepatit.te.ua
success-seo.com strana-solnca.ru
sukarame.net strongholdsb.ru
supervesti.ru svnuppsalaorebro.se
surintech.ac.th swinginwithme.ru
swinon.site sygraem.com
tabletkinaodchudzanie.com.pl tagil.zrus.org
taboola.com tam-gde-more.ru
taihouse.ru
takeflyte.com
tamada69.com
td-33.ru
td-l-market.ru td-l-market.ru
teenbbw.yopoint.in templates.franklinfire.co
thecoral.com.br theheroes.ru
thefarmergame.com thepokertimer.com
thegioixekhach.com thepornsex.org
themeforest.net timecrimea.ru
threecolumnblogger.com tizanidineformigraines.blogspot.com
tizanidineandgabapentin.blogspot.com tizanidineoral.blogspot.com
tizanidineingredients.blogspot.com tizanidinewithdrawal.blogspot.com
tizanidinesideeffects.blogspot.com tizanidinewithdrawalsymptoms.blogspot.com
tizanidinesideeffectsweightloss.blogspot.com tksn.ru
tizanidinestreetprice.blogspot.com todohr.com
tizanidinestreetvalue.blogspot.com
tmtrck.com
tomck.com tomck.com
toondinsey.com tongkatmadura.info
tracfone.com toonfamilies.net
track-rankings.online top-l2.com
trahic.ru torrentred.games
tubo360.com toys.erolove.in
tuckermktg.com trainoffend.ml
tula.howotorg.ru truck-rece.life
tvnewsclips.info trustedhealthtips.com
tvoystartup.ru trustedmaccleaner.com
ukrup.com tsyndicate.com
unisexjewelry.org twojebook.pl
url-img.link typer.one
uvozdeckych.info uac.net.au
vabasa.inwtrade.com ukrobstep.com
vancleefreplica.pw univerfiles.com
velpanex.ru uptime-gamma.net
veopornogratis.xxx uroffer.link
versaut.xxx-cam.webcam usbggettwku.ga
videochaty.ru utrolive.ru
videokrik.net valkiria-tk.ru
videotuber.ru vbtracker.net
viel.su vduplo.ru
vip-file.com vekzdorov.ru
virtualbb.com vepad.com
visa-pasport.ru videooko.weebly.com
vizag.kharkov.ua vitoriacabos.com
vpnmouse.com vkak.ru
vremya.eu voloo.ru
vsexkontakte.net vseigry.fun
vvpg.ru vtc.pw
vykupavto-krasnodar.ru w3javascript.com
walpaperlist.com warning.or.kr
wareseeker.com watchinf.com
website-speed-checker.site webnode.me
website-speed-up.top webshoppermac.com
websiteaccountant.de websites-reviews.com
wejdz-tu.pl weburok.com
wemarketing.se wetgames.ru
westen-v.life wholesalejerseychinaoutlet.com
westen-z.life wholesalejerseychinashop.com
wholinkstome.com wildnatureimages.com
wladimirpayen.com winterclassichockeyjerseys.com
wleuaprpxuvr.ga word-vorlagen.net
wnoz.de wsgames.ru
wordkeyhelper.com wygraj-skiny.win
works.if.ua wyniki-lista.pl
worldoffiles.ru
wormix-cheats.ru
writingservices17.blogspot.ru
wurr.voila.net
x-musics.com x-musics.com
x-stars.ru x5market.ru
x69ty.ru xblognetwork.com
xmladserver.com xmlinde.com
xn----7sbifcamovvfggw9d.xn--p1ai xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai
xn--1-8sbcpb0bdm8k6a.xn--p1ai xn-----7kcabaipgeakzcss7bjdqdwpfnhv.xn--p1ai
xn--80abgj3a5acid6ghs.top xn-----7kceclhb4abre1b4a0ccl2fxch1a.xn--p1ai
xn--80ajbshivpvn2i.xn--p1ai xn--80aanaardaperhcem4a6i.com
xn--80aodinpgi.xn--p1ai xn--b1adccaf1bzj.xn--p1ai
xolodremont.ru xn--d1acah0c.xn--p1ai
xpresscare.ru xnxxandxvideos.com
xsfetish.org xxart.ru
xsion.net xxxnatelefon.ru
xvideospornoru.com
xxxsiterips.xyz
y8games-free.com
yadro.ru
yellocloud.be
yellowads.men
yellowproxy.net yellowproxy.net
youbrainboost.asia yes-com.com
youdao.com yoluxuryevents.com
yougame.biz youdesigner.kz
youghbould.wordpress.com yuweng.info
youtubedownload.org
yurgorod.ru
za-music.mymobiplanet.com za-music.mymobiplanet.com
zagadki.in.ua zaim-pod-zalog-krasnodar.ru
zaimite.ru
zaloadi.ru zaloadi.ru
zarajbuilders.com zambini.ru
zed21.net zastroyka.org
zetgie.com.pl zojirushi-products.ru
zigarettenonlinekaufen2.drupalgardens.com zolotoy-lis.ru
zocaparj.kz zrelaya.pw
zootravel.com zryydi.com

484
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
007angels.com 1000n1.ru
00webcams.com 100dollars-seo.com
101billion.com 178evakuator178.ru
2020iscoming.info 1flag.co.za
5-steps-to-start-business.com 1stat.ru
7zap.com 2kata.ru
8gold.com 4kepics.com
9med.net 7makemoneyonline.com
addray.pro aa625d84f1587749c1ab011d6f269f7d64.com
adidas.frwebs.fr abovetherivernc.com
admanaerofoil.com acheterviagrafr24.com
acmebtn.ml
addbin.men
adf.ly
adrunnr.com
adsland.men
adtech.us adtech.us
adultfullhd.com aghanyna.com
advancedmassagebysara.com amigobulls.com
advocatemsk.ru amung.us
agroeconom.kz anapa-inns.ru
air-edem.ru android-vsem.org
alphavisions.net angel1777.kz
anal-acrobats.com apibrowsesmartne-a.akamaihd.net
andpolice.com apiwoodensealcom-a.akamaihd.net
anonymousfox.co ascat.porn
apiappenableinfo-a.akamaihd.net asiavirtualsolutions.com
apioutoboxnet-a.akamaihd.net astrochicks.com
app-ready.xyz avek.ru
arquapetrarca.info aviva-limoux.com
arvut.org azartniy-bonus.com
asdfg.pro azbuka-mo.ru
asscenihotosidea.blogspot.com bad-stars.net
b00kmarks.com bank.uz
bannerswap.com bavsac.com
betterscooter.com beachpics.com
bitporno.sx becuo.com
blog-hits.com bestofferswalkmydogouteveryday.gq
blogporn.in big-cash.net
blogqpot.com biteg.xyz
bochemit.com.ua bloke.com
c-english.ru bronzeaid-a.akamaihd.net
c-gainsbourg.com c1.onedmp.com
cartierlove.xyz carrauterie.be
christianlouboutinsaleonline.us cartierlove2u.xyz
compliance-barak.top cb.iphantom.com
compliance-checker-7.info cenoval.ru
contentlook.co chocoslim.pro
converse.ddsoldes.fr clarithromycin500mg.com
countbertwistdisp26.soup.io cleanmypc.com
creams.makeforum.eu clickintext.net
culmimg.pw clickzzs.nl
d2jsp.org clksite.com
decorazilla.com compliance-elena.top
deda-moroza-zakaz.ru dafatiri.com
deniven.1bb.ru damasarenai.info
detsky-nabytek.info danashop.ru
dikqlyremy.info data-ox.com
doctissimo.fr dealwifi.com
documentbase.net deutschehobbyhuren.net
dojki-hd.com directrev.com
dosug-lux.ru diy-handmade-ideas.com
dpihatinh.gov.vn doyouknowtheword-flummox.ml
dtm-spain.com dustyorate.com
e705.net dvr.biz.ua
earncash.com.ua dyshagi.ru
earthmagic.info empirepoker.com
ednorxmed.com ereko.ru
edwinkonijn.com.au eropho.net
eleimgo.pw erot.co
en.altezza.travel expolicenciaslatam.co
estelight.ru ezb.elvenmachine.com
expdom.com factorynightclub.com
exportshop.us falcon-images.blogspot.com
extremez.net feargames.ru
ezigarettekaufen1.hpage.com feel-planet.com
ezigarettenkaufen1.deviantart.com fireads.men
f-online.de fm-upgrade.ru
facebook-mobile.xyz fortevidyoze.net
familienzahnaerzte.com free-gluten.ru
farmprofi.net free-share-buttons.top
figuringmoneyout.com fsakhalin.ru
filkhbr.com ftns.ru
findclan.org fungirlsgames.net
flash4fun.com games.kolossale.ru
forpackningsutveckling.se ge0ip.org
foto-telok.net generalporn.org
fotzen-ficken.com gentamicineyedrops.blogspot.com
free411games.com getyourimage.club
freegamesplay.online go2album.com
freemaintenancesysforpcandmac.top
freeseedsonline.com
gambarxkata.co
gamesprite.me
gearcraft.us
getprismatic.com
girlsatgames.ru
glall.ru
golden-catalog.pro
goldenggames.com
good-mummy.ru
goodhumor24.com goodhumor24.com
gtopstats.com grupografico-pilar.com.ar
hccoder.info gta-top.ru
help.tpu.ru habermetre.com
helpmymacfaster.trade hamptonoaks.ca
homedo.fabpage.com handsandlegs.ru
hostnow.men hd-film.pl
hotdl.in headpress.ru
hulfingtonpost.com homemature.net
iklysha.ml housekuba.org
in-tandem.co impresagaia.it
incolors.club inclk.com
infazavr.ru ip-guide.com
infodocsportal.com ipostroika.ru
infogame.name isitpaleo.info
instasexyblog.com
int.search.mywebsearch.com
ipornox.xxx
istripper.com
itronics.ca
iwantmyfreecash.com
ix20.ru ix20.ru
iyasimasennka.com jamiembrown.com
jasonpartington.com javstock.com
jjbabskoe.ru jerseywholesalebizchina.com
johnrobertsoninc.com jyrxd.com
jovencitas.gratis kandidos.com
kakadu-interior.com.ua kerwinandcariza.com
kam-dom.ru keyword-suggestions.com
kemerovo.zrus.org kol-energo.ru
keywordglobal.co.za laapp.com
kirov.zrus.org lacave.ntic.fr
kosynka-games.ru law-check-eight.xyz
kredit-pod-zalog-krasnodar.ru law-check-nine.xyz
kupiproday.com.ua letolove.ru
kustanay.kz linkbuddies.com
landmania.ru loadopia.com
lasvegaslockandsafe.com lovasszovetseg.hu
linkredirect.biz m3gadownload.pl
livingcanarias.com macrotek.ru
lotto6888.com mainlinehobby.net
louboutinreplica.xyz marketingvici.com
mahnwachen-helfen.info max-p.men
manualterap.roleforum.ru megaapteka.ru
mcadamssupplyco.com minet.club
mebel-vstroika.ru
meendo-free-traffic.ga
metaxalonevstizanidine.blogspot.com
mobile.ok.ru
moetomnenie.com
mojpreskumanie.com
moncleroutletonline.win moncleroutletonline.win
monetizationking.net mp3films.ru
monetizer.com-01.site mrwhite.biz
monthlywinners.com mvpicton.co.uk
moonci.ru mydeathspace.com
mrcsa.com.au naperehresti.info
muztops.ru nastydollars.com
myanyone.net naturalpharm.com.ua
mycaf.it ngps1.ru
myftpupload.com nootrino.com
naphukete.ru nuker.com
notebook-pro.ru o-o-6-o-o.ru
oballergiya.ru ochistka-stokov.ru
optibuymac.com offf.info
partner-print.men okout.ru
partnersafe.men orgasmatrix.com
pateaswing.com outpersonals.com
photorepair.ru ownshop.cf
pivka.xyz paighambot.com
playfortuna-play.ru palocco.it
porngalleries.top partner-cdn.men
pornoblood.com partner-trustworthy.men
postclass.com pc-services.ru
praisong.net pdfprof.com
push-ad.com pdns.download
qcstrtvt.bloger.index.hr perl.dp.ua
qld10000.net pertlocogasilk.tk
recinziireale.com pornmania.pl
renecaovillasale.online porno-home365.com
resultshub-a.akamaihd.net potolokelekor.ru
rocketchange.ru pretty-mart.com
ros-ctm.ru pricheskaonline.ru
rospromtest.ru printdirectforless.com
royalvegascasino.com printie.com
ryetaw.com professionalwritingservices15.blogspot.ru
scanner-walt.top profit-opportunity.com
selectads.men prpops.com
seryeznie-znakomstva.ru pwwysydh.com
sevendays.com.ua ragecash.com
sexkrasivo.net ranksonic.info
spamnuker.com responsive-test.net
spin2016.cf rogervivierforsale.com
squidoo.com rus-teh.narod.ru
stanthonyscatholicchurch.org russkoe-zdorovie.ru
stard.shop rustic-quiver.win
streetfooduncovered.com s-forum.biz
stroy-portal22.ru sabaapress.com
subj.ukr-lit.com sarf3omlat.com
sudexpert66.ru sarm.tk
szucs.ru savefrom.com
taximytishi.ru saveindex.xyz
scanner-irvin.top
se.bnt-team.com
searchimpression.com
senger.atspace.co.uk
shillyourcoins.com
sisiynas.ru
sitechecker.pro
siteimprove.com
skatestick.bid
slavia.info
sledstvie-veli.net
social-s-ggg.xyz
sockshares.tv
songoo.wz.cz
spabali.org
space4update.pw
sps-shop.com
sr-rekneskap.no
steame.ru
streamin.to
studentguide.ru
suhanpacktech.com
svetlotorg.ru
svtrd.com
swingerseiten.com
tattoo33.ru
taxi-v-eisk.ru
theautoprofit.ml theautoprofit.ml
theendivechronicles.com tizanidinecapsules.blogspot.com
thepokertimer.com tizanidinedruginteractions.blogspot.com
timecrimea.ru tizanidinehcl2mg.blogspot.com
tizanidine6mg.blogspot.com trafficnetzwerk.de
tizanidineandgabapentin.blogspot.com treasuretrack-a.akamaihd.net
tizanidinesleepaid.blogspot.com trustedmaccleaner.com
tmearegion26.com turizm.bz
tongkatmadura.info ua.tc
torrent-newgames.com uptime-delta.net
track2.shop usacasino.com
tracksurf.daooda.com ussearche.cf
tri-slona.org
tupper-posuda.ru
twittrading.com
typer.one
u555u.info
v24s.net
vegascosmetics.ru
vepad.com
vgoloveboli.net vgoloveboli.net
vksex.ru videojam.tv
voronezh.xrus.org vids18.site
vseigru.one viteonlusarezzo.it
vzlom-na-zakaz.com viveresaniesnelli.it
walpaperlist.com wareseeker.com
waterpurifier.club webjam.com
webs.com webmasterhome.cn
webstatsdomain.org webmasters.stackexchange.com
website-stealer.nufaq.com
websiteexplorer.info
wellcome2slovenia.ru
wholesalejerseys-cheapest.com
whosonmyserver.com whosonmyserver.com
wildnatureimages.com wikes.20fr.com
wpsecurity.website wingsoffury2.com
writingservices17.blogspot.ru workle.website
ww1943.ru ws.ampower.me
www.ehscloud.cn x-lime.net
www.labves.ru xn----7sbifcamovvfggw9d.xn--p1ai
xboxster.ru xn--80aeahghtf8ac5i.xn--p1ai
xmlinde.com xn--b1addnj3cah.xn--p1ai
xn--80atua3d.xn--p1ai xn--l1aengat.xn--p1ai
xnxx-n.com xxxasianporn.net
xuki.us yoluxuryevents.com
xxlargepop.com youresponsive.com
xxxnatelefon.ru zapiszto.pl
yes-com.com
yougetsignal.com
yoursite.com
yuarra.pluto.ro
za-music.mymobiplanet.com
zakazfutbolki.com
zeroredirect10.com zeroredirect10.com
zetgie.com.pl zetmaster.ru
znakom.sibtest.ru zrizvtrnpale.tk
zscalerone.net

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.14 0; 10.1.0.118 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.06.3147 ### Version: V4.2022.06.3148
### Updated: Sat Jun 4 22:01:22 UTC 2022 ### Updated: Sun Jun 5 10:03:18 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.06.3147 ### Version: V4.2022.06.3148
### Updated: Sat Jun 4 22:01:22 UTC 2022 ### Updated: Sun Jun 5 10:03:18 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##