vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-05 04:00:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.06.3169 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-06-15 22:07:09 +00:00
parent f3ef5c028a
commit 36799d0cb8
46 changed files with 105566 additions and 105566 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.06.3168 #### Version: V4.2022.06.3169
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.06.3168 #### Version: V4.2022.06.3169
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.06.3168 #### Version: V4.2022.06.3169
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

13058
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.06.3168 #### Version: V4.2022.06.3169
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

13062
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

13062
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

13062
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

13062
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 1; 10.1.0.43 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 0; 10.1.0.43 0;
127.0.0.1 0; 127.0.0.1 0;

13062
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 1; 10.1.0.43 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 0; 10.1.0.43 0;
127.0.0.1 0; 127.0.0.1 0;

13062
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 1; 10.1.0.43 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 0; 10.1.0.43 0;
127.0.0.1 0; 127.0.0.1 0;

13062
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3168 3169

13062
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

13062
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.242.27.228 0; 20.245.26.250 0;

13062
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 1; 10.1.0.43 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 0; 10.1.0.43 0;
127.0.0.1 0; 127.0.0.1 0;

13062
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 1; 10.1.0.43 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 0; 10.1.0.43 0;
127.0.0.1 0; 127.0.0.1 0;

13062
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 1; 10.1.0.43 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 0; 10.1.0.43 0;
127.0.0.1 0; 127.0.0.1 0;

13062
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.242.27.228 0; 20.245.26.250 0;

13062
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.242.27.228 0; 20.245.26.250 0;

13062
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 1; 10.1.0.43 1;
127.0.0.1 1; 127.0.0.1 1;

18
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
BunnySlippers Ankit
DataCha0s LeechFTP
DigitalPebble Mb2345Browser
Mediatoolkitbot Page Analyzer
MegaIndex.ru Voil
Metauri WWWOFFLE
NetMechanic
PageAnalyzer
SpaceBison
Web Fuck Web Fuck
WebCopier
Zermelo
ips-agent

320
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
360Spider
404checker
80legs 80legs
ADmantX ALittle Client
ASPSeek Aboundex
Aboundexbot AiHitBot
AhrefsBot
Aipbot Aipbot
Alexibot
Alligator Alligator
AlphaBot
Anarchie Anarchie
Anarchy
Anarchy99 Anarchy99
Aspiegel
Atomseobot
AwarioRssBot AwarioRssBot
BBBike
BDCbot
BDFetch
BLEXBot BLEXBot
BackDoorBot BackWeb
BacklinkCrawler Badass
Bitacle BatchFTP
Black Hole Bolt
Blackboard BotALot
Blow
Brandprotect Brandprotect
Brandwatch
Buddy Buddy
BuiltBotTough
Bullseye Bullseye
BunnySlippers CSHttp
CODE87 Calculon
CazoodleBot ChinaClaw
Cegbfeieh
CheTeam
Chlooe Chlooe
Cocolyzebot Cliqzbot
Collector Cloud mapping
Copyscape Cosmos
Crawling at Home Project
CrazyWebCrawler
Crescent Crescent
DIIbot CrunchBot
DTS Agent Curious
DataCha0s Custo
DBLBot
DSearch
DatabaseDriverMysqli DatabaseDriverMysqli
Deusu
Devil
Digincore
DigitalPebble DigitalPebble
Dirbuster Disco
Discoverybot Discoverybot
DnBCrawler-Analytics Dispatch
DnyzBot
DomCopBot DomCopBot
DomainCrawler DomainSigmaCrawler
DomainStatsBot
Domains Project Domains Project
Dotbot
Dragonfly Dragonfly
EMail Wolf EMail Siphon
EasyDL EasyDL
Ebingbong
EirGrabber
EroCrawler EroCrawler
Exabot Evil
ExtLinksBot ExtLinksBot
Extreme Picture Finder Extreme Picture Finder
EyeNetIE EyeNetIE
FDM FDM
FemtosearchBot FHscan
FlashGet FrontPage
Flunky FyberSpider
Foobot G-i-g-a-b-o-t
Fuzz
Fyrebot
GT::WWW
GalaxyBot
GetRight
GetWeb
Getintent Getintent
Gigabot Gigabot
Go!Zilla Go-Ahead-Got-It
GoZilla
Grabber Grabber
GrapeFX GrapeshotCrawler
HMView HTTP::Lite
Haansoft Haansoft
Havij Havij
HonoluluBot Heritrix
HybridBot IRLbot
IDBTE4M
Id-search Id-search
IlseBot
Image Sucker
InfoNaviRobot
InfoTekies InfoTekies
Intelliseek InterGET
Internet Ninja JetCar
InternetSeer Jorgee
IstellaBot
JamesBOT
JikeSpider
Joomla
JustView JustView
Jyxobot Jyxobot
Kenjin Spider Kenjin Spider
Kinza Kinza
LNSpiderguy
Lanshanbot Lanshanbot
LeechFTP
LexiBot
Lftp
LibWeb LibWeb
LieBaoFast
Lightspeedsystems Lightspeedsystems
LinkScan
LinksManager
LinqiaMetadataDownloaderBot
LinqiaRSSBot LinqiaRSSBot
Lipperhey LinqiaScrapeBot
Lipperhey Spider Lipperhey Spider
Litemage_walker Lmspider
Ltx71 MIDown tool
MFC_Tear_Sample MJ12bot
MIIxpc Mag-Net
MSIECrawler Majestic-SEO
MTRobot Mata Hari
Magnet Mb2345Browser
Mail.RU_Bot MeanPath Bot
MarkMonitor MicroMessenger
Mass Downloader Minefield
Masscan Moblie Safari
Meanpathbot MolokaiBot
Mediatoolkitbot
Metauri
Microsoft Data Access
Mojeek
Morfeus Fucking Scanner
Mr.4x3 Mr.4x3
Msrabot Musobot
NICErsPRO NPbot
Nessus Name Intelligence
NetMechanic Nameprotect
Nettrack Navroad
Netvibes NearSite
Nibbler NetLyzer
Ninja NetSpider
Nuclei Netcraft
Niki-bot
NimbleCrawler
Not
Nutch
Octopus
Offline Explorer
Offline Navigator
OnCrawl OnCrawl
OrangeBot Openfind
OrangeSpider PageScorer
PECL::HTTP Picsearch
PageAnalyzer Piepmatz
PageGrabber ProPowerBot
Pandalytics Psbot
Papa Foto
Pavuk
Petalbot
Pump Pump
PxBroker
PyCurl
QueryN Metasearch QueryN Metasearch
RankActive RSSingBot
RankActiveLinkBot RankActiveLinkBot
Reaper RankingBot
Recorder RankingBot2
RedesScrapy Rankivabot
RankurBot
Re-re
ReGet
Ripper Ripper
SBIder Rogerbot
SEOkicks SEOkicks
SEOlyticsCrawler SEOkicks-Robot
SEOprofiler SEOprofiler
SISTRIX
SMTBot SMTBot
SalesIntelligent
ScanAlert ScanAlert
Scanbot Scanbot
ScoutJet
Scrapy
ScrepyBot ScrepyBot
SearchmetricsBot
Seekport Seekport
SemanticJuice
SemrushBot SemrushBot
SentiBot
Seomoz
Shodan Shodan
SiteCheckerBotCrawler
SiteSnagger SiteSnagger
Sitebeam
Siteimprove Siteimprove
Sitevigil
SlySearch
Snake
Snapbot Snapbot
Sociscraper
Sottopop
SpaceBison SpaceBison
SpankBot SpankBot
Spinn3r
Sqlmap Sqlmap
Sqlworm
Sqworm Sqworm
Sucker Steeler
Stripper
Sucuri Sucuri
SuperBot
SuperHTTP SuperHTTP
Swiftbot Surfbot
SurveyBot
Szukacz Szukacz
T0PHackTeam T0PHackTeam
T8Abot
Teleport Teleport
TeleportPro
Telesoft Telesoft
Telesphoreo Telesphorep
The Intraformant
TheNomad
TightTwatBot TightTwatBot
Trendiction Toata
TurnitinBot Toweyabot
Tracemyfile
TwengaBot
Typhoeus
URLy Warning URLy Warning
UnisterBot UnisterBot
VB Project Upflow
VCI
Vacuum
Vagabondo Vagabondo
VelenPublicWebCrawler VelenPublicWebCrawler
VeriCiteCrawler VoidEYE
Virusdie
Voil Voil
WASALive-Bot Voltron
WEBDAV WBSearchBot
WPScan WWW::Mechanize
WWW-Collector-E
WWWOFFLE
Wallpapers Wallpapers
WallpapersHD WeSEE
Web Fetch Web Enhancer
Web Pix WebCopier
Web Sauger WebEnhancer
WebBandit WebFetch
WebCollage
WebFuck
WebImageCollector WebImageCollector
WebLeacher WebLeacher
WebPix
WebReaper WebReaper
WebSauger
WebSucker
WebWhacker
WebZIP WebZIP
Webshag WebmasterWorldForumBot
Whack Website Quester
Whacker Whatweb
Widow Who.is Bot
Woobot WinHTTrack
Wotbox Wonderbot
Wprecon Wprecon
Xenu
Zade
Zermelo
Zeus Zeus
Zitebot Zitebot
ZmEu
ZumBot ZumBot
arquivo-web-crawler clark-crawler
arquivo.pt cognitiveseo
cah.io.community domainsproject.org
check1.exe
com.plumanalytics
crawl.sogou.com
crawler.feedback
demandbase-bot
evc-batch
heritrix
ips-agent ips-agent
oBot linkdexbot
page scorer mediawords
pcBrowser pcBrowser
s1z.ru
seocompany.store
seostar seostar
serpstatbot serpstatbot
sexsearcher sitechecker.pro
siteripz
sysscan sysscan
tAkeOut trendiction.com
ubermetrics-technologies.com voyagerx.com
webprosbot webmeup-crawler
x09Mozilla x09Mozilla
xpymep1.exe

322
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
360Spider 360Spider
404checker
80legs 80legs
ALittle Client ADmantX
Aboundexbot Aboundexbot
AfD-Verbotsverfahren AfD-Verbotsverfahren
AiHitBot
Aipbot Aipbot
AllSubmitter AllSubmitter
Alligator
AlphaBot AlphaBot
Anarchy Anarchie
Anarchy99 Anarchy99
Ankit Anthill
Atomseobot Atomseobot
AwarioRssBot Attach
BBBike AwarioSmartBot
BDCbot BDCbot
BDFetch BLEXBot
BackWeb
BacklinkCrawler
Barkrowler Barkrowler
BatchFTP BatchFTP
Battleztar Bazinga Battleztar Bazinga
BetaBot Bigfoot
Black Hole BlackWidow
Blow Blow
BlowFish
BotALot BotALot
Brandprotect
Brandwatch Brandwatch
BunnySlippers Buck
BuzzSumo Buddy
CCBot Bullseye
CazoodleBot CATExplorador
Cegbfeieh Cegbfeieh
CheTeam CherryPicker
CheeseBot
ChinaClaw ChinaClaw
Chlooe Citoid
Claritybot
Cloud mapping
Cocolyzebot Cocolyzebot
Cogentbot
Collector
Copier Copier
Copyscape
Cosmos
Crawling at Home Project Crawling at Home Project
Crescent CrazyWebCrawler
CrunchBot Custo
DBLBot CyotekWebCopy
DatabaseDriverMysqli Demon
Digincore Deusu
Dirbuster
Disco
Discobot
Discoverybot Discoverybot
Dispatch DomCopBot
DnBCrawler-Analytics DomainAppender
DnyzBot
DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
DomainStatsBot
Domains Project
Download Wonder
Drip Drip
EMail Siphon
EMail Wolf EMail Wolf
Ebingbong Ecxi
EroCrawler EirGrabber
Express WebPictures Evil
Extreme Picture Finder ExtLinksBot
Extractor
ExtractorPro
EyeNetIE
Ezooms
FemtosearchBot FemtosearchBot
Fimap Flunky
Fuzz Freeuploader
GT::WWW GalaxyBot
Genieo
GermCrawler GermCrawler
GetRight Getintent
GetWeb GrabNet
Gigabot Grabber
GoZilla GridBot
Grafula HEADMasterSEO
HMView HTTP::Lite
Hloader Haansoft
HonoluluBot HaosouSpider
IDBTE4M Harvest
IDBot Humanlinks
IRLbot
Iblog Iblog
Image Sucker Intelliseek
Indy Library Internet Ninja
InfoTekies Iskanie
Iria IstellaBot
JennyBot JamesBOT
JikeSpider JikeSpider
Keybot Translation-Search-Machine Jorgee
Keyword Density Kenjin Spider
LNSpiderguy LNSpiderguy
LWP::Simple LWP::Simple
Larbin Lanshanbot
LexiBot LeechFTP
LibWeb LibWeb
LinqiaMetadataDownloaderBot Libwhisker
Lightspeedsystems
LinksManager
Lipperhey
Lipperhey Spider
Lmspider
Ltx71
MIDown tool MIDown tool
MIIxpc
MJ12bot MJ12bot
MQQBrowser MSFrontPage
MTRobot MSIECrawler
Mata Hari Mag-Net
Magnet
Majestic SEO
Majestic12
Masscan
MeanPath Bot
Meanpathbot Meanpathbot
Mediatoolkitbot
MegaIndex.ru
Metauri Metauri
Microsoft Data Access
Microsoft URL Control Microsoft URL Control
Mister PiX Mozlila
Mojeek
Morfeus Fucking Scanner
Mr.4x3
Msrabot Msrabot
Musobot Musobot
Navroad NICErsPRO
Name Intelligence
Nameprotect
Nessus Nessus
Net Vampire Net Vampire
NetAnts NetAnts
NetSpider NetMechanic
Nettrack NetZIP
NextGenSearchBot Netcraft
Nibbler Niki-bot
Nikto NimbleCrawler
Nimbostratus Nimbostratus
Not Nutch
Nuclei Offline Explorer
OnCrawl Offline Navigator
OpenLinkProfiler
OpenVAS OpenVAS
Openfind
Openvas Openvas
OutfoxBot OrangeBot
OrangeSpider
PECL::HTTP PECL::HTTP
PHPCrawl PHPCrawl
Page Analyzer Pavuk
PageGrabber
PageThing.com
Panscient
PeoplePal PeoplePal
Pi-Monster Petalbot
Picsearch
PictureFinder PictureFinder
ProWebWalker Piepmatz
Pu_iN Pixray
Pump Probethenet
PxBroker PyCurl
QueryN Metasearch QueryN Metasearch
RSSingBot RSSingBot
RankActive
RankActiveLinkBot
RankingBot2
Rankivabot Rankivabot
ReGet Re-re
Reaper RebelMouse
RepoMonkey RepoMonkey
SEOkicks SEOkicks
SEOkicks-Robot
SEOstats SEOstats
SISTRIX SISTRIX
SMTBot SalesIntelligent
ScoutJet
Scrapy Scrapy
SearchmetricsBot Screaming
Seekport ScreenerBot
SemanticJuice Searchestate
Seomoz Siphon
SiteCheckerBotCrawler Site Sucker
SiteExplorer SiteExplorer
SmartDownload SiteSnagger
Sitebeam
Snake Snake
Snoopy Snapbot
SocialRankIOBot SocialRankIOBot
Sogou web spider
Sosospider
Sottopop
SpaceBison SpaceBison
Spanner Spbot
Spinn3r Spinn3r
SputnikBot
Sqlmap
Sqworm Sqworm
Steeler
Stripper Stripper
Surfbot Sucker
Swiftbot SuperHTTP
Szukacz TeleportPro
Teleport
Telesphoreo Telesphoreo
Telesphorep Trendiction
Thumbor
TightTwatBot
Toata
Toweyabot
Trendictionbot
True_Robot
TurnitinBot
TwengaBot TwengaBot
Typhoeus URLy.Warning
URLy Warning
V-BOT V-BOT
VCI VelenPublicWebCrawler
Vacuum
Virusdie Virusdie
WASALive-Bot VoidEYE
Voltron
WEBDAV WEBDAV
WISENutbot WPScan
WWWOFFLE
Wallpapers Wallpapers
WallpapersHD
WeSEE WeSEE
Web Auto Web Enhancer
Web Fuck Web Fuck
Web Sauger WebAuto
Web Sucker
WebBandit
WebCollage WebCollage
WebFetch WebEnhancer
WebSucker WebLeacher
WebZIP WebPix
WebReaper
WebStripper
Webalta Webalta
WebmasterWorldForumBot Webshag
Website Quester Webster
WebsiteExtractor WiseGuys Robot
WebsiteQuester Woobot
Widow Wotbox
WinHTTrack Xenu
Zade
Zauba
ZmEu ZmEu
ZoomBot
ZoominfoBot
ZyBorg
archive.org_bot
arquivo-web-crawler arquivo-web-crawler
arquivo.pt autoemailspider
check1.exe check1.exe
cognitiveseo coccocbot
com.plumanalytics crawler.feedback
crawl.sogou.com
dataforseo.com dataforseo.com
demandbase-bot demandbase-bot
eCatch gopher
evc-batch instabid
heritrix ips-agent
internetVista monitor isitwp.com
iubenda-radar
linkdexbot
magpie-crawler magpie-crawler
mediawords mediawords
muhstik-scan muhstik-scan
oBot
plumanalytics plumanalytics
polaris version
probe-image-size probe-image-size
ripz
s1z.ru s1z.ru
scalaj-http
seobility seobility
seoscanners
serpstatbot
sexsearcher sexsearcher
sogouspider sitechecker.pro
sysscan spyfu
trendiction.de ubermetrics-technologies.com
voyagerx.com
webgains-bot
webmeup-crawler
webpros.com webpros.com
x22Mozilla
xpymep1.exe
zauba.io

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
00it.com atdedinotuho.tk
aa9d046aab36af4ff182f097f840430d51.com factorynightclub.com
advokateg.ru kvartiry-remont.ucoz.ru
akvamaster.dp.ua makenahartlin.com
amateurgalls.com most-kerch.org
girlporn.ru mustwineblog.com
lonfon.xyz onstrapon.purplesphere.in
malls.com reeyanaturopathy.com
rome2rio.com xn----ctbbcjd3dbsehgi.xn--p1ai
xn--80aikhbrhr.net zoompegs.com

924
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
12masterov.com 03p.info
273-fz.ru 100dollars-seo.com
2ads.co.uk 101lesbian.xyz
2nt.ru 101raccoon.ru
5000-cotydzien.com 108shot.com
5i2.net 1314dh.com
70casino.online 1kinobig.ru
1webmaster.ml
20pascals.nl
4ur.click
8xv8.com
DomainStatsBot DomainStatsBot
a14download.com activepr.ru
adelly.bg ad-words.ru
adcash.com
adssafeprotected.com adssafeprotected.com
adtech.de adultactioncam.com
advancedsoftwaresupport.com adultmeetonline.info
aibolita.com aero2.ru
airlandsea.info agecheckadult.com
aktivator-windows10.blogspot.com ahrefs.com
akusherok.ru all4invest.info
alekseevec.ru alvtank.se
alert-fdm.xyz
algerianembassy.co.in
allnews.md
alltheviews.com
alphacarolinas.com
amehdaily.com amehdaily.com
amyfoxfitness.com anabolics.shop
anaseracresar.tk andpolice.com
android-systems.ru anti-virus-removal.info
anlimebel.kiev.ua apibetweenlinesn-a.akamaihd.net
anticrawler.org apimegabrowsebiz-a.akamaihd.net
antons-transporte.de
api.stathat.com
apiappenableinfo-a.akamaihd.net
apispringsmartne-a.akamaihd.net
apiwoodensealcom-a.akamaihd.net
appfixing.space appfixing.space
apple.com-cleaner.systems apptester.tk
arabseyes.com apxeo.info
arcadepages.com arabsexxxtube.com
arcteryxstore.online arcteryxstore.online
arenda-avtoprokat-krasnodar.ru astrochicks.com
arkartex.ru atmagroup.ru
asacopaco.tk auto-moto-elektronika.cz
asophoto.com autoplate.info
asrvvv-a.akamaihd.net aviva-limoux.com
atmovs.com awaybird.ru
avto-oligarh.ru baixar-musicas-gratis.com
avtovolop.ru baltgem.com
babieca.com bannerswap.com
bad-stars.net battleforupdating.site
bannerconnect.net beauty-b0x.pl
batanga.net becuo.com
belinka.com.ua beget.tech
bestmarriages.com bestadbid.com
bestmobilityscooterstoday.com bestpornuha.com
bestofpicture.com bestserials.com
betonka.pro bestwaystogetpaid.us
bestwebsiteawards.com
bestwrinklecreamnow.com
bezcmexa.ru bezcmexa.ru
bibys.com bezsporno.ru
bigpictures.club bharatdefencekavach.com
bildsuche.ru bidbuy.co.kr
binaryoptionscops.info big-cash.net
bizcheapjerseyswholesalechina.com bingoporno.com
biznesluxe.ru biographiya.com
bonzibuddi.com bird1.ru
brainxs.us bkns.vn
brandov.ru bloggen.be
browsepulse-a.akamaihd.net bmwhighperformers.com
buigas.00it.com bonky.biz
bumble.cheapwebsitehoster.com boom-celebs.com
burn-fat.ga brainsandeggs.blogspot.com
buyantiviralwp.com brainzpod.win
buyclomidonlaine.com buqayy0.livejournal.com
buyessayonline19.blogspot.ru businescoop.men
calvet.altervista.org buyhoverboard.com
carsdined.org buyparajumpers.online
children-learningreading.info buytizanidineonline.blogspot.com
buzzonclick.com
bycontext.com
candlewooddentalcentre.com.au
carabela.com.do
carina-sy.de
cars-modification.net
cartierlovestore.com
cartierreplica.pw
cartierreplica.win
casinos4dummies.co.uk
cayado.snn.gr
cheapmarmot.online
check-host.net
checkm8.com
chocolateslim-original.com chocolateslim-original.com
chorus.terakeet.com
christianlouboutinsaleonline.us christianlouboutinsaleonline.us
chuckguilford.com cibpenonptib.flu.cc
cinemaflix.website colobit.biz
cleansearch.net compliance-alexa.xyz
clickhype.com compliance-checker.info
clicksor.com compliance-jane.top
com-secure.download comsysnet.com
compliance-checker-7.info cookingmeat.ru
compliance-elena.top coslab.club
compliance-ivan.xyz counter.yadro.ru
compliance-walter.top creams.makeforum.eu
connectingsingles.com custodieva.ru
coolgramgoods.com cutt.us
coolwallpapers-hd.com czat.wp.pl
cornomase.win czeshop.info
country-chic.ru darkages.info
crd.clan.su davidovic.info
cupday.com delta-line.men
customchocolate.business-for-home.com demian.kz
cyse.tk deplim.com
dailyrank.net deutschland123.de
data.vtc.pw dfiles.me
dawlenie.com dimaka.info
daydream-studio.ru dispo.de
ddpills.com dktr.ru
decoratinghomes.ga dodlive.mil
delitime.info dokumentalkino.net
dfwu1013.info dostavka-v-ukrainu.ru
digitalfaq.com dtnlyss.com
divatraffic.com duawitchrarato.tk
doeco.ru econom.co
domaincdn.xyz ecxtracking.com
drugs-no-rx.info ege-russian.ru
drugstoreforyou.com elektronischezigarette2.mex.tl
drunkmoms.net elektrozigarette1.dreamwidth.org
dumpsmania24.com elektrozigarettekaufen2.blogse.nl
eaplay.ru elidelcream.weebly.com
easyukraine.com eloxal.ru
ekosmetyki.net.pl emediate.eu
election.interferencer.ru enginebay.ru
elektir.ru enhand.se
elektrischezigarette2.devhub.com eredijovon.com
elstal.com.pl erotik0049.com
eluxer.net escortslet.net
epngo.bz eshop4u.jp
ezigarettekaufen2.blox.pl espaceinventoristes.com
face.hostingx.eu etm-consult.de
famix.xyz evehemming.blogspot.com.au
fapgon.com exdocsfiles.com
fashion-mk.net expolicenciaslatam.co
favorcosmetics.com extlabs.io
fbdownloader.com extremepornos.net
extremez.net
ezigarettenkaufen1.deviantart.com
fable.in.ua
facebook-mobile.xyz
fastcrawl.com
fba-mexico.com
fetlifeblog.com
filesmonster.porn
filesvine.com filesvine.com
findpics.pw financepoints.eu
findthe.pet findanysex.com
fix-website-errors.com flashbannernow.com
flash4fun.com flirt4free.com
flavors.me
floating-share-buttons.com
floridahuntingfishingadventures.com floridahuntingfishingadventures.com
forex-procto.ru foodcrafts.website
fortevidyoze.net forms-mtm.ru
fotopop.club forodvd.com
franecki.net forzeronly.com
free-share-buttons.com free-today.com
free-video-tool.com freefoto.ca
free411games.com fuck-paid-share-buttons.xyz
freshdz.com fullgirl.ru
friendflnder.com fz139.ttk.ru
fsakhalin.ru gallictures.com
funcrushgames.com gambarkataku.co
gay-site.store gamewrath.com
gazobeton-p.com.ua gembird.com
gdebestkupit.ru get-free-traffic-now.com
gearcraft.us ghazel.ru
getlamborghini.ga glopages.ru
given2.com gobongo.info
goldadpremium.com gojiberriess.apishops.ru
google-liar.ru googlepositions.com
googlemare.com googleseo.com.tr
gourcy.altervista.org gosmeb.ru
gribkovye-zabolevaniya.com gox.com.ua
gstatey.net greatfind-a.akamaihd.net
guiadeserraazul.com
guildebzh.info
guitar-master.org
guod.me
guruofcasino.com
gwebtools.com.br gwebtools.com.br
gz2.bbsoldes.fr handsandlegs.ru
hannasolution.ru hard-porn.mobi
hayate.biz hasshe.com
hazardky.net hawaiielectriclight.com
hdxnxxtube.mobi hchha.com
healing-dysplasia.ru healgastro.com
heartofbeijing.blogspot.com healmytrauma.info
hermesbelts.xyz health-medical-portal.info
herokuapp.com heartofpayne.xyz
heroz.fr highspeed5.net
himazin.info histock.info
hkdiiohi.skyrock.com hitsbox.info
homegardenlova.com host-protection.com
horoshieokna.com hotelcrocenzi.sm
host-tracker.com hotenergy.ru
hothot.ru houdom.net
houseofrose.com
hoztorg-opt.ru hoztorg-opt.ru
humbmosquina.tk hplaserjetpdriver8y.pen.io
i-midias.net.br hreade.com
i4track.net ibmdatamanagement.co
icaseclub.ru icaseclub.ru
idc.com.ua ilovevitaly.com
igtools.club incep.imagine-ex.co
iklysha.ml increasewwwtraffic.info
ikritikimou.gr infodocsportal.com
ilmen.net insider.pro
impisr.edunsk.ru insta-add.pro
infektsii.com integritylandscapeservices.com
infowarcraft.ru interferencer.ru
intellego.info istanbulit.com
internetartfair.com itronics.ca
iqoption.pro iweblist.info
iron-age.info jerseysbizwholesalecheap.com
janettabridal.com jikoman.info
jav-idol.com johnrobertsoninc.com
javitas.info karaganda.xkaz.org
javspace.net kihi.gdn
jerseywholesalechinabiz.com kiwi237au.tk
jolic2.com kladrus.ru
justkillingti.me knowyournextmove.com
kadashihotel.com kongruan.com
kaidalibor.de konoplisemena.com
kam-dom.ru kovrenok.ru
kasino-money.pw krasivye-devushki.net
keywords-monitoring-success.com krasnodar.xrus.org
keywordsking.com krasula.pp.ua
kichenaid.ru
kino-rating.ru
kinohall.ru
kinosed.net
kndxbkdx.bloger.index.hr
konica.kz
konteiner24.com
koptims.tiu.ru
koral.se
kostenloser-sex.com
kredit-blog.ru kredit-blog.ru
kurbappeal.info lancheck.net
kurwa.win law-check-seven.xyz
labafydjxa.su lawyersinfo.org
lakomka.com.ua lazymanyoga.com
lalalove.ru
landreferat.ru
landscaping.center
law-check-two.xyz
le-clos-des-alouettes.com le-clos-des-alouettes.com
lecbter-relationships.ga lesbian.xyz
ledpolice.ru lifebyleese.com
legalrc.biz lionking-1994.blogspot.com
lemon-ade.site littlesunraiser.com
lexaprogeneric.link live-cam6.info
littleberry.ru liver-chintai.org
livejasmin.com lollypopgaming.com
loadingpages.me lombia.com
loadopia.com longadventure.com
locimge.pw louboutinshoes.xyz
long-beach-air-conditioning.com
longgreen.info
longlifelomilomi.info
lookingglassemb.com
lovasszovetseg.hu
love-baby.cz love-baby.cz
low-format.ru lumb.co
lowephotos.info m3gadownload.pl
ltvperf.com mafa-free.com
luchshie-topcasino.ru magnetpress.sk
luciddiagnostics.in mail.allnews24.in
m4ever.net maju.bluesalt.co
magicplayer-s.acestream.net makis.nu
makemoneyonline.com marblestyle.ru
mamasuper.prom.ua max-eclat.men
mb140.ru mega-bony-2017.pl
mebel-alait.ru
mebeldekor.com.ua
med-zdorovie.com.ua
medictube.ru
mega-polis.biz.ua mega-polis.biz.ua
meget.co.za megaapteka.ru
menstennisforums.com menetie.ru
milblueprint.com michaelkorsoutletstore.net
mir-business-24.ru middlerush-a.akamaihd.net
misandesign.se minecraft-neo.ru
mini-modus.ru
misterjtbarbers.com
mixx.com mixx.com
mnogolok.info mlf.hordo.win
moinozhki.com mmstat.com
mompussy.net mobifunapp.weebly.com
monthlywinners.com mobile-appster.ru
moomi-daeri.com mobplayer.net
morepoweronmac.trade modabutik.ru
morlat.altervista.org moncleroutletonline.win
moskva.nodup.ru moneyviking-a.akamaihd.net
movies-in-theaters.net moonci.ru
mozello.ru mossmesi.com
multgo.ru motherboard.vice.com
musezone.su mturkcontent.com
mustat.com musicpro.monster
muz-baza.net musicstock.me
muznachas-service.ru myanyone.net
mvpicton.co.uk mydeathspace.com
mwtpludn.review myftpupload.com
my-floor.in.ua mygameplus.com
mycaf.it myindospace.com
myhealthcare.com mylida.org
nakozhe.com myperiod.club
naphukete.ru mywallpaper.top
narco24.me nacap.ru
narutonaruto.ru naj-filmy24.pl
naturalpharm.com.ua najaden.no
newstaffadsshop.club namecrumilchlet.tk
naughtyconnect.com
nevansk.ru
nextlnk12.com
nodding-passion.tk nodding-passion.tk
nofreezingmac.work nofreezingmac.click
nordvpn.com november-lax.com
nottyu.xyz novgorod.xrus.org
novosti-hi-tech.ru novodigs.com
nrv.co.za nrjmobile.fr
nuker.com
nyfinance.ml nyfinance.ml
o-o-11-o-o.com obiavo.kz
o00.in obiavo.uz
o333o.com oddamzadarmo.eu
odywpjtw.bloger.index.hr onclickpredictiv.com
onclkads.com
oneclickfiles.com oneclickfiles.com
onlinetvseries.me onlinedomains.ru
onstrapon.purplesphere.in onlineku.com
oralsexfilme.net onlinemeetingnow.com
ons-add.men
optom-deshevo.ru
orion-code-access.net orion-code-access.net
ourville.info ostroike.org
owathemes.com ourtherapy.ru
owohho.com outshop.ru
oyster-green.com
painting-planet.com painting-planet.com
partnerline.men partnerads.men
percin.biz.ly percin.biz.ly
phuketscreen.com petedrummond.com
picquery.com photokitchendesign.com
pictures-and-images.com phpdevops.com
picturesfrom.com piluli.info
pinstake.com pinapchik.com
piulatte.cz pistonclasico.com
plastgran.com pixgood.com
ploenjitmedia.azurewebsites.net pl.id-forex.com
plohaya-kreditnaya-istoriya.ru podshipniki-nsk.ru
pochemychka.net popunder.ru
poisk-zakona.ru pornobrazzers.biz
pojdelo.weebly.com pornokorol.com
pons-presse.com
popander.mobi
poppen-nw.net
porno-home365.com
pornoblood.com
pornoklad.net
pornoplen.com
pornoreino.com
pornosexrolik.com
pornoslive.net
pornoted.com
porto.abuilder.net
prchecker.info
pregnant.guru
presleycollectibles.com presleycollectibles.com
princevc.com pretty-mart.com
printdirectforless.com princeadvantagesales.com
printie.com privatbank46.ru
privatamateure.com privatov-zapisi.ru
pro-tec.kz prlog.ru
profolan.pl prom23.ru
projectforte.ru promgirldresses.xyz
pronorm.fr prosperent.com
pssucai.info psvita.ru
pulseonclick.com qwesa.ru
push-ad.com radiogambling.com
pussysaga.com rapidgator-porn.ga
razorweb-a.akamaihd.net rasteniya-vs-zombi.ru
rcb101.ru rczhan.com
reelheroes.net realresultslist.com
registry-clean-up.net relax.ru
repeatlogo.co.uk remont-comp-pomosh.ru
risparmiocasa.bz.it rennlist.com
ruclicks.com resultshub-a.akamaihd.net
rusprostitute.com revistaindustria.com
sabaapress.com rff-cfal.info
sampleletters.net rfid-locker.co
scanmarine.info ric.info
scanner-mary.top ringporno.com
scrnet.biz.ua rockprogblog.com
search.xtconnect.com rubbed.us
searchwebknow-a.akamaihd.net russkie-gorki.ru
seeresultshub-a.akamaihd.net s.lollypopgaming.com
seo-tools-optimizing.com sajatvelemeny.com
seocheki.net samoiedo.it
serdcenebolit.com sanidumps.com
sex-foto.pw scanner-barak.top
sexblog.pw scanner-donald.top
sexkrasivo.net scanner-irvin.top
sexobzor.info scanner-jessica.top
sexy.babes.frontend-stack.top searchengineranker.email
sexyali.com searchinterneat-a.akamaihd.net
sfj-ror.no seksvideoonlain.com
siha.de sel-hoz.com
seofied.com
seotoolsagency.com
seozoom.it
sethrollins.net
sexpornotales.com
sexstream.pl
shillyourcoins.com
shisha-swag.de
si-unique.com
sildenafil-tadalafil.info
simple-image.com.ua simple-image.com.ua
simul.co simple-share-buttons.com
site-analyzer.com sisi-go.ru
siteheart.net siteripz.net
sitiz.club smartadserver.com
sladkoevideo.com snjack.info
slkrm.ru social-vestnik.ru
smart-scripts.com socialseet.ru
social-button.xyz solicita.info
social-s-iii.xyz sonata-arctica.wz.cz
socialtrade.biz space4update.pw
solitaire-game.ru
sovinsteel.ru
special-porn.com special-porn.com
spin2016.cf specialfinanceoffers.com
spl63.fr splendorsearch-a.akamaihd.net
sptslmtrafms.com srdrvp.com
stairliftsarea.com stackthatbucks.com
stats-public.grammarly.io startwp.org
stroicol.net statistici.ro
stroy-portal22.ru stats-collector.org
strv.se statustroll.com
studiofmp.com stmassage.ru
stuffpride.com stopnarco.ru
suchenindeutschland.com strag-invest.ru
sundrugstore.com streetfire.net
supermama.top strigkaomsk.ru
supernew.org
supers.com.ua supers.com.ua
surffoundation.nl superstarfloraluk.com
susanholtphotography.com surflinksmedical.com
svbur.ru
swimpool.ca swimpool.ca
syndicate.fun tam-gde-more.ru
tagil.zrus.org
takethatad.com
tampabaywatch.org
teenfuck.tv teenfuck.tv
templates.radiodigital.co teenporn18.net
the-torrent-tracker.blogspot.com terraclicks.com
thebluffs.com thebluenoodle.com
theendivechronicles.com thefds.net
theporndude.com themestotal.com
timdreby.com thepornsex.org
timer4web.com theprofitsmaker.net
tizanidinehcl2mgsideeffects.blogspot.com thesmartsearch.net
tizanidinepills.blogspot.com thetattoohut.com
tizanidinerestlesslegsyndrome.blogspot.com tizanidinehcl4mgtabinfo.blogspot.com
token-lab.org tizanidinehcl4mgtablet.blogspot.com
topsy.com topanasex.com
torrents-tracker.com topcar-krasnodar.ru
touchmods.fr torrentdownloadhub.com
tracfone.com trafficjunky.net
tradedeals.biz traffmonster.info
traffpartners.com trk-4.net
treasuretrack-a.akamaihd.net
trkdf.com trkdf.com
twojebook.pl truemfilelj.gq
tzritel.tk tuberkuleznik.ru
turizmus.us
tvteleport.ru
twittrading.com
typer.one
ukr-lit.com
ukrtextbook.com ukrtextbook.com
ultimatesetnewfreeallsoftupgradesystems.pw unitexindia.com
upproar.com
uprour.com
uptimechecker.com
url-img.link url-img.link
valoresito.com uroffer.link
vancleefreplica.pw usbggettwku.ga
vesnatehno.ru usdx.us
videospornogratisx.net v720hd.ru
vikistars.com vavilone.com
vip2ch.com veopornogratis.xxx
viven.host.sk viagra.pp.ua
viveresaniesnelli.it vintontech.info
vkmusics.ru
vkontarkte.com
vod.com.ua
vozbujdenie.com vozbujdenie.com
vrotike.ru web-betting.ru
vseigru.one weboptimizes.com
vtc.pw weburok.com
vykupavto-krasnodar.ru weclipart.com
vysigy.su weightatraining.com
vzlomfb.com whiteproduct.com
walkme.com wma-x.com
walpaperlist.com worldmusicfests.com
wareseeker.com wttavern.com
warezaccess.com ww2awards.info
watsonrealtycorp.com www.888.com
webjam.com www.arenda-yeisk.ru
website-analytics.online
website-datenbank.de
website-speed-check.site
website-stealer.nufaq.com
wellcome2slovenia.ru
whengirlsgowild.com
white-truck.life
wholesalejerseychinaoutlet.com
word-vorlagen.net
wordpresscore.com
www.solartek.ru www.solartek.ru
x-true.info wygraj-skiny.win
xboxster.ru x-mix.info
xep.info xbaboon.com
xingzi-vision.com xdoza.com
xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai xjrul.com
xn-----7kceclhb4abre1b4a0ccl2fxch1a.xn--p1ai xn-----6kcaabbafhu7cskl7akvongwpo7hvjj.xn--p1ai
xn----8sbhefaln6acifdaon5c6f4axh.xn--p1ai xn----itbkqkfiq.xn--p1ai
xn--80aaafbn2bc2ahdfrfkln6l.xn--p1ai xn--1-8sbcpb0bdm8k6a.xn--p1ai
xn--80aebbcbcdemfkhba4byaehoejh8dza3v.xn--p1ai xn--80ab4aa2g.xn--p1ai
xn--90acenikpebbdd4f6d.xn--p1ai xn--80adaggc5bdhlfamsfdij4p7b.xn--p1ai
xtube.com xn--80ajbshivpvn2i.xn--p1ai
xnxx699.com
xtubeporno.net xtubeporno.net
xxxtubesafari.com xxxkaz.org
xz618.com xxxsiterips.xyz
yatrk.xyz y8games-free.com
ynymnwbm.bloger.index.hr yeartwit.com
youandcredit.ru youporn-ru.com
your-bearings.com your-bearings.com
ytmnd.com youtoner.it
yur-p.ru yur-p.ru
yuweng.info zakon-ob-obrazovanii.ru
z-master.ru zarenica.net
zacreditom.ru zeroredirect10.com
zakonobosago.ru zojirushi-products.ru
zetmaster.ru zoomovies.org
znaturaloriginal.com zscalertwo.net
zophim.me zvooq.eu
zygophyceous.womanstars.site

488
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
1millionusd.xyz 108shot.com
abtasty.com 11235813.webzdarma.cz
adultwebhosting.info 18ps.ru
air-edem.ru 1bet.com
2015god.org
2itech.ru
7a2rnuey1tw9ar.ru
7makemoneyonline.com
actionnooz.com
adclickservice.com
adclickthru.net
adf.ly
adservme.com
adtech.us
aktobe.xkaz.org aktobe.xkaz.org
albamargroup.com alarmobninsk.ru
alborzan.com alta-realestate.com
albuteroli.com am-se.com
anapa-inns.ru apiqualitinknet-a.akamaihd.net
apimegabrowsebiz-a.akamaihd.net appmsr.org
appsecurityr.com artdestshop.eu
arclk.net atlasvkusov.ru
ascat.porn avtovolop.ru
asseenontv.ru azinoofficial777.ru
audiobangout.com
aviav.eu
avkzarabotok.com
avon-novosib.ru
azadnegar.com
baladur.ru baladur.ru
baoxaydung.com.vn bamo.xsl.pt
battle.net bausparen.bz.it
bdsmgalls.net beatifullandscape.co
beremenyashka.com betteroffers.review
best-way.men bjanshee.ru
bestofferswalkmydogouteveryday.gq blagovest-med.ru
betwinservice.com blogig.org
biketank.ga bnt-team.com
bitcoinremote.com bookmaker-bet.com
bizcheapjerseyswholesalechina.com brandov.ru
bloggedporn.com browserprotecter.com
bmsco.co bus-offer.com
braindaily.xyz camdenmemorials.com
buntube.net candcstuccoandstone.com
buzzzg.men celejihad.info
canrole.com censys.io
carson.getenjoyment.net cerev.info
cartierlove2u.xyz cezartabac.ro
cartierlovebracelet.xyz cheapjerseysap.com
cartierreplica.xyz checkm8.com
cartierwatch.xyz christianlouboutinsaleoutletonline.us
casinos4dummies.co.uk cjmc.info
cfacarrosserie74.com clarithromycin500mg.com
cherrypointplace.ca com-onlinesupport.site
cialgenisrx.com compliance-andrew.top
compliance-donald.xyz connectionstrenth.com
compliance-jane.top contentlook.co
copenergo.ru counciltally.com
coprofam.org covid-schutzmasken.de
crackguru.tk creatives.adbetclickin.pink
crd.clan.su demenageur.com
creditmoney.com.ua detalizaciya-tut.biz
cubs-jerseys.us dieswaene.com
cxpromote.com directivepub.com
data-mining.tk discountbarbour.online
day-news.info divci-hry.info
decenttools.com dodlive.mil
demian.kz dogclothing.store
detki-opt.ru domcran.net
dichvuvesinhhanoi.com domoysshop.ru
dimaka.info domznaniy.ru
disruptingdinnerparties.com eager-nash.188-93-233-196.plesk.page
domaincrawler.com
dotnetdotcom.org
drhomes.biz
dumuelave.xyz
dverimegapolis.ru
ekto.ee
el-nation.com el-nation.com
election.interferencer.ru elektronischezigarette2.mex.tl
elvel.com.ua energydiet24.ru
escortslet.net etotupo.ru
etur.ru euronis-free.com
europeanwatches.ru excitacion.info
evaashop.ru expresstoplivo.ru
event-tracking.com extractorandburner.com
ezigarettekaufen2.blox.pl extremez.net
favoritemoney.ru filmi-onlain.info
findclan.org floridahuntingfishingadventures.com
fireads.men forum.doctissimo.fr
fishingwholesale.us freegamesplay.online
foxjuegos.com fugarif.ga
front.ru fz139.ttk.ru
galleryawesome.com gambarkataku.co
games.kolossale.ru gamerscorps.com
gembird.com ge0ip.net
goodwinmetals.co gemara.com
gopixdatabase.com gettpromos.com
hamacapty.com globatur.ru
headpress.ru goodhumor24.com
histock.info gopro-online.info
homeart.space gourcy.altervista.org
hothor.se gov.yanao.ru
hunthillfarmtrust.org greamimgo.pw
hyhj.info gsbs.com.ua
ilikevitaly.com healbio.ru
healthcarestore.info
hetmanship.xyz
hledejvshopech.cz
hotblognetwork.com
hoverboard360.es
howtoclean.club
hundejo.com
hunterboots.online
ilo134uloh.com ilo134uloh.com
impisr.edunsk.ru
inboundlinks.win inboundlinks.win
incanto.in.ua increasewwwtraffic.info
infokonkurs.ru inet-traffic.com
instabid.tech infobanks.ru
instasexyblog.com inmate-locator.us
internetproviderstucson.com iphantom.com
iqoption.com ispac.org
iwanttodeliver.com itag.pw
jose.mulinohouse.co izhevsk.zrus.org
justprofit.xyz jav-way.site
kadashihotel.com jerseywholesalechinabiz.com
keywordsdoctor.com junketjuice.blogspot.com
kinobaks.com keywordsuggests.com
kinopolet.net kol-energo.ru
kinotorka.ru krasivoe-hd.com
klikbonus.com kreativperlen.ch
larger.io kurwa.win
lazy-z.com lacloop.info
le-clos-des-alouettes.com lapolis.it
likrot.com lifehacĸer.com
linkpulse.com liky.co.ua
liver-chintai.org lion.snapmap.us
liveshoppersmac.com lol-smurfs.com
m.b00kmarks.com low-format.ru
mainlinehobby.net m4ever.net
mantramusic.ru mackeeper-center.club
maranbrinfo.com.br marktforschung-stuttgart.com
meble-bogart.info max-eclat.men
meetlocalchicks.com mediafresh.online
megaindex.ru minyetki.ru
moi-glazki.ru mobile.ok.ru
monclerboots.xyz monclervests.xyz
monclerjacketsoutlet.win mp3ritm.top
monsterdivx.tv mrinsidesales.com
moomi-daeri.com msfsaar.de
mymobilemoneypages.com myftpupload.com
myseoconsultant.com myhealthcare.com
myyour.eu myindospace.com
nerudlogistik.ru negral.pluto.ro
newstaffadsshop.club nl.netlog.com
nfljerseysforsalewholesaler.com nlfjjunb5.ru
nikitabuch.com normalegal.ru
nippon-bearings.ru notaria-desalas.com
novatech.vn o00.in
o-o-8-o-o.com obiavo.ru
omgtnc.com online-x.ru
omoikiri-japan.ru ontargetseo.us
one.net.in owathemes.com
orenburg-gsm.ru ownshop.cf
osagonline.ru pandarastore.top
ostroike.org paperwritingservice17.blogspot.ru
oyster-green.com paradontozanet.ru
p-business.ru parajumpersoutlet.online
paleohub.info pc-services.ru
panchro.xyz
pattersonsweb.com
paydayonlinecom.com
pcimforum.com
pekori.to
penisvergrotendepillennl.ovh
pensplan4u.com pensplan4u.com
phormchina.com petedrummond.com
pictures-and-images.com phobia.us
pkr1hand.com phpdevops.com
poiskzakona.ru piccshare.com
pornhive.org pizza-imperia.com
pornodojd.ru pons-presse.com
portadd.men popmarker.com
portside.cc producm.ru
profolan.pl prosmibank.ru
ptr.ruvds.com proxyradar.com
rebrand.ly rapevideosmovies.com
reklama-i-rabota.ru redirlock.com
responsinator.com rff-cfal.info
rimedia.org rfid-locker.co
ringporno.com rock-cafe.info
rique.host.sk rome2rio.com
ru-dety.ru royal-investments.net
rus-pornuha.com rrutw.com
rustic-quiver.win rusenvironmental.net
s8-nowy-wygraj.comli.com saveriopiazza.it
sabizonline.com scanner-walt.top
sakhboard.ru schlampen-treffen.com
see-your-website-here.com school-diplomat.ru
seoheap.com se.bnt-team.com
seoholding.com seasonvar.ru
seorankinglinks.xyz securesmrt-dt.com
sexphoto.site sexiporno.net
shemalegalls.blogporn.in shopsellcardsdumps.com
shop-electron.ru signoredom.com
sientalyric.co simple-share-buttons.com
skanninge.se sims-sims.ru
smartshoppymac.com sindragosa.comxa.com
snegozaderzhatel.ru site3.free-share-buttons.com
soblaznu.net sloopyjoes.com
social-search.me slowmacfaster.trade
socialbuttons.xyz snjack.info
soochi.co snyatie-lomki-v-stacionare.ru
startwp.org social-button.xyz
stats-collector.org sonata-arctica.wz.cz
stroimajor.ru stair.registrydomainservices.com
stroydetali.ru stairliftsarea.com
stroyhelp-dv.ru steame.ru
suchenindeutschland.com strigkaomsk.ru
superoboi.com.ua surcentro.com
tarad.com taboola.com
template-kid.com tds-advert005.info
thebitcoincode.com telefonsexsofort.tk
tizanidinevssoma.blogspot.com telsis.com
tizanidinewithdrawal.blogspot.com thebluenoodle.com
top-deal.com.pl thejournal.ru
torrent-to-magnet.com thenews-today.info
tour-line.net timecrimea.ru
tracksz.co timer4web.com
trahic.ru tizanidinehclsideeffects.blogspot.com
tube8.com tn811.us
tv-spoty.info tnaionline.org
twittruth.com tongkatmadura.info
uchebavchehii.ru topsiteminecraft.com
ucoz.ru totu.info
ultimatesetnewfreeallsoftupgradesystems.pw track112.site
userequip.com trichizobswiv.agddns.net
vbikse.com uasb.ru
verymes.xyz umityangin.net
videonsk.com upupa.net
videotuber.ru url-extractor.xyz
vladhistory.com urzedowski.eu
voloslove.ru v720hd.ru
vroze.com vacuumcleanerguru.com
watchinf.com vip-parfumeria.ru
watsonrealtycorp.com viven.host.sk
we-are-gamers.com vsexkontakte.net
webradiology.ru warezkeeper.com
westum.se weburok.com
wholesalecheapjerseysfree.com wholesalejerseysgaa.com
wholesalejerseychinaoutlet.com wikes.20fr.com
wholesalejerseys-cheapest.com winx-play.ru
wildcattube.com wowcasinoonline.ooo
xn--c1acygb.xn--p1ai wzgyyq.com
xrus.org xerox-douglas.cf
xuki.us xnxx-n.com
xwatt.ru xtremeeagles.net
xxx-treker.ru your-bearings.com
yourads.website za-fun-offer.com
yourserverisdown.com zaimite.ru
zebramart.ru zakon-ob-obrazovanii.ru
zeroredirect10.com zawyna.ua
zhuravlev.info zebradudka.com
znakomstva-moskva77.ru zeroredirect11.com
znakomstva-piter78.ru znaniyapolza.ru
zoogdinsney.com zona-aqua.ru
zryydi.com zvuker.net

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.2.41 0; 10.1.0.43 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.06.3168 ### Version: V4.2022.06.3169
### Updated: Wed Jun 15 10:03:49 UTC 2022 ### Updated: Wed Jun 15 22:01:17 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.06.3168 ### Version: V4.2022.06.3169
### Updated: Wed Jun 15 10:03:49 UTC 2022 ### Updated: Wed Jun 15 22:01:17 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##