vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-09 05:54:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.09.2565 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-09-02 10:09:19 +00:00
parent 5df40eb77a
commit cefe14d5bf
46 changed files with 61262 additions and 61262 deletions
Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.09.2564 #### Version: V4.2021.09.2565
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.09.2564 #### Version: V4.2021.09.2565
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -11,7 +11,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2021.09.2564 #### Version: V4.2021.09.2565
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

7520
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2021.09.2564 #### Version: V4.2021.09.2565
#### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [624](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

7524
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

7524
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

7524
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

7524
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

7524
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

7524
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

7524
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2564 2565

7524
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

7524
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
13.67.176.123 0; 52.183.124.158 0;

7524
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

7524
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

7524
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

7524
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
13.67.176.123 0; 52.183.124.158 0;

7524
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
13.67.176.123 0; 52.183.124.158 0;

7524
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Aboundexbot Abonti
CheeseBot DittoSpyder
Gotit Larbin
Ltx71 Mozlila
Mr.4x3 NearSite
Netvibes Netcraft
Pavuk PECL::HTTP
RebelMouse SEOkicks-Robot
Voil ips-agent
WebsiteExtractor webprosbot

310
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
360Spider 01h4x.com
404enemy 404enemy
ALittle Client Aboundex
ASPSeek
Aboundexbot Aboundexbot
Acunetix
AfD-Verbotsverfahren AfD-Verbotsverfahren
AiHitBot AiHitBot
AlphaBot Alexibot
AllSubmitter
Alligator
Anarchie Anarchie
Anarchy99 Anarchy99
Asterias Apexoo
Attach
AwarioRssBot AwarioRssBot
BBBike AwarioSmartBot
BackStreet BDFetch
Backlink-Ceck BackDoorBot
Barkrowler BackWeb
BlackWidow BetaBot
Black Hole
Blackboard
Blow
BlowFish
Boardreader Boardreader
Brandprotect Brandwatch
Buck BuiltWith
BuiltBotTough
Bullseye Bullseye
BunnySlippers CATExplorador
BuzzSumo CazoodleBot
CCBot
CODE87
CheTeam
CheeseBot
ChinaClaw ChinaClaw
Chlooe
Cliqzbot
Cloud mapping Cloud mapping
Cogentbot Cogentbot
CopyRightCheck CopyRightCheck
Cosmos
CrunchBot
Curious
Custo Custo
DTS Agent CyotekWebCopy
DataCha0s DIIbot
Digincore DSearch
Dirbuster Deusu
DigitalPebble
Discobot Discobot
Discoverybot Discoverybot
Dispatch
DittoSpyder DittoSpyder
DomainStatsBot Domains Project
Dotbot EMail Wolf
Drip
EMail Siphon
EasyDL EasyDL
Ecxi Ebingbong
EroCrawler Evil
Extreme Picture Finder Express WebPictures
FemtosearchBot ExtLinksBot
FrontPage EyeNetIE
Fuzz FHscan
FyberSpider Foobot
Fyrebot G-i-g-a-b-o-t
GT::WWW GT::WWW
GetRight GetWeb
Getintent Getintent
Go!Zilla Gigablast
Gotit Go-Ahead-Got-It
GrapeshotCrawler GoZilla
Grabber
GridBot GridBot
HEADMasterSEO HEADMasterSEO
HTMLparser HMView
HTTP::Lite HTTP::Lite
Haansoft
HaosouSpider HaosouSpider
Heritrix Heritrix
IRLbot Hloader
Iblog Humanlinks
IlseBot
Image Fetch
Image Sucker Image Sucker
Indy Library InfoTekies
Intelliseek Intelliseek
InternetSeer
JamesBOT JamesBOT
Jbrofuzz
JennyBot JennyBot
JetCar JetCar
Joomla Jyxobot
Jorgee Kenjin Spider
Keyword Density
Kinza Kinza
Larbin Lanshanbot
LeechFTP LeechFTP
LibWeb
Libwhisker Libwhisker
LieBaoFast LieBaoFast
LinkextractorPro Likse
LinksManager LinkScan
LinqiaScrapeBot LinqiaRSSBot
Lipperhey Ltx71
Litemage_walker MFC_Tear_Sample
MSIECrawler MIIxpc
MJ12bot
MTRobot
Magnet Magnet
Majestic SEO
Majestic-SEO Majestic-SEO
Majestic12
MarkMonitor
MarkWatch MarkWatch
MauiBot
Mb2345Browser Mb2345Browser
Meanpathbot
Mediatoolkitbot
Microsoft Data Access Microsoft Data Access
Moblie Safari
Mojeek
Mojolicious Mojolicious
Morfeus Fucking Scanner
Mozlila Mozlila
Mr.4x3
NICErsPRO
Name Intelligence Name Intelligence
Navroad Nameprotect
Needle
Nessus
NetAnts NetAnts
NetSpider NetMechanic
NetZIP
Netcraft
Nettrack
Netvibes Netvibes
Nibbler
Niki-bot
Nikto Nikto
Nimbostratus NimbleCrawler
Ninja Nmap
Nuclei Nutch
Offline Navigator Octopus
Offline Explorer
OnCrawl OnCrawl
OpenVAS OpenVAS
OrangeBot OrangeSpider
OutclicksBot OutfoxBot
PECL::HTTP
PHPCrawl
POE-Component-Client-HTTP POE-Component-Client-HTTP
Page Analyzer
PageAnalyzer
PageGrabber PageGrabber
PageScorer PageScorer
Pandalytics PageThing.com
Pavuk
PeoplePal
Petalbot Petalbot
Pi-Monster
Picsearch Picsearch
PictureFinder
Piepmatz Piepmatz
Pimonster Pimonster
Pixray Pixray
PleaseCrawl ProPowerBot
Probethenet Psbot
PxBroker QueryN Metasearch
Quick-Crawler
RSSingBot RSSingBot
RankActive
RankActiveLinkBot RankActiveLinkBot
RankFlex
RankingBot
RankingBot2
Rankivabot Rankivabot
RankurBot Re-re
RealDownload
Reaper Reaper
RepoMonkey SBIder
Ripper SEOlyticsCrawler
RocketCrawler ScanAlert
Rogerbot Scanbot
SEOkicks
SEOstats
SISTRIX
Scrapy
Screaming
ScreenerBot ScreenerBot
Searchestate ScrepyBot
Semrush SentiBot
SeoSiteCheckup SeoSiteCheckup
Seomoz
Siphon
Site Sucker Site Sucker
SiteCheckerBotCrawler SiteCheckerBotCrawler
SiteExplorer SiteSucker
SiteLockSpider Sitebeam
SiteSnagger Sitevigil
Siteimprove Snoopy
SlySearch SocialRankIOBot
SmartDownload Sociscraper
Snake
Sosospider Sosospider
SpankBot Spammen
Spanner
Spbot
Spinn3r Spinn3r
Sqlworm Sqlworm
Sqworm
Sucuri Sucuri
SuperHTTP
Surfbot Surfbot
Teleport Suzuran
Telesoft T8Abot
Telesphorep Telesphorep
The Intraformant The Intraformant
Titan
Toata Toata
Tracemyfile Toweyabot
Trendictionbot Trendiction
TurnitinBot True_Robot
Typhoeus Turingos
URLy.Warning TwengaBot
Twice
UnisterBot UnisterBot
VB Project V-BOT
VCI VelenPublicWebCrawler
Vagabondo VeriCiteCrawler
Voltron Virusdie
Wallpapers WASALive-Bot
WeSEE WEBDAV
Web Collage WPScan
WWW-Collector-E
WWW::Mechanize
Wallpapers/3.0
Web Auto
Web Enhancer
Web Fetch Web Fetch
WebBandit Web Pix
Web Sucker
WebCollage WebCollage
WebCopier
WebEnhancer WebEnhancer
WebFetch WebFetch
WebGo IS WebGo IS
WebPix WebPix
WebSucker WebWhacker
WebZIP Webalta
WebsiteExtractor WebmasterWorldForumBot
WebsiteQuester Website Quester
Webster Whack
Whacker
Whatweb Whatweb
Who.is Bot Who.is Bot
Widow
WinHTTrack WinHTTrack
Wonderbot Woobot
Wprecon Wprecon
Xaldon WebSpider YoudaoBot
Zade Zade
Zauba Zermelo
Zeus
Zitebot Zitebot
ZmEu ZmEu
ZyBorg
archive.org_bot archive.org_bot
arquivo-web-crawler arquivo-web-crawler
clark-crawler arquivo.pt
crawl.sogou.com
crawler.feedback crawler.feedback
crawler4j crawler4j
demandbase-bot
eCatch
evc-batch evc-batch
gopher facebookscraper
heritrix instabid
internetVista monitor ips-agent
lwp-request lwp-trivial
magpie-crawler
mediawords mediawords
netEstate NE Crawler muhstik-scan
oBot oBot
seoscanners probe-image-size
ripz
s1z.ru
scalaj-http
scan.lol
seostar seostar
sogouspider serpstatbot
spyfu spyfu
sysscan
trendiction.de trendiction.de
ubermetrics-technologies.com voyagerx.com
webpros.com zauba.io
x09Mozilla
x22Mozilla
xpymep1.exe
zgrab

312
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
404enemy 360Spider
Abonti 80legs
AIBOT
Aboundex
Aboundexbot Aboundexbot
Acunetix
AhrefsBot
AiHitBot AiHitBot
Aipbot
Alexibot Alexibot
AlphaBot AllSubmitter
Apexoo Alligator
AwarioSmartBot Anarchie
Badass Anarchy99
Bigfoot Ankit
Aspiegel
Attach
BDCbot
BDFetch
BatchFTP
BetaBot
Bitacle Bitacle
Blow Blackboard
BlowFish
Boardreader
Bolt Bolt
Brandprotect
Brandwatch Brandwatch
Buck Buck
Buddy
Bullseye
BuzzSumo
CATExplorador CATExplorador
CODE87 Calculon
CazoodleBot
Cegbfeieh Cegbfeieh
CheeseBot CensysInspect
ChinaClaw ChinaClaw
Claritybot Cliqzbot
Cogentbot Cloud mapping
Copier Copier
Copyscape CopyRightCheck
CrazyWebCrawler CrazyWebCrawler
Crescent Curious
Custo
CyotekWebCopy CyotekWebCopy
DSearch DIIbot
DTS Agent
Demon Demon
Devil Deusu
Digincore
DigitalPebble
Dirbuster Dirbuster
Discobot Disco
Discoverybot Discoverybot
Dispatch DnyzBot
DittoSpyder
DomainSigmaCrawler DomainSigmaCrawler
DomainStatsBot
Download Wonder Download Wonder
Dragonfly Drip
EMail Wolf EMail Siphon
EasyDL Exabot
Ebingbong
EirGrabber
EroCrawler
Express WebPictures
Extractor Extractor
ExtractorPro ExtractorPro
Extreme Picture Finder Extreme Picture Finder
EyeNetIE EyeNetIE
FDM
FHscan FHscan
FemtosearchBot Fimap
Flunky Freeuploader
Foobot
FrontPage FrontPage
Fuzz Fuzz
FyberSpider Genieo
Fyrebot GermCrawler
GetWeb
Gigablast
Gigabot
Go!Zilla Go!Zilla
GoZilla
Grabber Grabber
Grafula Grafula
GrapeFX
HEADMasterSEO
HTMLparser
HTTP::Lite
HTTrack HTTrack
Haansoft Haansoft
Harvest Humanlinks
Havij IDBot
Heritrix IRLbot
IDBTE4M IlseBot
Iblog IndeedBot
Image Fetch Indy Library
Image Sucker
InfoNaviRobot
InfoTekies
InterGET InterGET
Iria
JamesBOT JamesBOT
Jbrofuzz Jbrofuzz
JetCar JetCar
JustView Jetty
Jyxobot Kenjin Spider
Kozmosbot Kinza
LWP::Simple LeechGet
Lanshanbot
LeechFTP
LexiBot LexiBot
Likse Lightspeedsystems
LinkScan LinkScan
LinkWalker Linkbot
Linkdexbot Linkdexbot
LinkextractorPro LinksManager
LinkpadBot LinqiaMetadataDownloaderBot
Lipperhey Spider LinqiaScrapeBot
Litemage_walker Litemage_walker
Lmspider
MIDown tool MIDown tool
MIIxpc MIIxpc
MJ12bot MJ12bot
MTRobot MSIECrawler
Mag-Net
Magnet Magnet
Mail.RU_Bot Majestic-SEO
Majestic SEO
Majestic12 Majestic12
MarkMonitor Mata Hari
Masscan
Mb2345Browser Mb2345Browser
Mister PiX Meanpathbot
Moblie Safari Metauri
Mojolicious Musobot
Morfeus Fucking Scanner
Msrabot
NPbot
Name Intelligence Name Intelligence
Nameprotect
Navroad
NearSite
Needle Needle
Net Vampire Nessus
NetAnts NetAnts
NetLyzer NetLyzer
Netcraft
Nettrack Nettrack
Netvibes Niki-bot
Nibbler
Nikto Nikto
Nimbostratus Nimbostratus
Nutch
Octopus
Offline Navigator
OnCrawl OnCrawl
OpenLinkProfiler PECL::HTTP
OpenVAS PHPCrawl
Openvas PageAnalyzer
OutclicksBot PageGrabber
PageScorer
PageThing.com
Pandalytics
Papa Foto
Pavuk Pavuk
PeoplePal PeoplePal
Petalbot
Picscout Picscout
Piepmatz Piepmatz
PleaseCrawl Pimonster
Pockey Pixray
Pu_iN Probethenet
RankActive Psbot
Pump
PyCurl
QueryN Metasearch
RankFlex RankFlex
RankingBot RankingBot
RankingBot2 RankingBot2
RankurBot ReGet
RealDownload RebelMouse
Reaper Recorder
RepoMonkey RedesScrapy
Ripper Ripper
SEOkicks-Robot RocketCrawler
SEOprofiler
SEOstats
SISTRIX
SMTBot
SalesIntelligent SalesIntelligent
ScanAlert ScanAlert
ScrepyBot ScoutJet
ScreenerBot
Searchestate Searchestate
SearchmetricsBot Seomoz
Semrush Shodan
SemrushBot
SeoSiteCheckup
Siphon
Site Sucker
SiteCheckerBotCrawler
SiteExplorer
SiteLockSpider SiteLockSpider
SiteSnagger SiteSucker
Sitebeam
Siteimprove Siteimprove
Sitevigil Sitevigil
SmartDownload
Snapbot Snapbot
SocialRankIOBot
Sogou web spider Sogou web spider
Spammen Sottopop
SpankBot SpankBot
Spbot
Spinn3r Spinn3r
Steeler Sqlworm
Stripper
Sucker Sucker
SuperBot SuperHTTP
Suzuran Swiftbot
Szukacz Szukacz
T0PHackTeam T0PHackTeam
TeleportPro
Telesoft
Telesphoreo
The Intraformant The Intraformant
Toata Titan
Toweyabot
Tracemyfile
Trendictionbot Trendictionbot
Turingos True_Robot
Typhoeus
Upflow Upflow
VB Project V-BOT
VoidEYE VelenPublicWebCrawler
Voil VidibleScraper
Virusdie
WASALive-Bot
WBSearchBot WBSearchBot
WEBDAV WWW-Collector-E
WISENutbot WWWOFFLE
WPScan Wallpapers/3.0
WWW::Mechanize WeSEE
Wallpapers Web Enhancer
Web Collage
Web Fetch Web Fetch
Web Pix Web Fuck
WebCollage Web Sauger
WebBandit
WebCopier
WebEnhancer
WebFetch
WebFuck WebFuck
WebGo IS
WebLeacher WebLeacher
WebReaper
WebSauger WebSauger
WebWhacker WebSucker
WebZIP Webshag
WebmasterWorldForumBot Who.is Bot
Whack Wonderbot
Whatweb Wprecon
Wotbox Xaldon_WebSpider
Zade
Zauba Zauba
Zeus Zermelo
ZoominfoBot Zitebot
ZumBot
ZyBorg ZyBorg
arquivo-web-crawler archive.org_bot
autoemailspider autoemailspider
backlink-check backlink-check
check1.exe cah.io.community
com.plumanalytics coccocbot-web
crawl.sogou.com cognitiveseo
crawler.feedback crawler.feedback
dataforseo.com
demandbase-bot
domainsproject.org domainsproject.org
facebookscraper eCatch
gopher gopher
ips-agent internetVista monitor
isitwp.com
lwp-trivial lwp-trivial
magpie-crawler meanpathbot
muhstik-scan muhstik-scan
oBot oBot
page scorer
pcBrowser pcBrowser
polaris version
probe-image-size
scalaj-http scalaj-http
seobility scan.lol
seoscanners
seostar
serpstatbot serpstatbot
sexsearcher tAkeOut
webpros.com trendiction.com
x09Mozilla trendiction.de
x22Mozilla ubermetrics-technologies.com
xpymep1.exe voyagerx.com

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
asscenihotosidea.blogspot.co.za 11pikachu.ru
bharatdefencekavach.com alice.it
franch.info g33.org
manualterap.roleforum.ru one-gear.com
networkad.net onlinemeetingnow.com
newstraveller.ru tivolibasket.it
pornonik.com uptime-alpha.net
relax.ru valueclick.com
reyel1985.webnode.fr varikoz24.com
seoholding.com vod.com.ua

942
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
0912701309f8ce.com
0lovespells0.blogspot.com 0lovespells0.blogspot.com
18ps.ru 1001watch.com.ua
20pascals.nl 108shot.com
24videos.tv 123locker.com
5000-cotydzien.com 1millionusd.xyz
888.com 1stat.ru
a96527gi.beget.tech 2pxg8bcf.top
abcdeg.xyz 3dracergames.com
absolute-s.ru 3w24.com
45en.ru
51.la
66cpwgln.space
7minuteworkout.com
7search.com
8lufu.com
8si.ru
9icmzvn6.website
aab94f698f36684c5a852a2ef272e031bb.com
aad1f4acb0a373420d9b0c4202d38d94fa.com
activecampaign.dreamhosters.com
adcash.com adcash.com
adrenalin-stk.ru adult-shop.com.ua
adtech.us adzerg.com
adultwebhosting.info afslankpillen2017nl.eu
airmaxshoes-2016.com agardomains.com
albuteroli.com ahhjf.com
alice.it akita.kz
all4wap.ru alf-img.com
allknow.info
allnews24.in
allsilver925.co.il
alot.com alot.com
alphahoverboards.com alpharma.net
alveris.ru altamayoztourism.com
alvtank.se ameblo.jp
anaksma.info ample-awards-today.us
antons-transporte.de ampower.me
apessay.com android-systems.ru
apibatbrowsecom-a.akamaihd.net animalrank.com
apiholdingmypage-a.akamaihd.net anjalika.co.in
apioutoboxnet-a.akamaihd.net anniemation.com
apiwoodensealcom-a.akamaihd.net ap.senai.br
appleid-verification.com apidigidocketnet-a.akamaihd.net
apiitsthirteende-a.akamaihd.net
apimountainbikei-a.akamaihd.net
apisecretsaucebi-a.akamaihd.net
appsaurus.com appsaurus.com
aramaicmedia.org arcadeplayhouse.com
arenda-avtoprokat-krasnodar.ru asdfz.pro
arendovalka.xyz atovh.local-finders.com
articlesdirectoryme.info avramstroy.ru
auto-zapchasti.org azbukadiets.ru
autobrennero.it baikaleminer.com
autotrends.today baixar-musicas-gratis.com
autovideobroadcast.com beachtoday.ru
avkzarabotok.info bedandbreakfast.com
avtoarenda.by belsetirehafi.tk
avtolombard-krasnodar.ru benchmarkcommunications.co.uk
azarttoto.com besthoro.ru
azbuka-mo.ru bestpornuha.com
baditri.com bestpriceninja.com
bankmib.ru bidbuy.co.kr
bankofthewext.com biglistofwebsites.com
beremenyashka.com bio-japan.net
bestprofits-there.com bjgugu.net.ua
big-boards.info blavia.00author.com
bigcities.org blockworld.ru
bio.trade-jp.net bloglag.com
bioscorp.ru blue-square.biz
bitcoins-live.ru
biznesrealnost.ru
blog.axant.it
bloke.com
bmw-ark.ru bmw-ark.ru
boc.kz bonvillan.com
bolezniorganov.ru brainxs.us
bonzibuddi.com
brainzpod.win brainzpod.win
break-the-chains.com bumascloset.com
bristolhostel.com businesn.men
bristolhotel.com buydissertation.net
brus.city buypanicdisorderpill.com
buketeg.xyz calc-for-credit.ru
buytizanidineonlinenoprescription.blogspot.com callawaygolfoutlet.online
c-gainsbourg.com calvet.altervista.org
cablecar.us camsex.xxx-cam.webcam
callmd5map.com canyougethighofftizanidine.blogspot.com
candcstuccoandstone.com carapics.com
cardsharp1.ru carina-sy.de
cartierlovebraceletreplica.xyz cartierreplica.win
cartierloveringreplica.xyz cartiertrinity.xyz
cartierreplica.pw cdncash.com
cartujano-pre.de cfcl.co.uk
casinos4dummies.co.uk chatroulette.world
cbox.ws cheapsergiorossi.online
cdncash.net christianlouboutinsaleonline.us
cellfun.mobi cialis-samples.com
chastnoeporno.com cialischstgerts.com
cheapjerseysbizwholesale.us cicaki.net
chimiver.info cindymatches.com
chocolateslim-en-espana.com citysecurity.nu
chocolateslim-en-france.com cleanpcnow.com
chocoslim.pro clevernt.com
christianlouboutinsaleoutletonline.us clickcash.com
ciekawinki.pl clothesforcash.com
ciproandtizanidine.blogspot.com cocyq.inwtrade.com
circlesl.com cometorussia.net
citetick.com
clash-clans.ru
classicakuhni.ru
claytransformations.info
cleansearch.net
clicksotrk.com
cocolyze.com
coin-hive.com
communisave.co.za communisave.co.za
compliance-brian.xyz compliance-alex.xyz
compliance-julia.top computernetworksonline.com
concordexoticrentals.com conboy.us
contentlook.co coolbar.pro
coolgamechannel.com copro.pw
coolingoods.com counciltally.com
copblock.org counterbot.com
craftinsta.ru countercrazy.com
customcollegeessays.net courtshipgift.com
cymbaltaandtizanidine.blogspot.com cracksplanet.com
cyse.tk crest3d.ru
dailyrank.net customcatchcan.com
damasarenai.info danashop.ru
daptravel.com darmebel.com.ua
daydream-studio.ru datract.com
devochki.top deluxedumps.com
dieta-personalna.pl deshevo-nedorogo.ru
diffbot.com dev-seo.blog
diusyawiga.tk div.as
dividendo.ru divisioncore.com
dnepr-avtospar.com.ua docsportal.net
doeco.ru dokfilms.net
dollartree.info downloaddy.net
domaineaneblanc.com downloadeer.net
dorratex.tn e-buyeasy.com
download-of-the-warez.blogspot.com
drillsaw.ru
drpornogratisx.xxx
dscaas.website
dtnlyss.com
dwomlink.info
e-commerce-seo.com e-commerce-seo.com
e-kwiaciarz.pl earthmagic.info
e2click.com editors.choice6912650.hulfingtonpost.com
earn-from-articles.com efnor-ac.com
eas-seo.com
easyshoppermac.com
ebonyporn.site
edelstahlschornstein-123.de
eets.net
ege-russian.ru ege-russian.ru
elektrozigarette2.wordpressy.pl electro-prom.com
enews.tech elite-sex-finders.com
erotik-kostenlos.net elitesportsadvisor.com
erotik0049.com elvenmachine.com
enbersoft.com
energydiet-info.ru
epicbrogaming.com
erotikchat-24.com erotikchat-24.com
espaceinventoristes.com exlarseva.webblog.es
etm-consult.de expolicenciaslatam.co
europages.com.ru facialporntube.com
euroskat.ru family1st.ca
extremez.net fanpagerobot.com
ezigarettekaufen.myblog.de fashionindeed.ml
f-loaded.de fealq.com
falcon-images.blogspot.com
falllow.gq
fashion-mk.net
favornews.com
felizporno.com felizporno.com
fenoyl.batcave.net femmesdenudees.com
ferieboligkbh.dk fickkontaktehobbyhuren.com
fickenbumsen.net fikasound.tk
filesclub.net fitfloponline.store
filmi-onlain.info folowsite.com
filunika.com.ru forms-mtm.ru
financehint.eu forsex.info
findpics.pw
fishtauto.ru
focalink.com
footballfarrago.com
foto-weinberger.at foto-weinberger.at
free-fb-traffic.com franecki.net
free-video-tool.com free-laptop-reward.com
freejabs.com free-social-buttons.com
freevpn.space freenom.link
freewlan.info fres-news.com
fuel-gas.com freshberry.com.ua
funtoonez.com freshsuperbloop.com
fz139.ttk.ru frvo.alptandem.ru
g.starmoe.xyz fugarif.ga
g7m.pl galeon.com
gambarkatabaru.com gemgrab-a.akamaihd.net
games.kolossale.ru getoutofdebtfree.org
gaming-journal.com ghazel.ru
gamingspark.com girls-ufa.ru
gdzkurokam.ru glicol.kz
ge0ip.com
get-your-social-buttons.info
gilbertbanda.net
gloverid.site
goforexvps.com goforexvps.com
goldpanningtools.com gomusix.com
green-tea.tv goo.ne.jp
greenidesign.co googst2.ru
gsbs.com.ua gopro-online.info
guidefs.ru gosreg.amchs.ru
gwhwpxbw.bloger.index.hr greamimgo.pw
handicapbathtubarea.com gribkovye-zabolevaniya.com
hannasolution.ru gsasearchengineranker.site
havepussy.com gsou.cf
hawaiisurf.com gta-top.ru
healgastro.com guiadeserraazul.com
helpmymacfaster.trade gunsvicceadadebt.tk
hetmanship.xyz guruofcasino.com
hildinghr.se hatedriveapart.com
historichometeam.com hchha.com
homedo.fabpage.com hdseriale.pl
hotgirlhdwallpaper.com heatpower.ru
houdom.net heliko.no
houseofgaga.ru hem.passagen.se
iboss.com herokuapp.com
idealtits.net highland-homes.com
igrovyeavtomaty777.ru highstairs-a.akamaihd.net
iklysha.ml hoverboard360.de
ilo134uloh.com hoverboard360.se
imgarcade.com i-midias.net.br
imicrovision.com iminent.com
inbabes.sexushost.com imperiafilm.ru
incest-ru.com infospot.pt
inclk.com infostatsvc.com
infoupdate.org innoslicon.com
intellektmedia.at instasexyblog.com
internet-apteka.ru int.search.mywebsearch.com
invest-pamm.ru iplogger.org
iron-age.info it-max.com.ua
istizanidineanarcoticdrug.blogspot.com itrevolution.cf
itronics.ca itservicesthatworkforyou.com
iusstf.org iwanttodeliver.com
ivanstroi.ru iweblist.info
izismile.com jamiembrown.com
javspace.net jav-fetish.com
jerseychinabizwholesale.com jav-way.site
javstock.com
javxxx18.com
jbl-charge.info
jerseywholesalechinabiz.com jerseywholesalechinabiz.com
johannesburgsingles.co.za jikoman.info
johnnyhaley.top johnrobertsoninc.com
justprofit.xyz kaidalibor.de
kaliningrad.zrus.org kamalsinha.com
kamorel.com kazinogames.lv
kashubadesign.ru keyword-suggestions.com
kenaba.su kino2018.cc
klosetkitten.com kollekcioner.ru
kolotiloff.ru kolotiloff.ru
krynica.info kongruan.com
kwzf.net kosynka-games.ru
lacasamorett.com kotaku.com
lafriore.ru kountrylife.com
law-check-eight.xyz krasnodar.xrus.org
letmacwork.world ks1234.com
lexaprogeneric.link kutikomi.net
likesdesign.com ladsup.com
linkbuddies.com landreferat.ru
linkredirect.biz larose.jb2c.me
lenvred.org
lexiangwan.com
liquimondo.com liquimondo.com
listiseltemournlan.gq lirunet.ru
littlesunraiser.com livefixer.com
liveinternet.ru liveu.infoteka.hu
localmatchbook.com
locatellicorretor.com.br locatellicorretor.com.br
lodki-pvh.dp.ua lonerangergames.com
lunamedia.co lost-alpha.ru
macdamaged.space louboutinbooties.xyz
madot.onlinewebshop.net louboutinreplica.pw
mail.allnews24.in luckyshop.net.ua
mailemedicinals.com magazintiande.ru
manimpotence.com magicplayer-s.acestream.net
manipulyator-peterburg.ru maglid.ru
mall.uk
malls.com
manualterap.roleforum.ru manualterap.roleforum.ru
marinetraffic.com maranbrinfo.com.br
marketingtechniques.info
marketingvici.com marketingvici.com
marketland.ml martlinker.com
mature.free-websites.com mebel-alait.ru
max-p.men meble-bogart.info
mearns-tractors.co.uk medicinacom.ru
med-zdorovie.com.ua medictube.ru
meddesk.ru meet-flirt-dating.com
mediafresh.online megaapteka.ru
medtherapy.ru millionare.com
meduza-consult.ru missclub.info
megaindex.ru mmgq.ru
megavolt.net.ua mobstarr.com
merryhouse.co.uk moncleronline.xyz
mesbuta.info moncleroutletonline.win
mhi-systems.ru montredemarque.nl
miniature.io moroccosurfadventures.com
mobicover.com.ua moviemail-online.co.uk
moesonce.com music.utrolive.ru
most-kerch.org my-aladin.com
muznachas-service.ru mydirtystuff.com
mybinaryoptionsrobot.com myiptest.com
myindospace.com na15.ru
myprintscreen.com nailsimg.com
mzdish.site news-readers.ru
naturalpharm.com.ua newstaffadsshop.club
neobux-bg.info nfljerseysforsalewholesaler.com
net-profits.xyz
networkad.net
newsperuse.com
nfljerseyscheapchinabiz.com
ningessaybe.me
nippon-bearings.ru nippon-bearings.ru
nisuturnetdgu.tk nl.netlog.com
nlfjjunb5.ru normalegal.ru
nodup.ru nyfinance.ml
nomuos.it obiavo.in
nootrino.com oivcvx.website
nordstar.pro okeinfo.online
nullrefer.com online-podarki.com
o00.in onlineporno.site
one-gear.com onlywoman.org
online-x.ru onthemarch.co
online7777.com optikremont.ru
onlinemegax.com origin-my.ru
onlyforemont.ru osagonline.ru
paperwritingservice17.blogspot.ru paparazzistudios.com.au
parajumpersoutlet.online paphoselectricianandplumber.com
patterntrader-en.com paramountmarble.co.uk
pattersonsweb.com pensplan.com
pekori.to perosan.com
penisvergrotendepillennl.ovh pertlocogasilk.tk
pepperstyle.ru petsblogroll.com
physfunc.ru photochki.com
pinstake.com pills24h.com
pinwallpaper.top pio.polytopesexempt.com
piuminiita.com pirateday.ru
pl.id-forex.com pixshark.com
plusnetwork.com pl-vouchers.com
popup-hgd.xyz playboyfiles.xblog.in
porno-dojki.net poem-paying.gq
polytopesexempt.com
popup.matchmaker.com
porno-transsexuals.ru porno-transsexuals.ru
pornoklad.ru pornohd1080.online
pornolook.net pornokorol.com
pornosok.ru pornosee.info
pornzone.tv pornoxxx.com.mx
portal-eu.ru portnoff.od.ua
portside.cc postclass.com
pr0fit-b0x.com pozdravleniya-c.ru
presleycollectibles.com pozdrawleniya.ru
preventheadacheguide.info predmety.in.ua
priceg.com projectforte.ru
printdirectforless.com promoforum.ru
prize44.com prosmibank.ru
professionalwritingservices15.blogspot.ru psvita.ru
purplesphere.in pukaporn.com
puzzleweb.ru purplestats.com
quick-seeker.com pushdata.sendpulse.com
r-e-f-e-r-e-r.com puzo2arbuza.ru
redhotfreebies.co.uk pyrodesigns.com.au
reklama1.ru qexyfu.bugs3.com
rent2spb.ru rating-casino2021.ru
ringporno.com rcb101.ru
royalads.net rczhan.com
royalcar-ufa.ru real-time-analytics.com
royalvegascasino.com realmonte.net
rusprostitute.com reckonstat.info
rutor.vip reimageplus.com
s1z.ru remedyotc.com
rentalcarnavi.info
responsive-test.net
ritlweb.com
rixpix.ru
rockma.se
rogervivierforsale.com
roofers.org.uk
rozalli.com
ruf777.com
rus-pornuha.com
ryetaw.com
safe-app.net safe-app.net
sale-japan.com saitevpatorii.com
sanatorrii.ru saltspray.ru
saugatuck.com samara.rosfirm.ru
sayyoethe.blogspot.co.za samoiedo.it
sbwealthsolutions.ca saveindex.xyz
scanner-ivan.top
scanner-jack.top scanner-jack.top
se.bnt-team.com search.pch.com
seasaltwithfood.com selectads.men
secret.xn--oogle-wmc.com semalt.com
seo-tools-optimizing.com seonetwizard.com
seoanalyses.com servicecenter.co.ua
shareyards.com sexkontakteao.info
shillyourcoins.com sexystrippe.info
shop.xz618.com shijian.ac.cn
shikiso.info
shoplvlv.us shoplvlv.us
shtaketniki.kz shymkent.xkaz.org
signal03.ru sigmund-freud.co.uk
siha.de simple-share-buttons.com
sildenafil-tadalafil.info simplepooltips.com
silverage.ru sisiynas.ru
simpoed.ufop.br sledstvie-veli.net
site5.com small-games.biz
siteimprove.com snsdeainavi.info
sitiz.club social-vestnik.ru
slimcdn.com softonicads.com
smart-scripts.com sonnikforme.ru
social-s-hhh.xyz spb-plitka.ru
socialmonkee.com special-porn.com
solnplast.ru spywareit.com
sophang8.com spywarespy.com
spasswelt.net squidoo.com
spb.afora.ru srdrvp.com
spb.ru stal-rulon.ru
specialfinanceoffers.com stathat.com
sptslmtrafms.com stauga.altervista.org
startufa.ru steebook.com
stpicks.com stpicks.com
strfls.com strigkaomsk.ru
stroymonolit.su stroiminsk.com
subj.ukr-lit.com stroydetali.ru
sukirgenk.dvrlists.com sugarkun.com
supermesta.ru sumo.com
surcentro.com superoboi.com.ua
szamponrevita.pl surgut.zrus.org
tanieaukcje.com.pl swimpool.ca
tech4master.com tabletkinaodchudzanie.com.pl
teenporn18.net tacbibirfa.tk
testbotprocessor44.com tackletarts.co
thepantonpractice.co.uk tagil.zrus.org
thexart.club tandvardshuset.net
timer4web.com tattoo33.ru
tizanidine4mg.blogspot.com tattooha.com
tizanidine4mgtablets.blogspot.com td-l-market.ru
tizanidinehcl2mgsideeffects.blogspot.com tds-advert002.info
tizanidinehydrochloride2mg.blogspot.com telesvoboda.ru
tizanidinevscyclobenzaprine.blogspot.com templates.radiodigital.co
tizanidinewithdrawal.blogspot.com thaisamkok.com
tn811.us thaismartloan.com
todohr.com thegameriders.com
top10-online-games.com theguardlan.com
top250movies.ru themestotal.com
torontoplumbinggroup.com thfox.com
torrents-tracker.com tizanidineandcipro.blogspot.com
tourismvictoria.com tizanidinestreetprice.blogspot.com
toyota.7zap.com tizanidinewithdrawalsymptoms.blogspot.com
tpu.ru tkanorganizma.ru
tksn.ru
tnctrx.com
tobeyouday.win
topanasex.com
topbestgames.com
topkarkas.com
trackmedia101.com
trafficmania.com trafficmania.com
trainoffend.ml trafficmp.com
trk-4.net trafficreceiver.club
tula.mdverey.ru trubywriting.com
tupper-shop.ru truck-addzilla.life
turizm.bz tversvet.ru
twinderbella.com ukkelberg.no
uhdtv.website
ukrobstep.com
ukrtvir.com.ua ukrtvir.com.ua
underthesite.com umnovocaminho.com
uroffer.link universals.com.ua
valuado.com uptime-gamma.net
vapeface.club urlcut.ru
vapsy.com usadacha.net
ventelnos.com vchulkah.net
veopornogratis.xxx viagroid.ru
veselokloun.ru viberdownload10.com
vesnatehno.ru videos-for-your-business.com
videochat.ph vigrx-original.ru
videochat.tv.br viktoria-center.ru
vip2ch.com vizitki.net
vipromoffers.com voluumtracker1.com
vitalads.net voprosotvet24.ru
vodkoved.ru vriel.batcave.net
vostoktrade.info
vote-up.ru
vpnmouse.com
vrotike.ru vrotike.ru
vut.com.ru vseigru.one
vvon.co.uk vsesubwaysurfers.com
vzlom-na-zakaz.com vsexkontakte.net
wallpaperstock.net watchmygf.net
wdfdocando.com watracker.net
web.cvut.cz waycash.net
webextract.profound.net wcb.su
webix.biz webcamdevochka.com
webpromotion.ae
website-speed-check.site
webstatsdomain.org
wedding0venues.tk
weddingdresses.xyz weddingdresses.xyz
wesharepics.info westen-v.life
wetgames.ru wildnatureimages.com
whatsappbot.flyland.ru wineration.com
whatzmyip.net workle.website
whos.amung.us worldlovers.ru
windowssearch-exp.com worldoffiles.ru
woman-h.ru wosik-dach.service-for-web.de
works.if.ua wovis.site
woweb.com.ua writersgroup580.web.fc2.com
wttavern.com ww1943.ru
xaylapdiendanang.com www.souvenirua.com
xlolitka.com www.timer4web.com
xmronta.com x-porno.video
xn--80aaaks3bbhabgbigamdr2h.xn--p1ai x-true.info
xn--80aanaardaperhcem4a6i.com xn------7cdbapdecfd4ak1bn0amjffj7afu3y.xn--p1ai
xn--80abgj3a5acid6ghs.top xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai
xn--b1ag5cfn.xn--p1ai xn----ctbbcjd3dbsehgi.xn--p1ai
xn--h1aakne2ba.xn--p1ai xn--80adaggc5bdhlfamsfdij4p7b.xn--p1ai
xuki.us xn--b1adccaf1bzj.xn--p1ai
xvideosporn.biz xn--lifehacer-1rb.com
xxart.ru xnxx-n.com
xxxguitars.com yoshkarola.zrus.org
yadro.ru youporn-ru.com
yatrk.xyz youresponsive.com
yes-com.com yoursearch.me
yginekologa.com youtuhe.com
yougame.biz yuarra.pluto.ro
yougetsignal.com yunque.pluto.ro
youporn-forum.ga zastroyka.org
yourporngay.com zawyna.ua
yourtemplatefinder.com zeroredirect12.com
zaapplesales.blogspot.com zeroredirect5.com
zakonobosago.ru zooggames.com
zlatnajesen.com zophim.me
zog.link
zojirushi-products.ru
zoominfo.com
zscaler.net
zverokruh-shop.cz
zvooq.eu

488
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
2pxg8bcf.top 12masterov.com
4k-player.pl 2020iscoming.info
7a2rnuey1tw9ar.ru 202ch.com
883zy.com 2728fb936f0.com
acads.net 5ws.dating-app.ru
acmebtn.ml 999webdesign.com
addbin.men aad1f4acb0a373420d9b0c4202d38d94fa.com
adnotbad.com aaronabel.com
adtiger.tk actionnooz.com
airlandsea.info adrunnr.com
algerianembassy.co.in adservme.com
allwidewallpapers.com adsfresh.men
anaksma.info alltheviews.com
andpolice.com animaltoplist.com
animenime.ru apispringsmartne-a.akamaihd.net
apartamentwroclaw.eu arpelsreplica.xyz
arabseyes.com
arewater.com
asmxsatadriverin.aircus.com
atofilms.com
autoseo-traffic.com
azadnegar.com
azazaporn.com azazaporn.com
babieca.com bang-hotties.com
bambi.ck.ua bcmp.org
barnfurnituremart.com becuo.com
bdsmgalls.net berrymall.ru
bestadbid.com bestofferswalkmydogouteveryday.gq
betonka.pro bestwebsitesawards.com
bimatoprost-careprost.com.ua bimmerpost.com
blockworld.ru blog.koorg.ru
bmwhighperformers.com blog.yam.com
bolezniorganov.ru bolegapakistan.com
bracketsmackdown.com brimstonehillfortress.org
brainboosting.club buqyxa.rincian.info
builtwith.com buttons-for-your-website.com
buqayy0.livejournal.com camel-beach.com
buron.pw carsplate.com
buscarfoto.com cartierlovebracelet.xyz
businesn.men cartierloveringreplica.xyz
callawaygolfoutlet.online centre-indigo.org.ua
carson.getenjoyment.net cerev.info
chocolateslim-original.com cheapmarmot.online
cialis-samples.com citysecurity.nu
click2pawn.com cleanallspyware.com
clickso.com cleansearch.net
compliance-checker.info coinsspb.com
compliance-fred.top compliance-ivan.top
confib.ifmo.ru
contentsexpress.com
cookie-law-enforcement-aa.xyz cookie-law-enforcement-aa.xyz
courtshipgift.com cowlmash.com
crazyboost.pro crackguru.tk
crd.clan.su cyse.tk
cryptoswap.biz danceuniverse.ru
data.vtc.pw delfin-aqua.com.ua
dellalimov.com digest-project.ru
deshevo-nedorogo.ru dlya-android.org
designdevise.com dogoimage.com
discovertreasure-a.akamaihd.net domainsphoto.com
donvito.unas.cz e-buyeasy.com
ednorxmed.com el-nation.com
ekaterinburg.xrus.org elektrischezi.canalblog.com
euronis-free.com erotikchat-24.com
extlabs.io escort-russian.com
ez8motelseaworldsandiego.com essaypro.com
ezigarettenshop2.postbit.com excaliburfilms.com
filmi-v.online fanoboi.com
fireads.men fashion-mk.net
floridamobilebillboards.com faz99.com
forexgb.ru fealq.com
forms-mtm.ru felizporno.com
free-share-buttons.com flipper.top
freelinkbuilding.website.tk floridahuntingfishingadventures.com
freshwallpapers.info forum.tvmir.org
frighteningremain.cf foto-basa.com
frvo.alptandem.ru frankofficial.ru
fungamelands.com freeuploader.ml
fz139.ttk.ru frivgame250.com
gael-s.ru fun-mobi.pl
gaygalls.net funcrushgames.com
gdzkurokam.ru gsasearchengineranker.top
gearsadspromo.club guruofcasino.com
gemara.com hem.passagen.se
getpopunder.com hikesearch.net
ghernnqr.skyrock.com hostnow.men
gheus.altervista.org htmlcorner.com
gigablast.com hydropump.su
globatur.ru igithab.com
hcate.com increasewwwtraffic.info
hentaiheroes.com infospot.pt
hermesbirkinhandbagoutlets.com intelhdgraphicsgtdrive6w.metroblog.com
hidefiles.org intervsem.ru
highspeed5.net iptool.xyz
host-protection.com irkutsk.online-podarki.com
hostingclub.lk javxxx18.com
innodgfdriverhm.aircus.com jikoman.info
intermesh.net kasino-money.pw
internet-apteka.ru keyword-suggestions.com
keywordhut.com kidd.reunionwatch.com
koopilka.com kolotiloff.ru
kovesszucs.atw.hu kongoultry.net
krasnodar.ru kraljeva-sutjeska.com
laparfumotec.com krasivoe-hd.com
law-check-two.xyz ks1234.com
life-instyle.com kung-fu-ru.com
liveu.infoteka.hu kursy-ege.ru
longadventure.com larger.io
m3gadownload.pl law-check-seven.xyz
macfix.life leboard.ru
mafa-free.com libertybilisim.com
magicalfind-a.akamaihd.net liverpool.gsofootball.com
mahnwachen-helfen.info lombardfinder.ru
master-muznachas.ru louisvuittonoutletstore.net
mcar.in.ua lsp-awak-perikanan.com
meccadumps.net lyrster.com
medicines-choice.com m0r0zk0-krava.ru
medicineseasybuy.com mac-shield.com
meetlocalchicks.com magicdiet.gq
menstennisforums.com mansparskats.com
misterjtbarbers.com maximilitary.ru
mlvc4zzw.space maxthon.com
mobot.site medi-fitt.hu
meds-online24.com
megaindex.ru
methodsmarketing.com
mezaruk.info
middlerush-a.akamaihd.net
minyetki.ru
mirzonru.net
mixed-wrestling.ru
mobsfun.net mobsfun.net
monclerboots.xyz mojpreskumanie.com
morefastermac.trade moncleroutletonline.win
muschisexbilder.com morepoweronmac.trade
myanyone.net mttwtrack.com
myfreemp3.eu mustwineblog.com
myxdate.info muz-baza.net
myyour.eu muz-tracker.net
nakozhe.com my-bc.ru
narkologiya-peterburg.ru myliveblog.ru
nextlnk12.com mypets.by
ngps1.ru mywallpaper.top
nrv.co.za na-telefon.biz
oda.as nyfinance.ml
of-ireland.info o-o-8-o-o.ru
offf.info obiavo.com
onescreen.cc odoratus.net
oneshotdate.com oecnhs.info
opinionreelle.com ohmyrings.com
outclicks.net onemantrip.com
onlinedomains.ru
onlinefilmz.net
ons-add.men
ossmalta.com
ownshop.cf ownshop.cf
page2rss.com ownshop.win
panchro.xyz paceform.com
partnerline.men pcimforum.com
peugeot-club.org pechikamini.ru
pinapchik.com photo.houseofgaga.ru
pistonclasico.com php-market.ru
pix-hd.com pixell.club
plugingeorgia.com pobeiranie.pl
pornohub.me poems.com.ua
pornosok.ru pojdelo.weebly.com
pornowarp.info polytopesexempt.com
pozdrawleniya.ru pornhive.org
printie.com pozdravleniya-c.ru
privacylocationforloc.com priscilarodrigues.com.br
privhosting.com pushdata.sendpulse.com
pro-poly.ru puteshestvennik.com
pulse33.ru qwesa.ru
purchasepillsnorx.com rapevideosmovies.com
pwwysydh.com razorweb-a.akamaihd.net
qexyfu.bugs3.com rebrand.ly
qor360.com recipedays.com
razyboard.com reeyanaturopathy.com
rcb101.ru refererx.com
registry-cleaner.net
renecaovillasale.online renecaovillasale.online
renewablewealth.com royalads.net
rimedia.org roznica.com.ua
rixpix.ru rp9.ru
rospromtest.ru rusexy.xyz
rubbed.us sarm.tk
sbtdesign.co.uk satoristudio.net
scanner-george.top scalerite.co.za
scanner-nelson.top scottbywater.com
search-error.com security60-e.com
searchinquire.com shijian.ac.cn
seksotur.ru similardeals.net
servisural.ru simul.co
sevenstars7.com sitevalued.com
sex-dating.co sk.golden-praga.ru
sexad.net slowmacfaster.trade
sexkrasivo.net sluganarodu.ru
sexs-foto.top snow.nvr163.com
sfd-chess.ru soblaznu.net
shakhtar-doneck.ru soc-proof.su
shtora66.ru socialmadesimple.com
shymkent.xkaz.org soft1.ru
smartshoppymac.com soochi.co
socas.pluto.ro sovinsteel.ru
social-buttons.com space2019.top
soda.media sps-shop.com
souvenirua.com stanthonyscatholicchurch.org
star61.de stpolice.com
start.myplaycity.com studiofaca.com
steebook.com sucsesofinspiration.com
steelmaster.lv superfish.com
sudexpert66.ru tam-gde-more.ru
supers.com.ua tattoo33.ru
surintech.ac.th taxi-v-eisk.ru
szamponrevita.pl taximytishi.ru
tattooreligion.ru td-l-market.ru
teenforporn.com technopellet.gr
telegraf.by telsis.com
theallgirlarcade.com terrafootwear.us
timecrimea.ru thecounter.com
tizanidine4mg.blogspot.com theheroes.ru
tizanidinenarcotic.blogspot.com tizanidineingredients.blogspot.com
toon-families.com tnctrx.com
tonerbox.kz
top-karkas.ru top-karkas.ru
track112.site top-l2.com
trafficreceiver.club torrents.cd
trk-4.net torture.ml
trucri.me trichizobswiv.agddns.net
turist-strani.ru tubeline.biz
turvgori.ru underthesite.com
uhdtv.website varikoz24.com
ukrtextbook.com varikozdok.ru
ukrup.com vipromoffers.com
unblocksit.es virus-schutzmasken.de
uvozdeckych.info vod.com.ua
velpanex.ru volgograd.xrus.org
vertaform.com vote-up.ru
via-energy-acquistare.com vpnmouse.com
virtuagirl.com vysigy.su
vizag.kharkov.ua watsonrealtycorp.com
vkontarkte.com webix.me
weblibrary.win webtherapy.ru
webmonetizer.net whatzmyip.net
website-stealer.nufaq.com wheelchairliftsarea.com
wedding-salon.net whos.amung.us
wellcome2slovenia.ru williamrobsonproperty.com
www.wohnkabinen-shop.de wordkeyhelper.com
xedserver.com worldinternetauthority.com
xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai wpthemedetector.co.uk
xn--hxazdsfy.blogspot.com x5market.ru
xzlive.com xchat26.myfreecams.com
yaaknaa.info xmnb.net
yellowproxy.net xxxsiterips.xyz
yellowstonevisitortours.com yourmovies.pl
yoshkarola.zrus.org ytmnd.com
youbloodyripper.com zahvat.ru
zaobao.com.sg zero1.it
zeg-distribution.com zocaparj.kz
zhuravlev.info
zigzog.ru
zoomovies.org
zynax.ua

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.198 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.09.2564 ### Version: V4.2021.09.2565
### Updated: Wed Sep 1 22:01:26 UTC 2021 ### Updated: Thu Sep 2 10:03:31 UTC 2021
### Bad Bot Count: 624 ### Bad Bot Count: 624
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.09.2564 ### Version: V4.2021.09.2565
### Updated: Wed Sep 1 22:01:26 UTC 2021 ### Updated: Thu Sep 2 10:03:31 UTC 2021
### Bad Bot Count: 624 ### Bad Bot Count: 624
################################################### ###################################################
### Version Information ## ### Version Information ##