vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-10 06:15:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.09.2572 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-09-05 10:10:08 +00:00
parent 6236cbdbc6
commit 2c637c3e4d
46 changed files with 76060 additions and 76060 deletions
Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.09.2571 #### Version: V4.2021.09.2572
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.09.2571 #### Version: V4.2021.09.2572
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -11,7 +11,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2021.09.2571 #### Version: V4.2021.09.2572
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

9372
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2021.09.2571 #### Version: V4.2021.09.2572
#### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7078](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

9376
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9376
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

9376
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9376
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 1; 10.1.1.40 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 0; 10.1.1.40 0;
127.0.0.1 0; 127.0.0.1 0;

9376
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 1; 10.1.1.40 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 0; 10.1.1.40 0;
127.0.0.1 0; 127.0.0.1 0;

9376
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 1; 10.1.1.40 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 0; 10.1.1.40 0;
127.0.0.1 0; 127.0.0.1 0;

9376
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2571 2572

9376
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

9376
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.149.40.206 0; 52.179.100.214 0;

9376
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 1; 10.1.1.40 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 0; 10.1.1.40 0;
127.0.0.1 0; 127.0.0.1 0;

9376
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 1; 10.1.1.40 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 0; 10.1.1.40 0;
127.0.0.1 0; 127.0.0.1 0;

9376
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 1; 10.1.1.40 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 0; 10.1.1.40 0;
127.0.0.1 0; 127.0.0.1 0;

9376
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.149.40.206 0; 52.179.100.214 0;

9376
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.149.40.206 0; 52.179.100.214 0;

9376
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 1; 10.1.1.40 1;
127.0.0.1 1; 127.0.0.1 1;

18
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
AhrefsBot 360Spider
BDFetch HMView
Fyrebot HTMLparser
Page Analyzer Mister PiX
Pandalytics
Piepmatz Piepmatz
PyCurl SurveyBot
Quick-Crawler Upflow
SearchmetricsBot Virusdie
Web Fuck crawler4j
serpstatbot

300
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
360Spider 404enemy
404checker 80legs
AIBOT ALittle Client
Abonti Abonti
Aboundex AiHitBot
AhrefsBot
Alexibot
AllSubmitter AllSubmitter
Alligator
AlphaBot AlphaBot
Anarchie
Anarchy Anarchy
Anarchy99 Aspiegel
Ankit AwarioSmartBot
BBBike BackDoorBot
BDCbot Backlink-Ceck
Bandit Badass
Barkrowler Barkrowler
Battleztar Bazinga Battleztar Bazinga
Bigfoot
Bitacle Bitacle
Blackboard Blow
Bolt Bolt
BotALot
Brandprotect
Brandwatch
Buck Buck
BuiltBotTough BuzzSumo
BuiltWith CATExplorador
CCBot CazoodleBot
ChinaClaw CheTeam
Chlooe CheeseBot
Claritybot
Cliqzbot
Cloud mapping Cloud mapping
Cocolyzebot Copier
Cosmos Copyscape
CrazyWebCrawler Crawling at Home Project
Curious CrunchBot
Custo
CyotekWebCopy
DBLBot DBLBot
DIIbot DIIbot
Discoverybot Demon
Digincore
Dirbuster
Disco
Dispatch
DittoSpyder DittoSpyder
DomCopBot DnyzBot
DomainAppender
DomainCrawler
DomainStatsBot DomainStatsBot
Domains Project Dotbot
Download Wonder
Drip Drip
ECCP/1.0 ECCP/1.0
EMail Siphon EMail Wolf
EasyDL
Ebingbong Ebingbong
Ecxi
EirGrabber EirGrabber
EroCrawler EroCrawler
Evil
Exabot Exabot
Express WebPictures Express WebPictures
ExtLinksBot ExtLinksBot
Extractor Extractor
Extreme Picture Finder Extreme Picture Finder
EyeNetIE Ezooms
FemtosearchBot FemtosearchBot
FlashGet
Foobot Foobot
Freeuploader FrontPage
Fuzz
Fyrebot Fyrebot
GT::WWW G-i-g-a-b-o-t
GermCrawler GermCrawler
Getintent
Gigabot
Go!Zilla Go!Zilla
Go-Ahead-Got-It Go-Ahead-Got-It
GrabNet
Grabber
GrapeFX GrapeFX
GrapeshotCrawler GrapeshotCrawler
HMView HMView
HTMLparser HTMLparser
HTTrack Harvest
Haansoft HybridBot
Heritrix IDBTE4M
IDBot
Id-search
IlseBot IlseBot
Iskanie Image Fetch
JamesBOT InfoNaviRobot
InternetSeer
IstellaBot
Jbrofuzz
JennyBot JennyBot
Jetty JetCar
JikeSpider Kenjin Spider
Joomla Keyword Density
Jorgee Leap
Kinza
Kozmosbot
LNSpiderguy
Larbin
LeechGet LeechGet
LexiBot LexiBot
Libwhisker LibWeb
LieBaoFast LieBaoFast
LinkpadBot Likse
LinksManager LinkScan
Lmspider Linkbot
MSFrontPage Linkdexbot
Magnet LinqiaMetadataDownloaderBot
Lipperhey
Ltx71
MSIECrawler
Mag-Net
Mail.RU_Bot
Majestic SEO
Majestic-SEO Majestic-SEO
Majestic12 Majestic12
MarkMonitor
MarkWatch MarkWatch
Mass Downloader Mass Downloader
Masscan Masscan
Meanpathbot Mb2345Browser
Microsoft Data Access MeanPath Bot
Metauri
Microsoft URL Control Microsoft URL Control
Mister PiX Mister PiX
Mojeek
Mojolicious Mojolicious
Morfeus Fucking Scanner Morfeus Fucking Scanner
Msrabot NetAnts
NPbot NetMechanic
Navroad Netvibes
NearSite Nibbler
NetLyzer Niki-bot
NetSpider Nimbostratus
NetZIP Nutch
Nettrack Openvas
Nikto PageGrabber
Nmap PageScorer
Offline Navigator
Openfind
OutclicksBot
POE-Component-Client-HTTP
Pandalytics
Panscient
Pi-Monster
Picscout
Picsearch Picsearch
Pixray Pixray
PleaseCrawl PleaseCrawl
Psbot Probethenet
Pump Pu_iN
QueryN Metasearch PyCurl
Quick-Crawler
RSSingBot RSSingBot
RankActive RankActive
RankFlex RankFlex
RankingBot2
Rankivabot
RankurBot RankurBot
Re-re Recorder
ReGet
RedesScrapy RedesScrapy
RepoMonkey
Ripper Ripper
Rogerbot
SBIder SBIder
SEOlyticsCrawler SEOkicks
SEOprofiler SEOkicks-Robot
SMTBot ScanAlert
Scanbot ScoutJet
Screaming ScrepyBot
Searchestate Searchestate
SentiBot SearchmetricsBot
SeobilityBot Semrush
Seomoz
Shodan Shodan
Siphon Siphon
SiteCheckerBotCrawler
SiteExplorer SiteExplorer
SiteLockSpider SiteLockSpider
SiteSucker
Sitebeam
Sitevigil Sitevigil
SlySearch SlySearch
Snoopy Snake
Sqlmap Snapbot
Sqlworm Sottopop
Sqworm SpaceBison
Sucker Spammen
Sucuri Steeler
SuperBot Stripper
SurveyBot
Suzuran Suzuran
Swiftbot Szukacz
T0PHackTeam T0PHackTeam
T8Abot Teleport
TeleportPro Telesoft
Telesphoreo Telesphoreo
Telesphorep Telesphorep
TightTwatBot TheNomad
Thumbor
Titan Titan
Toweyabot True_Robot
Tracemyfile Turnitin
Trendictionbot TwengaBot
URLy Warning UnisterBot
URLy.Warning Upflow
VCI V-BOT
VeriCiteCrawler Vacuum
VidibleScraper VelenPublicWebCrawler
Virusdie
VoidEYE
WASALive-Bot WASALive-Bot
WBSearchBot WBSearchBot
WWW-Collector-E WEBDAV
WWW-Mechanize
WWWOFFLE WWWOFFLE
Wallpapers Wallpapers/3.0
WallpapersHD WeSEE
Web Auto
Web Enhancer Web Enhancer
Web Fetch
Web Fuck Web Fuck
Web Pix
Web Sucker Web Sucker
WebAuto
WebCollage
WebEnhancer WebEnhancer
WebFetch
WebFuck WebFuck
WebImageCollector WebGo IS
WebLeacher
WebSauger WebSauger
WebStripper WebStripper
Webalta WebZIP
Webshag Webshag
Website Quester
WebsiteExtractor
Webster Webster
Whack Whack
Whacker Who.is Bot
WiseGuys Robot WiseGuys Robot
Wonderbot Wonderbot
Woobot
Wotbox Wotbox
YoudaoBot Wprecon
Xaldon WebSpider
Xenu
Zade Zade
Zauba Zauba
Zermelo Zermelo
Zitebot Zitebot
ZumBot ZmEu
ZoomBot
ZoominfoBot
ZyBorg ZyBorg
archive.org_bot
arquivo-web-crawler arquivo-web-crawler
arquivo.pt cognitiveseo
cah.io.community dataforseo.com
check1.exe facebookscraper
clark-crawler
crawl.sogou.com
demandbase-bot
domainsproject.org
eCatch
gopher gopher
heritrix heritrix
instabid instabid
internetVista monitor
lwp-request
lwp-trivial
meanpathbot meanpathbot
oBot mediawords
muhstik-scan
netEstate NE Crawler
page scorer page scorer
pcBrowser
probe-image-size
ripz
seobility
seostar
sexsearcher
sitechecker.pro
sogouspider
spyfu
sysscan sysscan
tAkeOut
trendiction.com trendiction.com
ubermetrics-technologies.com
voyagerx.com
webmeup-crawler webmeup-crawler
x09Mozilla webpros.com
zgrab webprosbot

306
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 360Spider
404checker 404enemy
ALittle Client 80legs
ASPSeek Aboundex
Aboundexbot
Acunetix Acunetix
AiHitBot AfD-Verbotsverfahren
AllSubmitter
AlphaBot AlphaBot
Anarchy Ankit
Anthill Apexoo
Asterias
Attach
BBBike
BDFetch
BackDoorBot BackDoorBot
BackStreet Badass
BacklinkCrawler Blackboard
Bandit BotALot
Barkrowler Brandprotect
BlackWidow Buck
Blow BuiltBotTough
Bolt BuiltWith
Brandwatch
Buddy
BunnySlippers BunnySlippers
Calculon BuzzSumo
CATExplorador
CazoodleBot CazoodleBot
Cegbfeieh
CensysInspect
CherryPicker CherryPicker
Chlooe ChinaClaw
Claritybot Claritybot
Cliqzbot
Cloud mapping Cloud mapping
Collector Cocolyzebot
Copyscape Cogentbot
Copier
CopyRightCheck
Craftbot
Crawling at Home Project
CrazyWebCrawler
Crescent
CrunchBot CrunchBot
Curious DTS Agent
DBLBot Devil
Digincore Digincore
DigitalPebble
Dirbuster
Discoverybot
Dispatch Dispatch
DittoSpyder
DnyzBot DnyzBot
DomCopBot DomCopBot
DomainAppender DomainAppender
DomainStatsBot DomainCrawler
Domains Project DomainSigmaCrawler
Dotbot
Download Wonder Download Wonder
EMail Wolf Dragonfly
EasyDL EasyDL
EirGrabber
Evil
Exabot
ExtLinksBot ExtLinksBot
ExtractorPro
EyeNetIE
FHscan
FemtosearchBot FemtosearchBot
Fimap
Firefox/7.0
FlashGet FlashGet
Flunky Flunky
Foobot Foobot
Freeuploader
FrontPage FrontPage
Fyrebot
G-i-g-a-b-o-t G-i-g-a-b-o-t
GT::WWW
Genieo
GetRight GetRight
Getintent Gigabot
Go-Ahead-Got-It GoZilla
Gotit Grafula
GrabNet HMView
Grabber HTMLparser
GridBot HTTP::Lite
HEADMasterSEO
HTTrack
HaosouSpider
Harvest Harvest
Heritrix
Humanlinks Humanlinks
IDBTE4M HybridBot
IDBot IDBot
IRLbot IRLbot
Iblog Iblog
IlseBot Id-search
Image Fetch IndeedBot
Indy Library InfoNaviRobot
Intelliseek InfoTekies
Internet Ninja InterGET
InternetSeer InternetSeer
Iskanie Iria
JOC Web Spider IstellaBot
JamesBOT JamesBOT
JetCar
Jorgee
JustView JustView
Kenjin Spider Jyxobot
Keyword Density Kinza
Kozmosbot
LNSpiderguy LNSpiderguy
Larbin LWP::Simple
LibWeb Leap
Libwhisker Libwhisker
LieBaoFast Likse
LinkScan
LinkWalker LinkWalker
Linkbot LinksManager
LinkpadBot LinqiaMetadataDownloaderBot
LinqiaScrapeBot Lmspider
Lipperhey
Litemage_walker
Ltx71 Ltx71
MFC_Tear_Sample MFC_Tear_Sample
Mag-Net MIIxpc
Majestic SEO MJ12bot
MQQBrowser
MTRobot
Mail.RU_Bot
Majestic12 Majestic12
Masscan MarkWatch
Mass Downloader
MauiBot
Mb2345Browser
MeanPath Bot MeanPath Bot
MegaIndex.ru Meanpathbot
Moblie Safari MicroMessenger
Microsoft Data Access
Mojolicious Mojolicious
Morfeus Fucking Scanner
Mozlila Mozlila
Msrabot NPbot
Musobot Navroad
NICErsPRO
NearSite NearSite
Needle
Nessus Nessus
NetMechanic NetMechanic
Netcraft NetZIP
Netvibes NextGenSearchBot
Nibbler
Niki-bot Niki-bot
Nikto
NimbleCrawler NimbleCrawler
Nimbostratus Nimbostratus
Nuclei Ninja
Octopus Nmap
Offline Explorer Nutch
Offline Navigator OutclicksBot
OpenVAS
Openvas
OrangeBot
OrangeSpider
PECL::HTTP
PHPCrawl PHPCrawl
POE-Component-Client-HTTP Page Analyzer
PageAnalyzer
PageGrabber PageGrabber
PageScorer PageScorer
PageThing.com
Pandalytics
Panscient Panscient
Pavuk PeoplePal
Pi-Monster Pi-Monster
Picsearch Picsearch
Pixray
PleaseCrawl PleaseCrawl
Pockey Pockey
ProWebWalker ProPowerBot
Probethenet
Psbot Psbot
Pu_iN
Pump Pump
PyCurl PxBroker
Quick-Crawler Quick-Crawler
RSSingBot RankActiveLinkBot
RankingBot2 RankingBot2
Reaper Rankivabot
Recorder RankurBot
RocketCrawler Re-re
RedesScrapy
SBIder
SEOkicks SEOkicks
SEOkicks-Robot
SISTRIX SISTRIX
SMTBot SMTBot
SalesIntelligent ScanAlert
Scrapy Scrapy
SearchmetricsBot SearchmetricsBot
Semrush
SemrushBot SemrushBot
Seomoz Seomoz
Site Sucker SiteCheckerBotCrawler
Sitebeam SiteExplorer
SiteLockSpider
Siteimprove Siteimprove
Sociscraper SlySearch
Snoopy
Sogou web spider Sogou web spider
Sosospider Sosospider
Spammen Sottopop
SpankBot SpankBot
Spbot Sqlworm
SputnikBot
Sqworm
Steeler Steeler
SurveyBot SurveyBot
Swiftbot Swiftbot
Teleport Szukacz
TeleportPro
Telesphoreo Telesphoreo
TheNomad
Thumbor Thumbor
Toata Trendiction
Tracemyfile
True_Robot
Turnitin Turnitin
TurnitinBot Twice
Typhoeus Upflow
URLy.Warning VB Project
VCI
Vacuum Vacuum
Vagabondo VeriCiteCrawler
VelenPublicWebCrawler
VidibleScraper VidibleScraper
Voil Voltron
WASALive-Bot WISENutbot
WEBDAV
WPScan WPScan
WWW-Collector-E WWW-Collector-E
WWWOFFLE
Wallpapers
Wallpapers/3.0
WallpapersHD
Web Auto
Web Enhancer Web Enhancer
Web Sucker Web Fetch
WebGo IS WebAuto
WebReaper WebCollage
WebSucker WebFetch
WebWhacker WebFuck
WebSauger
WebStripper
Webalta
Webster
Whack
Whacker Whacker
Who.is Bot
Widow
Wonderbot Wonderbot
Zitebot Wotbox
Wprecon
Xaldon WebSpider
Xenu
Zauba
ZmEu ZmEu
ZoominfoBot ZoominfoBot
ZumBot
ZyBorg
arquivo.pt arquivo.pt
autoemailspider backlink-check
cah.io.community cah.io.community
check1.exe
clark-crawler
coccocbot-web coccocbot-web
crawler4j crawl.sogou.com
dataforseo.com facebookscraper
demandbase-bot
eCatch
ips-agent
lwp-request
lwp-trivial lwp-trivial
magpie-crawler mediawords
meanpathbot netEstate NE Crawler
muhstik-scan
page scorer page scorer
pcBrowser probe-image-size
plumanalytics
polaris version
ripz
satoristudio.net satoristudio.net
siteripz scan.lol
spyfu seobility
seoscanners
seostar
sitechecker.pro
sysscan sysscan
trendiction.com
trendiction.de trendiction.de
ubermetrics-technologies.com
voyagerx.com voyagerx.com
webmeup-crawler
webpros.com webpros.com
webprosbot
x09Mozilla x09Mozilla
zgrab zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
7a2rnuey1tw9ar.ru aihelen.net
acunetix-referrer.com allblogroll.com
danceuniverse.ru f5mtrack.com
j33x.com healmytrauma.info
myprintscreen.com ks1234.com
searchencrypt.com rent2spb.ru
soc-proof.su rustic-quiver.win
vpnmouse.com video-chat.love
wetgames.ru websitevaluebot.com
youtuhe.com zmoda.hostreo.com

924
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
007angels.com 00author.com
01apple.com 0n-line.tv
101flag.ru 101billion.com
13tabs.com 1adult.com
18ps.ru 1bet.com
1j7740kd.website 20pascals.nl
34.gs 23kw.ru
3w24.com 2drittel.de
4kpics.rocks 3th.co.in
5ws.dating-app.ru 57883.net
76brighton.co.uk 72-news.com
7makemoneyonline.com 7fon.ru
8si.ru 7search.com
98oi.ru 991mostfm.co.id
999webdesign.com 9icmzvn6.website
a1security.com.ua aa9bd78f328a6a41279d0fad0a88df1901.com
acheterviagrafr24.com abwa.tk
actulite.com accessoires-mode.in
addbin.men adamoads.com
addblueoff.com.ua adclickthru.net
adman.se adserver-e7.com
adobereader-free.ru afora.ru
adrunnr.com agecheckadult.com
adultnet.in aibolita.com
advertisingtag.net akita.kz
adzpower.com alert-hgd.xyz
afftrack001.com alfa9.com
afslankpillen2017nl.eu alfabot.xyz
agroeconom.kz allwomen.info
alevinefesleri.com alphaforum.ru
amateurmatch.com alphavisions.net
amt-k.ru alt-servis.ru
amung.us amazingpic.net
analnoeporno.tv android-style.com
anaseracresar.tk angry-fermi-7633.arukascloud.io
angkortours.vn animaltoplist.com
animal-drawings.com anonymizeme.pro
animebox.com.ua apibrowsesmartne-a.akamaihd.net
apiholdingmypage-a.akamaihd.net
apioutoboxnet-a.akamaihd.net apioutoboxnet-a.akamaihd.net
appmsr.org apple.com-cleaner.systems
apple.com-webbrowsing-security.review
apps-analytics.net apps-analytics.net
arabsexxxtube.com arabgirls.adultgalls.com
arcadepages.com arenda-avtoprokat-krasnodar.ru
ardimobileinfo.ml articlesdirectoryme.info
arendovalka.xyz asmxsatadriverin.aircus.com
arkartex.ru atlasvkusov.ru
autobrennero.it atmovs.com
awaybird.ru aviav.ru.com
ayodhya.co avtointeres.ru
balla-boo.se avtolombard-krasnodar.com
belinka.com.ua baladur.ru
belsetirehafi.tk bannerspace.com
bensbargains.net bannerswap.com
berrymall.ru beachfix.co
bestimagecoollibrary.com belstaffstore.online
bestssaker.com bestdooz.com
betshuckclean.com bestdraws.com
bg6s0.com bestpornuha.com
bio-optomarket.ru bestwrinklecreamnow.com
biznesrealnost.ru betteroffers.review
bjgugu.net.ua bget.ru
bkns.vn bio-market.kz
bloggerads.net bio.trade-jp.net
blogporn.in biteg.xyz
bmsco.co bizcheapjerseyswholesalechina.com
bmw.afora.ru biznesluxe.ru
bolegapakistan.com blackplanet.com
bonanza-fortune.men blog.koreadaily.com
braindaily.xyz bloggen.be
brandedleadgeneration.com bloggers.nl
brk-rti.ru bluesalt.co
brains2.biz
bravegirlsclub.com
briomotor.co
bscodecs.com bscodecs.com
buckspar.gq
budilneg.xyz
builtwith.com
bumascloset.com bumascloset.com
business-suggested.tk buyk.host.sk
buy-cheap-pills-order-online.com buynorxx.com
buyessaynow.biz bvlgariwallet.xyz
buyscabiescream.com bwlx.prepedu.cn
buzzonclick.com
bytimedance.ru bytimedance.ru
c1.onedmp.com cacheimages.com
cah.io.community
calcularpagerank.com calcularpagerank.com
candy-glam-hp.com cam-kontakte.org
carstrends2015.com camel-beach.com
cb.iphantom.com capture-room.com
cbox.ws carabela.com.do
checktext.org cartierreplica.win
chiblackhawks-jerseys.com cartierreplica.xyz
casinorewards.info
cayado.snn.gr
cctva.tv
cdnnetwok.xyz
cenokos.ru
charmstroy.info
cheapjerseysa.com
checkhit.com
checkm8.com
chimiver.info chimiver.info
chocolateslim-original.com chinacheapelitejerseys.com
christianlouboutinreplica.pw cialischstgerts.com
cialiswithoutadoctor.net cialisndbrx.com
co.lumb.co cl.s7.exct.net
cocyq.inwtrade.com clickbank.net
compliance-fred.top cocolyze.com
compliance-irvin.top coin-hive.com
compliance-walt.top com-secure.download
converse.ddsoldes.fr compliance-don.xyz
cookie-law-enforcement-aa.xyz compliance-elena.top
compliance-ivan.top
conciergegroup.org
cookielawblog.wordpress.com cookielawblog.wordpress.com
coolwallpapers4k.info coolingoods.com
copblock.org cougfan.info
counciltally.com counterbot.com
cowlmash.com covadhosting.biz
cpajunkies.com cowblog.fr
creativehutindia.com credit.co.ua
customboxes4less.com dailyfinancefix.com
daooda.com dailyrank.net
datract.com dangphoto.trade
ddsoldes.fr dealitright.click
decoratinghomes.ga deanmoore.ie
defenderxtactical.com destinationrealestate.com
dentfidemountpreach.tk diesel-parts28.ru
deplim.com digitalfaq.com
designdevise.com doeco.ru
dfiles.me dokumentalkino.net
diegolopezcastan.com domaineaneblanc.com
dieta-personalna.pl download-of-the-warez.blogspot.com
diffbot.com downloadme.life
digilander.libero.it downloadmefiranaratb1972.xpg.com.br
diminishedvalueoforegon.com doxyporno.com
directrev.com drunkenstepfather.com
discovertreasure-a.akamaihd.net duawitchrarato.tk
dms-sw.ru eandsgallery.com
dominateforex.ml earthmagic.info
download-wallpaper.net easytuningshop.ru
drugspowerstore.com ecxtracking.com
dustyorate.com ed-shop01.ru
eduserver.net eduserver.net
ekspertmed.com ee77ee.com
elektrischeziga.livejournal.com elektrozigarettekaufen1.devhub.com
elektronischezigarette2.mex.tl eloxal.ru
elidelcream.weebly.com emergencyneeds.org
eluxer.net eng-lyrics.com
eredijovon.com eropho.com
errorfixing.space este-line.com.ua
etur.ru etotupo.ru
europages.com.ru eu-cookie-law.info
ewebarticle.info eywords-monitoring-your-success.com
ezigarettekaufen.myblog.de faptitans.com
ezigarettekaufen2.mpbloggar.se fast-wordpress-start.com
ezigaretteshop2.vefblog.net fastfixing.tech
facecup.top fatfasts-4tmz.com
falllow.gq ferrotodo.com
falopicm.pw ff30236ddef1465f88547e760973d70a.com
fanoboi.com fickenbumsen.net
fettefrauen.net fickkontakte.org
filmbokep69.com filesmonster.porn
flowwwers.com filmgo.ru
forensicpsychiatry.ru financehint.eu
forex.osobye.ru fix-website-errors.com
forminecrafters.ru flex4launch.ru
fortevidyoze.net flirt4free.com
fortunejack.com floridamobilebillboards.com
forum.doctissimo.fr foxweber.com
forum20.smailik.org frbizlist.com
foto-basa.com freakycheats.com
fototravel.eu free-fb-traffic.com
frameimage.org free-floating-buttons.com
free-laptop-reward.com free-gluten.ru
free-share-buttons.com free-share-buttons.com
free-share-buttons.top freelifetimefuckbook.com
fuckbuddybestgilf.info gambarxkata.co
funnel.co.za gameprimary.com
fxgallery.com gamevalue7.weebly.com
galaxyflowers.ru gdebestkupit.ru
gay.adultgalls.com gearsadspromo.club
generalporn.org germes-trans.com
getresponse.com getmiro.com
glicol.kz getridofstretchmarks.org
globatur.ru global-ics.co.za
gne8.com gnuetella.com
gok-kasten.net goodhumor24.com
golden-praga.ru gratis-sexkontakte.com
gotcher.us gratuitbaise.com
gpms.org.my greendream.com.ua
greatgrace.ru
greatidea.marketing
greenidesign.co greenidesign.co
grizzlysgrill.com guardlink.com
grtyi.com gwhwpxbw.bloger.index.hr
gsmtlf.ru hacktougroup.ru
gta-club.ru
guiadeserraazul.com
gyffu.com
haikuware.com haikuware.com
hamilton.ca hatdc.org
hdapp1008-a.akamaihd.net hermesreplica.win
headpharmacy.com herokuapp.com
hobbyhuren-datenbank.com histock.info
hobild.net historichometeam.com
homelygarden.com hitmuzik.ru
horoshieokna.com hmywwogw.bloger.index.hr
hosting-tracker.com homedecoguide.info
hotxnights.info hotkeys.com
hoverboard360.de
hoverboard360.es hoverboard360.es
howopen.ru hoverboardforsaledirect.com
i-hobot.ru humanelydrew.com
hvd-store.com
id-forex.com id-forex.com
idegenvezeto.eu igru-xbox.net
ie.57883.net imagefinder.site
ifmo.ru imagerydatabase.com
igadgetsworld.com impresagaia.it
imagecoolpub.com infektsii.com
imguramx.pw infospot.pt
imitex-plus.ru inspiring-desperate.tk
incanto.in.ua integritylandscapeservices.com
inome.com.ua
interior-stickers.ru
ipchicken.com ipchicken.com
iplusbit.blogspot.co.za
iptool.xyz
irunfar.com
islandminingsupply.wordpress.com islandminingsupply.wordpress.com
ispac.org istizanidineanarcoticdrug.blogspot.com
itbc.kiev.ua itch.io
j33x.com iyasimasennka.com
jabimgo.pw javrip.net
janavibekken.no jerseysbizwholesalecheap.com
japan-bearings.ru johannesburgsingles.co.za
jbl-charge.info journeydownthescale.info
jewelryandfiligree.com justkillingti.me
jjbabskoe.ru kakablog.net
joy-penguin.com kamorel.com
kerch.site keyhantercume.com
keywordblocks.com
keywords-monitoring-your-success.com keywords-monitoring-your-success.com
kichenaid.ru
kino-filmi.com kino-filmi.com
kino-fun.ru kinosed.net
kinopolet.net kinozapas.com
kladrus.ru kollesa.ru
kochanelli.com konturkrasoty.ru
konteiner24.com kozhniebolezni.com
kountrylife.com
krafte.ru
kraljeva-sutjeska.com kraljeva-sutjeska.com
krasivoe-hd.com kustanay.kz
kupiproday.com.ua landreferat.ru
lacave.ntic.fr
laexotic.com
landoftracking.com
legionalpha.com legionalpha.com
lennatin.info lennatin.info
letmacworkfaster.site lerporn.info
letslowbefast.today letsrepair.in
lexiangwan.com lida-ru.com
liffa.ru lifehacĸer.com
light.ifmo.ru
likrot.com likrot.com
liky.co.ua linkredirect.biz
linkbuddies.com locksmith.jp
lirunet.ru lonerangergames.com
listiseltemournlan.gq luchshie-topcasino.ru
liumimgx.pw lunamedia.co
livefixer.com m4ever.net
livejasmin.com macnewtech.com
localflirtbuddies.com magda-gadalka.ru
loftdigital.eu manifestyourmillion.com
lolnex.us manyresultshub-a.akamaihd.net
louboutinreplica.xyz marmotstore.online
lovi-moment.com.ua master-muznachas.ru
lrac.info mcar.in.ua
luciddiagnostics.in meddesk.ru
makedo.ru mediafresh.online
maladot.com megaapteka.ru
mansparskats.com menstennisforums.com
mapquestz.us metallosajding.ru
marcoislandvacations.net michellblog.online
marcowebonyodziezowe.pl mobilemedia.md
mastershef.club moneyviking-a.akamaihd.net
mathgym.com.au moroccosurfadventures.com
maxxtor.eu mrsdalloways.com
mebel-ekb.com msk-diplomat.com
meccadumps.net
medi-fitt.hu
medicines-choice.com
megaindex.ru
megavolt.net.ua
metascephe.com
microsearch.ru
mielec.pl
milblueprint.com
mini-modus.ru
mirtorrent.net
mister-shop.com
moi-glazki.ru
monclercoats.xyz
monclerjacketsoutlet.win
morf.snn.gr
mostantikor.ru
mp3s.club
mrinsidesales.com
mtmtv.info
musflashtv.com
musicdaddy.net musicdaddy.net
mxgetcode.com musicvidz.ru
mydirtyhobby.com mystats.xyz
mydoctorok.ru na-telefon.biz
myfreemp3.eu najaden.no
myhitmp3.club naphukete.ru
mypets.by nardulan.com
narosty.com narkologiya-voronezh.ru
national-today-winning-winner.club naturalshair.site
netanalytics.xyz net-radar.com
nikitsyringedrivelg.pen.io newhairstylesformen2014.com
njkmznnb.ru nfljerseys.online
novgorod.xrus.org nicefloor.co.uk
o-o-6-o-o.com nightvision746.weebly.com
offer.wpsecurity.website njpalletremoval.com
oktube.ru nl.netlog.com
onclkads.com noclegonline.info
onefilms.net nofreezingmac.work
onlinadverts.com nosecret.com.ua
online-sbank.ru nucia.biz.ly
online-x.ru oakleyglassesonline.us
onlinewritingjobs17.blogspot.ru ochistka-stokov.ru
onlywoman.org offf.info
openmultipleurl.com omgtnc.com
ostrovtaxi.ru onescreen.cc
ownshop.cf onlineserialy.ru
pcads.ru onstrapon.purplesphere.in
perfectpracticeweb.com optikremont.ru
phormchina.com otbelivanie-zubov.com
photochki.com oxotl.com
phrcialiled.com ozas.net
pistonclasico.com pagesense.com
pk-services.ru palocco.it
pl.id-forex.com panamaforbeginners.com
placid-rounded-coast.glitch.me paradontozanet.ru
pladform.ru partner-cdn.men
partner-print.men
pateaswing.com
pay2me.pl
pcimforum.com
percin.biz.ly
perfection-pleasure.ru
photokitchendesign.com
phpdevops.com
pintattoos.com
pisanieprac.info
piski.top
pizda.lol
pl-top.pl
plastweb.ru plastweb.ru
platinumdeals.gr pochtovyi-index.ru
plohaya-kreditnaya-istoriya.ru pogruztehnik.ru
pobeiranie.pl porndairy.in
polimga.pw pornoelita.info
popup-hgd.xyz printingpeach.com
porno-home365.com prokotov.com
pornoted.com promodj.com
portside.cc proposal-engine.com
portside.xyz prostitutki-odessa.org
pricheskaonline.ru proxyradar.com
prizeestates.cricket pst2017.onlinewebshop.net
productarium.com psvita.ru
projectforte.ru puserving.com
prostitutki-astana.org putevka24.ru
qcstrtvt.bloger.index.hr qualitymarketzone.com
quick-seeker.com rankings-analytics.com
radiogambling.com rasteniya-vs-zombi.ru
rczhan.com razorweb-a.akamaihd.net
redirectingat.com rednise.com
regdefense.com registrydomainservices.com
registry-cleaner.net responsive-test.net
reklama-i-rabota.ru reward-survey.net
reklamuss.ru rialp.getenjoyment.net
replica-watch.ru rique.host.sk
riders.ro robotixix.com
ritlweb.com rocis.site
russkoe-zdorovie.ru romhacking.ru
sammlungfotos.online ruf777.com
santasgift.ml ryetaw.com
sarafangel.ru s.lollypopgaming.com
scanner-elena.top sabizonline.com
scat.porn salutmontreal.com
scopich.com sanidumps.com
search-goo.com scanner-fred.top
searchlock.com seansonline24.pl
seccioncontrabajo.com searchmywindow-a.akamaihd.net
selectads.men seksotur.ru
seocheckupx.com seoanalyses.com
seofied.com seogadget.ru
seokicks.de seoprofiler.com
seonetwizard.com serving.adbetclickin.pink
service-core.ru sex-tracker.de
sex-dating.co sexkrasivo.net
sexflirtbook.com sexkvartal.com
sexxdate.net sexphoto.site
sexy-screen-savers.com
sexystrippe.info
sexytrend.ru sexytrend.ru
shakhtar-doneck.ru
shodanhq.com shodanhq.com
shopperifymac.com siteexpress.co.il
sierraapps.com siteonomy.com
siha.de slonechka.ru
silverage.ru
sindragosa.comxa.com
skutecznetabletkinaporostwlosow.pl
sky-mine.ru
slim.sellany.ru
sms2x2.ru
snworks.com snworks.com
socialbuttons.xyz
soft1.ru soft1.ru
space2019.top softlinesolutions.me
space4updating.win sonata-arctica.wz.cz
speeddream.xyz spammen.de
sportbetfair.com sport-video-obzor.ru
sps-shop.com spravka-medosmotr.ru
srecorder.com spymac.net
strigkaomsk.ru srgwebmail.nl
sucsesofinspiration.com sstroy44.ru
sugvant.ru stal-rulon.ru
sukarame.net stardevine.com
summerlinhomes411.info statistici.ro
sundrugstore.com stretchingabuckblog.com
superiends.org stroilka.info
supernew.org stroiminsk.com
surffoundation.nl studiokamyk.com.pl
survival.betteroffers.review sugarkun.com
t3chtonic.com svetlotorg.ru
svetodiodoff.ru
tamada69.com tamada69.com
tecspb.ru tanieaukcje.com.pl
textads.men tecnoteakviareggio.it
thebitcoincode.com teenbbw.yopoint.in
thefds.net telefonsexkostenlos.tk
thegolfclub.info telegraf.by
themeforest.net tengohydar.tk
thfox.com the-trader.net
time-japan.ru the-usa-games.blogspot.com
titelhelden.eu thegameriders.com
tizanidine6mg.blogspot.com theplacetoupdating.pw
tizanidinedruginteractions.blogspot.com ticketsys.inetwd.com
tizanidinehcl4mgtabinfo.blogspot.com timhost.ru
tizanidineandcipro.blogspot.com
tizanidineandgabapentin.blogspot.com
tizanidinecapsules.blogspot.com
tizanidineformigraines.blogspot.com
tizanidineforopiatewithdrawal.blogspot.com
tizanidinehcl4mgisitanarcotic.blogspot.com
tizanidinehcl4mgtablet.blogspot.com tizanidinehcl4mgtablet.blogspot.com
tizanidineusedfor.blogspot.com tizanidinepain.blogspot.com
tkanorganizma.ru tizanidinepills.blogspot.com
tizanidinerecreationaluse.blogspot.com
tizanidinestreetprice.blogspot.com
tizanidinevssoma.blogspot.com
tizanidinevsvalium.blogspot.com
token-lab.org token-lab.org
top10-online-games.com tongkatmadura.info
topappspro.com toondinsey.com
topkarkas.com torrents-tracker.com
topshef.ru touchmods.fr
torrents.cd toy-shop.top
tour-line.net trafficjunky.net
tracklead.net
tradgardspartner.se
traffic2money.com
trafficnetzwerk.de trafficnetzwerk.de
tri-slona.org trafficstars.com
tupper-posuda.ru trainoffend.ml
tupper-shop.ru ttrraacckkrr.com
twincitiescarservice.com turbo-suslik.org
twittruth.com twodollarshows.com
tx41tclega.ru
uchebavchehii.ru uchebavchehii.ru
uptime-alpha.net ultramart.biz
url2image.com um-razum.ru
unece.org
uni.me
url-extractor.xyz
url-img.link
urll.eu
urzedowski.eu urzedowski.eu
usiad.net usiad.net
usswrite.com valkiria-tk.ru
utiblog.fr valueclick.com
v720hd.ru vapomnoncri.tk
vabasa.inwtrade.com vasileostrovsky-rayon.ru
vashsvet.com viddyoze.com
vektorpress.ru videositename.com
veles.shop videtubs.pl
ventelnos.com vipps.com.my
viandpet.com vkak.ru
videochaty.ru vladimir.zrus.org
vip.51.la voloomoney.com
vitalads.net vpnhowto.info
vitanail.ru vsesubwaysurfers.com
vod.com.ua vzlom-na-zakaz.com
vonradio.com
vsdshnik.com
wallpapers-best.com
watracker.net watracker.net
wdrake.com weboptimizes.com
webalan.ru website-speed-up.top
webcamdevochka.com website-stealer.nufaq.com
webscouter.net wejdz-tu.pl
websitevaluebot.com whereiskentoday.com
westvilletowingservices.co.za win-spy.com
whatistizanidine2mg.blogspot.com winner7777.net
whatzmyip.net wma-x.com
where-toget.com
wholesalejerseychinashop.com
wjgony.com
workle.website
worldis.me
worldtraveler.world
wowas31.ucoz.ru wowas31.ucoz.ru
www.bookmaker-bets.com wrona.it
www.jbetting.com www.timer4web.com
x-mix.info x-lime.com
xedserver.com xblog.in
xgames-04.com xn----7sbaaabaei0cc8aj5bj0bncejx.xn--p1ai
xmladserver.com xn----8sblgmbj1a1bk8l.xn----161-4vemb6cjl7anbaea3afninj.xn--p1ai
xmlinde.com xn----9sbebi2bvzr7h.xn--p1ai
xn------7cdbapdecfd4ak1bn0amjffj7afu3y.xn--p1ai xn--h1aakne2ba.xn--p1ai
xn----ctbigni3aj4h.xn--p1ai
xn--80aaaks3bbhabgbigamdr2h.xn--p1ai
xn--80aebbcbcdemfkhba4byaehoejh8dza3v.xn--p1ai
xn--l1aengat.xn--p1ai
xn--q1a.xn--b1aube0e.xn--c1acygb.xn--p1ai
xpresscare.ru xpresscare.ru
yellowfootprints.com xtubeporno.net
yoopsie.com xuki.us
youjizz.com xvideosporn.biz
youtubedownload.org yadro.ru
yugo-star.ru yes-com.com
zeroredirect.com yoshkarola.zrus.org
zigarettenonlinekaufen.tumblr.com yottos.com
zigarettenonlinekaufen1.blox.pl ytmnd.com
znaniyapolza.ru zaapplesales.blogspot.com
znaturaloriginal.com zatjmuzu.info
zynax.ua zebramart.ru
zetgie.com.pl
zigarettenonlinekaufen2.drupalgardens.com
znakomstva-piter78.ru
zophim.me
zrelaya.pw

488
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
101billion.com 123movies.love
123any.com 12masterov.com
1flag.co.za 2000k.ru
4istoshop.com 273-fz.ru
4senses.co 2nt.ru
66cpwgln.space 4replicawatch.net
70casino.online 72-news.com
a14download.com 777-club.ru
aaronabel.com 7zap.com
adssafeprotected.com 9med.net
agysya.ru aab94f698f36684c5a852a2ef272e031bb.com
alf-img.com absolute-s.ru
alice.it acc.eu.org
allcredits.su accessoires-mode.in
alyeskaresort.com admatic.com.tr
ameblo.jp adventureparkcostarica.com
ananumous.ru albamargroup.com
all4wap.ru
allwomen.info
altamayoztourism.com
amateurlivechat.org
ampower.me
android-style.com android-style.com
apartmentbay.ru
apidigidocketnet-a.akamaihd.net apidigidocketnet-a.akamaihd.net
apioutoboxnet-a.akamaihd.net apper.de
appmsr.org appiq.mobi
arabgirls.adultgalls.com asiavirtualsolutions.com
asiengirls.net auto-zapchasti.org
atlasvkusov.ru autolombard-krasnodar.ru
australia-opening-times.com avtovolop.ru
avek.ru babs.com.ua
avtolombard-krasnodar.com bagcionderlab.com
azbaseimages.net batanga.net
azinoofficial777.ru betteroffers.review
banksrf.ru bio.trade-jp.net
bannerspace.com biteg.xyz
beachpics.com bkns.vn
bear.gotcher.us blog-hits.com
begalka.xyz bloggedporn.com
bestofupload.info bocoarchives.org
bezcmexa.ru briomotor.co
bjgugu.net.ua bristolhostel.com
blogos.kz buyclomidonlaine.com
brk-rti.ru c1.onedmp.com
bvlgaribracelet.xyz candy-glam-hp.com
cakesplus.com.au canoncdriverq3.pen.io
cbb1smartlist12.click canrioloadm.gq
cleanersoft.com cardul.ru
clicksotrk.com casite-513049.cloudaccess.net
coinsspb.com cctva.tv
com-cleaner.systems cdnanalytics.xyz
cometorussia.net cheapbelstaff.online
compliance-alex.top cheapsergiorossi.online
conboy.us chlooe.com
cowblog.fr chocolateslim-original.com
demenageur.com christianlouboutinshoes.xyz
dfwu1019.info comeondog.info
divisioncore.com compliance-checker.info
doctissimo.fr compliance-fred.top
dojki-hd.com compliance-irvin.top
eandsgallery.com compliance-jack.top
eas-seo.com compliance-jessica.top
eduinfosite.com concealthyself.com
eets.net constantaservice.net
crafthubs.com
crirussian.ru
damedingel.ya.ru
darodar.com
deadlinkchecker.com
detmebel.su
dinosaurus.site
djstools.com
dlya-android.org
dokumentalkino.net
download-of-the-warez.blogspot.com
downloaddy.net
doxysexy.com
drugs-no-rx.info
e705.net
earthmagic.info
ecommerce-seo.com
ecommerce-seo.org
election.interferencer.ru
electrik-avenue.com electrik-avenue.com
elektrischezigarette1.blog.pl elektronischezigarette2.mex.tl
elektrozigaretten1.tumblr.com empis.magix.net
elstal.com.pl enbersoft.com
enskedesquashclub.se encodable.com
erotag.com eropho.com
erotische-geschichten-xxl.com erotiktreff24.info
essaypro.com f5mtrack.com
evehemming.blogspot.com.au falcon-images.blogspot.com
ezigaretteshop2.mywapblog.com faracontrol.ir
failingmarriege.blogspot.com fetroshok.ru
farmingworm.com fickenprivat.info
fertilitetsradgivningen.se fm-upgrade.ru
figuringmoneyout.com fortunejack.com
firesub.pl free-video-tool.com
footbalive.org freza-sverlo.ru
franecki.net froggytube.com
freecamdollars.com frustrated-favorable.gq
freeseedsonline.com fungamelands.com
freewhatsappload.com fxgallery.com
freewlan.info get-seo-domain.com
gambarkatabaru.com get-your-social-buttons.info
getpopunder.com getaclueamerica.com
getprismatic.com gigixo.com
grand-chlen.ru girlsatgames.ru
graphics8.info glopages.ru
gratis-sexkontakte.com greatgrace.ru
gsasearchengineranker.site gstatey.net
gta-club.ru guigyverpo.cf
hayate.biz gyffu.com
highland-homes.com hem.passagen.se
hotgirlhdwallpaper.com homedecoguide.info
hoverboard360.de hostsshop.ru
hoverboard360.se humanorightswatch.org
hunterboots.online ideoworld.org
hybrid.ru incep.imagine-ex.co
ideibiznesa2015.ru insta-add.pro
ie.57883.net invitefashion.com
ilbe.club iplogger.org
imagecoolpub.com isistaylorporn.info
incomekey.net istmira.ru
investpamm.ru james13prix.info
janettabridal.com japan-bearings.ru
jasonpartington.com justdating.online
jikoman.info kalb.ru
jrpmakati.com keyword-suggestions.com
justucalling32211123456789.tk keywords-monitoring-your-success.com
ketrzyn.pl keywordsdoctor.com
keywordsuggest.org kino2018.cc
kimcurlrvsms.com kometars.xyz
kino-rf.ru kursy-ege.ru
kinogonew.ru lerporn.info
kmd-pto.ru lizace.com
kopihijau.info lost-alpha.ru
krafte.ru lowephotos.info
ledis.top luckyshop.net.ua
legalrc.biz macresource.co.uk
lernur.net magicplayer-s.acestream.net
letsart.ru mebel-vstroika.ru
lifehacĸer.com med-zdorovie.com.ua
liran-locks.com mega-bony2017.pl
littlesexdolls.com mega-polis.biz.ua
littlesunraiser.com meltwater.com
locatellicorretor.com.br metascephe.com
lombia.co mitrasound.ru
maximpartnerspr.com modnie-futbolki.net
mazda-roadsters.com monclerboots.xyz
mebeldekor.com.ua moneytop.ru
middlerush-a.akamaihd.net montredemarque.nl
mister-shop.com mowser.com
mmoguider.ru mycaf.it
moyaterapiya.ru myfreecams.com
moz.com myhydros.org
mp3downloadhq.com
mp3ringtone.info
mrlmedia.net
msfsaar.de
musflashtv.com
mustat.com
muz-baza.net
mvpicton.co.uk
myanyone.net
myperiod.club myperiod.club
nakozhe.com netanalytics.xyz
nina.az nhl09.ru
no-rx.info nippon-bearings.ru
norththeface.store nomuos.it
o00.in oconto.ru
oil-td.ru onlinemegax.com
olgacvetmet.com openmultipleurl.com
ooo-olni.ru page2rss.com
orhonit.com pagesense.com
ourville.info paidonlinesites.com
ozas.net paparazzistudios.com.au
pacificair.com paphoselectricianandplumber.com
parlament.biz parajumpersoutlet.online
pendelprognos.se partner-cdn.men
pillscheap24h.com partnersafe.men
pk-services.ru photo.houseofgaga.ru
pkr1hand.com pic-re.blogspot.com
playfortuna-play.ru picquery.com
pornoforadult.com popserve.adscpm.net
poshiv-chehol.ru popunder.net
prefersurvey.net porno-home365.com
priscilarodrigues.com.br pornoklad.ru
prod2016.com preconnubial.usuby.site
prostitutki-almata.org printdirectforless.com
pshare.biz prizesbook.online
pulse33.ru prostitutki-kharkova.org
qexyfu.bugs3.com pssucai.info
ragecash.com pyramidlitho.webs.com
randalljhoward.com quizzitch.net
reeyanaturopathy.com rabot.host.sk
reining.lovasszovetseg.hu rabotaetvse.ru
remontvsamare.su raisedseo.com
robertefuller.com rankia.com
rating-casino2021.ru
razamicroelectronics.com
rcb101.ru
registrysweeper.com
rentaremotecomputer.com
responsive-test.net
rollercoin.com rollercoin.com
roofers.org.uk rp9.ru
ruspdd.com s.lollypopgaming.com
salutmontreal.com sajatvelemeny.com
sardinie.us seo-tools-optimizing.com
scanner-irvin.top service.adtech.us
scrapy.org sexfreepornoxxx.com
seasaltwithfood.com sexgalleries.top
secretscook.ru sexsearch.com
seocheki.net sharpchallenge.com
serpstat.com shoppytoolmac.com
servethis.com simul.co
sex-foto.pw
sexvideo-sex.com
sexy.babes.frontend-stack.top
shareyards.com
sientalyric.co
siteheart.net siteheart.net
small-games.biz snts.shell-pmr.ru
socialmediasuggest.com social-buttons.com
soft1.ru social-buttons.xyz
sosdepotdebilan.com sogimlecal.tk
spacash.com soheavyblog.com
sr-rekneskap.no staff.prairiesouth.ca
stair.registrydomainservices.com stal-rulon.ru
steebook.com stpolice.com
stockspmb.info tattoo33.ru
suture.co telegraf.by
sweepstakes.rewardit.com theendivechronicles.com
taqplayer.info thegameriders.com
taxi-v-eisk.ru tn811.us
templates.radiodigital.co topkarkas.com
tengohydar.tk trafficmp.com
tgtclick.com trafficnetzwerk.de
thefarmergame.com tuberkulezanet.ru
thegamerznetwork.com tytoona.com
tizanidinefibromyalgia.blogspot.com videochat.mx
tmearegion26.com videochaty.ru
top250movies.ru virtuagirl.com
topbestgames.com vltai.com
torrents.cd vut.com.ru
tpu.ru vykupavto-krasnodar.ru
tracksurf.daooda.com warningzscaler.heraeus.com
traxdom.ru website-speed-checker.site
trucri.me weburok.com
trustedhealthtips.com welcomeauto.ru
ttrraacckkrr.com wholesalejerseysgaa.com
tv-spoty.info wildnatureimages.com
twu.com.ua xchangetrak.com
uchetunet.su xn--l1aengat.xn--p1ai
ugguk.online xxxhdvideo.site
us-america.ru xxxnatelefon.ru
vabasa.inwtrade.com yadro.ru
vchulkah.net zaimhelp.ru
videtubs.pl zambini.ru
voloomoney.com znakomstva-piter78.ru
vsesubwaysurfers.com zocaparj.kz
wejdz-tu.pl zojirushi-products.ru
wesharepics.com zrizvtrnpale.tk
whatsappdownload10.com
wholesalecheapjerseysfree.com
win-spy.com
winx-play.ru
woodyguthrie.se
wordkeyhelper.com
www.labves.ru
xclicks.net
xn--80ahvj9e.xn--p1ai
xn--d1abj0abs9d.in.ua
youjizz.com
zeleznobeton.ru
zog.link
zooggames.com
zzbroya.com.ua

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.205 0; 10.1.1.40 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.09.2571 ### Version: V4.2021.09.2572
### Updated: Sat Sep 4 22:01:29 UTC 2021 ### Updated: Sun Sep 5 10:04:27 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.09.2571 ### Version: V4.2021.09.2572
### Updated: Sat Sep 4 22:01:29 UTC 2021 ### Updated: Sun Sep 5 10:04:27 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##