vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-03 19:20:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.05.3084 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-05-04 10:09:34 +00:00
parent 8a87bece5c
commit 6d8a4d0042
46 changed files with 94767 additions and 94767 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.05.3083 #### Version: V4.2022.05.3084
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.05.3083 #### Version: V4.2022.05.3084
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.05.3083 #### Version: V4.2022.05.3084
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

11712
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.05.3083 #### Version: V4.2022.05.3084
#### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7094](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

11716
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11716
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

11716
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11716
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 1; 10.1.0.138 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 0; 10.1.0.138 0;
127.0.0.1 0; 127.0.0.1 0;

11716
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 1; 10.1.0.138 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 0; 10.1.0.138 0;
127.0.0.1 0; 127.0.0.1 0;

11716
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 1; 10.1.0.138 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 0; 10.1.0.138 0;
127.0.0.1 0; 127.0.0.1 0;

11716
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3083 3084

11716
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

11716
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.152.141.103 0; 23.101.112.253 0;

11716
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 1; 10.1.0.138 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 0; 10.1.0.138 0;
127.0.0.1 0; 127.0.0.1 0;

11716
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 1; 10.1.0.138 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 0; 10.1.0.138 0;
127.0.0.1 0; 127.0.0.1 0;

11716
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 1; 10.1.0.138 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 0; 10.1.0.138 0;
127.0.0.1 0; 127.0.0.1 0;

11716
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.152.141.103 0; 23.101.112.253 0;

11716
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.152.141.103 0; 23.101.112.253 0;

11716
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 1; 10.1.0.138 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
G-i-g-a-b-o-t Aspiegel
GrapeFX Blackboard
GridBot CCBot
Masscan Copier
Piepmatz Pandalytics
SurveyBot Screaming
WWW-Mechanize ScreenerBot
WebBandit lwp-trivial
Zeus mediawords
cah.io.community pcBrowser

294
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
360Spider AIBOT
80legs
ADmantX
ASPSeek ASPSeek
Abonti Aboundex
Acunetix AfD-Verbotsverfahren
AhrefsBot AhrefsBot
Aipbot Alligator
Alexibot Anarchy
AllSubmitter Atomseobot
AlphaBot Attach
Anarchie
Anthill
Apexoo
Asterias
AwarioSmartBot AwarioSmartBot
BBBike
BDCbot
BDFetch BDFetch
BackDoorBot BLEXBot
BackStreet BacklinkCrawler
Barkrowler
Battleztar Bazinga Battleztar Bazinga
Bitacle Black Hole
BlackWidow Bolt
Blow BuiltBotTough
BlowFish BuiltWith
Boardreader CATExplorador
BotALot
Buddy
Bullseye
BunnySlippers
BuzzSumo
CCBot CCBot
CODE87 CSHttp
CensysInspect Chlooe
CheeseBot
Citoid
Cliqzbot
Cloud mapping
Cocolyzebot Cocolyzebot
Collector Collector
CopyRightCheck Copier
Copyscape
Craftbot
Crawling at Home Project
CrazyWebCrawler CrazyWebCrawler
Crescent
CrunchBot
Curious Curious
CyotekWebCopy Custo
DIIbot DIIbot
DSearch DSearch
DTS Agent
Demon Demon
Deusu Deusu
Digincore Devil
DigitalPebble Discobot
Dirbuster
Dispatch Dispatch
DittoSpyder
DnBCrawler-Analytics
DnyzBot
DomCopBot DomCopBot
DomainStatsBot DomainAppender
Dotbot DomainSigmaCrawler
Download Wonder
Drip Drip
EasyDL ECCP/1.0
Ebingbong EMail Wolf
Express WebPictures Ecxi
ExtractorPro ExtLinksBot
Ezooms FDM
FHscan FHscan
FlashGet FemtosearchBot
Fimap
Firefox/7.0
Flunky
G-i-g-a-b-o-t G-i-g-a-b-o-t
GetWeb GermCrawler
Go!Zilla GetRight
Getintent
Gigabot
Go-Ahead-Got-It
GoZilla
Gotit
GrabNet GrabNet
GrapeshotCrawler GridBot
HEADMasterSEO
HTMLparser HTMLparser
HTTP::Lite HTTP::Lite
HybridBot Harvest
IRLbot Heritrix
IlseBot HonoluluBot
Humanlinks
IDBot
Image Fetch Image Fetch
Image Sucker Image Sucker
IndeedBot
Indy Library Indy Library
InfoNaviRobot
Intelliseek Intelliseek
InterGET InternetSeer
Iria
Iskanie Iskanie
JOC Web Spider IstellaBot
JamesBOT JetCar
JennyBot Kenjin Spider
Jorgee Keybot Translation-Search-Machine
Kinza Kinza
Larbin LNSpiderguy
Leap LWP::Simple
LeechFTP LeechFTP
LexiBot Lftp
LibWeb LibWeb
Libwhisker
LieBaoFast LieBaoFast
Lightspeedsystems
Likse
LinkScan LinkScan
Linkbot LinkWalker
LinqiaRSSBot
LinqiaScrapeBot LinqiaScrapeBot
Lipperhey
Ltx71 Ltx71
MFC_Tear_Sample MJ12bot
MQQBrowser MQQBrowser
MSFrontPage
MTRobot MTRobot
Majestic SEO Mag-Net
Majestic-SEO Majestic-SEO
MarkWatch
Masscan Masscan
Mata Hari Mata Hari
MauiBot Mb2345Browser
MegaIndex.ru MeanPath Bot
Mediatoolkitbot
Metauri Metauri
Microsoft Data Access Microsoft Data Access
Minefield
Mister PiX Mister PiX
Moblie Safari Moblie Safari
Mojolicious
Mozlila
Msrabot Msrabot
NPbot Name Intelligence
Nameprotect
NearSite
Needle Needle
NetAnts NetAnts
NetMechanic NetLyzer
NetZIP NetZIP
Netcraft
Nettrack Nettrack
Netvibes
Nibbler Nibbler
Not NimbleCrawler
Nmap
Nuclei Nuclei
Offline Explorer Nutch
OpenVAS Octopus
Openfind OnCrawl
OutfoxBot PHPCrawl
POE-Component-Client-HTTP
PageAnalyzer PageAnalyzer
Panscient PageGrabber
Papa Foto
PeoplePal PeoplePal
Petalbot Picsearch
Picscout
PictureFinder PictureFinder
Pixray
PleaseCrawl
ProPowerBot
ProWebWalker
Probethenet
Psbot
Pu_iN Pu_iN
PxBroker PxBroker
PyCurl
Quick-Crawler Quick-Crawler
RankActiveLinkBot RankActive
RankFlex RankFlex
RankingBot
RankingBot2 RankingBot2
RebelMouse RankurBot
Re-re
ReGet
RealDownload
Reaper
Recorder
RedesScrapy
RepoMonkey
Rogerbot Rogerbot
SBIder SEOkicks-Robot
SEOkicks SEOlyticsCrawler
SEOprofiler Scanbot
SISTRIX
SMTBot
ScanAlert
Scrapy
Screaming Screaming
ScreenerBot ScreenerBot
Searchestate Seekport
SearchmetricsBot Semrush
SeoSiteCheckup SemrushBot
Shodan Shodan
Site Sucker SiteExplorer
SiteSucker SiteSnagger
SlySearch SlySearch
Snake SmartDownload
Snoopy Snoopy
SocialRankIOBot
Sociscraper
Sogou web spider Sogou web spider
Sosospider Sosospider
SpankBot SpaceBison
Spanner Spanner
Spbot Spbot
SputnikBot Spinn3r
Steeler Sqlmap
Swiftbot Surfbot
Szukacz SurveyBot
T8Abot T8Abot
Teleport Teleport
Trendictionbot Thumbor
Tracemyfile
Turingos Turingos
Turnitin Turnitin
Typhoeus
URLy Warning URLy Warning
VelenPublicWebCrawler Vacuum
VeriCiteCrawler VeriCiteCrawler
VoidEYE VoidEYE
Voil Voil
Voltron
WASALive-Bot
WBSearchBot WBSearchBot
WWW-Mechanize WWW-Mechanize
Wallpapers
Wallpapers/3.0 Wallpapers/3.0
WallpapersHD WeSEE
Web Collage
Web Enhancer Web Enhancer
Web Pix Web Fetch
Web Sauger Web Fuck
WebBandit
WebCollage WebCollage
WebCopier WebCopier
WebEnhancer WebFetch
WebFuck WebLeacher
WebPix WebPix
WebReaper
WebSauger WebSauger
WebZIP WebStripper
Webalta WebSucker
WebmasterWorldForumBot WebWhacker
Website Quester Website Quester
Who.is Bot WebsiteQuester
Whacker
Whatweb
Widow
WinHTTrack WinHTTrack
Woobot Woobot
Wotbox Wotbox
Wprecon Xaldon_WebSpider
Xenu Xenu
Zermelo ZmEu
Zitebot ZoominfoBot
arquivo-web-crawler
arquivo.pt arquivo.pt
backlink-check cah.io.community
check1.exe
clark-crawler clark-crawler
cognitiveseo dataforseo.com
demandbase-bot demandbase-bot
domainsproject.org domainsproject.org
eCatch
gopher
heritrix
instabid instabid
internetVista monitor internetVista monitor
isitwp.com
linkdexbot linkdexbot
magpie-crawler magpie-crawler
meanpathbot meanpathbot
mediawords mediawords
muhstik-scan muhstik-scan
plumanalytics netEstate NE Crawler
polaris version oBot
page scorer
probe-image-size probe-image-size
seostar ripz
serpstatbot seobility
sexsearcher
sitechecker.pro
sogouspider sogouspider
spyfu
sysscan sysscan
ubermetrics-technologies.com x22Mozilla
voyagerx.com zauba.io
webmeup-crawler
xpymep1.exe

304
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
360Spider 360Spider
404checker
80legs 80legs
ALittle Client ADmantX
Aboundexbot ASPSeek
AfD-Verbotsverfahren Abonti
AhrefsBot Aboundex
AiHitBot AiHitBot
Anarchie AllSubmitter
AlphaBot
Anarchy Anarchy
Anthill Ankit
Apexoo
Atomseobot
Attach Attach
AwarioRssBot AwarioSmartBot
BBBike
BDCbot BDCbot
BDFetch BDFetch
BackDoorBot BetaBot
Backlink-Ceck Black Hole
Bandit BlackWidow
Battleztar Bazinga Blow
BlowFish BotALot
Boardreader
Buck Buck
Buddy Buddy
BuiltBotTough BuiltWith
Bullseye
BunnySlippers BunnySlippers
BuzzSumo
CCBot
CODE87 CODE87
CSHttp
Calculon
Cegbfeieh Cegbfeieh
CensysInspect
CheeseBot CheeseBot
Chlooe CherryPicker
Citoid Claritybot
Copier Cliqzbot
Copyscape Cloud mapping
Cocolyzebot
Collector
Cosmos Cosmos
CrunchBot Craftbot
Crawling at Home Project
Crescent
Curious Curious
Custo CyotekWebCopy
DTS Agent
DataCha0s DataCha0s
DatabaseDriverMysqli Deusu
Demon Digincore
DigitalPebble DigitalPebble
DnyzBot Discobot
DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
EMail Wolf Domains Project
Ecxi Dotbot
Evil Drip
Extractor
EyeNetIE
FHscan FHscan
FemtosearchBot
Fimap Fimap
Flunky FlashGet
FrontPage Freeuploader
Fuzz Fuzz
FyberSpider FyberSpider
G-i-g-a-b-o-t Fyrebot
Genieo GT::WWW
GetWeb GetWeb
Gigabot Go!Zilla
Go-Ahead-Got-It
GoZilla
GrabNet GrabNet
GrapeFX Grabber
GrapeshotCrawler
GridBot GridBot
HMView HEADMasterSEO
HTTP::Lite
Haansoft Haansoft
HonoluluBot Havij
Humanlinks IRLbot
IDBTE4M
IlseBot IlseBot
InfoTekies Image Sucker
Indy Library
InfoNaviRobot
Intelliseek Intelliseek
InterGET Iria
Internet Ninja
InternetSeer
Iskanie
JennyBot
JetCar
JikeSpider JikeSpider
Joomla Joomla
Jorgee Jorgee
JustView JustView
Jyxobot Jyxobot
Kinza Keybot Translation-Search-Machine
Kozmosbot Kozmosbot
Lanshanbot LNSpiderguy
Leap
LexiBot LexiBot
Lftp
Libwhisker
LieBaoFast LieBaoFast
Lightspeedsystems
LinkWalker LinkWalker
LinkextractorPro LinkextractorPro
LinkpadBot LinkpadBot
LinksManager LinksManager
LinqiaMetadataDownloaderBot Lipperhey Spider
MIIxpc Litemage_walker
MJ12bot Ltx71
MQQBrowser MQQBrowser
Mag-Net MSFrontPage
Majestic12 MSIECrawler
MarkWatch Magnet
Mass Downloader
Masscan Masscan
MeanPath Bot Mata Hari
Mb2345Browser
Meanpathbot
MegaIndex.ru MegaIndex.ru
MicroMessenger Mojeek
Mojolicious Mojolicious
MolokaiBot
Morfeus Fucking Scanner Morfeus Fucking Scanner
Needle Musobot
NetAnts NICErsPRO
NetLyzer Nessus
Net Vampire
NetMechanic NetMechanic
Netvibes NetZIP
NextGenSearchBot NextGenSearchBot
NimbleCrawler Nibbler
Niki-bot
Nimbostratus Nimbostratus
Ninja
Not
Offline Navigator Offline Navigator
OnCrawl OpenVAS
Openfind Openvas
PECL::HTTP OutfoxBot
PHPCrawl PageAnalyzer
POE-Component-Client-HTTP
Page Analyzer
PageThing.com PageThing.com
Panscient
Papa Foto Papa Foto
Pavuk Pavuk
PeoplePal PeoplePal
Petalbot Pi-Monster
Picscout
Picsearch
PictureFinder PictureFinder
Piepmatz Pimonster
Pockey ProPowerBot
Psbot
Pu_iN Pu_iN
Pump PxBroker
PyCurl PyCurl
Quick-Crawler
RSSingBot RSSingBot
RankActive RankActive
RankActiveLinkBot RankFlex
RankingBot RankingBot
RankingBot2 RankingBot2
RankurBot Re-re
Recorder Recorder
RedesScrapy RedesScrapy
SEOkicks-Robot SBIder
SEOkicks
SEOlyticsCrawler
SEOprofiler
SEOstats
SMTBot
ScanAlert
Scanbot
Scrapy Scrapy
Semrush ScrepyBot
Searchestate
SemanticJuice
SentiBot
Seomoz Seomoz
SiteCheckerBotCrawler SiteCheckerBotCrawler
SiteLockSpider
SiteSnagger
SiteSucker SiteSucker
Siteimprove SmartDownload
Sitevigil Snapbot
Snake Snoopy
Sociscraper SpankBot
Sosospider
Spanner Spanner
Spinn3r Spbot
SputnikBot Sqlmap
Sqworm Sqlworm
Steeler
Sucker Sucker
SuperBot Sucuri
Surfbot
SurveyBot
Swiftbot Swiftbot
Szukacz T0PHackTeam
TeleportPro Telesphoreo
Telesphorep TightTwatBot
The Intraformant Titan
TheNomad Trendictionbot
Toweyabot
Tracemyfile
Trendiction
Turingos Turingos
Turnitin TwengaBot
Twice
Typhoeus
Vagabondo Vagabondo
VeriCiteCrawler VeriCiteCrawler
VoidEYE Virusdie
WPScan WWW-Mechanize
WWWOFFLE
Wallpapers/3.0 Wallpapers/3.0
Web Auto WallpapersHD
WeSEE
Web Collage Web Collage
Web Pix Web Fuck
Web Sucker Web Sucker
WebEnhancer WebBandit
WebFetch
WebFuck WebFuck
WebImageCollector
WebLeacher WebLeacher
WebSauger WebWhacker
WebStripper
WebSucker
WebZIP
Webalta
Website Quester Website Quester
Webster WebsiteQuester
Whacker Whack
Whatweb Who.is Bot
Widow Widow
WiseGuys Robot WiseGuys Robot
Wprecon Woobot
Xaldon_WebSpider Wotbox
YoudaoBot Xaldon WebSpider
Xenu
Zauba Zauba
Zermelo Zermelo
Zeus
Zitebot Zitebot
ZumBot ZoomBot
adscanner adscanner
autoemailspider arquivo.pt
backlink-check
cah.io.community cah.io.community
check1.exe clark-crawler
coccocbot
cognitiveseo cognitiveseo
crawl.sogou.com com.plumanalytics
dataforseo.com crawler.feedback
evc-batch demandbase-bot
gopher gopher
ips-agent
isitwp.com isitwp.com
iubenda-radar
linkdexbot linkdexbot
lwp-request
lwp-trivial lwp-trivial
mediawords magpie-crawler
muhstik-scan meanpathbot
netEstate NE Crawler pcBrowser
page scorer
plumanalytics
polaris version
probe-image-size probe-image-size
ripz ripz
satoristudio.net s1z.ru
scan.lol seobility
seocompany.store
seoscanners seoscanners
serpstatbot
sexsearcher sexsearcher
spyfu sogouspider
tAkeOut sp_auditbot
trendiction.com sysscan
trendiction.de
voyagerx.com voyagerx.com
webgains-bot
webpros.com
webprosbot webprosbot
x22Mozilla x22Mozilla
xpymep1.exe zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
1001watch.com.ua artel116.ru
birzha-truda.eu keywordsuggests.com
delayreferat.ru lankarns.com
gcup.ru lombia.com
istizanidineanarcoticdrug.blogspot.com mygameplus.ru
mzdish.site produkto.net
repeatlogo.co.uk prohoster.info
seo-smm.kz sadaholding.com
youandcredit.ru trichizobswiv.agddns.net
zolotoy-lis.ru za-music.mymobiplanet.com

922
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
01apple.com 0lovespells0.blogspot.com
08800.top
12bet.com
1flag.co.za
1kinobig.ru 1kinobig.ru
1millionusd.xyz 24videos.tv
2020iscoming.info 2girls1cup-free.com
24-ak.ru 4pp13.com
7makemoneyonline.com 4senses.co
7xc4n.com 8gold.com
991mostfm.co.id 991mostfm.co.id
a.frcls.fr aa08daf7e13b6345e09e92f771507fa5f4.com
aa9bd78f328a6a41279d0fad0a88df1901.com abcguru.xyz
abcdeg.xyz academia-nsk.org
adhome.biz ad-words.ru
adsland.men adclickthru.net
adultfriendfinder.com adtology.com
adviceforum.info adultwebhosting.info
aero2.ru advertisingtag.net
afslankpillen2017nl.eu advocatemsk.ru
agadelha.com.br advokat-grodno.by
ahrntal.verymes.xyz advokateg.ru
aktobe.xkaz.org aimayangzhi.com
akusherok.ru alfa9.com
akvamaster.dp.ua all4invest.info
albumsuper.info allboard.xobor.de
allblogroll.com allkrim.com
allornamenti.com
alloysteel.ru alloysteel.ru
allproblog.com allwidewallpapers.com
allstatesugarbowl.org aloofly.com
alphahoverboards.com amospalla.es
amazingninja.com animali.deagostinipassion.it
ameblo.top
amehdaily.com
analytics-ads.xyz
anonymousfox.co anonymousfox.co
api.stathat.com apibatbrowsecom-a.akamaihd.net
apiappenableinfo-a.akamaihd.net app5.letmacworkfaster.world
apibetweenlinesn-a.akamaihd.net apper.de
apidiamondatanet-a.akamaihd.net appfixing.space
apimountainbikei-a.akamaihd.net appiq.mobi
apispringsmartne-a.akamaihd.net applicationg29.com
appleid-verification.com applyneedy.xyz
asdfz.pro appmsr.org
asrvvv-a.akamaihd.net appsaurus.com
atleticpharm.org apxeo.info
atofilms.com arabgirls.adultgalls.com
audiobangout.com architecturebest.com
auto.rusvile.lt arkartex.ru
autolombard-krasnodar.ru artpicso.com
autorn.ru asdfg.pro
avitocars.ru asia-forum.ru
avkzarabotok.info atley.eu.pn
atovh.local-finders.com
automatic-seo.com
avramstroy.ru avramstroy.ru
avtoarenda.by ayerbo.xhost.ro
ayakino.net azartmsl.com
azte.ch azte.ch
bangla-chat-uk-paralud.ga backuperwebcam.weebly.com
bankofthewext.com baksman.com
bannerbank.ru balois.worldbreak.com
barnfurnituremart.com battleforupdating.site
bbtec.net beauty-things.com
beachtoday.ru berdasovivan.ru
beatifullandscape.co best-coupon-offer.com
bestimagecoollibrary.com best-seo-solution.com
bestmobilityscooterstoday.com bestdraws.com
bestwaystogetpaid.us bestmouthwash.club
bestofpicture.com
bestwebsitesawards.com bestwebsitesawards.com
bestwrinklecreamnow.com betwinservice.com
betteroffers.review bidr.trellian.com
beyan.host.sk big-cash.net
bget.ru bigpictures.club
bigcareer.info bimatoprost-careprost.com
bigcities.org binomo.com
bimmerpost.com bitporno.sx
bin-brokers.com biznesluxe.ru
bitcoinpile.com bloggedporn.com
bjanshee.ru boanonihaca.tk
blogig.org board.f00d.de
bmsco.co bonanza-fortune.men
bnt-team.com bonkers.name
bombla.org bookmark4you.com.biz
bongiornos.info boole.onlinewebshop.net
brainxs.us braintobe.win
brandedleadgeneration.com briomotor.co
brateg.xyz buntube.net
brothers-smaller.ru businesn.men
bryansk.zrus.org business-suggested.tk
buckspar.gq bwlx.prepedu.cn
buqyxa.rincian.info bzero1jewelry.net
burger-tycoon.com cand.jp
c-gainsbourg.com candcstuccoandstone.com
calstaterealty.us
capture-room.com
cardiosport.com.ua cardiosport.com.ua
cardsharp1.ru
cartierbraceletsreplica.pw
cartierlovebracelet.xyz
ccbill.com ccbill.com
celebsopics.com centraletermice.us
chastnoeporno.com cercacamion.it
chatroulette.life cerev.info
cheap-trusted-backlinks.com cezartabac.ro
checkhit.com cheap-pills-norx.com
chocolateslim-en-espana.com christianlouboutinshoes.xyz
christianlouboutinreplica.win ciceros.co
cialisovercounteratwalmartusa.com citysecurity.nu
cienum.fr cjs.com.ru
cinemaenergy-hd.ru clickbank.net
civilwartheater.com clickcash.com
cleansearch.net
clodo.ru
club-musics.ru
cntravelre.com cntravelre.com
compliance-don.xyz coderstate.com
compliance-george.xyz compliance-alex.xyz
cookielawblog.wordpress.com comsysnet.com
coolgamechannel.com conocer-sanabria.com
customcollegeessays.net cookie-law-enforcement-aa.xyz
dairyindia.in cpajunkies.com
darkbooks.org crazyprotocol.com
demon-tweeks.com creditwell.ru
detalizaciya-tut.biz crystalslot.com
detki-opt.ru culturevie.info
detoxmed24.ru cypernhuset.se
detsky-nabytek.info daisye.top
dev78.dioniqlabb.se dame-ns.kz
dichvuvesinhhanoi.com danceuniverse.ru
dieswaene.com danschawbel.com
dipstar.org data-mining.tk
davebestdeals.com
dbmkfhqk.bloger.index.hr
dealitright.click
deutschland123.de
dfiles.me
diarioaconcagua.com
divci-hry.info
divku.ru
dneprsvet.com.ua dneprsvet.com.ua
doctormakarova.ru docsportal.net
doktoronline.no domaincdn.xyz
dorratex.tn domaincheck.io
downloadeer.net dscaas.website
doxysexy.com dvd-famille.com
dynainbox.com dyshagi.ru
e-buyeasy.com e-avon.ru
eas-seo.com earl-brown.info
edge.sharethis.com easyshoppermac.com
electrik-avenue.com ee77ee.com
elektroniksigaraankara.info election.interferencer.ru
elektrozigarette2.webs.com elektrischezigarette1.onsugar.com
elektrozigaretten1.tumblr.com elektrischezigarette2.devhub.com
elektrozigarette1.dreamwidth.org
elidelcream.weebly.com
elvel.com.ua elvel.com.ua
energydiet24.ru emergencyneeds.org
englishgamer.com energydiet-info.ru
eonpal.com erot.co
erotik-kostenlos.net european-torches.ru
eshop4u.jp evehemming.blogspot.com.au
essay-writing.work ewebarticle.info
etotupo.ru exhibitionplus.eu
euronis-free.com exportshop.us
europages.com.ru eyes-on-you.ga
excaliburfilms.com eywords-monitoring-your-success.com
exchanges-bet.com ezigarettenshop1.yolasite.com
expolicenciaslatam.co ezofest.sk
ezaz.info face.hostingx.eu
f0815.de family1st.ca
f1nder.org favorcosmetics.com
faithe.top feel-planet.com
fapgon.com fillmewithhappiness.com
fatmaelgarny.com film-one.ru
feargames.ru filmfanatic.com
felizporno.com financetip.eu
fenoyl.batcave.net food.dtu.dk
fertilitetsradgivningen.se foxinsocks.ru
fetishinside.com franecki.net
filefilter.weebly.com franklinfire.co
filerockstar298.weebly.com free-floating-buttons.com
fivedwld.cf
fix-website-errors.com
flash4fun.com
forex-indextop20.ru
forsex.info
free-laptop-reward.com free-laptop-reward.com
freshsuperbloop.com free-stock-illustration.com
funtoonez.com freegamesplay.online
fxgallery.com freemags.cc
gameonasia.com freetangodownload.com
gamevalue7.weebly.com ftns.ru
getmyads24.com fxund.us
greatdealshop.com gael-s.ru
grizzlysgrill.com galleryawesome.com
gsmlab.pl gamesprite.me
gtopstats.com generalporn.org
hahashka.ru gepezz.info
haikuware.com gidonline.one
hd720kino.ru gifspics.com
hentai-manga.porn goodprotein.ru
herehloadibs.cf googlepositions.com
hermesbelts.xyz googleseo.com.tr
hermesreplica.pw googlsucks.com
hidefiles.org goosefishpost.bid
himazin.info gotcher.us
histats.com greendream.com.ua
hmywwogw.bloger.index.hr growboxbl.ru
hobild.net gsasearchengineranker.pw
homedo.fabpage.com gsmtlf.ru
homegardenlova.com guildebzh.info
hoste.octopis.com hacktougroup.ru
hostsshop.ru handsandlegs.ru
hotchatdate.com hannasolution.ru
hoztorg-opt.ru hdxnxxtube.mobi
hustoon.over-blog.com heartofbeijing.blogspot.com
hyiphunter.org hermesreplica.win
hystersister.com histock.info
igrovyeavtomaty777.ru home-task.com
ilo134uloh.com hornymatches.com
imadedinner.net hotblog.top
hotelcrocenzi.sm
hplaserjetpdriver8y.pen.io
hptwaakw.blog.fc2.com
hspline.com
htmlcorner.com
huhn.altervista.org
iamsport.org
ico.re
ictizanidinehcl4mg.blogspot.com
ifmo.ru
igithab.com
importchinacoach-teach.com
in-tandem.co
inclk.com
infobanks.ru infobanks.ru
infowarcraft.ru intellego.info
intellektmedia.at investzalog.ru
interior-stickers.ru iplogger.org
invivo.hu irkutsk.zrus.org
iomoio.net iron-age.info
irkutsk.online-podarki.com isitwp.com
iskalko.ru islamtoday.co.za
itrevolution.cf istizanidineanarcoticdrug.blogspot.com
ivanovo.zrus.org javitas.info
iwantedmoney.com jeremyeaton.co
iwanttodeliver.com johannesburgsingles.co.za
janavibekken.no jolic2.com
japfm.com kaliningrad.zrus.org
javrip.net keenoutlet.online
job.icivil.ir keywordhut.com
joessmogtestonly.com keywordteam.net
justprofit.xyz kino-rf.ru
kareliatobacco.ru konica.kz
katjimej.blog.fc2.com kopihijau.info
kazrent.com koral.se
kemerovo.zrus.org kostenlos-sexvideos.com
keywordglobal.co.za kskjco.club
kinky-fetishes.com ku6.com
kinostorm.net kutikomi.net
kiskinhouse.com kwzf.net
klikbonus.com lacapilla.info
kliksaya.com
koleso24.com.ua
krynica.info
kumuk.info
kvartir-remont.biz
labvis.host.sk
ladov.ru
lafourmiliaire.com
lakomka.com.ua lakomka.com.ua
lanasshop.ru
lancheck.net lancheck.net
lankarns.com landscapebackgrounds.blogspot.com
law-check-eight.xyz law-check-eight.xyz
leadn.pl leadn.pl
letmacwork.world letmacwork.world
li-er.ru letolove.ru
linkdebrideur.xyz lifeinsurancekb.com
liver-chintai.org live-cam6.info
liverpool.gsofootball.com locustdesign.co
liveshoppersmac.com longlifelomilomi.info
loadingpages.me lostaruban.com
lodki-pvh.dp.ua lovasszovetseg.hu
lolnex.us
lunamedia.co
lunchrock.co
m0r0zk0-krava.ru
mackeeper-land-672695126.us-east-1.elb.amazonaws.com
magda-gadalka.ru
magiadinamibia.blogspot.com magiadinamibia.blogspot.com
makedo.ru makedo.ru
makemoneyonline.com makis.nu
marblestyle.ru mathgym.com.au
med-zdorovie.com.ua mazda-roadsters.com
mega-bony-2017.pl me-ke.com
mezaruk.info meds-online24.com
megaindex.ru
mhi-systems.ru mhi-systems.ru
middlerush-a.akamaihd.net michaelkorsoutlet.store
mir-limuzinov.ru mobilierland.com
mirzonru.net mokrayakiska.com
mncrftpcs.com monclervests.xyz
mobicover.com.ua morocco-nomad-excursions.com
moncleroutletonline.xyz mtmtv.info
moneytop.ru mug-na-chas-moscow.ru
morepoweronmac.trade muizre.ru
moscow-clining.ru musicktab.com
most.gov.iq my-bc.ru
moz.com my-cash-bot.co
myblogregistercm.tk myxdate.info
mydownlodablefiles.com mzdish.site
mypets.by narkologiya-belgorod.ru
narkologiya-penza.ru nash-krym.info
naturalbreakthroughsresearch.com nastroyke.net
naturalpharm.com.ua netanalytics.xyz
nero-us.com netcheckcdn.xyz
nerudlogistik.ru new7ob.com
nextbackgroundcheck.gq
nextconseil.com nextconseil.com
nl.netlog.com nikitabuch.com
notaria-desalas.com nlfjjunb5.ru
nudo.ca nomuos.it
nvformula.ru nonews.co
november-lax.com
nsatc.net
nvssf.com
obiavo.kz
obuv-kupit.ru obuv-kupit.ru
okayimage.com odoratus.net
omoikiri-japan.ru oksrv.com
one-gear.com olvanto.ru
onemactrckr.com onthemarch.co
online-zaymy.ru ooo-olni.ru
onlineku.com open-odyssey.org
optikremont.ru openlibrary.org
ostrovtaxi.ru openmultipleurl.com
pandroid.co ops.picscout.com
panicatack.com ourtherapy.ru
paparazzistudios.com.au ozas.net
partners-ship.pro parajumpersoutlet.online
pensplan.com park.above.com
picturesify.com pcgroup.com.uy
pills24h.com perfectplanned.com
pinup-casino1.ru photo.houseofgaga.ru
piski.top photokitchendesign.com
pivka.xyz php-market.ru
phpdevops.com
pic-re.blogspot.com
piulatte.cz
piuminiita.com
pixgood.com pixgood.com
platezhka.net pokemon-go-play.online
platinumdeals.gr popunder.ru
polimga.pw porndl.org
popugauka.ru pornhub-ru.com
popup-hgd.xyz porno-home365.com
pornodojd.ru
pornogad.com pornogad.com
pornoplen.com pornoinn.com
pornosok.ru pornosmola.info
posible.net privetsochi.ru
pospr.waw.pl
pr0fit-b0x.com
pretty-mart.com
princevc.com
prizeestates.cricket prizeestates.cricket
professionaldieselcare.com pron.pro
proftests.net proprostatit.com
progress-upakovka.ru prostitutki-almata.org
promodj.com queerspace.com
propranolol40mg.blogspot.com
prostitutki-kiev.org
prostitutki-novgoroda.org
proxyradar.com
ptr.ruvds.com
quelle.ru quelle.ru
quizzitch.net quit-smoking.ga
rabot.host.sk ranking2017.ga
rapidgator-porn.ga ranksonic.info
realnye-otzyvy.info razvratnoe.org
rebrand.ly recipedays.com
redirlock.com redbottomheels.xyz
relax.ru rednise.com
renewablewealth.com reklamuss.ru
rentaremotecomputer.com
repeatlogo.co.uk
replica-watch.ru
resant.ru resant.ru
rethinkwasteni.info responsinator.com
rubanners.com restaurantlescampi.com
ric.info
ricorsogiustizia.org
rocis.site
rock-cafe.info
ros-ctm.ru
rosbalt.com.ua
royalvegascasino.com
ru-dety.ru
ruf777.com
rus-teh.narod.ru rus-teh.narod.ru
rvi.biz sa-live.com
s-iwantyou.com sarm.tk
sakhboard.ru scanner-george.top
sammlungfotos.online scanner-jane.top
sanjosestartups.com scanner-jess.top
sape.top scanner-jessica.top
sarahmilne.top scrapy.org
sbdl.no seadragonherbery.com
sbprabooks.com search.alot.com
scalerite.co.za searchimage.co
scanner-julianna.top searchimpression.com
scripted.com searchwebknow-a.akamaihd.net
scrnet.biz.ua seemoreresultshu-a.akamaihd.net
search-goo.com serptehnika.ru
searchkut.com service.adtech.us
security60-e.com sevendays.com.ua
seo-2-0.com sevenstars7.com
seocheckupx.com sexpartygirls.net
seoexperimenty.ru sextracker.de
seoholding.com sexyteens.hol.es
sexpornotales.net
sextracker.com
shiksabd.com shiksabd.com
shitmovs.com shopfishing.com.ua
shop.acim.org shopvilleroyboch.com.ua
shoppingmiracles.co.uk shtora66.ru
shoppytoolmac.com sierraapps.com
shopwme.ru silvercash.com
si-unique.com simon3.ru
signoredom.com sirpornogratis.xxx
signx.info sitiz.club
similardeals.net sitopreferito.it
site.ru sklad-24.ru
siteheart.net sladkoevideo.com
sitevaluation.com slowmac.tech
slujbauborki.ru smailik.org
smart-balancewheel.com sms2x2.ru
snegozaderzhatel.ru snow.nvr163.com
sosdepotdebilan.com soft-terminal.ru
sotkal.lark.ru sophang8.com
spacash.com sortthemesitesby.com
spasswelt.net sovetogorod.ru
special-porn.com space4updating.win
sports-supplements.us spaceshipad.com
sribno.net standardchartered-forex.com
stanthonyscatholicchurch.org staynplay.net
strangeduckfilms.com stop-gepatit.te.ua
stroilka.info strfls.com
studentguide.ru stroiminsk.com
stuff-about-money.com sugarlyflex.pw
styro.ru surfbuyermac.com
suggest-keywords.com svensk-poesi.com
sunflowerdrawingpaintings.blogspot.com swinger-mobil.net
sweepstakes.rewardit.com tanieaukcje.com.pl
swsociety.se
t3chtonic.com
taqywu51.soup.io taqywu51.soup.io
taranerymagesswa.blogspot.com
taraz.xkaz.org taraz.xkaz.org
templates.radiodigital.co tastyfoodideas.com
testbotprocessor44.com td-33.ru
thebestphotos.eu teesdaleflyballclub.co.uk
thebluffs.com thegalerie.eu
thenews-today.info theheroes.ru
tilido.com theprofitsmaker.net
time-japan.ru thewebsitetemplate.info
tizanidinefibromyalgia.blogspot.com timhost.ru
tizanidinehclsideeffects.blogspot.com tizanidineorflexeril.blogspot.com
tizanidinevscyclobenzaprine.blogspot.com tizanidinesleepaid.blogspot.com
topcar-krasnodar.ru tongkatmadura.info
topbestgames.com
topquality.cf
toproadrunner5.info
topsy.com topsy.com
torrent-to-magnet.com track112.site
totu.info trackzapper.com
tracklead.net triumf-realty.ru
tradgardspartner.se
trafficstars.com
traffmonster.info
truebeauty.cc truebeauty.cc
trustedmaccleaner.com tt-ipd.info
turn-up-life.life tube8.com
tvand.ru tubeline.biz
twiclub.in tuckermktg.com
twittruth.com twelvevisionspartyofcolorado.com
ua-company.ru twittrading.com
uhodzalijami.ru ukrtextbook.com
ukkala.xyz ukrtvir.com.ua
ukrtvory.in.ua
ultimatesetnewfreeallsoftupgradesystems.pw
umg-stroy.ru
unacittaconte.org unacittaconte.org
unimodemhalfduplefw.pen.io uptime-as.net
unitygame3d.com urengoy.pro
unmaroll.ya.ru url-img.link
upproar.com urzedowski.eu
us-america.ru vacances-voyages.info
valsalud.com vbtracker.net
videochat.mx video-camer.com
videochat.ph video-chat.love
vip-parfumeria.ru videochat.bz
videogamesecrets.com
videotuber.ru
vigrx-original.ru
viktoria-center.ru
viteonlusarezzo.it
vkonche.com
vladimir.xrus.org
vodkoved.ru vodkoved.ru
vonradio.com voloomoney.com
vostoktrade.info voprosotvet24.ru
vroze.com vsdshnik.com
vseuznaem.com vykupavto-krasnodar.ru
vuclip.com wapsite.me
vvon.co.uk warezaccess.com
vzglyadriv.kg watchmyfb.pl
vzlomsn.org wdrake.com
wallpaperaccess.com website-speed-checker.site
wardreapptokone.tk website-speed-up.top
webalan.ru websiteexplorer.info
weblibrary.win whiteproduct.com
weburok.com wholesalejerseychinashop.com
wesharepics.info wingsofrefuge.net
westermarkanjou.se wnhjavlhezp.gq
wetgames.ru womama.ru
white-truck.life wordkeyhelper.com
wholesalejerseys-cheapest.com worldhistory.biz
wildnatureimages.com worldis.me
wonderfulflowers.biz wpthemedetector.co.uk
writersgroup580.web.fc2.com xblognetwork.com
writingservices17.blogspot.ru
wsgames.ru
wygraj-skiny.win
xgftnlrt.bloger.index.hr
xjlottery.com
xmnb.net
xn----7sbaaabaei0cc8aj5bj0bncejx.xn--p1ai
xn----8sblgmbj1a1bk8l.xn----161-4vemb6cjl7anbaea3afninj.xn--p1ai xn----8sblgmbj1a1bk8l.xn----161-4vemb6cjl7anbaea3afninj.xn--p1ai
xn--90acenikpebbdd4f6d.xn--p1ai xn----9sbubg3ambdfl1j.xn--p1ai
xn--b1ag5cfn.xn--p1ai xn--80ajjbdhgmudixfjc8c5a9df8b.xn--p1ai
xn--d1abj0abs9d.in.ua xn--d1aifoe0a9a.top
xn--e1afanlbnfckd7c3d.xn--p1ai xn--e1aggki3c.xn--80adxhks
xxx-treker.ru xnxxandxvideos.com
xxxguitars.com xxxmania.top
yaderenergy.ru xxxsiterips.xyz
yhit.press yellowfootprints.com
yoluxuryevents.com youjizz.com
yoursite.com youporn-ru.com
youthreaders.com youresponsive.com
youtube-downloader.savetubevideo.com yousense.info
za-music.mymobiplanet.com zacreditom.ru
zaimhelp.ru
zarenica.net zarenica.net
zoogdinsney.com zeikopay.com
zvooq.eu zocaparj.kz
zoogdiesney.com
zrelaya.pw
zrizvtrnpale.tk

470
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
08800.top 24videos.tv
5u.com 2itech.ru
9med.net 4inn.ru
a1security.com.ua 51unlim.ru
aa625d84f1587749c1ab011d6f269f7d64.com 5ws.dating-app.ru
abtasty.com adelly.bg
acheterviagrafr24.com
adf.ly
admatic.com.tr admatic.com.tr
affrh2015.com adpremium.org
ahrefs.com advokat-grodno.by
agysya.ru
akuhni.by akuhni.by
alert-jdh.xyz alert.scansafe.net
alldezservice.kz aliexpresscashback.club
allforminecraft.ru alloysteel.ru
allknow.info alphacarolinas.com
allwidewallpapers.com amoi.tn
antons-transporte.de apimossnetinfo-a.akamaihd.net
ap.senai.br apioutoboxnet-a.akamaihd.net
apccargo.com apiqualitinknet-a.akamaihd.net
artefakct.com atlant-auto.info
atelielembrancaqueencanta.com.br autoloans.com
automobile-spec.com aviapanda.ru
bankcrediti.ru aviav.co
bestofferswalkmydogouteveryday.gq avon-novosib.ru
bget.ru awaybird.ru
bizlist.com.de axbocz.net
blackcurranthumidifiers.site azartmix.com
blackplanet.com baladur.ru
blog-hits.com bayareaaeroclub.org
blpmovies.com bcmp.org
brainsandeggs.blogspot.com bimmerpost.com
braip.com.br biznesschat.net
brillianty.info bloggerads.net
bristolhotel.com bmwhighperformers.com
cakesplus.com.au brandov.ru
breastaugmentation.co.za
buchananshardware.com
buntube.net
businesn.men
buy-forum.ru
calcularpagerank.com
callawaygolfstore.online
cand.jp
candycrushshop.com candycrushshop.com
cartierlovestore.xyz carmuffler.net
cartierreplica.xyz cartechnic.ru
castingbank.ru cartierlovebracelet.xyz
cb.iphantom.com cartierring.xyz
centrdebut.ru cellfun.mobi
cheapmoncler.win cezartabac.ro
christianlouboutinshoes.xyz cheatcode-lita12.rhcloud.com
clean-start.net chlooe.com
compliance-don.top ci.ua
compliance-elena.top citizenclsdriveri7.pen.io
conciergegroup.org clickbanksites.info
cobaltpro.ru
cocyq.inwtrade.com
compliance-don.xyz
cookie-law-enforcement-ee.xyz cookie-law-enforcement-ee.xyz
creams.makeforum.eu copro.pw
daptravel.com cosmetic.donna7753191.ru
cowlmash.com
creativehutindia.com
crest3d.ru
cscwtalkto.site
csgo4.win
dataloading.net
dawlenie.com dawlenie.com
descargar-musica-gratis.net
dev78.dioniqlabb.se dev78.dioniqlabb.se
diminishedvalueoforegon.com doreenblog.online
dkvorota.ru election.interferencer.ru
domainanalyzing.xyz elektrozigaretten1.postbit.com
downloaddy.net elvenmachine.com
drillsaw.ru executehosting.com
drugspowerstore.com f012.de
dvr.biz.ua family1st.ca
dvrlists.com fbdownloader.com
ec-file.info foto-weinberger.at
ekspresihati.info frbizlist.com
ekto.ee free-today.com
elektrischezigarette2.devhub.com freemaintenancesysforpcandmac.top
elektrischezigaretten2.beeplog.com galaxyflowers.ru
elexies.info gallery.rennlist.com
embedle.com gamerscorps.com
evogarage.com goldenggames.com
ezigarettenkaufen2.dreamwidth.org gsasearchengineranker.pw
face.hostingx.eu happy.new.yeartwit.com
fakehandbags.xyz
ferieboligkbh.dk
filmi-v.online
findanysex.com
finuse.com
flooringinstallation-edmonton.com
forum-engineering.ru
forum.tvmir.org
foxweber.com
free-social-buttons.com
freesitetest.com
froggytube.com
fx-brokers-review.com
gazobeton-p.com.ua
gcup.ru
get-free-traffic-now.com
girls-ufa.ru
global-ics.co.za
globatur.ru
glopages.ru
googst2.ru
guardlink.org
hatdc.org
hazardky.net
hdhc.ru
hdpixent.com
healthcarestore.info
helvetia.com.ua
highspeed5.net
histats.com
hobild.net
houseofrose.com houseofrose.com
ilovevitaly.com hoverboard360.at
imagine-ex.co howmuchdoestizanidinecost.blogspot.com
imgarit.pw hplaserjetpdriver8y.pen.io
imgata.com hully.altervista.org
insurple.com ictizanidinehcl4mg.blogspot.com
interesnie-faktu.ru inspiring-desperate.tk
intim-uslugi.info interior-stickers.ru
invitefashion.com ipornox.xxx
ivanstroi.ru isitwp.com
ix20.ru istizanidineanopiate.blogspot.com
jackwolfskinoutlet.online itag.pw
jaxcube.info itservicesthatworkforyou.com
jerseyssportsshop.com javstock.com
karaganda.xkaz.org joessmogtestonly.com
kit-opt.ru joingames.org
krassh.ru karadene.com
kretpicf.pw kerwinandcariza.com
keywordteam.net
kiskinhouse.com
kurbappeal.info
kuzinsp.ru
lapolis.it lapolis.it
larger.io linkredirect.biz
laserpen.club
leadn.pl
leftofcentrist.com
linkwithin.com
live-sexcam.tk live-sexcam.tk
liveinternet.ru llastbuy.ru
lkbennettstore.online lookingglassemb.com
lockerz.com louisvuittonoutletstore.net
locksmith.jp macdamaged.space
map028.com magicplayer-s.acestream.net
marmitaco.cat manifestyourmillion.com
master-muznachas.ru manyresultshub-a.akamaihd.net
masthopehomes.com marinetraffic.com
matchpal-a.akamaihd.net mebelcomplekt.ru
mecash.ru med-zdorovie.com.ua
menstennisforums.com megahdporno.net
metarip.ru meget.co.za
mister-shop.com metabar.ru
mixed-wrestling.ru mikozstop.com
mnimmigrantrights.net mmostrike.ru
mobplayer.ru mobplayer.net
monthlywinners.com monclercheap.xyz
mypets.by mouselink.co
nationalbreakdown.com mrcsa.com.au
newstraveller.ru mug-na-chas-moscow.ru
nhl17coins.exblog.jp myphotopipe.com
nordvpn.com napalm51.nut.cc
novgorod.xrus.org narkologiya-penza.ru
nrjmobile.fr narosty.com
obiavo.uz oakridgemo.com
olvanto.ru obuv-kupit.ru
onemantrip.com ochistka-stokov.ru
ontargetseo.us oecnhs.info
onthemarch.co oivcvx.website
ops.picscout.com panchro.co.uk
orsonet.ru pc-services.ru
osb.se11.ru pepperstyle.ru
ozas.net
par-fallen.ga
paramountmarble.co.uk
pdns.cz
perfectplanned.com perfectplanned.com
pigrafix.at pic2fly.com
pihl.se picphotos.net
play-movie.pl picswe.com
porngalleries.top pintattoos.com
pornoforadult.com plohaya-kreditnaya-istoriya.ru
pornosok.ru pogodnyyeavarii.gq
portal-eu.ru pornonik.com
preg.marketingvici.com pornstartits.xblog.in
profitfx.online promodj.com
projefrio.com.br prosmibank.ru
prostitutki-kharkova.org pushdata.sendpulse.com
prostitutki-kiev.org pussyspace.net
qitt.ru putevka24.ru
quangcaons.com quangcaons.com
quickbuck.com raisedseo.com
razyboard.com remmling.de
rennlist.com research.ifmo.ru
risparmiocasa.bz.it richinvestmonitor.com
rockingclicks.com robotixix.com
rockprogblog.com rockprogblog.com
royalcar-ufa.ru roll123.com
rozalli.com royalads.net
rvzr-a.akamaihd.net ruscopybook.com
ryetaw.com saneyes.com
samo-soznanie.ru sarf3omlat.com
satellite.maps.ilovevitaly.com scanner-alexa.top
sbf441.com scrapinghub.com
seeingmeerkat.com screentoolkit.com
screpy.com
sdelai-prosto.ru
searchinquire.com
semp.net
seocheckupx.com
seoheap.com
seorank.info
seorankinglinks.us
sex-dating.co
sexkrasivo.net
sexobzor.info
sexs-foto.com
sexsaoy.com
sextracker.de sextracker.de
sexy-screen-savers.com shell-pmr.ru
sexy.babes.frontend-stack.top shitting.pro
sharebutton.org shohanb.com
sigmund-freud.co.uk signx.info
skincrate.net slayerlife.com
sluganarodu.ru smartadserver.com
social-buttons.com snjatie-geroinovoy-lomki.ru
speedup-my.site social-vestnik.ru
stardevine.com socialbuttons.xyz
steamoff.net soda.media
streha-metalko.si softomix.com
summerlinhomes411.info spbchampionat.ru
surflinksmedical.com speeddream.xyz
thaismartloan.com sport-video-obzor.ru
theautoprofit.ml spywarenuker.com
thecoral.com.br stockspmb.info
thegameriders.com t-bygg.com
tilido.com tackletarts.co
timhost.ru tandvardshuset.net
tizanidinehcl2mgsideeffects.blogspot.com taxi-v-eisk.ru
tizanidinesideeffects.blogspot.com teenforporn.com
tk-assortiment.ru teguh.info
token-lab.org telegraf.by
tozup.com tizanidinecost.blogspot.com
tizanidinehydrochloride2mg.blogspot.com
tizanidineshowupondrugtest.blogspot.com
tonerbox.kz
top-deal.com.pl
topappspro.com
torture.ml
track112.site
trafficjunky.net
trailer.cinemaflix.website
tri-slona.org tri-slona.org
trustedmaccleaner.com tsstcorpcddvdwshbbdriverfb.aircus.com
tube8.com turbodsp.com
tubo360.com typimga.pw
ugguk.online uvozdeckych.info
umnovocaminho.com velen.io
valkiria-tk.ru veles.shop
vipms.ru vertaform.com
virtuagirl.com videochat.world
virtualbb.com videokrik.net
vseigry.fun videtubs.pl
vykupavto-krasnodar.ru vipsexfinders.com
webpromotion.ae vita.com.hr
webtherapy.ru vizitki.net
woman-tampon.ru vrotike.ru
worldtraveler.world vtcdns.com
wosik-dach.service-for-web.de webcamdevochka.com
wufak.com webix.me
ww1943.ru whereverdesperate.gq
wygraj-teraz.com ws.ampower.me
xn--h1aakne2ba.xn--p1ai www.solartek.ru
xolodremont.ru xblognetwork.com
xrus.org xn----7sbbahaq9bb5afgiqfliv4m.xn--p1ai
youradulthosting.com xxxasianporn.net
youraticles.pl yaderenergy.ru
yourporn.com youdesigner.kz
yoursearch.me yurgorod.ru
ytmnd.com zigzog.ru
zambini.ru
zone-kev717.info
zoodrawings.com
zoolubimets.ru
zootravel.com zootravel.com

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.219 0; 10.1.0.138 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.05.3083 ### Version: V4.2022.05.3084
### Updated: Tue May 3 22:01:34 UTC 2022 ### Updated: Wed May 4 10:03:40 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.05.3083 ### Version: V4.2022.05.3084
### Updated: Tue May 3 22:01:34 UTC 2022 ### Updated: Wed May 4 10:03:40 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##