vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-04 19:50:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.08.3291 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-08-15 10:09:33 +00:00
parent c8142a4175
commit d4afbd626e
46 changed files with 80972 additions and 80972 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.08.3290 #### Version: V4.2022.08.3291
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.08.3290 #### Version: V4.2022.08.3291
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.08.3290 #### Version: V4.2022.08.3291
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

9984
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.08.3290 #### Version: V4.2022.08.3291
#### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7097](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

9988
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9988
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

9988
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

9988
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 1; 10.1.1.44 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 0; 10.1.1.44 0;
127.0.0.1 0; 127.0.0.1 0;

9988
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 1; 10.1.1.44 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 0; 10.1.1.44 0;
127.0.0.1 0; 127.0.0.1 0;

9988
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 1; 10.1.1.44 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 0; 10.1.1.44 0;
127.0.0.1 0; 127.0.0.1 0;

9988
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3290 3291

9988
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

9988
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.168.24.51 0; 20.253.195.140 0;

9988
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 1; 10.1.1.44 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 0; 10.1.1.44 0;
127.0.0.1 0; 127.0.0.1 0;

9988
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 1; 10.1.1.44 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 0; 10.1.1.44 0;
127.0.0.1 0; 127.0.0.1 0;

9988
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 1; 10.1.1.44 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 0; 10.1.1.44 0;
127.0.0.1 0; 127.0.0.1 0;

9988
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.168.24.51 0; 20.253.195.140 0;

9988
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.168.24.51 0; 20.253.195.140 0;

9988
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 1; 10.1.1.44 1;
127.0.0.1 1; 127.0.0.1 1;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Claritybot BacklinkCrawler
CopyRightCheck Battleztar Bazinga
FDM DTS Agent
Mozlila Dotbot
PHPCrawl Fuzz
Rogerbot JennyBot
SemrushBot LinqiaScrapeBot
Tracemyfile Mojeek
Vagabondo Semrush
magpie-crawler SuperHTTP

328
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
404checker 404enemy
Abonti Abonti
Aboundexbot AllSubmitter
Acunetix Alligator
AfD-Verbotsverfahren
AlphaBot AlphaBot
Anarchie Anthill
Aspiegel
Asterias
Atomseobot Atomseobot
Attach
AwarioRssBot
AwarioSmartBot AwarioSmartBot
BDCbot BBBike
BDFetch
BackWeb BackWeb
Backlink-Ceck BacklinkCrawler
Badass Badass
Barkrowler Barkrowler
BetaBot BetaBot
Bigfoot Black Hole
Bitacle Bolt
BlackWidow Brandprotect
Blackboard Brandwatch
Boardreader
Buck
Buddy Buddy
BuiltBotTough BuiltBotTough
BuiltWith Bullseye
BunnySlippers
BuzzSumo BuzzSumo
CATExplorador CATExplorador
CCBot Cegbfeieh
CSHttp
Calculon
CensysInspect
CheeseBot
CherryPicker CherryPicker
ChinaClaw ChinaClaw
Citoid Citoid
Claritybot Cliqzbot
Cocolyzebot
Cogentbot Cogentbot
Collector Collector
Crawling at Home Project CopyRightCheck
Crescent Cosmos
DSearch Curious
DTS Agent DBLBot
Digincore DataCha0s
DigitalPebble Deusu
Devil
Discobot Discobot
DittoSpyder DnBCrawler-Analytics
DnyzBot DnyzBot
DomCopBot
DomainAppender DomainAppender
DomainStatsBot DomainSigmaCrawler
Dotbot Dotbot
ECCP/1.0 Download Wonder
EMail Siphon Dragonfly
EMail Wolf EMail Wolf
EirGrabber EasyDL
Ecxi
Evil
Exabot
ExtLinksBot ExtLinksBot
EyeNetIE Extreme Picture Finder
Ezooms FHscan
FDM Fimap
Firefox/7.0
FlashGet
Foobot Foobot
Freeuploader FrontPage
Fuzz Genieo
G-i-g-a-b-o-t GetWeb
GT::WWW Gigabot
GetRight Go!Zilla
Getintent
Go-Ahead-Got-It
GoZilla GoZilla
Gotit GrabNet
GrapeshotCrawler Grabber
Grafula
GrapeFX
GridBot GridBot
HMView HMView
HTMLparser
HTTrack HTTrack
Haansoft
Havij Havij
IRLbot Heritrix
IlseBot HonoluluBot
Image Sucker IDBTE4M
IndeedBot IDBot
Iblog
Image Fetch
InfoNaviRobot
Intelliseek Intelliseek
InternetSeer Internet Ninja
Iskanie JOC Web Spider
Jyxobot Jbrofuzz
Keyword Density Jetty
Kenjin Spider
Keybot Translation-Search-Machine
Kinza Kinza
Larbin Kozmosbot
Lanshanbot
Leap
Lftp Lftp
LibWeb LibWeb
Lightspeedsystems
Likse Likse
LinkextractorPro LinkWalker
LinksManager Linkbot
LinqiaScrapeBot
Lipperhey Spider
Litemage_walker Litemage_walker
Lmspider Lmspider
Ltx71
MFC_Tear_Sample MFC_Tear_Sample
MIDown tool MIDown tool
MIIxpc
MJ12bot
MQQBrowser MQQBrowser
MSFrontPage MSFrontPage
Majestic-SEO MSIECrawler
MarkMonitor Magnet
Mata Hari Majestic SEO
MauiBot MauiBot
Mb2345Browser
MeanPath Bot MeanPath Bot
Meanpathbot Meanpathbot
Metauri Mediatoolkitbot
MicroMessenger Microsoft URL Control
Minefield Mojeek
Moblie Safari
Mojolicious Mojolicious
NPbot Morfeus Fucking Scanner
Navroad Msrabot
NetAnts NearSite
NetLyzer Net Vampire
NetSpider Niki-bot
NetZIP
Netcraft
Nettrack
Nikto Nikto
Nimbostratus Nuclei
Not Nutch
Octopus
Offline Explorer Offline Explorer
Offline Navigator Offline Navigator
OnCrawl OpenLinkProfiler
Openfind OpenVAS
Openvas Openvas
OrangeBot OutclicksBot
OrangeSpider
OutfoxBot OutfoxBot
PECL::HTTP
PHPCrawl PHPCrawl
PageAnalyzer POE-Component-Client-HTTP
PageThing.com PageScorer
PeoplePal Panscient
Petalbot Papa Foto
Picscout
Picsearch Picsearch
Pimonster Piepmatz
Pixray Pockey
Probethenet Pu_iN
PyCurl RSSingBot
QueryN Metasearch RankActive
Quick-Crawler RankFlex
RankingBot2 RankingBot
RealDownload RankurBot
RebelMouse Re-re
Rogerbot Reaper
RedesScrapy
RepoMonkey
SBIder SBIder
SEOkicks-Robot SEOkicks
SEOprofiler SEOprofiler
SISTRIX ScanAlert
Screaming ScoutJet
SearchmetricsBot Scrapy
SemrushBot SentiBot
Sitebeam Shodan
Siteimprove Snake
Sitevigil Sosospider
SocialRankIOBot
Sottopop Sottopop
SpaceBison SpaceBison
Spbot Spanner
Spinn3r
SputnikBot SputnikBot
Sqworm
Steeler Steeler
Stripper Sucuri
SurveyBot SuperBot
Suzuran Surfbot
T0PHackTeam TeleportPro
T8Abot TheNomad
Teleport
Telesoft
Telesphoreo
The Intraformant
TightTwatBot TightTwatBot
Titan Titan
Toata Toweyabot
Trendiction Tracemyfile
Turingos UnisterBot
Turnitin Upflow
TwengaBot V-BOT
URLy.Warning VCI
VelenPublicWebCrawler VelenPublicWebCrawler
VeriCiteCrawler VeriCiteCrawler
Virusdie Voil
WASALive-Bot
WBSearchBot WBSearchBot
WEBDAV
WISENutbot WISENutbot
WPScan WPScan
WWW-Collector-E WWW-Mechanize
WWW::Mechanize
Wallpapers/3.0 Wallpapers/3.0
WallpapersHD WallpapersHD
WeSEE
Web Auto
Web Collage
Web Fetch
Web Fuck Web Fuck
Web Sucker Web Sauger
WebEnhancer WebAuto
WebGo IS WebImageCollector
WebPix WebLeacher
WebReaper WebReaper
WebSauger
WebSucker WebSucker
WebWhacker WebWhacker
WebmasterWorldForumBot WebZIP
Webshag
Website Quester
WebsiteExtractor
WebsiteQuester
Webster
Who.is Bot
Widow Widow
WiseGuys Robot WinHTTrack
Wonderbot
Wotbox
Wprecon Wprecon
Xaldon WebSpider Xaldon WebSpider
Xaldon_WebSpider
Xenu Xenu
YoudaoBot YoudaoBot
Zermelo Zeus
Zitebot ZumBot
ZyBorg
adscanner adscanner
arquivo.pt arquivo-web-crawler
autoemailspider autoemailspider
backlink-check backlink-check
cah.io.community
check1.exe check1.exe
coccocbot crawl.sogou.com
cognitiveseo crawler.feedback
demandbase-bot crawler4j
eCatch dataforseo.com
facebookscraper
gopher gopher
heritrix heritrix
linkdexbot instabid
lwp-request internetVista monitor
ips-agent
isitwp.com
lwp-trivial lwp-trivial
magpie-crawler meanpathbot
pcBrowser muhstik-scan
s1z.ru probe-image-size
scalaj-http ripz
seocompany.store seostar
sexsearcher serpstatbot
sitechecker.pro
siteripz
sogouspider
sp_auditbot
spyfu spyfu
sysscan sysscan
tAkeOut
trendiction.com
trendiction.de trendiction.de
webpros.com ubermetrics-technologies.com
webmeup-crawler
webprosbot webprosbot
x22Mozilla xpymep1.exe
zgrab

308
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
404checker 01h4x.com
ADmantX ASPSeek
ALittle Client
Abonti
Aboundexbot Aboundexbot
AfD-Verbotsverfahren Acunetix
Aipbot AhrefsBot
AllSubmitter Alligator
AlphaBot
Anarchie Anarchie
Anarchy99 Anarchy
Anthill Ankit
Aspiegel Apexoo
Atomseobot Asterias
Attach AwarioSmartBot
BBBike BDFetch
BDCbot
BLEXBot
BackDoorBot BackDoorBot
BackStreet BackStreet
Black Hole BackWeb
Backlink-Ceck
BacklinkCrawler
Bandit
BatchFTP
BetaBot
BlackWidow BlackWidow
Boardreader BlowFish
Brandprotect Brandprotect
Buck Brandwatch
BuzzSumo Buddy
CATExplorador BunnySlippers
Calculon CSHttp
CazoodleBot CheeseBot
CheTeam CherryPicker
Chlooe Chlooe
Citoid Collector
Claritybot Curious
Cliqzbot DSearch
Cloud mapping
Cocolyzebot
Cosmos
Crawling at Home Project
CrazyWebCrawler
CrunchBot
DTS Agent DTS Agent
DatabaseDriverMysqli DataCha0s
Discoverybot Demon
Devil
Digincore
DigitalPebble
Disco
Discobot
Dispatch Dispatch
DnBCrawler-Analytics DomainCrawler
DomCopBot Dotbot
DomainAppender
Download Wonder Download Wonder
Dragonfly
Drip Drip
EMail Wolf ECCP/1.0
EasyDL EMail Siphon
Ebingbong
EirGrabber
EroCrawler
Evil
Exabot
ExtLinksBot ExtLinksBot
ExtractorPro Extreme Picture Finder
EyeNetIE Ezooms
FemtosearchBot FemtosearchBot
FlashGet Flunky
Freeuploader
FrontPage
Fuzz Fuzz
FyberSpider FyberSpider
Genieo
Getintent
Gigabot
Go-Ahead-Got-It Go-Ahead-Got-It
Gotit
GrabNet GrabNet
GrapeFX GrapeshotCrawler
HMView HMView
HTTP::Lite Harvest
Haansoft Heritrix
HaosouSpider
Havij
HonoluluBot
Humanlinks
HybridBot HybridBot
IDBTE4M
IDBot IDBot
IRLbot
Iblog Iblog
Id-search Id-search
IlseBot IlseBot
Image Fetch Image Fetch
Image Sucker
IndeedBot IndeedBot
Indy Library Indy Library
InfoTekies
InterGET InterGET
Internet Ninja
IstellaBot IstellaBot
JOC Web Spider Jbrofuzz
JetCar
JikeSpider JikeSpider
JustView JustView
Kenjin Spider Kenjin Spider
Keybot Translation-Search-Machine Keyword Density
Kinza Kozmosbot
LWP::Simple LWP::Simple
Lanshanbot LeechFTP
Leap LexiBot
Lightspeedsystems Lftp
LinkWalker Libwhisker
LieBaoFast
LinkextractorPro LinkextractorPro
LinkpadBot
LinksManager LinksManager
LinqiaScrapeBot LinqiaMetadataDownloaderBot
Lmspider Lipperhey
Litemage_walker
MFC_Tear_Sample MFC_Tear_Sample
MIIxpc
MJ12bot MJ12bot
MQQBrowser Mail.RU_Bot
MSFrontPage Majestic-SEO
Majestic12 Mata Hari
MarkMonitor
MarkWatch
Mb2345Browser Mb2345Browser
Mediatoolkitbot Mediatoolkitbot
Metauri MicroMessenger
Microsoft Data Access Microsoft Data Access
Microsoft URL Control Microsoft URL Control
Mister PiX Minefield
Mojeek Mojeek
Mojolicious
MolokaiBot MolokaiBot
Mozlila
Msrabot Msrabot
NICErsPRO Musobot
NPbot Name Intelligence
Nameprotect NetLyzer
Navroad NextGenSearchBot
Nessus Niki-bot
NetAnts
NetMechanic
NetSpider
Nikto Nikto
Not NimbleCrawler
Nmap
Nuclei
Nutch Nutch
Octopus Offline Navigator
OnCrawl OnCrawl
OpenVAS
Openvas
OrangeSpider
OutclicksBot OutclicksBot
POE-Component-Client-HTTP POE-Component-Client-HTTP
PageScorer Page Analyzer
PageAnalyzer
PageGrabber
PageThing.com PageThing.com
Papa Foto Pandalytics
Pavuk Pavuk
Pi-Monster
Picscout
Picsearch
PictureFinder
PleaseCrawl PleaseCrawl
Probethenet ProWebWalker
Psbot Pu_iN
Pump RankActiveLinkBot
RankFlex
RankingBot RankingBot
Rankivabot RankingBot2
Re-re RepoMonkey
RedesScrapy
Ripper Ripper
SBIder
SEOkicks SEOkicks
SEOkicks-Robot SEOkicks-Robot
SEOprofiler SEOprofiler
SEOstats SEOstats
SMTBot SMTBot
ScoutJet Scanbot
ScreenerBot Scrapy
Searchestate ScrepyBot
SearchmetricsBot SearchmetricsBot
Seekport
Semrush
SemrushBot SemrushBot
SentiBot SeobilityBot
Shodan Seomoz
Siphon Site Sucker
SiteCheckerBotCrawler SiteCheckerBotCrawler
SiteExplorer SiteExplorer
SiteLockSpider
Sitebeam Sitebeam
SmartDownload Siteimprove
Snoopy Sitevigil
SlySearch
Snapbot
SocialRankIOBot SocialRankIOBot
SpankBot Sosospider
Spbot Sottopop
Spinn3r SpaceBison
Spammen
Sqlmap Sqlmap
Stripper Steeler
Sucuri Sucker
SuperHTTP SurveyBot
T0PHackTeam Telesphorep
TeleportPro The Intraformant
Telesoft Thumbor
Titan
Toweyabot
Tracemyfile Tracemyfile
Trendiction
Trendictionbot Trendictionbot
True_Robot
Turnitin Turnitin
TwengaBot TwengaBot
Twice Twice
Typhoeus URLy Warning
URLy.Warning URLy.Warning
Vagabondo Upflow
VeriCiteCrawler VB Project
VidibleScraper
VoidEYE VoidEYE
Voil
WBSearchBot WBSearchBot
WEBDAV WEBDAV
WISENutbot WPScan
WWW-Collector-E
Wallpapers/3.0 Wallpapers/3.0
WallpapersHD Web Collage
Web Auto Web Fuck
Web Fetch Web Pix
Web Sauger Web Sauger
Web Sucker WebCopier
WebAuto WebGo IS
WebEnhancer WebImageCollector
WebFetch WebStripper
WebReaper WebSucker
WebWhacker
WebmasterWorldForumBot WebmasterWorldForumBot
Website Quester
Whacker Whacker
Widow Widow
WinHTTrack Wprecon
Wotbox Xenu
YoudaoBot
Zeus Zeus
Zitebot
ZoomBot
ZoominfoBot ZoominfoBot
ZumBot
archive.org_bot archive.org_bot
arquivo-web-crawler arquivo-web-crawler
arquivo.pt autoemailspider
cah.io.community cah.io.community
coccocbot coccocbot
cognitiveseo
com.plumanalytics
crawler.feedback
crawler4j crawler4j
dataforseo.com
demandbase-bot demandbase-bot
evc-batch domainsproject.org
facebookscraper eCatch
instabid internetVista monitor
lwp-request iubenda-radar
linkdexbot
lwp-trivial lwp-trivial
muhstik-scan
netEstate NE Crawler netEstate NE Crawler
page scorer page scorer
pcBrowser plumanalytics
probe-image-size
ripz ripz
satoristudio.net satoristudio.net
scalaj-http scan.lol
seobility
seoscanners seoscanners
seostar sexsearcher
sitechecker.pro
siteripz siteripz
sogouspider
spyfu
sysscan sysscan
tAkeOut trendiction.com
voyagerx.com
webgains-bot
webmeup-crawler
webprosbot webprosbot
xpymep1.exe
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
datract.com adpremium.org
filunika.com.ru damasarenai.info
hoverboard360.se gogps.me
incomekey.net itronics.ca
ixora.pro lollypopgaming.com
keywordglobal.co.za mafa-free.com
m-google.xyz okroshki.ru
magicdiet.gq tako3.com
scrapinghub.com usacasino.com
txxx.com xn----7sbaaabaei0cc8aj5bj0bncejx.xn--p1ai

922
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
00webcams.com 101raccoon.ru
24-ak.ru 108shot.com
2kata.ru 12masterov.com
1688.com
2015god.org
2020iscoming.info
214jaluwobapef.cf
2itech.ru
3th.co.in
4k-player.pl
4kpics.rocks 4kpics.rocks
4ur.click 66cpwgln.space
7minuteworkout.com 8558l.jobs.net
7xc4n.com a.frcls.fr
7zap.com aa81bf391151884adfa3dd677e41f94be1.com
aa8780bb28a1de4eb5bff33c28a218a930.com aaa38852e886ac4af1a3cff9b47cab6272.com
aac52006ec82a24e08b665f4db2b5013f7.com aasoldes.fr
aad1f4acb0a373420d9b0c4202d38d94fa.com absolutelyanalog.com
abcdefh.xyz acmebtn.ml
above.com ad-words.ru
acheterviagrafr24.com adinterax.com
adf.ly agardomains.com
adminshop.com
adnotbad.com
adrenalin-stk.ru
adservme.com
adultfriendfinder.com
ageofclones.com ageofclones.com
ahmedabadwebs.com agysya.ru
aksonural.ru air-edem.ru
alert-fjg.xyz alarmobninsk.ru
allesohnegirls.net alert-hgd.xyz
allnews24.in alfa-img.com
alphacarolinas.com allblogroll.com
anaseracresar.tk alldezservice.kz
android-style.com alt-servis.ru
animal36.com am-se.com
animalrank.com amyfoxfitness.com
animenime.ru animal-drawings.com
anticrawler.org anonymous-redirect.com
apartmentratings.com apartmentbay.ru
apiwebwebgetcom-a.akamaihd.net apibatbrowsecom-a.akamaihd.net
arabgirls.adultgalls.com apidigidocketnet-a.akamaihd.net
arendas.net apiluckyleapnet-a.akamaihd.net
arvut.org apiwoodensealcom-a.akamaihd.net
asscenihotosidea.blogspot.co.za apparel-offer.com
atofilms.com apple.com-webbrowsing-security.review
aucoinhomes.com apple.com-webbrowsing-security.science
auto.rusvile.lt arcadeplayhouse.com
autobudpostach.club arcteryxsale.online
autonew.biz artefakct.com
availit.weebly.com asiavirtualsolutions.com
asophoto.com
asrv-a.akamoihd.net
atmagroup.ru
autochoixspinelli.com
aviav.eu aviav.eu
axbocz.net avtolombard-krasnodar.com
azazu.ru azlex.uz
azbaseimages.net bad-stars.net
back.dog baersaratov.ru
bannerpower.com bang-hotties.com
benchmarkcommunications.co.uk battlecarnival.su
berlininsl.com beauty-b0x.pl
bestkfiledxd.cf becuo.com
bestmarriages.com berdasovivan.ru
bestwrinklecreamnow.com best-businessman.ru
betteroffers.review best-seo-solution.com
betune.onlinewebshop.net bestplacetobuyeriacta.jw.lt
beztuberkuleza.ru beta.hotkeys.com
birzha-truda.eu betonka.pro
biznesschat.net betterscooter.com
blockety.co bg6s0.com
boleznikogi.com bif-ru.info
boole.onlinewebshop.net binomo.com
braindaily.xyz bio-market.kz
buckspar.gq bio.trade-jp.net
burkesales.com biographiya.com
buyk.host.sk bitcoins-live.ru
bytimedance.ru biteg.xyz
call-of-duty.info bkns.vn
capture-room.com bobba.dzaba.com
cardiosport.com.ua bonkers.name
cartierbraceletsreplica.pw brandehk.dk
cartierreplica.top bscodecs.com
buntube.net
burger-tycoon.com
buttons-for-your-website.com
buyfriend.ru
cactussoft.biz
cand.jp
canrioloadm.gq
canrole.com
carrauterie.be
cartierlovestore.com
cartierwatch.xyz
cascadelink.org cascadelink.org
casinos4dummies.co.uk cbox.ws
ccjp.eu chatroulette.life
cdncash.net chatroulette.si
chatroulette.video cheapestjerseysonwholesale.com
cheapwebsitehoster.com chiblackhawks-jerseys.com
check-this-out-now.online cialis-samples.com
chee-by.biz cialischmrx.com
chiptuninger.com classiquebijoux.ru
christianlouboutinoutlet.win cleansearch.net
cicaki.net clodo.ru
clean-virus-mac.com clubfashionista.com
click2pawn.com cmd.kz
clickcash.com cobaltpro.ru
clickzzs.nl coindirect.io
coin-hive.com
com-secure.download
com-supportcenter.website
commoncrawl.org commoncrawl.org
compiko.info compliance-alexa.xyz
compliance-elena.xyz computernetworksonline.com
continent-e.tv cookie-law-enforcement-hh.xyz
copypaste.traderzplanet.in cpabegins.ru
cottageofgrace.com crackzplanet.com
countbertwistdisp26.soup.io crazyboost.pro
creativehutindia.com credit.co.ua
creditmoney.com.ua culturevie.info
customerpromos-a.akamaihd.net customerguru.in
cxpromote.com daymusam.com
damasarenai.info dealwifi.com
daneshetabiat.com dear-diary.ru
data-ox.com dekorkeramik.ru
ddsoldes.fr delitime.info
deanmoore.ie derevesendeco.com
dellalimov.com div.as
deluxedumps.com divisioncore.com
demenageur.com diy-handmade-ideas.com
dentalpearls.com.au doeco.ru
depositfiles-porn.ga doreenblog.online
detskie-zabavi.ru dostavka-v-krym.com
dev78.dioniqlabb.se downloadeer.net
distver.ru downloadkakaotalk.com
divatraffic.com downloadmefiranaratb1972.xpg.com.br
djonwatch.ru dumpsmania24.com
docs4all.com dumuelave.xyz
doctissimo.fr elektrozigarettekaufen2.blogse.nl
documentbase.net elitedollars.com
dogsrun.net empathica.com
dojki-hd.com
dokumentalkino.net
domashneeruporno.com
dotmass.top
doyouknowtheword-flummox.ml
dprtb.com
dvervmoskvu.ru
earthmagic.info
ekobata.ru
ekto.ee
empirepoker.com
enginebay.ru
enskedesquashclub.se
ereko.ru
erotik-kostenlos.net erotik-kostenlos.net
erotiktreff24.info erotik0049.com
esnm.ru etm-consult.de
everytalk.tv etur.ru
ewebarticle.info eu-cookie-law.blogspot.com
eyessurgery.ru europages.com.ru
f00kclan.de everypony.ru
f012.de exdocsfiles.com
famix.xyz ezigarettekaufen2.blox.pl
ezigarettekaufen2.mpbloggar.se
ezrvrentals.com
f0815.de
farmprofi.net
fashion-mk.net
filefilter.weebly.com filefilter.weebly.com
filkhbr.com filesdatabase.net
filmetricsasia.com fireads.men
financepoints.eu
flowwwers.com flowwwers.com
flytourisme.org forexunion.net
focalink.com forum-engineering.ru
folowsite.com forumprofi.de
forensicpsychiatry.ru free-deals.faith
forums.toucharcade.com freecamdollars.com
fotosfotos.eu freewareseek.com
freejabs.com freewhatsappload.com
froggytube.com froggytube.com
frustrated-favorable.gq fungamelands.com
funnypica.com fxgallery.com
futbolkisales.ru gael-s.ru
fxtips.ru game-top.su
fyl.com.ru gay.adultgalls.com
fym.com.ru generousdeal-a.akamaihd.net
gcup.ru girlgamerdaily.com
gdzkurokam.ru gktt.ru
ge0ip.org glass-msk.ru
gearsadspromo.club golden-catalog.pro
geilehausfrauen.net goodwinmetals.co
genericlowlatencyasiodriverhq.aircus.com gox.com.ua
geoads.com greamimgo.pw
get-seo-domain.com greatzip.com
getdot.ru gsasearchenginerankerdiscount.com
getridofstretchmarks.org guigyverpo.cf
gettpromos.com gunsvicceadadebt.tk
girlporn.ru habermetre.com
glopages.ru hannasolution.ru
goblacked.com hdapp1008-a.akamaihd.net
gok-kasten.net heartofpayne.xyz
golmau.host.sk hidemyass.com
gomusix.com highland-homes.com
goodnightjournal.com hmmm.cz
googglet.com hobild.net
google-liar.ru hoste.octopis.com
gosmeb.ru
growboxbl.ru
gsasearchengineranker.space
gta-top.ru
guardlink.org
guarrasdelporno.xxx
hanink.biz.ly
hanwei.us
hasshe.com
havepussy.com
hd-film.pl
heatpower.ru
highstairs-a.akamaihd.net
hkdiiohi.skyrock.com
homedo.fabpage.com
hostsshop.ru hostsshop.ru
hotblog.top hothor.se
husky-shop.cz hotkeys.com
hyipmanager.in hoverboard360.at
hystersister.com hoverboard360.es
iamsport.org humanorightswatch.org
hundejo.com
hvd-store.com
idealtits.net
imagerydatabase.com imagerydatabase.com
images-free.net imagine-ex.co
imfamous.info impisr.ru
indo-export.ru incanto.in.ua
indiasourcemart.in
infowarcraft.ru
inspiring-desperate.tk inspiring-desperate.tk
interfucks.net
internetproviderstucson.com internetproviderstucson.com
iqbazar.ru intim-uslugi.info
islandminingsupply.wordpress.com investpamm.ru
ispac.org itag.pw
itis4you.com javcoast.com
janavibekken.no javidol.site
jasonpartington.com job.icivil.ir
jav-fetish.com kalb.ru
javstock.com kerwinandcariza.com
jerseysbizwholesalecheap.com kevblog.top
jerseyswholesalechinalimited.com kino-fun.ru
jobgirl24.ru
kaz.kz
kiinomaniak.pl
kimcurlrvsms.com
kino-key.info kino-key.info
kinoduh.ru kliksaya.com
klejonka.info komputers-best.ru
kometars.xyz kotaku.com
komp-pomosch.ru kovrenok.ru
kopihijau.info kskjco.club
kurbappeal.info kumuk.info
lanadelreyfans.us kutikomi.net
languagecode.com lasvegaslockandsafe.com
laparfumotec.com lego4x4.ru
lapolis.it letto.by
laptoper.net lider-zhaluzi.kiev.ua
laserpen.club lincolntheatre.com
lazy-z.com
leadn.pl
leboard.ru
lerporn.info
letmacworkfaster.site
lezbiyanki.net
libertybilisim.com
lifehacĸer.com
lineavabit.it
linkbolic.com linkbolic.com
locimge.pw littlesexdolls.com
locustdesign.co liver-chintai.org
losangeles-ads.com liveu.infoteka.hu
m.b00kmarks.com lkbennettoutlet.online
macnewtech.com lollypopgaming.com
magiadinamibia.blogspot.com louboutinbooties.xyz
magicdiet.gq love-baby.cz
magicplayer-s.acestream.net low-format.ru
magnetic-bracelets.ru magazin-pics.ru
mansiondelrio.co mansparskats.com
manyresultshub-a.akamaihd.net mb140.ru
markjaybeefractal.com mcnamaratech.com
marmitaco.cat mecash.ru
marsgatan.com meds-online24.com
med-zdorovie.com.ua megagrabber.ru
medispainstitute.com.au mere.host.sk
meduza-consult.ru michaelkorsoutlet.store
minyetki.ru migente.com
mobicover.com.ua militarysale.pro
mobot.site misswell.net
moetomnenie.com mobsfun.net
mojowhois.com moncleroutletonline.win
moneyviking-a.akamaihd.net morf.snn.gr
music.utrolive.ru morpicert.pw
musicspire.online most-kerch.org
musicstock.me movpod.in
mydeathspace.com mrcsa.com.au
myfreetutorials.com mttwtrack.com
musicktab.com
musicpro.monster
muztops.ru
mxgetcode.com
myftpupload.com
mygameplus.com
myhitmp3.club
myhydros.org myhydros.org
myonigroup.com myphotopipe.com
nacap.ru nacap.ru
nakozhe.com namenectar.com
nationalbreakdown.com nastroyke.net
neodownload.webcam national-today-winning-winner.club
naval.jislaaik.com
negral.pluto.ro
nelc.edu.eg
nero-us.com
netanalytics.xyz
netcheckcdn.xyz netcheckcdn.xyz
offtopic.biz next-dentists.tk
ok.ru niki-mlt.ru
okonich.com.ua no-fuel.org
one.net.in novgorod.xrus.org
onescreen.cc nucia.biz.ly
online-podarki.com nyfinance.ml
online-x.ru obiavo.in
oddamzadarmo.eu
oecnhs.info
oklogistic.ru
okout.ru
oksrv.com
olgacvetmet.com
oneminutesite.it
onlinadverts.com
online-sbank.ru
onlywoman.org onlywoman.org
ooo-olni.ru ops.picscout.com
openstat.com oralsexfilme.net
orion-code-access.net orel-reshka.net
osb.se11.ru ownshop.cf
ozoz.it ozoz.it
paceform.com panicatack.com
paclitor.com paperwritingservice17.blogspot.ru
parajumpersoutlet.online partnersafe.men
partner-pop.men pawli.eu
partner-print.men pay2me.pl
partner-trustworthy.men pcgroup.com.uy
patol01.pw pearlisland.ru
perl.dp.ua pelfind.me
pensplan.com
perfection-pleasure.ru
pestomou.info pestomou.info
picturesmania.com photorepair.ru
pigrafix.at picturesboss.com
pirateday.ru pihl.se
pizdopletka.club piratecams.com
pizza-imperia.com pixelrz.com
plastweb.ru plastgranar.nu
pogodnyyeavarii.gq pliks.pl
pokerniydom.ru pooleroadmedicalcentre.co.uk
polimga.pw popup-jdh.xyz
popads.net popup.matchmaker.com
popup-fdm.xyz porndairy.in
poquoson.org pornhubforum.tk
pornobrazzers.biz porno-video-chati.ru
pornoelita.info porno.simple-image.com.ua
pornogad.com pornofiljmi.com
pornoxxx.com.mx pornoinn.com
portal-eu.ru pornoklad.ru
postclass.com pornokorol.com
predmety.in.ua pornosemki.info
predominant-invent.tk potoideas.us
productarium.com primedice.com
produkto.net privatefx-in.ru
prostitutki-kharkova.org promalp-universal.ru
prostitutki-tolyatti.org prostitutki-belgoroda.org
prostitutki-rostova.org
prostitutki-tyumeni.org
prostitutki-yaroslavlya.org
prpops.com
pskcijdc.bloger.index.hr pskcijdc.bloger.index.hr
psoriasis-file.trade push-ad.com
pst2017.onlinewebshop.net pwwysydh.com
pulse33.ru qualitymarketzone.com
puzzleweb.ru quangcaons.com
pxhdwsm.com qwesa.ru
qwertty.net r-e-f-e-r-e-r.com
r-control.ru rangjued.com
rapidsites.pro rankscanner.com
rcb101.ru ranksonic.com
rebrand.ly razorweb-a.akamaihd.net
recinziireale.com rebuildermedical.com
recordpage-a.akamaihd.net recipedays.ru
refudiatethissarah.info redirectingat.com
registry-clean-up.net redirectme.net
relatodelpresente.com.ar remont-komputerov-notebook.ru
renewablewealth.com rosbalt.com.ua
rent2spb.ru
rentehno.ru
reward-survey.net
rique.host.sk
robertefuller.com
rocketchange.ru
rossmark.ru rossmark.ru
rozalli.com ruscopybook.com
rus-teh.narod.ru sack.net
russkoe-zdorovie.ru sady-urala.ru
salmonfishingsacramentoriver.com saneitconsulting.com
salutmontreal.com sbdl.no
samoiedo.it scanner-josh.top
saneyes.com scanner-julia.top
sashagreyblog.ga
saugatuck.com
scalerite.co.za
scanner-barak.top
scanner-elena.top
scanner-olga.top
scanspyware.net
schoolfiles.net
scrapinghub.com scrapinghub.com
sdsjweb.com scrapy.org
seasaltwithfood.com sdi-pme.com
senger.atspace.co.uk search-error.com
seoheap.com seo-platform.com
seogadget.ru
seomarketings.online
seoprofiler.com seoprofiler.com
seorankinglinks.com seorankinglinks.xyz
service.adtech.fr serving.adbetclickin.pink
sex.hotblog.top sexcamamateurchat.com
sexy-pings.com sexs-foto.top
ship-marvel.co.ua sextracker.be
shopvilleroyboch.com.ua sexuria.net
shtaketniki.kz sexy-screen-savers.com
sierraapps.com shareyards.com
silktide.com shemalegalls.blogporn.in
sinel.info shmetall.com.ua
sitevalued.com skanninge.se
sk.golden-praga.ru smstraf.ru
sklad-24.ru snomer1.ru
slayerlife.com sobecjvuwa.com.ru
smadihome.com soblaznu.net
smartpet.ru socas.pluto.ro
snsdeainavi.info sockshares.tv
socialtrade.biz softlinesolutions.me
solinf.co softomix.net
special-porn.com sophang8.com
spitfiremusic.com space4update.pw
sprttrack.com spammen.de
stackthatbucks.com spb.ru
stal-rulon.ru stal-rulon.ru
startufa.ru starpages.net
statistici.ro
staynplay.net
stillmiracle.com
stockquotes.wooeb.com stockquotes.wooeb.com
stockspmb.info store-rx.com
stop-zavisimost.com strangeduckfilms.com
stroicol.net stretchingabuckblog.com
stroy-portal22.ru stroilka.info
studentguide.ru sudexpert66.ru
stuff-about-money.com supers.com.ua
subj.ukr-lit.com superstarfloraluk.com
surcentro.com swagbucks.com
susanholtphotography.com
svbur.ru
swiped.su swiped.su
szamponrevita.pl t-bygg.com
tackletarts.co tackletarts.co
td-l-market.ru tam-gde-more.ru
taraz.xkaz.org
tastyfoodideas.com
taximytishi.ru
technopellet.gr
tecnoteakviareggio.it
telefonsexkostenlos.tk telefonsexkostenlos.tk
tfxiq.com templates.radiodigital.co
theallgirlarcade.com thefds.net
thejournal.ru
thenetinfo.com
thepantonpractice.co.uk
theporndude.com theporndude.com
tizanidineandhydrocodone.blogspot.com tizanidinedosage.blogspot.com
tizanidinehcl4mgtab.blogspot.com tizanidinedrugtest.blogspot.com
tizanidineinteractions.blogspot.com
tizanidinenarcotic.blogspot.com tizanidinenarcotic.blogspot.com
tizanidinevscyclobenzaprine.blogspot.com tizanidinevscyclobenzaprine.blogspot.com
tomatis.gospartner.com toonfamilies.net
tooplay.com top-karkas.ru
top-deal.com.pl topflownews.com
top10-online-games.com toproadrunner5.info
topappspro.com torrents-tracker.com
topvidos.ru touchmods.fr
torrentdownloadhub.com tracfone.com
trackmedia101.com tracksz.co
triplepanda.xyz trafficstars.com
triumf-realty.ru trustedmaccleaner.com
trustl.life tvnewsclips.info
try-rx.com tvoystartup.ru
tupper-posuda.ru twincitiescarservice.com
tvory.predmety.in.ua u-cheats.ru
twittrading.com ua.tc
tytoona.com uchil.net
uac.net.au uggsale.online
ufa.xrus.org ukrtextbook.com
ukkala.xyz ukrtvory.in.ua
ukr-lit.com
ukrobstep.com
umg-stroy.ru umg-stroy.ru
unacittaconte.org
upproar.com
uzungil.com uzungil.com
vclicks.net valmetrundan.se
vegascosmetics.ru vertaform.com
veles.shop video-chat.in
vellings.info videokrik.net
venerologiya.com vipps.com.my
veopornogratis.xxx vksaver-all.ru
vereo.eu
via-energy-cumpara.com
video-chat.love
videochat.ph
videotuber.ru
vipcallsgirls.com
viven.host.sk
vizag.kharkov.ua
vksex.ru vksex.ru
voditeltrezviy.ru vodaodessa.com
vsdshnik.com voltrknc1.com
vvon.co.uk vpnmouse.com
vzlomsn.org vut.com.ru
wallpaperstock.net vykupavto-krasnodar.ru
vysigy.su
wahicbefa31.soup.io
walkme.com
wallpapersdesk.info
warningzscaler.heraeus.com
watch-movies.ru watch-movies.ru
wcb.su watchinf.com
weburok.com weboptimizes.com
wesharepics.info website-datenbank.de
whipme.yopoint.in worldlovers.ru
winterclassichockeyjerseys.com worldtraveler.world
woman-h.ru wpthemedetector.co.uk
writingservices17.blogspot.ru wttavern.com
wwwadultcheck.com ww1943.ru
xaijo.com www.ehscloud.cn
xlolitka.com www.timer4web.com
xn----7sbbahaq9bb5afgiqfliv4m.xn--p1ai wyniki-lista.pl
xn----9sbebi2bvzr7h.xn--p1ai x-diesel.org
xn--80aagddcgkbcqbad7amllnejg6dya.xn--p1ai xn--80adaggc5bdhlfamsfdij4p7b.xn--p1ai
xn--80aodinpgi.xn--p1ai xn--80ajbshivpvn2i.xn--p1ai
xn--e1afanlbnfckd7c3d.xn--p1ai xn--b1agm2d.net
xn--oogle-wmc.com xn--q1a.xn--b1aube0e.xn--c1acygb.xn--p1ai
xolodremont.ru xwatt.ru
xtremeeagles.net yebocasino.com
yaminecraft.ru yoopsie.com
youdesigner.kz yougame.biz
youporn-ru.com yougetsignal.com
yourserverisdown.com zaloro.com
youtube-downloader.savetubevideo.com
yugk.net
z-master.ru
zaloadi.ru
zarabotat-na-sajte.ru zarabotat-na-sajte.ru
zeroredirect12.com zeg-distribution.com
zionstar.net zerocash.msk.ru
znakomstva-moskva77.ru zeroredirect6.com
zigarettenonlinekaufen1.blox.pl
zirondelli.it
zoogdiesney.com zoogdiesney.com
zoogdinsney.com
zoosexart.com
zrizvtrnpale.tk
zvetki.ru

490
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
000free.us 2kata.ru
2rich4bitches.com 57883.net
4webmasters.com 76brighton.co.uk
8lufu.com 883zy.com
aa8b68101d388c446389283820863176e7.com aa9bd78f328a6a41279d0fad0a88df1901.com
aaronabel.com adanih.com
absugars.com adclickservice.com
adtech.us adssafeprotected.com
adultmeetonline.info adtech.fr
advocatemsk.ru adult-shop.com.ua
afmuseum.com animalia-life.club
aghanyna.com anticrawler.org
alfa-img.com apiwebwebgetcom-a.akamaihd.net
allknow.info arenda-avtoprokat-krasnodar.ru
aludecor.info as5000.com
alyeskaresort.com autoplate.info
anime.dougasouko.com avon-ukraine.com
architecturebest.com azartmix.com
arquapetrarca.info baksman.com
asmxsatadriverin.aircus.com beachpics.com
asophoto.com belinka.com.ua
autogrep.ru bestcalovebracelet.cn
autotrends.today bestofpicture.com
bangla-chat-uk-paralud.ga bigcities.org
batanga.net bioca.org
beclean-nn.ru birzha-truda.eu
bezsporno.ru blogos.kz
bfz.biz blogsfunky672.weebly.com
big-cash.net bradleylive.xyz
blackplanet.com buehne-fuer-menschenrechte.de
blog.f00kclan.de buy-forum.ru
blogig.org buynorxx.com
bobinoz.com cablecar.us
bosefux.esy.es carloans.com
braindaily.xyz casinorewards.info
candcstuccoandstone.com certifywebsite.win
candlewooddentalcentre.com.au checkpagerank.net
cartierring.xyz chocolateslim-en-france.com
cashonads.com clevernt.com
censys.io clicksor.com
cfcl.co.uk commoncrawl.org
cheapjerseysa.com compliance-barak.top
ciekawinki.pl compliance-barak.xyz
cl.s7.exct.net compliance-viktor.top
cleanersoft.com converse.ddsoldes.fr
club-samodelkin.ru cookielawblog.wordpress.com
codq.info courtshipgift.com
compliance-nelson.top cracksplanet.com
compliance-willy.top customcatchcan.com
coop-gamers.ru den-noch24.ru
coslab.club design-lands.ru
customerpromos-a.akamaihd.net dieswaene.com
cwetochki.ru dirtpics.pw
czat.wp.pl div.as
data.vtc.pw djonwatch.ru
day-news.info docsportal.net
delayreferat.ru dostavka-v-ukrainu.ru
demian.kz ege-russian.ru
detoxmed24.ru egovaleo.it
dinkolove.ya.ru elektrischezigaretten1.blogse.nl
dogclothing.store elektrischezigaretten2.beeplog.com
domain2008.com en.altezza.travel
dominateforex.ml exonline.info
elektrischezigarette1.onsugar.com extads.net
elektrozigarette1.dreamwidth.org extener.org
elektrozigaretten2.yn.lt ezigarettekaufen1.hpage.com
elstal.com.pl ezigarettenshop2.postbit.com
engines-usa.com fanrto.com
eropornosex.ru feel-planet.com
estibot.com fellowshipoftheminds.com
everypony.ru filesmonster.porn
exct.net film-one.ru
executehosting.com fishingwholesale.us
expertnaya-ocenka.ru flash4fun.com
eyessurgery.ru forum.tvmir.org
fable.in.ua foto-sisek.porngalleries.top
fanoboi.com freelifetimefuckbook.com
figuringmoneyout.com galeria-zdjec.com
fireads.men gameonasia.com
fotosfotos.eu gardene.ru
free-share-buttons.blogspot.com gateway.zscalertwo.net
free-video-tool.com gaz-voshod.ru
gamerscorps.com gg.zzyjxs.com
gcup.ru girlspicsa.com
geile-lelly.eu gorabagrata.ru
germes-trans.com gourcy.altervista.org
gg-arena.ru gox.com.ua
gheus.altervista.org gsasearchengineranker.top
go2mike.ru habermetre.com
goldadpremium.com hackers-crackers.tk
golfresa.lucania.se hausfrauensex18.com
googglet.com hccoder.info
graphid.com healing-dysplasia.ru
gsasearchengineranker.xyz hem.passagen.se
guod.me highspeed5.net
gwebtools.com.br homeinns.com
healbio.ru hostnow.men
highstairs-a.akamaihd.net howtoclean.club
hledejvshopech.cz hptwaakw.blog.fc2.com
hopeonthestreet.co.uk iccornacircri.cf
housediz.com impisr.edunsk.ru
hut1.ru investpamm.ru
iboss.com janettabridal.com
ido3.com javrip.net
ilmen.net jerseyschinabizwholesale.us
imgata.com kaac.ru
iminent.com kadashihotel.com
inboundlinks.win
incanto.in.ua
infobabki.ru
intermesh.net
intim-uslugi.info
investmac.com
inzn.ru
iqoption-bin.com
itsdp3.com
jo24news.com
juliadiets.com
kamalsinha.com
kazinogames.lv
kchaxton.com kchaxton.com
keywords-monitoring-your-success.com kerwinandcariza.com
kichenaid.ru kicknights.gq
komputers-best.ru kinofak.net
klin3952.ru
konkursov.net
kwzf.net kwzf.net
lakomka.com.ua landreferat.ru
lampokrat.ws letmacwork.world
levaquin750mg.blogspot.com limads.men
linkdebrideur.xyz limtu.ifmo.ru
littleberry.ru lubetube.com
littlecity.ch lyrster.com
livejasmin.com m-google.xyz
lkbennettstore.online manifestation.betteroffers.review
locatellicorretor.com.br manyresultshub-a.akamaihd.net
lookingglassemb.com masturbate.co.uk
madot.onlinewebshop.net matematikus.info
malwareremovalcenter.com mature.free-websites.com
maximpartnerspr.com mbiologi.ru
meble-bogart.info mebelcomplekt.ru
medizinreisen.de medictube.ru
medkletki.ru memberty.com
mex-annushka.ru mexicotravelnet.com
mhtr.be mikozstop.com
mil-stak.com minecraft-rus.org
mirzonru.net minegam.com
mlf.hordo.win mokrayakiska.com
mmoguider.ru monclerjacketsoutlet.pw
moyakuhnia.ru moomi-daeri.com
mp3downloadhq.com motherboard.vice.com
mybinaryoptionsrobot.com mp3ritm.top
mydeathspace.com mttwtrack.com
national-today-winning-winner.club myshopmatemac.com
needtosellmyhousefast.com naperehresti.info
nero-us.com neodownload.webcam
netvouz.com nfljerseys.online
new-apps.ru norththeface.store
newfilmsonline.ru obiavo.su
nextbackgroundcheck.gq okayimage.com
no-rx.info okroshki.ru
oecnhs.info olgacvetmet.com
offtime.ru online-sbank.ru
ok-ua.info online-templatestore.com
onlineporno.site onlineku.com
pagesense.com openfrost.net
parajumpersoutlet.online oranga.host.sk
phuketscreen.com osoznanie-narkotikam.net
pijoto.net paceform.com
pizdeishn.com partner-stop.men
pochemychka.net picmoonco.pw
pogodnyyeavarii.gq picswe.com
porno-raskazy.ru pinkduck.ga
prefersurvey.net pirateday.ru
prointer.net.ua pobeiranie.pl
prom23.ru poisk-zakona.ru
promgirldresses.xyz poppen-nw.net
py100.ru porndairy.in
quebec-bin.com porno-transsexuals.ru
quick-seeker.com pornoforadult.com
r-e-f-e-r-e-r.com portside.cc
positive2b.ru
postclass.com
privacyassistant.net
projefrio.com.br
quelle.ru
questionmarque.ch
raavidesigns.com raavidesigns.com
rangoman.date ranking2017.ga
realresultslist.com razvratnoe.org
recipedays.ru recordpage-a.akamaihd.net
rennlist.com remorcicomerciale.ro
rent2spb.ru rentalcarnavi.info
risparmiocasa.bz.it rocketchange.ru
sarafangel.ru royalads.net
seadragonherbery.com ruscopybook.com
search.xtconnect.com rvtv.ru
seksvideoonlain.com rvzr-a.akamaihd.net
seorussian.ru s-iwantyou.com
sexpartygirls.net s1z.ru
sextracker.be saneitconsulting.com
sexy.babes.frontend-stack.top sbricur.com
shinikiev.com.ua scottbywater.com
sitechecker.pro seansonline24.pl
social-buttons.com searchimpression.com
soft-terminal.ru seobility.net
sogimlecal.tk seryeznie-znakomstva.ru
sohoindia.net sharebutton.net
soietvousmaime.fr shop-electron.ru
sotechco.co signx.info
spaceshipad.com sildenafil-tadalafil.info
spasswelt.net simple-share-buttons.com
spywareit.com slim.sellany.ru
sribno.net socialseet.ru
stackthatbucks.com spyfu.com
staynplay.net sr-rekneskap.no
stmassage.ru stairliftsarea.com
sucsesofinspiration.com sukarame.net
sumo.com tedxrj.com
suralink.com telefonsex-ohne0900.net
taranerymagesswa.blogspot.com thefotosgratis.eu
thecoral.com.br thejournal.ru
tiandeural.ru thelottosecrets.com
time-japan.ru tizanidineandgabapentin.blogspot.com
tizanidinedrugtest.blogspot.com tizanidinehcl2mgsideeffects.blogspot.com
tn811.us tizanidinehcl4mgtablet.blogspot.com
tobeyouday.win tnaionline.org
tongkatmadura.info trafaret74.ru
tooplay.com trudogolik.net
torontoplumbinggroup.com ttrraacckkrr.com
tubeoffline.com
turkeyreport.tk turkeyreport.tk
turvgori.ru tvoystartup.ru
udayavani.com uchetunet.su
ural-buldozer.ru unitexindia.com
urccvfmc.bloger.index.hr unrealcommander.org
usbggettwku.ga ussearche.cf
vandrie-ict.nl valsalud.com
videositename.com valueclick.com
videtubs.pl ventelnos.com
vinsit.ru voluumtrk.com
voditeltrezviy.ru voprosotvet24.ru
wdfdocando.com vseuznaem.com
website-audit.com.ua w3data.co
weightatraining.com webadvance.club
whatsappdownload10.com webcamtalk.net
whereverdesperate.gq wemedinc.com
wineitudes.wordpress.com westsextube.com
wovis.site
x69ty.ru
xchat26.myfreecams.com
xdoza.com
xerox-douglas.cf xerox-douglas.cf
xn----8sblgmbj1a1bk8l.xn----161-4vemb6cjl7anbaea3afninj.xn--p1ai xjlottery.com
xn--80aafb2a.xn--p1ai xn----9sbebi2bvzr7h.xn--p1ai
xn--80aanaardaperhcem4a6i.com xn--80aaagvmjabrs1aoc9luc.xn--p1ai
yellowads.men xn--80aeahghtf8ac5i.xn--p1ai
zrus.org xn--b1ag5cfn.xn--p1ai
zvooq.eu xnxx699.com
xnxxandxvideos.com
xtraffic.plus
xvideosporn.biz
yatrk.xyz
yellowproxy.net
youraticles.pl
zeroredirect7.com
zvuker.net

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.84 0; 10.1.1.44 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.08.3290 ### Version: V4.2022.08.3291
### Updated: Sun Aug 14 22:01:23 UTC 2022 ### Updated: Mon Aug 15 10:03:44 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.08.3290 ### Version: V4.2022.08.3291
### Updated: Sun Aug 14 22:01:23 UTC 2022 ### Updated: Mon Aug 15 10:03:44 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##