vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-04 19:50:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.04.3028 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-04-07 07:18:28 +00:00
parent 79b0be2654
commit 8f2b94f952
52 changed files with 88929 additions and 88919 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.04.3027 #### Version: V4.2022.04.3028
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.04.3027 #### Version: V4.2022.04.3028
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

2
README.md
View file

@ -14,7 +14,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.04.3027 #### Version: V4.2022.04.3028
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

10976
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

2
_generator_lists/bad-user-agents.list
View file

@ -241,7 +241,6 @@ Likse
LinkScan LinkScan
LinkWalker LinkWalker
Linkbot Linkbot
linkdexbot
LinkextractorPro LinkextractorPro
LinkpadBot LinkpadBot
LinksManager LinksManager
@ -591,6 +590,7 @@ internetVista\ monitor
ips-agent ips-agent
isitwp.com isitwp.com
iubenda-radar iubenda-radar
linkdexbot
lwp-request lwp-request
lwp-trivial lwp-trivial
magpie-crawler magpie-crawler

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.04.3027 #### Version: V4.2022.04.3028
#### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7092](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [637](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)

10982
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

10982
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

10982
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/_robots_input/robots-input.txt
View file

@ -241,7 +241,6 @@ Likse
LinkScan LinkScan
LinkWalker LinkWalker
Linkbot Linkbot
Linkdexbot
LinkextractorPro LinkextractorPro
LinkpadBot LinkpadBot
LinksManager LinksManager
@ -591,6 +590,7 @@ internetVista monitor
ips-agent ips-agent
isitwp.com isitwp.com
iubenda-radar iubenda-radar
linkdexbot
lwp-request lwp-request
lwp-trivial lwp-trivial
magpie-crawler magpie-crawler

10982
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

10982
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

10982
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -302,7 +302,6 @@
"~*(?:\b)LinkScan(?:\b|)" 0; "~*(?:\b)LinkScan(?:\b|)" 0;
"~*(?:\b)LinkWalker(?:\b|)" 0; "~*(?:\b)LinkWalker(?:\b|)" 0;
"~*(?:\b)Linkbot(?:\b|)" 0; "~*(?:\b)Linkbot(?:\b|)" 0;
"~*(?:\b)Linkdexbot(?:\b|)" 0;
"~*(?:\b)LinkextractorPro(?:\b|)" 0; "~*(?:\b)LinkextractorPro(?:\b|)" 0;
"~*(?:\b)LinkpadBot(?:\b|)" 0; "~*(?:\b)LinkpadBot(?:\b|)" 0;
"~*(?:\b)LinksManager(?:\b|)" 0; "~*(?:\b)LinksManager(?:\b|)" 0;
@ -652,6 +651,7 @@
"~*(?:\b)ips-agent(?:\b|)" 0; "~*(?:\b)ips-agent(?:\b|)" 0;
"~*(?:\b)isitwp.com(?:\b|)" 0; "~*(?:\b)isitwp.com(?:\b|)" 0;
"~*(?:\b)iubenda-radar(?:\b|)" 0; "~*(?:\b)iubenda-radar(?:\b|)" 0;
"~*(?:\b)linkdexbot(?:\b|)" 0;
"~*(?:\b)lwp-request(?:\b|)" 0; "~*(?:\b)lwp-request(?:\b|)" 0;
"~*(?:\b)lwp-trivial(?:\b|)" 0; "~*(?:\b)lwp-trivial(?:\b|)" 0;
"~*(?:\b)magpie-crawler(?:\b|)" 0; "~*(?:\b)magpie-crawler(?:\b|)" 0;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

10982
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3027 3028

10982
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

10982
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
13.88.115.177 0; 104.45.218.19 0;

10982
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

10982
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

10982
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -302,7 +302,6 @@
"~*(?:\b)LinkScan(?:\b|)" 0; "~*(?:\b)LinkScan(?:\b|)" 0;
"~*(?:\b)LinkWalker(?:\b|)" 0; "~*(?:\b)LinkWalker(?:\b|)" 0;
"~*(?:\b)Linkbot(?:\b|)" 0; "~*(?:\b)Linkbot(?:\b|)" 0;
"~*(?:\b)Linkdexbot(?:\b|)" 0;
"~*(?:\b)LinkextractorPro(?:\b|)" 0; "~*(?:\b)LinkextractorPro(?:\b|)" 0;
"~*(?:\b)LinkpadBot(?:\b|)" 0; "~*(?:\b)LinkpadBot(?:\b|)" 0;
"~*(?:\b)LinksManager(?:\b|)" 0; "~*(?:\b)LinksManager(?:\b|)" 0;
@ -652,6 +651,7 @@
"~*(?:\b)ips-agent(?:\b|)" 0; "~*(?:\b)ips-agent(?:\b|)" 0;
"~*(?:\b)isitwp.com(?:\b|)" 0; "~*(?:\b)isitwp.com(?:\b|)" 0;
"~*(?:\b)iubenda-radar(?:\b|)" 0; "~*(?:\b)iubenda-radar(?:\b|)" 0;
"~*(?:\b)linkdexbot(?:\b|)" 0;
"~*(?:\b)lwp-request(?:\b|)" 0; "~*(?:\b)lwp-request(?:\b|)" 0;
"~*(?:\b)lwp-trivial(?:\b|)" 0; "~*(?:\b)lwp-trivial(?:\b|)" 0;
"~*(?:\b)magpie-crawler(?:\b|)" 0; "~*(?:\b)magpie-crawler(?:\b|)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

10982
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
13.88.115.177 0; 104.45.218.19 0;

10982
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
13.88.115.177 0; 104.45.218.19 0;

10982
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 1; 10.1.0.71 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -302,7 +302,6 @@
"~*(?:\b)LinkScan(?:\b|)" 0; "~*(?:\b)LinkScan(?:\b|)" 0;
"~*(?:\b)LinkWalker(?:\b|)" 0; "~*(?:\b)LinkWalker(?:\b|)" 0;
"~*(?:\b)Linkbot(?:\b|)" 0; "~*(?:\b)Linkbot(?:\b|)" 0;
"~*(?:\b)Linkdexbot(?:\b|)" 0;
"~*(?:\b)LinkextractorPro(?:\b|)" 0; "~*(?:\b)LinkextractorPro(?:\b|)" 0;
"~*(?:\b)LinkpadBot(?:\b|)" 0; "~*(?:\b)LinkpadBot(?:\b|)" 0;
"~*(?:\b)LinksManager(?:\b|)" 0; "~*(?:\b)LinksManager(?:\b|)" 0;
@ -652,6 +651,7 @@
"~*(?:\b)ips-agent(?:\b|)" 0; "~*(?:\b)ips-agent(?:\b|)" 0;
"~*(?:\b)isitwp.com(?:\b|)" 0; "~*(?:\b)isitwp.com(?:\b|)" 0;
"~*(?:\b)iubenda-radar(?:\b|)" 0; "~*(?:\b)iubenda-radar(?:\b|)" 0;
"~*(?:\b)linkdexbot(?:\b|)" 0;
"~*(?:\b)lwp-request(?:\b|)" 0; "~*(?:\b)lwp-request(?:\b|)" 0;
"~*(?:\b)lwp-trivial(?:\b|)" 0; "~*(?:\b)lwp-trivial(?:\b|)" 0;
"~*(?:\b)magpie-crawler(?:\b|)" 0; "~*(?:\b)magpie-crawler(?:\b|)" 0;

18
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
ASPSeek 80legs
Dotbot Calculon
Crescent
Extreme Picture Finder
Jorgee Jorgee
MQQBrowser Majestic SEO
MSIECrawler SeobilityBot
Psbot SputnikBot
RocketCrawler SuperHTTP
SBIder magpie-crawler
Telesphorep
coccocbot

282
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
360Spider 404checker
80legs 80legs
AIBOT ASPSeek
Aboundexbot Abonti
Aipbot Acunetix
Anarchy AfD-Verbotsverfahren
Ankit AiHitBot
Apexoo Alexibot
AwarioSmartBot AlphaBot
BBBike Anarchie
Anthill
Aspiegel
BLEXBot BLEXBot
BackStreet
BackWeb
Backlink-Ceck Backlink-Ceck
Bandit Badass
Battleztar Bazinga BetaBot
BlackWidow Bolt
Blackboard Buck
Buddy Buddy
BuiltBotTough BunnySlippers
CATExplorador CATExplorador
Calculon
Cegbfeieh
CheTeam CheTeam
Chlooe CheeseBot
Citoid Cliqzbot
Copyscape Cocolyzebot
Crawling at Home Project Cogentbot
CrazyWebCrawler CrazyWebCrawler
Crescent CrunchBot
Custo
CyotekWebCopy
DBLBot
DIIbot DIIbot
DTS Agent DTS Agent
DataCha0s DigitalPebble
DatabaseDriverMysqli
Demon
Disco
Dispatch Dispatch
DnBCrawler-Analytics DittoSpyder
DomCopBot DomCopBot
DomainAppender DomainCrawler
DomainSigmaCrawler
Domains Project Domains Project
Dotbot
Download Wonder
Dragonfly Dragonfly
Drip
EMail Siphon EMail Siphon
EirGrabber Ebingbong
EroCrawler
Exabot Exabot
Express WebPictures Express WebPictures
Extractor FDM
EyeNetIE FemtosearchBot
Fimap
FlashGet
Flunky Flunky
Freeuploader
FrontPage
Fuzz Fuzz
FyberSpider
Fyrebot
G-i-g-a-b-o-t G-i-g-a-b-o-t
Go-Ahead-Got-It GalaxyBot
Grafula Genieo
Go!Zilla
GrabNet
GrapeFX GrapeFX
GrapeshotCrawler GridBot
HEADMasterSEO
HMView HMView
HTTP::Lite
Haansoft Haansoft
Harvest Heritrix
HonoluluBot Hloader
HybridBot
IDBTE4M IDBTE4M
IDBot Iblog
Id-search
IlseBot IlseBot
IndeedBot IndeedBot
InfoTekies InfoTekies
Iskanie InterGET
IstellaBot Iria
JOC Web Spider JOC Web Spider
JamesBOT
Jbrofuzz Jbrofuzz
JennyBot JennyBot
JetCar JetCar
Jetty Joomla
JustView
Kenjin Spider Kenjin Spider
Keyword Density LWP::Simple
Lanshanbot
Leap Leap
LeechFTP
LexiBot
Lftp Lftp
LibWeb Libwhisker
LieBaoFast LieBaoFast
Lightspeedsystems
Likse Likse
LinkWalker LinkScan
LinkextractorPro Linkbot
LinkpadBot LinkpadBot
Lipperhey Lipperhey
Lipperhey Spider
Litemage_walker
Ltx71
MIIxpc
MJ12bot MJ12bot
MTRobot MSFrontPage
Mag-Net Mag-Net
Magnet
MarkMonitor
MarkWatch MarkWatch
Masscan
Mata Hari Mata Hari
Mb2345Browser MeanPath Bot
Meanpathbot
Mediatoolkitbot
MegaIndex.ru
Metauri Metauri
Microsoft URL Control
Minefield
Moblie Safari Moblie Safari
Mojeek Morfeus Fucking Scanner
Mojolicious
Mozlila Mozlila
Musobot Msrabot
NPbot NPbot
Nameprotect Nameprotect
Navroad
NearSite NearSite
Needle
Nessus Nessus
Net Vampire
NetAnts
NetZIP
Netcraft Netcraft
Nettrack Nettrack
NextGenSearchBot
Niki-bot Niki-bot
NimbleCrawler
Nimbostratus Nimbostratus
Not
Offline Explorer Offline Explorer
OrangeSpider Offline Navigator
OnCrawl
Openfind
OutclicksBot OutclicksBot
PECL::HTTP PECL::HTTP
PHPCrawl PHPCrawl
PageGrabber PageGrabber
PageScorer
Pavuk Pavuk
PeoplePal PeoplePal
Picsearch Pi-Monster
Pimonster
Pixray
PleaseCrawl
ProPowerBot ProPowerBot
ProWebWalker
Pump
RSSingBot
RankActive
RankFlex
RankingBot RankingBot
Re-re Re-re
ReGet ReGet
RebelMouse RealDownload
RedesScrapy Reaper
Recorder
RepoMonkey RepoMonkey
SEOstats SEOkicks
ScanAlert SEOlyticsCrawler
Scrapy SISTRIX
Searchestate Screaming
SemanticJuice SemanticJuice
Semrush SeoSiteCheckup
SentiBot
Seomoz Seomoz
Shodan Siphon
Site Sucker Site Sucker
SiteSnagger SiteCheckerBotCrawler
Sitebeam SiteExplorer
SiteLockSpider
Siteimprove
SlySearch SlySearch
Snake SmartDownload
Sociscraper Sociscraper
Sosospider Sogou web spider
Sottopop Sottopop
SpaceBison
Spammen Spammen
Spanner Spinn3r
SputnikBot SputnikBot
Surfbot Sqlmap
Swiftbot Sqworm
Steeler
Stripper
Sucuri
SuperBot
SuperHTTP
Suzuran
T8Abot T8Abot
Telesphorep The Intraformant
Thumbor
Titan
Tracemyfile
Trendiction Trendiction
Trendictionbot
Turingos
Turnitin Turnitin
TurnitinBot
Twice Twice
Typhoeus
URLy.Warning
Upflow Upflow
VB Project
Vacuum Vacuum
Vagabondo VeriCiteCrawler
VoidEYE Voil
WASALive-Bot
WBSearchBot WBSearchBot
WEBDAV
WPScan
WWW-Mechanize
WWW::Mechanize WWW::Mechanize
WWWOFFLE WWWOFFLE
WeSEE Wallpapers/3.0
WallpapersHD
Web Auto Web Auto
Web Fetch Web Fetch
Web Fuck Web Pix
Web Sucker Web Sucker
WebAuto WebBandit
WebCollage WebCollage
WebEnhancer WebFuck
WebFetch WebGo IS
WebImageCollector
WebLeacher WebLeacher
WebPix WebPix
WebReaper
WebSauger WebSauger
WebWhacker WebStripper
WebZIP WebmasterWorldForumBot
Webalta
Webshag
Whack Whack
Whatweb Whatweb
Who.is Bot Who.is Bot
Widow WinHTTrack
Woobot WiseGuys Robot
Wotbox Wprecon
Xaldon WebSpider
Xaldon_WebSpider Xaldon_WebSpider
Xenu Xenu
YoudaoBot YoudaoBot
Zade
Zauba Zauba
ZoomBot Zeus
Zitebot
ZoominfoBot ZoominfoBot
adscanner ZyBorg
arquivo-web-crawler arquivo-web-crawler
autoemailspider
cah.io.community cah.io.community
clark-crawler clark-crawler
coccocbot coccocbot
crawl.sogou.com crawler.feedback
crawler4j dataforseo.com
domainsproject.org
eCatch
evc-batch evc-batch
gopher linkdexbot
heritrix
instabid
internetVista monitor
lwp-request lwp-request
lwp-trivial
magpie-crawler magpie-crawler
muhstik-scan meanpathbot
ripz mediawords
scalaj-http netEstate NE Crawler
oBot
polaris version
probe-image-size
satoristudio.net
seobility seobility
seostar seostar
serpstatbot sitechecker.pro
siteripz spyfu
sp_auditbot
trendiction.de trendiction.de
voyagerx.com voyagerx.com
webmeup-crawler webgains-bot
x09Mozilla webpros.com
x22Mozilla
xpymep1.exe xpymep1.exe
zgrab zauba.io

320
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 404checker
360Spider 404enemy
80legs
ADmantX ADmantX
AIBOT AIBOT
Abonti
Aboundex Aboundex
Aboundexbot
AhrefsBot AhrefsBot
AiHitBot AiHitBot
Alexibot Alexibot
Ankit AlphaBot
Anarchy
Anthill Anthill
Apexoo Aspiegel
Asterias
AwarioSmartBot AwarioSmartBot
BackWeb BackDoorBot
BackStreet
Backlink-Ceck Backlink-Ceck
Badass Badass
BatchFTP Bandit
Barkrowler
Bigfoot
Black Hole
BlackWidow
Blow Blow
BlowFish
Boardreader
Bolt
BotALot BotALot
Brandprotect Brandprotect
Brandwatch
Buck Buck
Bullseye CATExplorador
CODE87 CCBot
CazoodleBot CSHttp
CensysInspect CensysInspect
ChinaClaw Cliqzbot
Chlooe Cocolyzebot
Cogentbot
Collector
CopyRightCheck CopyRightCheck
Copyscape
Cosmos Cosmos
Craftbot Craftbot
Crawling at Home Project
Crescent Crescent
DIIbot Curious
CyotekWebCopy
DSearch
DatabaseDriverMysqli DatabaseDriverMysqli
Demon Digincore
Dirbuster DigitalPebble
Disco Discoverybot
Discobot
DnyzBot DnyzBot
DomainAppender
DomainStatsBot
Download Wonder Download Wonder
ECCP/1.0 Dragonfly
Ebingbong Drip
EirGrabber EMail Siphon
EroCrawler EMail Wolf
Evil Ecxi
Exabot ExtLinksBot
Express WebPictures Extractor
ExtractorPro
Ezooms
FDM
Fimap
Firefox/7.0 Firefox/7.0
Flunky Fuzz
Foobot
Freeuploader
FyberSpider FyberSpider
Fyrebot
GT::WWW
GalaxyBot GalaxyBot
Genieo
GetRight GetRight
GetWeb GetWeb
Go-Ahead-Got-It
Grabber Grabber
GrapeFX GridBot
GrapeshotCrawler
HEADMasterSEO
HTTP::Lite
HTTrack HTTrack
Haansoft Haansoft
HaosouSpider HaosouSpider
Harvest Harvest
Havij Havij
Heritrix
Hloader
HonoluluBot HonoluluBot
HybridBot Humanlinks
IDBTE4M IDBTE4M
IRLbot IRLbot
Iblog InfoTekies
Id-search Intelliseek
IndeedBot InterGET
Iskanie Internet Ninja
IstellaBot JennyBot
JOC Web Spider
JamesBOT
JetCar JetCar
Jetty Jetty
Jorgee Joomla
Kenjin Spider JustView
Keyword Density Keybot Translation-Search-Machine
Kinza
LNSpiderguy LNSpiderguy
LWP::Simple LWP::Simple
LibWeb Lanshanbot
Libwhisker Larbin
Linkdexbot LeechFTP
LinkextractorPro LexiBot
Lftp
Lightspeedsystems
LinkScan
Linkbot
LinkpadBot
LinksManager
LinqiaRSSBot LinqiaRSSBot
Lipperhey Lipperhey
Lipperhey Spider Lipperhey Spider
Lmspider
Ltx71 Ltx71
MFC_Tear_Sample
MIDown tool
MIIxpc MIIxpc
MJ12bot MTRobot
Magnet Mag-Net
Mail.RU_Bot Mail.RU_Bot
MauiBot MarkWatch
Mata Hari
Mb2345Browser Mb2345Browser
Meanpathbot MeanPath Bot
Mediatoolkitbot MegaIndex.ru
Microsoft Data Access Minefield
Mojolicious Mister PiX
MolokaiBot Moblie Safari
Morfeus Fucking Scanner
Mr.4x3 Mr.4x3
Nameprotect Musobot
Nessus NearSite
Net Vampire
NetAnts NetAnts
NetSpider Netcraft
Nettrack
Netvibes Netvibes
NextGenSearchBot Nibbler
NimbleCrawler NimbleCrawler
Ninja Ninja
Nmap Nuclei
OpenVAS Nutch
OrangeSpider Octopus
Page Analyzer Offline Explorer
PageAnalyzer Openfind
Openvas
OrangeBot
OutfoxBot
Pandalytics
Papa Foto Papa Foto
PeoplePal PeoplePal
Petalbot
Pi-Monster
Picscout Picscout
Picsearch Pimonster
Pockey Psbot
ProPowerBot PyCurl
Probethenet Quick-Crawler
Pump
PxBroker
RSSingBot RSSingBot
RankActiveLinkBot Rankivabot
Re-re Re-re
Reaper ReGet
RedesScrapy RebelMouse
RepoMonkey
Ripper Ripper
RocketCrawler RocketCrawler
Rogerbot SBIder
SEOkicks
SEOkicks-Robot SEOkicks-Robot
SEOprofiler SEOprofiler
SISTRIX SEOstats
SMTBot
SalesIntelligent
ScanAlert
Screaming Screaming
SearchmetricsBot ScreenerBot
SentiBot
SeobilityBot
Shodan
Siphon
Site Sucker
SiteCheckerBotCrawler
SiteExplorer
SiteLockSpider SiteLockSpider
SiteSnagger SiteSnagger
SmartDownload SiteSucker
Siteimprove
Snapbot Snapbot
Spammen Snoopy
SpankBot SocialRankIOBot
Sociscraper
Sogou web spider
Sosospider
Spanner Spanner
Spinn3r Steeler
SputnikBot
Sqworm
Stripper
Sucker Sucker
Sucuri
SuperBot SuperBot
Suzuran SuperHTTP
T8Abot SurveyBot
TheNomad Swiftbot
Titan T0PHackTeam
TeleportPro
Telesphorep
The Intraformant
TightTwatBot
Toata
Toweyabot Toweyabot
Trendiction Tracemyfile
True_Robot True_Robot
Turnitin TurnitinBot
TwengaBot URLy Warning
Twice URLy.Warning
UnisterBot
Upflow Upflow
VCI VB Project
Vagabondo Vacuum
VelenPublicWebCrawler VelenPublicWebCrawler
VoidEYE
Voil Voil
WEBDAV Voltron
WBSearchBot
WISENutbot
WPScan
WWW-Collector-E
WWW-Mechanize WWW-Mechanize
WWWOFFLE WWW::Mechanize
Web Auto Web Auto
Web Collage
Web Enhancer
Web Fetch Web Fetch
Web Sauger Web Fuck
Web Pix
Web Sucker Web Sucker
WebFetch WebAuto
WebFuck WebBandit
WebLeacher WebCopier
WebSauger WebImageCollector
WebWhacker WebPix
WebReaper
WebZIP WebZIP
Webalta Website Quester
WebsiteExtractor WebsiteExtractor
Whack WinHTTrack
Whatweb
WiseGuys Robot WiseGuys Robot
Wonderbot Wonderbot
Xaldon_WebSpider Woobot
YoudaoBot Wprecon
Zauba Xaldon WebSpider
Zermelo Xenu
ZoomBot
ZoominfoBot ZoominfoBot
ZyBorg ZyBorg
adscanner clark-crawler
arquivo-web-crawler
arquivo.pt
autoemailspider
backlink-check
coccocbot
crawl.sogou.com crawl.sogou.com
crawler.feedback eCatch
crawler4j
dataforseo.com
demandbase-bot
domainsproject.org
evc-batch
iubenda-radar
magpie-crawler magpie-crawler
meanpathbot muhstik-scan
mediawords page scorer
netEstate NE Crawler
plumanalytics plumanalytics
ripz
s1z.ru s1z.ru
satoristudio.net satoristudio.net
seocompany.store scalaj-http
serpstatbot seobility
spyfu seoscanners
trendiction.de seostar
ubermetrics-technologies.com sexsearcher
voyagerx.com webmeup-crawler
webgains-bot
webprosbot webprosbot
x09Mozilla
x22Mozilla x22Mozilla
xpymep1.exe zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
adultnet.in 2015god.org
ausergrubhof.info aa08daf7e13b6345e09e92f771507fa5f4.com
funponsel.com gribokstop.com
gateway.zscalerone.net larose.jb2c.me
gobongo.info ros-ctm.ru
goldenggames.com social-fun.ru
indo-export.ru topsy.com
lkbennettoutlet.online vroze.com
longadventure.com yoluxuryevents.com
nevansk.ru you-shall-not-pass.is74.ru

932
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
00webcams.com 00author.com
03e.info 10bet.com
08800.top 11pikachu.ru
1bet.com 123locker.com
1flag.co.za 1kdailyprofit.me
2kata.ru 1kinobig.ru
3w24.com 20pascals.nl
404.mn 300richmond.co.nz
5elementov.ru 4ur.click
7fon.ru 6128786.com
7makemoneyonline.com 8lufu.com
83net.jp aad1f4acb0a373420d9b0c4202d38d94fa.com
991mostfm.co.id acgs.tk
aa14ab57a3339c4064bd9ae6fad7495b5f.com addblueoff.com.ua
aanapa.ru adktrailmap.com
aasoldes.fr admeasures.com
abclauncher.com advokateg.ru
abiente.ru
absugars.com
academiacsmendoza.org
acc.eu.org
adf.ly
adidas.frwebs.fr
advokateg.xyz
agecheckadult.com agecheckadult.com
agreda.pluto.ro agroeconom.kz
agysya.ru
ahrntal.verymes.xyz
aimayangzhi.com aimayangzhi.com
akama.com
alert-jdh.xyz
alessandraleone.com
alfa-medosmotr.ru alfa-medosmotr.ru
allbizne.co.ua allwomen.info
alldubai.biz animal-drawings.com
alyeskaresort.com
amospalla.es
anal-acrobats.hol.es
ananumous.ru
anti-virus-removal.info anti-virus-removal.info
apibatbrowsecom-a.akamaihd.net
apidiamondatanet-a.akamaihd.net
apiitsthirteende-a.akamaihd.net
apilinkswiftco-a.akamaihd.net apilinkswiftco-a.akamaihd.net
appartement-stumm.at apparel-offer.com
appfixing.space apple.com-webbrowsing-security.review
apxeo.info apptester.tk
aramaicmedia.org
architecturebest.com
ardimobileinfo.ml ardimobileinfo.ml
arenda-avtoprokat-krasnodar.ru arvut.org
arendatora.ru asrvrep-a.akamaihd.net
astana.xxxkaz.org astana.xxxkaz.org
atmovs.com audiobangout.com
ausmepa.org.au
autodo.info
avek.ru
aviapanda.ru aviapanda.ru
awaybird.ru avto-oligarh.ru
azartmix.com avtocredit-legko.ru
b2b-lounge.com azbukadiets.ru
babyguns.ru azbukafree.com
bag77.ru baditri.com
banki76.ru bdsmgalls.net
bbsoldes.fr beauty-bracelet.com
beachtoday.ru beauty-things.com
bedandbreakfast.com begalka.xyz
bestmarriages.com belsetirehafi.tk
berdasovivan.ru
besthomemadepornsites.com
bestsexyblog.com
bet365.com bet365.com
beyan.host.sk betterhdporn.com
blogqpot.com betterscooter.com
bolitgorlo.net bigpenisguide.com
bonvillan.com blavia.00author.com
boostmyppc.com bloglag.com
bosefux.esy.es bmwhighperformers.com
brandehk.dk boom-celebs.com
buybest1.biz brimstonehillfortress.org
buzz-porno.info briomotor.co
buzzzg.men btnativenav.com
cacheimages.com btvn.ru
callcafe.info buckspar.gq
cand.jp budpost.com.ua
buntube.net
businesn.men
business-made-fun.com
buy-cheap-online.info
buyviagraa.blogspot.com
candlewooddentalcentre.com.au candlewooddentalcentre.com.au
cars-modification.net candycrushshop.com
carson.getenjoyment.net canrioloadm.gq
cartierbracelet.xyz carapics.com
cardul.ru
cartechnic.ru
cartierbraceletsreplica.pw
cartierlove2u.xyz cartierlove2u.xyz
cartierlovebracelet.xyz casinorewards.info
castingbank.ru cayado.snn.gr
censys.io cheapjerseysa.com
centrdebut.ru check-this-out-now.online
cfcl.co.uk checktext.org
chatroulette.world chiblackhawks-jerseys.com
cheapestjerseysonwholesale.com chimiver.info
cheapmoncler.win chinese-amezon.com
cheapwebsitehoster.com christianlouboutinreplica.win
cherrypointplace.ca cibpenonptib.flu.cc
chocolateslim-u-srbiji.com cicaki.net
chuckguilford.com ciceros.co
cialis-samples.com citysecurity.nu
cialiswithoutadoctor.net
ciekawinki.pl
cigarpass.com
civilwartheater.com civilwartheater.com
cleanpcnow.com classicakuhni.ru
cleanersoft.com
clickso.com
clicksor.net clicksor.net
clubfashionista.com co.lumb.co
coldfilm.ru coinsspb.com
compliance-andrew.xyz colehaanoutlet.store
compliance-josh.top com-onlinesupport.host
connectionstrenth.com compliance-alex.top
coprofam.org compliance-fred.xyz
coslab.club computernetworksonline.com
covadhosting.biz constantaservice.net
cookie-law-enforcement-ff.xyz
cool-wedding.net
counter.yadro.ru
country-chic.ru
covid-schutzmasken.de covid-schutzmasken.de
cowblog.fr cowlmash.com
craftburg.ru crazyboost.pro
crazyprotocol.com credit.co.ua
crd.clan.su crystalslot.com
creditmoney.com.ua cscwtalkto.site
creditwell.ru custom-electric-guitar.com
crest3d.ru customsua.com.ua
daisye.top cv.wallhade.co
day-news.info cyse.tk
dentalpearls.com.au ddpills.com
destinationrealestate.com dfwu1019.info
detskie-zabavi.ru dgfitness.co
deutschland123.de digitalfaq.com
dimfour.com diminishedvalueoforegon.com
dimkino.ru dmmspy.com
discovertreasure-a.akamaihd.net docstoc.com
dokfilms.net dokfilms.net
domain2008.com
domainroam.win domainroam.win
drugspowerstore.com dostavka-v-ukrainu.ru
dotmass.top
doxysexy.com
draniki.org
dtm-spain.com dtm-spain.com
dvd-famille.com e-avon.ru
dverimegapolis.ru e-poker-2005.com
eduserver.net e2click.com
eets.net earncash.com.ua
ekn-art.se easytuningshop.ru
elektrozigarette1.dreamwidth.org edge.sharethis.com
elvenmachine.com elektroniksigaraankara.info
energydiet24.ru elmacho.xyz
envaseslotusama.com en.altezza.travel
enbersoft.com
enpolis.ru
enskedesquashclub.se
epngo.bz
eredijovon.com eredijovon.com
eropho.net eropho.com
eropornosex.ru eropornosex.ru
erotiktreff24.info erotikstories.ru
escortslet.net es5.com
eu-cookie-law.blogspot.com euroskat.ru
eurosamodelki.ru everypony.ru
extremez.net exci.ru
eyessurgery.ru
ezigarettekaufen1.hpage.com ezigarettekaufen1.hpage.com
ezigarettenkaufen1.pagina.gr ezigarettekaufen2.yolasite.com
ezofest.sk f-loaded.de
f07.de f012.de
familyholiday.ml falcoware.com
fast-wordpress-start.com falllow.gq
fastcrawl.com fashionindeed.ml
fefo.gdn ferrotodo.com
femdom.twiclub.in findpics.pw
feorina.ru findsexguide.com
filmgo.ru findthe.pet
filmi-onlain.info fingerprintjs.com
financehint.eu fivedwld.cf
flowersbazar.com flprog.com
forex-procto.ru food.dtu.dk
forzeronly.com foodcrafts.website
fotzen-ficken.com forex-indextop20.ru
frameimage.org forminecrafters.ru
free-fb-traffic.com forms-mtm.ru
free-share-buttons.com formulaantiuban.com
free-share-buttons.top forsex.info
free-social-buttons.xyz forum.tvmir.org
freevpn.space foto-telok.net
fres-news.com foxydeal.com
ftns.ru franecki.net
fugarif.ga frequiry.com
gamerextra.com frustrated-favorable.gq
fuckbuddybestgilf.info
fungamelands.com
futbolkisales.ru
gamebackyard.com
gazobeton-p.com.ua
ge0ip.org ge0ip.org
generousdeal-a.akamaihd.net geileweiber.tk
gettpromos.com ghostvisitor.com
gifspics.com
girlporn.ru
gk-atlant.info gk-atlant.info
gkvector.ru golmau.host.sk
globalsurfari.com growmyfunds.ca
gojiberriess.apishops.ru guigyverpo.cf
goodprotein.ru guod.me
grizzlysgrill.com hamptonoaks.ca
gsasearchengineranker.site hdpixion.com
gsasearchenginerankerdiscount.com healgastro.com
gsasearchenginerankerhelp.com health-medical-portal.info
gsmtlf.ru heartofbeijing.blogspot.com
gta-top.ru helicalpile.us
gungamesz.com hentaiheroes.com
hahashka.ru hesteel.pl
handicapvantoday.com highspeed5.net
hasfun.com hit-men.men
hasshe.com hmmm.cz
hauleddes.com
haveinc.xyz
hccoder.info
hebr.myddns-flir.com
herokuapp.com
hetmanship.xyz
heygidday.biz
hildinghr.se
histats.com
hkladys.com
homemade.gq homemade.gq
hotkeys.com hop.clickbank.net
htmlcorner.com hornymatches.com
icaseclub.ru hotel-mkad.ru
id-forex.com hotloans.ru
ifmo.ru hotsocialz.com
ihc2015.info hptwaakw.blog.fc2.com
images.gyffu.com hspline.com
infektsii.com ideashome.id
informatiecentro.be ilikevitaly.com
innodgfdriverhm.aircus.com imgarcade.com
instabid.tech imguramx.pw
intim-uslugi.info imperia31.ru
iradiology.ru infogame.name
irkutsk.zrus.org insomniagamingfestival.com
irunfar.com intellektmedia.at
iscblog.info interior-stickers.ru
itis4you.com invitefashion.com
ix20.ru invivo.hu
japfm.com iqoption.pro
ixora.pro
izismile.com
jav-fetish.site jav-fetish.site
javidol.site javatex.co.id
jerseyssportsshop.com jerseywholesalebizchina.com
johnnyhaley.top jetsli.de
johnrobertsoninc.com
joyceblog.top
jumptap.com jumptap.com
kabbalah-red-bracelets.com justucalling32211123456789.tk
keywordglobal.co.za karusel-market.ru
keywordteam.net kazan.zrus.org
kidskunst.info keenoutlet.online
kinozapas.org kinky-fetishes.com
kleine-titten.biz kinocccp.net
klejonka.info kinoduh.ru
kol-energo.ru konoplisemena.com
konkursowo-24.pl koptims.tiu.ru
kopihijau.info krasivoe-hd.net
kosmetyki.tm.pl krassh.ru
kosynka-games.ru krasula.pp.ua
krasnodar.zrus.org lakomka.com.ua
kskjco.club laptoper.net
lacapilla.info lasvegaslockandsafe.com
ladov.ru laudit.ru
ladsup.com laulini.soclog.se
leadwayau.com ledis.top
lego4x4.ru li-er.ru
lescinq.com lineshops.biz
letsart.ru liveinternet.ru
likrot.com localmatchbook.com
linkbuddies.com
linkdebrideur.xyz
lion.misba.us
livefixer.com
lizace.com
loginduepunti.it
lookingglassemb.com lookingglassemb.com
low-format.ru lotto6888.com
lulea-auktionsverk.se ltvperf.com
lunchrock.co lumb.co
mabdoola.blogspot.com lyrster.com
magnetic-bracelets.ru m0r0zk0-krava.ru
mall.uk macrotek.ru
marktforschung-stuttgart.com madot.onlinewebshop.net
mebel-alait.ru magazin-pics.ru
mega-bony-2017.pl magnetpress.sk
megakino.net mainhp.net
mejoresfotos.eu make-money-online.com
mesto-x.com mararoom.ru
michellblog.online marcoislandvacations.net
microstatic.pl marcowebonyodziezowe.pl
mindeyegames.com marmitaco.cat
minecraft-rus.org mbiologi.ru
mixtapetorrent.com meble-bogart.info
mobioffertrck.com medi-fitt.hu
mockupui.com megahdporno.net
mojowhois.com metallosajding.ru
mole.pluto.ro metaxalonevstizanidine.blogspot.com
moncleroutletonline.win minharevisao.com
monclervests.xyz misswell.net
morefastermac.trade mixx.com
moroccosurfadventures.com mmofreegames.online
mowser.com moesonce.com
moz.com monetizer.com-01.site
mundoaberrante.com morlat.altervista.org
music7s.me movies-in-theaters.net
musicspire.online msk-diplomat.com
myfreecams.com mustat.com
mymercy.info myyour.eu
mywallpaper.top naval.jislaaik.com
nacap.ru nelc.edu.eg
nalogovyy-kodeks.ru nero-us.com
narosty.com nfljerseyscheapbiz.us
nevansk.ru nippon-bearings.ru
newsperuse.com noumeda.com
nfvsz.com npoet.ru
nsatc.net nuit-artisanale.com
o-o-11-o-o.com obiavo.by
o-o-8-o-o.com
obiavo.com
ochistka-stokov.ru ochistka-stokov.ru
officedocuments.net offtopic.biz
ohmyrings.com onlinemeetingnow.com
okout.ru onlinetvseries.me
oneshotdate.com ooo-olni.ru
online-zaymy.ru opinionreelle.com
onlinefilmz.net otbelivanie-zubov.com
osnova3.ru outshop.ru
ostrovtaxi.ru oynat.info
ownshop.win paccohichetoti.ml
ozoz.it paclitor.com
paradontozanet.ru pathwhelp.org
partybunny.ru paydayonlinecom.com
pc-test.net perso.wanadoo.es
pechikamini.ru pestomou.info
pepperstyle.ru photochki.com
perederni.net physfunc.ru
photosaga.info piccdata.com
picsearch.com picturesfrom.com
pintattoos.com picturesify.com
pistonclasico.com pinsdaddy.com
pizdopletka.club pinup-casino1.ru
pobeiranie.pl pixell.club
poisk-zakona.ru pizza-imperia.com
pooleroadmedicalcentre.co.uk plugingeorgia.com
poppen-nw.net pochemychka.net
pornhub-ru.com podshipniki-nsk.ru
pornobrazzers.biz podshipniki-ntn.ru
pornofeuer.com poligon.com
pornogad.com pornosok.ru
pornohub.me pornstartits.xblog.in
pornosee.info pornzone.tv
prizeestates.cricket preconnubial.usuby.site
prizefestival.mobi printdirectforless.com
produkto.net pro-okis.ru
projefrio.com.br prodess.ru
prosmibank.ru prointer.net.ua
proxyradar.com prostitutki-odessa.org
puserving.com purchasepillsnorx.com
pussyfleet.com pussysaga.com
puzo2arbuza.ru qor360.com
pyramidlitho.webs.com rangoman.date
qld10000.net rank-checker.online
quit-smoking.ga realizmobi.com
raavidesigns.com rebelmouse.com
rabotaetvse.ru reelheroes.net
ragecash.com registrationdomainsite.com
randalljhoward.com respublica-otel.ru
rangjued.com restorator-msk.ru
ranksays.com reversing.cc
ranksignals.com rewardpoll.com
rating-bestcasino.com reyel1985.webnode.fr
realting-moscow.ru rospromtest.ru
redbottomheels.xyz runtnc.net
registry-clean-up.net rutor.vip
remont-fridge-tv.ru rvi.biz
remontvsamare.su sad-torg.com.ua
rentalcarnavi.info saddiechoua.com
resultshub-a.akamaihd.net santasgift.ml
rfserial.net scmor.ilxc.cc
riralmolamsaca.tk sdi-pme.com
risparmiocasa.bz.it senger.atspace.co.uk
rostov.xrus.org serptehnika.ru
rrutw.com sexreliz.com
ruclicks.com sexvideo-sex.com
ruf777.com sexxdate.net
rukino.org sherlock.se
ruscoininvest.company ship-marvel.co.ua
rusexy.xyz shopfishing.com.ua
russia-today-video.ru shymkent.xkaz.org
samolet.fr silvermature.net
sbf441.com simple-share-buttons.com
sc-specialhost.com simplepooltips.com
scanner-barak.top sisiynas.ru
scanner-donald.top sixcooler.de
searchengineranker.email slonechka.ru
segol.tv sluganarodu.ru
seofied.com slujbauborki.ru
seokicks.de smadihome.com
seomarketings.online smokewithrabbits.com
serialsway.ucoz.ru
servisural.ru
sethrollins.net
sexkrasivo.net
sexphoto.site
sexuria.net
sexvporno.ru
sexyali.com
sexyebonyteen.com
sharebutton.net
sharebutton.to
sim-service.net
site-speed-check.site
site3.free-share-buttons.com
slimcdn.com
sms2x2.ru sms2x2.ru
smsactivator.ru
snomer1.ru
snowplanes.com snowplanes.com
soc-econom-problems.ru soaksoak.ru
solution4u.com soblaznu.net
sonata-arctica.wz.cz socas.pluto.ro
spacash.com social-buttons.xyz
srdrvp.com social-s-hhh.xyz
staff.prairiesouth.ca socialbuttons.xyz
soda.media
softlinesolutions.me
space2019.top
space4updating.win
ssconstruction.co
stackthatbucks.com
startwp.org startwp.org
stickers-market.ru staticfs.host
stmassage.ru streetfooduncovered.com
strana-solnca.ru streha-metalko.si
strfls.com
stroicol.net stroicol.net
stroiminsk.com stroyhelp-dv.ru
stroiminsk.org
stroyplus.ru stroyplus.ru
success-seo.com subj.ukr-lit.com
sukarame.net surffoundation.nl
superinterstitial.com svetlotorg.ru
superstats.com szamponrevita.pl
survival.betteroffers.review taaaak.com
takeflyte.com tamada69.com
tattoo33.ru teesdaleflyballclub.co.uk
tecspb.ru tehngr.ru
tedxrj.com the-trader.net
thaisamkok.com thebluenoodle.com
themeforest.net thelottosecrets.com
timecrimea.ru thepantonpractice.co.uk
tizanidineduringpregnancy.blogspot.com theprofitsmaker.net
tizanidinepills.blogspot.com thetardistimes.ovh
tizanidinewithdrawal.blogspot.com titslove.yopoint.in
tmtrck.com tizanidinecost.blogspot.com
top-study.work tizanidinedosageforsleep.blogspot.com
torrnada.ru tksn.ru
toys.erolove.in toon-families.com
tradgardspartner.se toondinsey.com
trafficmp.com topsiteminecraft.com
tramadolandtizanidine.blogspot.com torrentgamer.net
tubeline.biz torrents.cd
uasb.ru tour-line.net
urdoot.win toursmaps.com
urlcut.ru traffictrade.life
urll.eu tri-slona.org
us-america.ru triumf-realty.ru
ussearche.cf trustaffs.com
tryrating.com
tula.howotorg.ru
tvand.ru
tversvet.ru
uggsale.online
ugogo.info
ukr-lit.com
ultimateclassicrock.com
unece.org
unpredictable.ga
uprour.com
uptime-delta.net
ustion.ru
utrolive.ru
uzporno.mobi
v24s.net
v720hd.ru
vacuumscleaner.com
validdomain.xyz validdomain.xyz
vbabule.net vandrie-ict.nl
ventelnos.com vashsvet.com
vereo.eu vduplo.ru
vezuviy.su venta-prom.ru
via-energy-cumpara.com veopornogratis.xxx
video-camer.com via-energy-order.com
videooko.weebly.com videochat.ph
viel.su videositename.com
vip.51.la vikistars.com
vipsexfinders.com vilingstore.net
vizitki.net vip2ch.com
wallpapersist.com viveresaniesnelli.it
waycash.net vk-mus.ru
webmasterhome.cn vkontarkte.com
website-analytics.online vladhistory.com
website-stealer.nufaq.com vodkoved.ru
weburlopener.com waterefficiency.co
wedding0venues.tk waterpurifier.club
weightatraining.com watsonrealtycorp.com
whatsappdownload10.com we-are-gamers.com
whatsupinfoley.com webinstantservice.com
williamrobsonproperty.com webix.me
winterclassichockeyjerseys.com webpromotion.ae
wiosenny-bon-1500.pl website-datenbank.de
woman-h.ru website-speed-up.site
woman-tampon.ru websiteaccountant.de
wosik-dach.service-for-web.de wechatdownload10.com
wovis.site weprik.ru
wowcasinoonline.ooo wetgames.ru
www.arenda-yeisk.ru whatsappbot.flyland.ru
www.bookmaker-bets.com white-truck.life
x-rates.ru whiteproduct.com
xingzi-vision.com wildnatureimages.com
xlovecam.com wineration.com
xn----7sbaphztdjeboffeiof6c.xn--p1ai worldhistory.biz
xn----9sbebi2bvzr7h.xn--p1ai woweb.com.ua
xn--80aaaks3bbhabgbigamdr2h.xn--p1ai ww2awards.info
xn-----6kccaibs5cb8afhjrfmix2n.xn--p1ai
xn----7sbaaabaei0cc8aj5bj0bncejx.xn--p1ai
xn--80aanaardaperhcem4a6i.com xn--80aanaardaperhcem4a6i.com
xn--b1ag5cfn.xn--p1ai xn--80aikhbrhr.net
xn--d1abj0abs9d.in.ua xnxx-n.com
xn--oogle-wmc.com yachts-cruise.info
y8games-free.com yellowstonevisitortours.com
yellowads.men yoshkarola.zrus.org
yginekologa.com za-music.mymobiplanet.com
youandcredit.ru zaloro.com
youradexchange.com zarajbuilders.com
yousense.info zebradudka.com
za-fun-offer.com zverokruh-shop.cz
zakon-ob-obrazovanii.ru zvezdagedon.ru
zakonobosago.ru zx6.ru
zarabotok--doma.ru
zhorapankratov7.blogspot.com
zigarettenonlinekaufen.tumblr.com

486
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
03p.info 0daymusic.org
1688.com 1-99seo.com
1kinobig.ru 11pikachu.ru
4k-player.pl 1314dh.com
883zy.com 1xbet4you.com
aa9d046aab36af4ff182f097f840430d51.com 24videos.tv
aadroid.net 2itech.ru
abclauncher.com 2rich4bitches.com
akuhni.by 6hopping.com
all4invest.ru 7minuteworkout.com
all4wap.ru DomainStatsBot
allforminecraft.ru academiacsmendoza.org
allkrim.com aerogo.com
alpinism.ru afftrack001.com
am-se.com amazingpic.net
anonymousfox.co an-donut.com
apelsinnik.website android4fun.org
apibatbrowsecom-a.akamaihd.net apiappenableinfo-a.akamaihd.net
apimountainbikei-a.akamaihd.net
apiqualitinknet-a.akamaihd.net
appmsr.org
aramaicmedia.org aramaicmedia.org
architecturebest.com artdestshop.eu
asrvvv-a.akamaihd.net asdfz.pro
autoloans.com asrv-a.akamoihd.net
bambi.ck.ua autoua.top
belsetirehafi.tk avon-novosib.ru
best-seo-software.xyz beatifullandscape.co
bestdraws.com bestmobilityscooterstoday.com
bestofpicture.com bestofferswalkmydogouteveryday.gq
bfz.biz besttorrentknifta.weebly.com
bgviagrachrx.com bidbuy.co.kr
bimmerpost.com bitporno.sx
bjetjt.com bonkers.name
bkgr.se breastaugmentation.co.za
bloglag.com bronzeaid-a.akamaihd.net
blogs.rediff.com bus-offer.com
blue-square.biz buybest1.biz
bmwhighperformers.com buyparajumpers.online
bocoarchives.org buypillsorderonline.com
boo-arts.com buyscabiescream.com
buyantiviralwp.com cbox.ws
c-gainsbourg.com cheapbelstaff.online
callmd5map.com cherubinimobili.it
catalogs-parts.com children-learningreading.info
citizenclsdriveri7.pen.io cleanmypc.com
cmhomestayagency.com compliance-george.xyz
coaching-netz.info compliance-mark.top
compliance-alex.xyz coolwallpapers4k.info
compliance-barak.top covetnica.com
compliance-olga.top dailyrank.net
cookie-law-enforcement-ff.xyz datract.com
coolyarddecorations.com demenageur.com
crazyprotocol.com detective01.ru
creams.makeforum.eu digital-sale.su
creatives.adbetclickin.pink dikx.gdn
creditwell.ru dkvorota.ru
dandiyabeats.in doctormakarova.ru
daptravel.com domainroam.win
ddlmega.net dostavimvdom.ru
doeco.ru doublepimp.com
dojki-hd.com download-of-the-warez.blogspot.com
domination.ml drillsaw.ru
dosugrostov.site drugstoreforyou.com
drev.biz eaplay.ru
drugs-no-rx.info easyshoppermac.com
e-biznes.info eets.net
ekobata.ru efkt.jp
elektrischezigaretten1.blogse.nl ekaterinburg.xrus.org
elexies.info elidelcream.weebly.com
englishdictionaryfree.com emailaccountlogin.co
etur.ru empis.magix.net
exct.net errorfixing.space
expdom.com essay-edu.biz
extrabot.com etenininrade.ga
ezigarettekaufen2.blox.pl eyelike.com.ua
f-online.de f1nder.org
falcoware.com facebook-mobile.xyz
ferrotodo.com fakehandbags.xyz
fil.ru fashion.stellaconstance.co
filmgo.ru fetroshok.ru
finuse.com finstroy.net
fivedwld.cf flauntyoursite.com
flashbannernow.com flirt4free.com
fm-upgrade.ru forensicpsychiatry.ru
free-gluten.ru foxjuegos.com
freefoto.ca free-share-buttons.com
front.to freshberry.com.ua
gamblingnerd.com funtoonez.com
ghostvisitor.com gagrasector.ru
gidonline.one gameonasia.com
gorabagrata.ru gamewrath.com
gpirate.com gateway.zscalertwo.net
gratis-sexkontakte.com ghazel.ru
greamimgo.pw global-ics.co.za
guigyverpo.cf gomusix.com
guildebzh.info
gwebtools.com
gz2.bbsoldes.fr
hatedriveapart.com hatedriveapart.com
ictizanidinehcl4mg.blogspot.com hawaiielectriclight.com
ideibiznesa2015.ru helicalpile.us
iframe-toloka.com hikesearch.net
ihc2015.info howmuchdoestizanidinecost.blogspot.com
imagui.eu hplaserjetpdriver8y.pen.io
imk.com.ua hystersister.com
incanto.in.ua ideawheel.com
informatiecentro.be ideoworld.org
infoupdate.org impisr.ru
iskalko.ru increasewwwtraffic.info
ispac.org infodocsportal.com
japan-bearings.ru investpamm.ru
jerseywholesalechinabiz.com ipchicken.com
jikoman.info iqbazar.ru
jwcialislrt.com izoll.ru
kinoduh.ru javrip.net
kirov.zrus.org jpcycles.com
klumba55.ru kakablog.net
kosmetyki.tm.pl kamorel.com
laapp.com kenaba.su
labvis.host.sk ketoanhanoi.info
lancheck.net keywordbasket.com
letmacworkfaster.world keywordteam.net
libertybilisim.com kimcurlrvsms.com
lignofix.ua kinoplen.ru
linkwithin.com konturkrasoty.ru
locustdesign.co kosynka-games.ru
magicalfind-a.akamaihd.net krasnodar.zrus.org
maglid.ru lafourmiliaire.com
maladot.com lampokrat.ws
medanestesia.ru law-check-nine.xyz
mikozstop.com lietaer.com
minharevisao.com lion.conboy.us
liveu.infoteka.hu
louisvuittonoutletstore.net
m5home.ru
marcowebonyodziezowe.pl
maximilitary.ru
mediawhirl.net
mega-polis.biz.ua
mikrobiologies.ru
mir-business-24.ru mir-business-24.ru
mmoguider.ru mojpregled.com
mmstat.com moncleronline.xyz
mobplayer.ru moncleroutletonline.win
mp3s.club
multgo.ru multgo.ru
musezone.ru myhydros.org
mustat.com myonigroup.com
my-floor.in.ua myshopmatemac.com
mystats.xyz najaden.no
naperehresti.info narkologiya-orel.ru
netanalytics.xyz news-speaker.com
netoil.no niki-mlt.ru
nomuos.it nordstar.pro
normalegal.ru notfastfood.ru
norththeface.store onefilms.net
noscrapleftbehind.co onlyporno.ru
notasprensa.info oops-cinema.ru
nudejapan.net optom-deshevo.ru
nudo.ca ostrovtaxi.ru
nuker.com pc-virus-d0l92j2.pw
olvanto.ru pepperstyle.ru
onlinadverts.com perfectpracticeweb.com
onlinewritingjobs17.blogspot.ru photostudiolightings.com
osoznanie-narkotikam.net picscout.com
ourtherapy.ru pixgood.com
partner-high.men platezhka.net
partner-trustworthy.men playfortuna-play.ru
pcimforum.com poems.com.ua
pdn-4.com pojdelo.weebly.com
pdns.cz pokemongooo.ml
perederni.net polimga.pw
pierrehardysale.online pons-presse.com
pinkduck.ga
pony-business.com pony-business.com
popugauka.ru porndroids.com
pourvous.info privacylocationforloc.com
pricheskaonline.ru privatamateure.com
produkto.net prizefestival.mobi
prokotov.com professionaldieselcare.com
promgirldresses.xyz prombudpostach.com.ua
proposal-engine.com prostitutki-novgoroda.org
prostitutki-almata.org puzzleweb.ru
prostitutki-astana.org qpypcx.com
psa48.ru rangoman.date
push-ad.com rewardit.com
puteshestvennik.com rozalli.com
pyramidlitho.webs.com ruscoininvest.company
rankexperience.com russkoe-zdorovie.ru
rasteniya-vs-zombi.ru savetubevideo.com
resellerclub.com scanner-george.top
reversing.cc scanner-julia.top
rfserial.net scrapinghub.com
robotixix.com
roll123.com
rootandroid.org
runstocks.com
s-iwantyou.com
s1z.ru
saecsa.co
sanatorrii.ru
sarafangel.ru
sarahmilne.top
scanner-mark.top
seccioncontrabajo.com
secretscook.ru secretscook.ru
seeresultshub-a.akamaihd.net seofirmreviewsus.info
seoexperimenty.ru seorank.info
sex-sex-sex5.com serw.clicksor.com
sex-tracker.com sethrollins.net
sexkontakte-seite.com sexsaoy.com
shillyourcoins.com sexystrippe.info
sitevaluation.com shisha-swag.de
skachat-besplatno-obrazcy.ru sibtest.ru
skatestick.bid sinestesia.host.sk
soda.media soft-terminal.ru
srdrvp.com soietvousmaime.fr
stairliftsarea.com spabali.org
students-cheapskate.ml sport-video-obzor.ru
spywarespy.com
staff.prairiesouth.ca
stardevine.com
studiokamyk.com.pl studiokamyk.com.pl
svetlotorg.ru subj.ukr-lit.com
szamponrevita.pl theporndude.com
taqplayer.info tizanidineshowupondrugtest.blogspot.com
thenews-today.info tizanidinesideeffectsweightloss.blogspot.com
tizanidineandcipro.blogspot.com tnctrx.com
tizanidineformigraines.blogspot.com torrents-tracker.com
traffic2money.com toys.erolove.in
turizmus.us trackzapper.com
u555u.info trafficnetzwerk.de
uhod-za-sobakoj.ru truck-addzilla.life
uni.me tzritel.tk
urll.eu ugguk.online
urlopener.blogspot.com.au usacasino.com
vapsy.com vavilone.com
via-energy-cumpara.com vesnatehno.com
video-production.com videochat.ph
videosbox.ru videooko.weebly.com
vilingstore.net videtubs.pl
vitalads.net vtc.pw
vltai.com vysigy.su
vpnmouse.com walkme.com
vzlomsn.org walpaperlist.com
waterefficiency.co website-stealer.nufaq.com
webs.com websiteaccountant.de
website-speed-check.site woman-orgasm.ru
website-speed-up.top wygraj-skiny.win
wpthemedetector.co.uk xbaboon.com
wstroika.ru xclicks.net
xboxster.ru xcombear.ru
xn--d1aifoe0a9a.top xn----7sbifcamovvfggw9d.xn--p1ai
xxxasianporn.net xn--80aodinpgi.xn--p1ai
yopoint.in xxxnatelefon.ru
yourothersite.com yoopsie.com
youtubologia.it yoursearch.me
yugk.net yoursite.com
za-music.mymobiplanet.com yuarra.pluto.ro
zaapplesales.blogspot.com yun56.co
zacreditom.ru zambini.ru
zaloadi.ru zazagames.org
zona-aqua.ru zdesformula.ru
zrizvtrnpale.tk zeroredirect11.com
zooggames.com

14
dev-tools/test_units/ratelimittest-beta.txt
View file

@ -1,11 +1,21 @@
<html> <!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><html>
<head><title>503 Service Temporarily Unavailable</title></head> <head><title>503 Service Temporarily Unavailable</title></head>
<body> <body>
<center><h1>503 Service Temporarily Unavailable</h1></center> <center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center> <hr><center>nginx/1.18.0 (Ubuntu)</center>
</body> </body>
</html> </html>
ml><html> <html>
<head><title>503 Service Temporarily Unavailable</title></head> <head><title>503 Service Temporarily Unavailable</title></head>
<body> <body>
<center><h1>503 Service Temporarily Unavailable</h1></center> <center><h1>503 Service Temporarily Unavailable</h1></center>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.240 0; 10.1.0.71 0;
127.0.0.1 0; 127.0.0.1 0;

8
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.04.3027 ### Version: V4.2022.04.3028
### Updated: Wed Apr 6 22:01:21 UTC 2022 ### Updated: Thu Apr 7 07:12:39 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -501,8 +501,6 @@ User-agent: LinkWalker
Disallow:/ Disallow:/
User-agent: Linkbot User-agent: Linkbot
Disallow:/ Disallow:/
User-agent: Linkdexbot
Disallow:/
User-agent: LinkextractorPro User-agent: LinkextractorPro
Disallow:/ Disallow:/
User-agent: LinkpadBot User-agent: LinkpadBot
@ -1201,6 +1199,8 @@ User-agent: isitwp.com
Disallow:/ Disallow:/
User-agent: iubenda-radar User-agent: iubenda-radar
Disallow:/ Disallow:/
User-agent: linkdexbot
Disallow:/
User-agent: lwp-request User-agent: lwp-request
Disallow:/ Disallow:/
User-agent: lwp-trivial User-agent: lwp-trivial

8
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.04.3027 ### Version: V4.2022.04.3028
### Updated: Wed Apr 6 22:01:21 UTC 2022 ### Updated: Thu Apr 7 07:12:39 UTC 2022
### Bad Bot Count: 637 ### Bad Bot Count: 637
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -501,8 +501,6 @@ User-agent: LinkWalker
Disallow:/ Disallow:/
User-agent: Linkbot User-agent: Linkbot
Disallow:/ Disallow:/
User-agent: Linkdexbot
Disallow:/
User-agent: LinkextractorPro User-agent: LinkextractorPro
Disallow:/ Disallow:/
User-agent: LinkpadBot User-agent: LinkpadBot
@ -1201,6 +1199,8 @@ User-agent: isitwp.com
Disallow:/ Disallow:/
User-agent: iubenda-radar User-agent: iubenda-radar
Disallow:/ Disallow:/
User-agent: linkdexbot
Disallow:/
User-agent: lwp-request User-agent: lwp-request
Disallow:/ Disallow:/
User-agent: lwp-trivial User-agent: lwp-trivial