vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-01 18:19:55 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2025.01.4983 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2025-01-19 06:40:00 +00:00
parent b3c20aec7f
commit 997e356f2a
40 changed files with 71088 additions and 70952 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2025.01.4982 #### Version: V4.2025.01.4983
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [668](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [675](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2025.01.4982 #### Version: V4.2025.01.4983
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [668](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [675](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -14,9 +14,9 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2025.01.4982 #### Version: V4.2025.01.4983
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [668](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [675](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

11780
_generator_lists/bad-ip-addresses.list
View file

File diff suppressed because it is too large Load diff

14
_generator_lists/bad-user-agents.list
View file

@ -1,10 +1,3 @@
SiteAuditBot
SemrushBot-BA
SemrushBot-SI
SemrushBot-SWA
SplitSignalBot
SemrushBot-OCOB
SemrushBot-FT
01h4x.com 01h4x.com
360Spider 360Spider
404checker 404checker
@ -428,6 +421,11 @@ SeekportBot
SemanticJuice SemanticJuice
Semrush Semrush
SemrushBot SemrushBot
SemrushBot-BA
SemrushBot-FT
SemrushBot-OCOB
SemrushBot-SI
SemrushBot-SWA
SentiBot SentiBot
SenutoBot SenutoBot
SeoCherryBot SeoCherryBot
@ -436,6 +434,7 @@ SeobilityBot
Seomoz Seomoz
Shodan Shodan
Siphon Siphon
SiteAuditBot
SiteCheckerBotCrawler SiteCheckerBotCrawler
SiteExplorer SiteExplorer
SiteLockSpider SiteLockSpider
@ -463,6 +462,7 @@ Spbot
Spider_Bot Spider_Bot
Spider_Bot/3.0 Spider_Bot/3.0
Spinn3r Spinn3r
SplitSignalBot
SputnikBot SputnikBot
Sqlmap Sqlmap
Sqlworm Sqlworm

4
_google_analytics_ghost_spam/README.md
View file

@ -3,9 +3,9 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2025.01.4982 #### Version: V4.2025.01.4983
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [668](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [675](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

11793
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

11793
conf.d/globalblacklist-testing-version.conf
View file

File diff suppressed because it is too large Load diff

11793
conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

7
dev-tools/_robots_input/robots-input.txt
View file

@ -421,6 +421,11 @@ SeekportBot
SemanticJuice SemanticJuice
Semrush Semrush
SemrushBot SemrushBot
SemrushBot-BA
SemrushBot-FT
SemrushBot-OCOB
SemrushBot-SI
SemrushBot-SWA
SentiBot SentiBot
SenutoBot SenutoBot
SeoCherryBot SeoCherryBot
@ -429,6 +434,7 @@ SeobilityBot
Seomoz Seomoz
Shodan Shodan
Siphon Siphon
SiteAuditBot
SiteCheckerBotCrawler SiteCheckerBotCrawler
SiteExplorer SiteExplorer
SiteLockSpider SiteLockSpider
@ -456,6 +462,7 @@ Spbot
Spider_Bot Spider_Bot
Spider_Bot/3.0 Spider_Bot/3.0
Spinn3r Spinn3r
SplitSignalBot
SputnikBot SputnikBot
Sqlmap Sqlmap
Sqlworm Sqlworm

2
dev-tools/buildnumber
View file

@ -1 +1 @@
4982 4983

11793
dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

11793
dev-tools/globalblacklist.template
View file

File diff suppressed because it is too large Load diff

2
dev-tools/nginxv1.txt
View file

@ -1 +1 @@
nginx version: nginx/1.18.0 (Ubuntu) nginx version: nginx/1.24.0 (Ubuntu)

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.157.4.87 0; 20.25.193.37 0;

11793
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.5 1; 10.1.0.152 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.5 0; 10.1.0.152 0;
127.0.0.1 0; 127.0.0.1 0;

11793
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.5 1; 10.1.0.152 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.5 0; 10.1.0.152 0;
127.0.0.1 0; 127.0.0.1 0;

11793
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.5 1; 10.1.0.152 1;
127.0.0.1 1; 127.0.0.1 1;

7
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -482,6 +482,11 @@
"~*(?:\b)SemanticJuice(?:\b|)" 0; "~*(?:\b)SemanticJuice(?:\b|)" 0;
"~*(?:\b)Semrush(?:\b|)" 0; "~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0; "~*(?:\b)SemrushBot(?:\b|)" 0;
"~*(?:\b)SemrushBot-BA(?:\b|)" 0;
"~*(?:\b)SemrushBot-FT(?:\b|)" 0;
"~*(?:\b)SemrushBot-OCOB(?:\b|)" 0;
"~*(?:\b)SemrushBot-SI(?:\b|)" 0;
"~*(?:\b)SemrushBot-SWA(?:\b|)" 0;
"~*(?:\b)SentiBot(?:\b|)" 0; "~*(?:\b)SentiBot(?:\b|)" 0;
"~*(?:\b)SenutoBot(?:\b|)" 0; "~*(?:\b)SenutoBot(?:\b|)" 0;
"~*(?:\b)SeoCherryBot(?:\b|)" 0; "~*(?:\b)SeoCherryBot(?:\b|)" 0;
@ -490,6 +495,7 @@
"~*(?:\b)Seomoz(?:\b|)" 0; "~*(?:\b)Seomoz(?:\b|)" 0;
"~*(?:\b)Shodan(?:\b|)" 0; "~*(?:\b)Shodan(?:\b|)" 0;
"~*(?:\b)Siphon(?:\b|)" 0; "~*(?:\b)Siphon(?:\b|)" 0;
"~*(?:\b)SiteAuditBot(?:\b|)" 0;
"~*(?:\b)SiteCheckerBotCrawler(?:\b|)" 0; "~*(?:\b)SiteCheckerBotCrawler(?:\b|)" 0;
"~*(?:\b)SiteExplorer(?:\b|)" 0; "~*(?:\b)SiteExplorer(?:\b|)" 0;
"~*(?:\b)SiteLockSpider(?:\b|)" 0; "~*(?:\b)SiteLockSpider(?:\b|)" 0;
@ -517,6 +523,7 @@
"~*(?:\b)Spider_Bot(?:\b|)" 0; "~*(?:\b)Spider_Bot(?:\b|)" 0;
"~*(?:\b)Spider_Bot/3.0(?:\b|)" 0; "~*(?:\b)Spider_Bot/3.0(?:\b|)" 0;
"~*(?:\b)Spinn3r(?:\b|)" 0; "~*(?:\b)Spinn3r(?:\b|)" 0;
"~*(?:\b)SplitSignalBot(?:\b|)" 0;
"~*(?:\b)SputnikBot(?:\b|)" 0; "~*(?:\b)SputnikBot(?:\b|)" 0;
"~*(?:\b)Sqlmap(?:\b|)" 0; "~*(?:\b)Sqlmap(?:\b|)" 0;
"~*(?:\b)Sqlworm(?:\b|)" 0; "~*(?:\b)Sqlworm(?:\b|)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.5 0; 10.1.0.152 0;
127.0.0.1 0; 127.0.0.1 0;

11793
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.157.4.87 0; 20.25.193.37 0;

11793
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.157.4.87 0; 20.25.193.37 0;

11793
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

File diff suppressed because it is too large Load diff

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.5 1; 10.1.0.152 1;
127.0.0.1 1; 127.0.0.1 1;

7
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -482,6 +482,11 @@
"~*(?:\b)SemanticJuice(?:\b|)" 0; "~*(?:\b)SemanticJuice(?:\b|)" 0;
"~*(?:\b)Semrush(?:\b|)" 0; "~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0; "~*(?:\b)SemrushBot(?:\b|)" 0;
"~*(?:\b)SemrushBot-BA(?:\b|)" 0;
"~*(?:\b)SemrushBot-FT(?:\b|)" 0;
"~*(?:\b)SemrushBot-OCOB(?:\b|)" 0;
"~*(?:\b)SemrushBot-SI(?:\b|)" 0;
"~*(?:\b)SemrushBot-SWA(?:\b|)" 0;
"~*(?:\b)SentiBot(?:\b|)" 0; "~*(?:\b)SentiBot(?:\b|)" 0;
"~*(?:\b)SenutoBot(?:\b|)" 0; "~*(?:\b)SenutoBot(?:\b|)" 0;
"~*(?:\b)SeoCherryBot(?:\b|)" 0; "~*(?:\b)SeoCherryBot(?:\b|)" 0;
@ -490,6 +495,7 @@
"~*(?:\b)Seomoz(?:\b|)" 0; "~*(?:\b)Seomoz(?:\b|)" 0;
"~*(?:\b)Shodan(?:\b|)" 0; "~*(?:\b)Shodan(?:\b|)" 0;
"~*(?:\b)Siphon(?:\b|)" 0; "~*(?:\b)Siphon(?:\b|)" 0;
"~*(?:\b)SiteAuditBot(?:\b|)" 0;
"~*(?:\b)SiteCheckerBotCrawler(?:\b|)" 0; "~*(?:\b)SiteCheckerBotCrawler(?:\b|)" 0;
"~*(?:\b)SiteExplorer(?:\b|)" 0; "~*(?:\b)SiteExplorer(?:\b|)" 0;
"~*(?:\b)SiteLockSpider(?:\b|)" 0; "~*(?:\b)SiteLockSpider(?:\b|)" 0;
@ -517,6 +523,7 @@
"~*(?:\b)Spider_Bot(?:\b|)" 0; "~*(?:\b)Spider_Bot(?:\b|)" 0;
"~*(?:\b)Spider_Bot/3.0(?:\b|)" 0; "~*(?:\b)Spider_Bot/3.0(?:\b|)" 0;
"~*(?:\b)Spinn3r(?:\b|)" 0; "~*(?:\b)Spinn3r(?:\b|)" 0;
"~*(?:\b)SplitSignalBot(?:\b|)" 0;
"~*(?:\b)SputnikBot(?:\b|)" 0; "~*(?:\b)SputnikBot(?:\b|)" 0;
"~*(?:\b)Sqlmap(?:\b|)" 0; "~*(?:\b)Sqlmap(?:\b|)" 0;
"~*(?:\b)Sqlworm(?:\b|)" 0; "~*(?:\b)Sqlworm(?:\b|)" 0;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Crawling at Home Project EyeNetIE
FDM Go-Ahead-Got-It
Foobot InterGET
Navroad Larbin
NetMechanic Majestic12
SuperHTTP Niki-bot
TightTwatBot SEOkicks-Robot
Vacuum SenutoBot
omgili Snapbot
s1z.ru VCI

172
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
360Spider 80legs
ALittle Client Abonti
ASPSeek Aboundex
AiHitBot Aboundexbot
AllSubmitter Acunetix
Badass BacklinksExtendedBot
Bitacle Bolt
Brandwatch Cegbfeieh
CheTeam
Copier Copier
Crescent CopyRightCheck
CyotekWebCopy DBLBot
DIIbot Demon
DTS Agent DigitalPebble
Disco DnBCrawler-Analytics
DittoSpyder
DomainCrawler
DomainSigmaCrawler
Domains Project Domains Project
Ebingbong Evil
Ecxi ExtractorPro
Express WebPictures FacebookBot
FDM FlashGet
Fuzz Foobot
GT::WWW GalaxyBot
Getintent Gigabot
GoZilla Gotit
GrapeshotCrawler Grafula
Haansoft HMView
HybridBot Heritrix
Iblog InfoNaviRobot
IlseBot
InfoTekies
Information Security Team InfraSec Scanner Information Security Team InfraSec Scanner
InternetSeer Internet Ninja
Jbrofuzz
JetCar JetCar
Jorgee Jetty
LeechGet Jyxobot
Kinza
LieBaoFast
LinksManager LinksManager
MIDown tool Lipperhey
Mag-Net Ltx71
MarkWatch Meanpathbot
Mata Hari Mediatoolkitbot
MegaIndex.ru Mojeek
Microsoft URL Control
Mister PiX
Moblie Safari
Mojolicious Mojolicious
NetMechanic Mr.4x3
OrangeSpider Name Intelligence
PageAnalyzer Needle
PleaseCrawl Ninja
Psbot Page Analyzer
QueryN Metasearch PageGrabber
Piepmatz
Pixray
ProPowerBot
PxBroker
Quick-Crawler Quick-Crawler
Re-re
RealDownload RealDownload
Recorder RebelMouse
Screaming SBIder
SeoSiteCheckup SEOkicks-Robot
Shodan Seekport
SemrushBot-SI
SeoCherryBot
SiteSnagger
Siteimprove Siteimprove
SlySearch Spammen
Spbot SurveyBot
Spinn3r Swiftbot
Sqworm Szukacz
Suzuran T0PHackTeam
T8Abot
Teleport Teleport
TheNomad Thumbor
Trendictionbot TightTwatBot
Turingos Vacuum
Twice VoidEYE
Typhoeus
URLy Warning
URLy.Warning
VelenPublicWebCrawler
WASALive-Bot WASALive-Bot
WPScan WPScan
Web Collage WWW-Collector-E
WebFetch WebAuto
WebCollage
WebEnhancer
WebGo IS WebGo IS
WebImageCollector WebPix
WebSauger WebReaper
WinHTTrack
Wonderbot Wonderbot
YoudaoBot Xaldon WebSpider
Zermelo adscanner
ZoominfoBot coccocbot
archive.org_bot com.plumanalytics
arquivo.pt dataforseo.com
crawler.feedback eCatch
linkfluence gopher
openai heritrix
ripz iubenda-radar
seostar lwp-trivial
serpstatbot meanpathbot
tAkeOut muhstik-scan
x09Mozilla pcBrowser
xpymep1.exe satoristudio.net
seocompany.store
siteripz
trendiction.de
webgains-bot

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
1-99seo.com avkzarabotok.com
agysya.ru christianlouboutinsaleoutletonline.us
evogarage.com insider.pro
fermersovet.ru insurple.com
fsalas.com m5home.ru
ihc2015.info mafa-free.com
larger.io rusprostitute.com
pooleroadmedicalcentre.co.uk sitopreferito.it
softxaker.ru symbaloo.com
top-karkas.ru trk-4.net

198
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
03e.info 11235813.webzdarma.cz
3th.co.in 3th.co.in
4k-player.pl 3xforum.ro
adrunnr.com 76brighton.co.uk
aktobe.xkaz.org 7makemoneyonline.com
alfa-medosmotr.ru activepr.ru
allbizne.co.ua adnotbad.com
amt-k.ru affrh2015.com
animaltoplist.com akusherok.ru
anonymizeme.pro ameblo.jp
apiportalmorecom-a.akamaihd.net antons-transporte.de
arendakvartir.kz applicationg29.com
ayakino.net arendas.net
azlex.uz bankiem.pl
beta.hotkeys.com besthomemadepornsites.com
beyan.host.sk bestimagecoollibrary.com
bizlist.com.de bestofupload.info
bryansk.zrus.org big-boards.info
buypillsorderonline.com billiard-classic.com.ua
carstrends2015.com blpmovies.com
cartierbraceletsreplica.pw boostmyppc.com
cb.iphantom.com cah.io.community
cgi2.nintendo.co.jp cardsdumps.com
chiblackhawks-jerseys.com caulderwoodkennels.com
chuckguilford.com cheapmoncler.win
decorationspcs.com circlesl.com
doeco.ru clips.ua.ac.be
editors.choice6912650.hulfingtonpost.com copblock.org
escort-russian.com cranly.net
ezofest.sk cxpromote.com
free411games.com dfwu1019.info
frustrated-favorable.gq domoysshop.ru
galleryawesome.com ec-file.info
get-free-social-traffic.com erotikchat-24.com
gotcher.us essayassist.com
gratis-sexkontakte.com etotupo.ru
hasshe.com fenoyl.batcave.net
heliko.no film-one.ru
himazin.info fun-mobi.pl
hotkeys.com goodnightjournal.com
ifirestarter.ru hdhc.ru
iframe-toloka.com hermesbracelets.xyz
images-free.net horoshieokna.com
iptool.xyz hotcore.info
istizanidinelikexanax.blogspot.com impresagaia.it
javspace.net jerseyssportsshop.com
karting196.ru jimmychoosale.online
kovrenok.ru jovencitas.gratis
liran-locks.com kino-ecran.ru
liupis.com laapp.com
long-beach-air-conditioning.com lafourmiliaire.com
monitorwebsitespeed.com liky.co.ua
nelc.edu.eg live-sexcam.tk
neobux-bg.info marwer.info
office2web.com medictube.ru
online-sbank.ru merryhouse.co.uk
online.ktc45.ru mnogolok.info
onlineku.com nmrk.ru
par-fallen.ga novosti-hi-tech.ru
pl.aasoldes.fr onblastblog.online
ploenjitmedia.azurewebsites.net onlineporno.site
poem-paying.gq oz-offers.com
polybuild.ru pattersonsweb.com
prod2016.com pintattoos.com
prostitutki-astana.org platinumdeals.gr
prostitutki-tyumeni.org pokemongooo.ml
puzzleweb.ru pornosmola.info
ranksignals.com ranksonic.org
razleton.com rock-cafe.info
rebuildermedical.com royal-betting.net
reimageplus.com rumamba.com
replica-watch.ru sax-sex.com
rightenergysolutions.com.au selectads.men
royalads.net seoexperimenty.ru
savingsslider-a.akamaihd.net serviporno.com
scanner-jane.top simon3.ru
seansonline24.pl social-fun.ru
secretscook.ru sosdepotdebilan.com
securesmrt-dt.com sport-video-obzor.ru
sel-hoz.com sribno.net
seofirmreviewsus.info stop-zavisimost.com
sexy.babes.frontend-stack.top svarbit.com
shell-pmr.ru tecspb.ru
smartpet.ru telefonsex-ohne0900.net
startufa.ru timer4web.com
strongsignal-a.akamaihd.net tizanidinedrugtest.blogspot.com
svensk-poesi.com tsstcorpcddvdwshbbdriverfb.aircus.com
tako3.com tsyndicate.com
telefonsexkostenlos.tk turist-strani.ru
titan-cloud.life ublaze.ru
titelhelden.eu ukrobstep.com
top10-way.com vapeface.club
truck-rece.life veerotech.com
videotuber.ru vesnatehno.com
www.wohnkabinen-shop.de vk-mus.ru
xn--b1adccaf1bzj.xn--p1ai vmnmvzsmn.over-blog.com
xvideosporn.biz writingservices17.blogspot.ru
ynymnwbm.bloger.index.hr xn--80ahdheogk5l.xn--p1ai
youjizz.com yopoint.in
zed21.net zytpirwai.net

10
dev-tools/test_units/ratelimittest-master.txt
View file

@ -878,4 +878,14 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html> </html>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.5 0; 10.1.0.152 0;
127.0.0.1 0; 127.0.0.1 0;

20
dev-tools/tmp/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2025.01.4982 ### Version: V4.2025.01.4983
### Updated: Mon Jan 6 10:04:07 UTC 2025 ### Updated: Sun Jan 19 06:36:20 UTC 2025
### Bad Bot Count: 668 ### Bad Bot Count: 675
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -861,6 +861,16 @@ User-agent: Semrush
Disallow:/ Disallow:/
User-agent: SemrushBot User-agent: SemrushBot
Disallow:/ Disallow:/
User-agent: SemrushBot-BA
Disallow:/
User-agent: SemrushBot-FT
Disallow:/
User-agent: SemrushBot-OCOB
Disallow:/
User-agent: SemrushBot-SI
Disallow:/
User-agent: SemrushBot-SWA
Disallow:/
User-agent: SentiBot User-agent: SentiBot
Disallow:/ Disallow:/
User-agent: SenutoBot User-agent: SenutoBot
@ -877,6 +887,8 @@ User-agent: Shodan
Disallow:/ Disallow:/
User-agent: Siphon User-agent: Siphon
Disallow:/ Disallow:/
User-agent: SiteAuditBot
Disallow:/
User-agent: SiteCheckerBotCrawler User-agent: SiteCheckerBotCrawler
Disallow:/ Disallow:/
User-agent: SiteExplorer User-agent: SiteExplorer
@ -931,6 +943,8 @@ User-agent: Spider_Bot/3.0
Disallow:/ Disallow:/
User-agent: Spinn3r User-agent: Spinn3r
Disallow:/ Disallow:/
User-agent: SplitSignalBot
Disallow:/
User-agent: SputnikBot User-agent: SputnikBot
Disallow:/ Disallow:/
User-agent: Sqlmap User-agent: Sqlmap

20
robots.txt/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2025.01.4982 ### Version: V4.2025.01.4983
### Updated: Mon Jan 6 10:04:07 UTC 2025 ### Updated: Sun Jan 19 06:36:20 UTC 2025
### Bad Bot Count: 668 ### Bad Bot Count: 675
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -861,6 +861,16 @@ User-agent: Semrush
Disallow:/ Disallow:/
User-agent: SemrushBot User-agent: SemrushBot
Disallow:/ Disallow:/
User-agent: SemrushBot-BA
Disallow:/
User-agent: SemrushBot-FT
Disallow:/
User-agent: SemrushBot-OCOB
Disallow:/
User-agent: SemrushBot-SI
Disallow:/
User-agent: SemrushBot-SWA
Disallow:/
User-agent: SentiBot User-agent: SentiBot
Disallow:/ Disallow:/
User-agent: SenutoBot User-agent: SenutoBot
@ -877,6 +887,8 @@ User-agent: Shodan
Disallow:/ Disallow:/
User-agent: Siphon User-agent: Siphon
Disallow:/ Disallow:/
User-agent: SiteAuditBot
Disallow:/
User-agent: SiteCheckerBotCrawler User-agent: SiteCheckerBotCrawler
Disallow:/ Disallow:/
User-agent: SiteExplorer User-agent: SiteExplorer
@ -931,6 +943,8 @@ User-agent: Spider_Bot/3.0
Disallow:/ Disallow:/
User-agent: Spinn3r User-agent: Spinn3r
Disallow:/ Disallow:/
User-agent: SplitSignalBot
Disallow:/
User-agent: SputnikBot User-agent: SputnikBot
Disallow:/ Disallow:/
User-agent: Sqlmap User-agent: Sqlmap