vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-02 18:50:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2023.05.3717 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2023-05-01 04:41:11 +00:00
parent ed95057e1d
commit b2d0916888
37 changed files with 371 additions and 346 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2023.05.3716 #### Version: V4.2023.05.3717
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [639](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [640](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2023.05.3716 #### Version: V4.2023.05.3717
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [639](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [640](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -14,9 +14,9 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2023.05.3716 #### Version: V4.2023.05.3717
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [639](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [640](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
_google_analytics_ghost_spam/README.md
View file

@ -3,9 +3,9 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2023.05.3716 #### Version: V4.2023.05.3717
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [639](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [640](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

7
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:14 UTC 2023 ### Updated: Mon May 1 04:37:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

7
conf.d/globalblacklist-testing-version.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2023.05.3716 ### Version: V3.2023.05.3717
### Updated: Mon May 1 04:23:15 UTC 2023 ### Updated: Mon May 1 04:37:17 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

7
conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:14 UTC 2023 ### Updated: Mon May 1 04:37:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

1
dev-tools/_robots_input/robots-input.txt
View file

@ -404,6 +404,7 @@ ScrepyBot
Searchestate Searchestate
SearchmetricsBot SearchmetricsBot
Seekport Seekport
SeekportBot
SemanticJuice SemanticJuice
Semrush Semrush
SemrushBot SemrushBot

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3716 3717

7
dev-tools/globalblacklist-testing.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2023.05.3716 ### Version: V3.2023.05.3717
### Updated: Mon May 1 04:23:15 UTC 2023 ### Updated: Mon May 1 04:37:17 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

7
dev-tools/globalblacklist.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:14 UTC 2023 ### Updated: Mon May 1 04:37:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
157.55.189.33 0; 40.86.28.128 0;

7
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:14 UTC 2023 ### Updated: Mon May 1 04:37:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.119 1; 10.1.0.44 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.119 0; 10.1.0.44 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:14 UTC 2023 ### Updated: Mon May 1 04:37:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.119 1; 10.1.0.44 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.119 0; 10.1.0.44 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:14 UTC 2023 ### Updated: Mon May 1 04:37:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.119 1; 10.1.0.44 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -465,6 +465,7 @@
"~*(?:\b)Searchestate(?:\b|)" 0; "~*(?:\b)Searchestate(?:\b|)" 0;
"~*(?:\b)SearchmetricsBot(?:\b|)" 0; "~*(?:\b)SearchmetricsBot(?:\b|)" 0;
"~*(?:\b)Seekport(?:\b|)" 0; "~*(?:\b)Seekport(?:\b|)" 0;
"~*(?:\b)SeekportBot(?:\b|)" 0;
"~*(?:\b)SemanticJuice(?:\b|)" 0; "~*(?:\b)SemanticJuice(?:\b|)" 0;
"~*(?:\b)Semrush(?:\b|)" 0; "~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0; "~*(?:\b)SemrushBot(?:\b|)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.119 0; 10.1.0.44 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:14 UTC 2023 ### Updated: Mon May 1 04:37:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
157.55.189.33 0; 40.86.28.128 0;

7
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:14 UTC 2023 ### Updated: Mon May 1 04:37:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
157.55.189.33 0; 40.86.28.128 0;

7
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:14 UTC 2023 ### Updated: Mon May 1 04:37:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -553,6 +553,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3; "~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SeekportBot(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.119 1; 10.1.0.44 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -465,6 +465,7 @@
"~*(?:\b)Searchestate(?:\b|)" 0; "~*(?:\b)Searchestate(?:\b|)" 0;
"~*(?:\b)SearchmetricsBot(?:\b|)" 0; "~*(?:\b)SearchmetricsBot(?:\b|)" 0;
"~*(?:\b)Seekport(?:\b|)" 0; "~*(?:\b)Seekport(?:\b|)" 0;
"~*(?:\b)SeekportBot(?:\b|)" 0;
"~*(?:\b)SemanticJuice(?:\b|)" 0; "~*(?:\b)SemanticJuice(?:\b|)" 0;
"~*(?:\b)Semrush(?:\b|)" 0; "~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0; "~*(?:\b)SemrushBot(?:\b|)" 0;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
ALittle Client CCBot
Aspiegel EyeNetIE
Bandit IDBTE4M
Devil LinkWalker
FrontPage MegaIndex.ru
Grabber Metauri
Kinza NetZIP
Octopus Titan
T8Abot trendiction.com
Telesphorep voyagerx.com

160
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
404enemy AIBOT
ADmantX Alligator
Acunetix Anarchy
Anarchie AwarioRssBot
Ankit BlackWidow
Apexoo Boardreader
BetaBot
Brandprotect
Buddy
CazoodleBot CazoodleBot
ChinaClaw CherryPicker
Chlooe DBLBot
Citoid DatabaseDriverMysqli
Claritybot Deusu
Devil Dirbuster
Discoverybot
DnBCrawler-Analytics DnBCrawler-Analytics
DomainAppender DomainAppender
EMail Wolf
EasyDL
EirGrabber EirGrabber
Evil FDM
ExtractorPro Firefox/7.0
FHscan
GT::WWW GT::WWW
GetRight Getintent
GridBot Go!Zilla
Go-Ahead-Got-It
GoZilla
GrabNet
HEADMasterSEO
HTMLparser HTMLparser
HybridBot Humanlinks
IlseBot Internet Ninja
Image Fetch Iskanie
InfoNaviRobot Kinza
InfoTekies Larbin
JamesBOT LibWeb
JetCar LinkScan
Keybot Translation-Search-Machine Linkbot
LeechGet LinkextractorPro
Litemage_walker LinkpadBot
Majestic-SEO Lipperhey Spider
MarkMonitor MQQBrowser
Meanpathbot
Mediatoolkitbot
Minefield
Moblie Safari Moblie Safari
Mr.4x3 Mojeek
Nuclei Nameprotect
Offline Explorer Nessus
NetSpider
Nettrack
Octopus
Offline Navigator Offline Navigator
Openfind OpenLinkProfiler
Openvas OpenVAS
Panscient OutclicksBot
Pavuk Page Analyzer
PyCurl Papa Foto
RankActiveLinkBot Pi-Monster
RankingBot PleaseCrawl
RankingBot2 Pockey
RankurBot Probethenet
Reaper Pu_iN
RedesScrapy Recorder
ScoutJet RepoMonkey
ScreenerBot Ripper
SEOkicks
ScanAlert
Scanbot
ScrepyBot ScrepyBot
SemanticJuice
Semrush Semrush
SemrushBot Seomoz
SentiBot Sitebeam
SiteLockSpider
Sociscraper Sociscraper
Steeler
Sucker Sucker
T8Abot T8Abot
Toweyabot Telesphorep
Turingos Titan
V-BOT Trendictionbot
VCI
Vacuum
Virusdie
WEBDAV
WISENutbot WISENutbot
WWWOFFLE
WeSEE
Web Enhancer Web Enhancer
WebAuto
WebCopier WebCopier
WebReaper WebImageCollector
Whatweb WebWhacker
Wotbox Website Quester
Xaldon_WebSpider WebsiteExtractor
YoudaoBot Xaldon WebSpider
Zauba Zauba
ZoomBot
ZoominfoBot ZoominfoBot
archive.org_bot archive.org_bot
backlink-check arquivo-web-crawler
com.plumanalytics gopher
heritrix heritrix
instabid netEstate NE Crawler
internetVista monitor oBot
isitwp.com scan.lol
iubenda-radar seobility
lwp-trivial sexsearcher
page scorer spyfu
plumanalytics webgains-bot
s1z.ru
sysscan
voyagerx.com
webmeup-crawler
x09Mozilla

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
groupmoney.ru board.f00d.de
harmonyglen.us dnepr-avtospar.com.ua
letolove.ru favoritemoney.ru
mosrif.ru kskjco.club
nootrino.com massage-info.nl
picturesify.com repeatlogo.co.uk
rome2rio.com shoesonlinebuy.cn
seorankinglinks.xyz waycash.net
tanieaukcje.com.pl wheelchairliftsarea.com
zarabotat-na-sajte.ru womensterritory.ru

200
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
100dollars-seo.com 007angels.com
123locker.com 1kinobig.ru
1millionusd.xyz 7search.com
300richmond.co.nz aa625d84f1587749c1ab011d6f269f7d64.com
3dracergames.com aaronabel.com
4replicawatch.net accessoires-mode.in
alta-realestate.com acunetix-referrer.com
am15.net adman.gr
asdfz.pro adprotect.net
ayeartoforget.com agadelha.com.br
azartmsl.com anniemation.com
beztuberkuleza.ru apibrowsesmartne-a.akamaihd.net
blavia.00author.com app5.letmacworkfaster.world
boo-arts.com appiq.mobi
brainsandeggs.blogspot.com atmovs.com
budpost.com.ua auto-moto-elektronika.cz
cartiertrinity.xyz autoua.top
cherrypointplace.ca aviav.ru
chorus.terakeet.com babyguns.ru
christianlouboutinshoes.xyz blue-square.biz
clickintext.net bonzibuddi.com
commoncrawl.org buchananshardware.com
compliance-nelson.top candycrushshop.com
covadhosting.biz cashonads.com
covetnica.com chastnoeporno.com
creativehutindia.com cheapestjerseysonwholesale.com
dailyfinancefix.com chelnytruck.ru
davebestdeals.com customcatchcan.com
deluxedumps.com detskie-konstruktory.ru
dividendo.ru downloadme.life
djekxa.ru e-c.al
eaplay.ru edudocs.net
elektrischezi.canalblog.com elektronischezigarettekaufen2.tumblr.com
fastfixing.tech eloconcream.blogspot.com
fix-website-errors.com eropho.net
friendflnder.com esopini.com
fuckmill.com everflora.ru
gaz-voshod.ru eywords-monitoring-your-success.com
goldenggames.com fapgon.com
gwebtools.com.br floridamhca.org
histock.info gk170.ru
inet-traffic.com graphid.com
infazavr.ru gsasearchengineranker.site
iplusbit.blogspot.co.za gsasearchengineranker.top
iron-age.info hamptonoaks.ca
jbl-charge.info hd720kino.ru
johannesburgsingles.co.za howmuchdoestizanidinecost.blogspot.com
kopihijau.info iamsport.org
law-check-seven.xyz insider.pro
lineavabit.it intellekt21.ru
lomza.info iopeninghours.co.uk
losangeles-ads.com kakadu-interior.com.ua
makedo.ru kinogonew.ru
maladot.com konkursov.net
medicinacom.ru kosmetyki.tm.pl
ngps1.ru kursy-ege.ru
nhl17coinsforps3.gratisblog.biz leftofcentrist.com
openmediasoft.com limtu.ifmo.ru
pic2fly.com linkrr.com
platesauto.com michaelkorsoutletstore.net
prointer.net.ua mp3ringtone.info
proposal-engine.com openmultipleurl.com
prostitutki-belgoroda.org pihl.se
reklama-i-rabota.ru pizdeishn.com
rockingclicks.com porngalleries.top
rubbed.us prombudpostach.com.ua
russintv.fr promoheads.com
sampleletters.net prostitutki-kharkova.org
scanner-don.top pukaporn.com
scanner-jack.top qwesa.ru
seorankinglinks.us ritlweb.com
service.adtech.us running-line.ru
shop.xz618.com s-forum.biz
sibdevice.ru scanner-jessica.top
site3.free-share-buttons.com sex.hotblog.top
sklad-24.ru sexystrippe.info
sovetogorod.ru sisi-go.ru
spitfiremusic.com siteimprove.com
spywareit.com snomer1.ru
stevemonsen.com socialbuttons.xyz
superoboi.com.ua soft-program.com
support.nopeas.sk softwaretrend.net
teenfuck.tv solution4u.com
telegraf.by sport7777.net
thebestweightlosspills.ovh strana-krasoty.ru
tootoo.to testbotprocessor44.com
topsiteminecraft.com tizanidinecapsules.blogspot.com
ultimateclassicrock.com top10registrycleaners.com
urengoy.pro trafficmp.com
videojam.tv undergroundcityphoto.com
wholesalejerseychinaoutlet.com valoresito.com
worldinternetauthority.com vapomnoncri.tk
x5market.ru versaut.xxx-cam.webcam
xn--c1acygb.xn--p1ai wait3sec.org
xtubeporno.net whatsappdownload10.com
zahvat.ru xjrul.com
zaim-pod-zalog-krasnodar.ru xlovecam.com
zigarettenonlinekaufen2.drupalgardens.com youtubologia.it
zrelaya.pw zeroredirect7.com
zs2vm.top znakomstva-piter78.ru

181
dev-tools/test_units/ratelimittest-master.txt
View file

@ -8,93 +8,6 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html>ml>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><html> </html><html>
<head><title>503 Service Temporarily Unavailable</title></head> <head><title>503 Service Temporarily Unavailable</title></head>
<body> <body>
@ -494,6 +407,90 @@
<hr><center>nginx/1.18.0 (Ubuntu)</center> <hr><center>nginx/1.18.0 (Ubuntu)</center>
</body> </body>
</html> </html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
<!DOCTYPE html> <!DOCTYPE html>
<html> <html>
<head> <head>
@ -584,4 +581,14 @@
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1> <h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p> <p>This is merely used for testing !!!</p>
</body> </body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html> </html>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.119 0; 10.1.0.44 0;
127.0.0.1 0; 127.0.0.1 0;

8
dev-tools/tmp/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:16 UTC 2023 ### Updated: Mon May 1 04:37:18 UTC 2023
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -827,6 +827,8 @@ User-agent: SearchmetricsBot
Disallow:/ Disallow:/
User-agent: Seekport User-agent: Seekport
Disallow:/ Disallow:/
User-agent: SeekportBot
Disallow:/
User-agent: SemanticJuice User-agent: SemanticJuice
Disallow:/ Disallow:/
User-agent: Semrush User-agent: Semrush

8
robots.txt/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2023.05.3716 ### Version: V4.2023.05.3717
### Updated: Mon May 1 04:23:16 UTC 2023 ### Updated: Mon May 1 04:37:18 UTC 2023
### Bad Bot Count: 639 ### Bad Bot Count: 640
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -827,6 +827,8 @@ User-agent: SearchmetricsBot
Disallow:/ Disallow:/
User-agent: Seekport User-agent: Seekport
Disallow:/ Disallow:/
User-agent: SeekportBot
Disallow:/
User-agent: SemanticJuice User-agent: SemanticJuice
Disallow:/ Disallow:/
User-agent: Semrush User-agent: Semrush