vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-10 06:15:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2024.10.4836 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2024-10-26 05:51:09 +00:00
parent e1e2d14eec
commit b1cfec8203
37 changed files with 286 additions and 268 deletions
Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2024.10.4835 #### Version: V4.2024.10.4836
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [667](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [668](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2024.10.4835 #### Version: V4.2024.10.4836
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [667](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [668](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -14,9 +14,9 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2024.10.4835 #### Version: V4.2024.10.4836
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [667](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [668](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

2
_generator_lists/bad-user-agents.list
View file

@ -1,4 +1,3 @@
everyfeed-spider
01h4x.com 01h4x.com
360Spider 360Spider
404checker 404checker
@ -608,6 +607,7 @@ demandbase-bot
domainsproject.org domainsproject.org
eCatch eCatch
evc-batch evc-batch
everyfeed-spider
facebookscraper facebookscraper
gopher gopher
heritrix heritrix

4
_google_analytics_ghost_spam/README.md
View file

@ -3,9 +3,9 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2024.10.4835 #### Version: V4.2024.10.4836
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [667](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [668](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

7
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:17 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -760,6 +760,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

7
conf.d/globalblacklist-testing-version.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2024.10.4835 ### Version: V3.2024.10.4836
### Updated: Fri Oct 25 22:01:18 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -757,6 +757,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

7
conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:17 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -760,6 +760,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

1
dev-tools/_robots_input/robots-input.txt
View file

@ -607,6 +607,7 @@ demandbase-bot
domainsproject.org domainsproject.org
eCatch eCatch
evc-batch evc-batch
everyfeed-spider
facebookscraper facebookscraper
gopher gopher
heritrix heritrix

2
dev-tools/buildnumber
View file

@ -1 +1 @@
4835 4836

7
dev-tools/globalblacklist-testing.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2024.10.4835 ### Version: V3.2024.10.4836
### Updated: Fri Oct 25 22:01:18 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -757,6 +757,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

7
dev-tools/globalblacklist.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:17 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -760,6 +760,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
172.183.122.254 0; 172.171.255.193 0;

7
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:17 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -760,6 +760,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.48 1; 10.1.0.64 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.48 0; 10.1.0.64 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:17 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -760,6 +760,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.48 1; 10.1.0.64 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.48 0; 10.1.0.64 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:17 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -760,6 +760,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.48 1; 10.1.0.64 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -668,6 +668,7 @@
"~*(?:\b)domainsproject.org(?:\b|)" 0; "~*(?:\b)domainsproject.org(?:\b|)" 0;
"~*(?:\b)eCatch(?:\b|)" 0; "~*(?:\b)eCatch(?:\b|)" 0;
"~*(?:\b)evc-batch(?:\b|)" 0; "~*(?:\b)evc-batch(?:\b|)" 0;
"~*(?:\b)everyfeed-spider(?:\b|)" 0;
"~*(?:\b)facebookscraper(?:\b|)" 0; "~*(?:\b)facebookscraper(?:\b|)" 0;
"~*(?:\b)gopher(?:\b|)" 0; "~*(?:\b)gopher(?:\b|)" 0;
"~*(?:\b)heritrix(?:\b|)" 0; "~*(?:\b)heritrix(?:\b|)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.48 0; 10.1.0.64 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:17 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -760,6 +760,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
172.183.122.254 0; 172.171.255.193 0;

7
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:17 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -760,6 +760,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
172.183.122.254 0; 172.171.255.193 0;

7
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:17 UTC 2024 ### Updated: Sat Oct 26 05:47:29 UTC 2024
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -760,6 +760,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)domainsproject.org(?:\b)" 3; "~*(?:\b)domainsproject.org(?:\b)" 3;
"~*(?:\b)eCatch(?:\b)" 3; "~*(?:\b)eCatch(?:\b)" 3;
"~*(?:\b)evc-batch(?:\b)" 3; "~*(?:\b)evc-batch(?:\b)" 3;
"~*(?:\b)everyfeed-spider(?:\b)" 3;
"~*(?:\b)facebookscraper(?:\b)" 3; "~*(?:\b)facebookscraper(?:\b)" 3;
"~*(?:\b)gopher(?:\b)" 3; "~*(?:\b)gopher(?:\b)" 3;
"~*(?:\b)heritrix(?:\b)" 3; "~*(?:\b)heritrix(?:\b)" 3;

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.48 1; 10.1.0.64 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -668,6 +668,7 @@
"~*(?:\b)domainsproject.org(?:\b|)" 0; "~*(?:\b)domainsproject.org(?:\b|)" 0;
"~*(?:\b)eCatch(?:\b|)" 0; "~*(?:\b)eCatch(?:\b|)" 0;
"~*(?:\b)evc-batch(?:\b|)" 0; "~*(?:\b)evc-batch(?:\b|)" 0;
"~*(?:\b)everyfeed-spider(?:\b|)" 0;
"~*(?:\b)facebookscraper(?:\b|)" 0; "~*(?:\b)facebookscraper(?:\b|)" 0;
"~*(?:\b)gopher(?:\b|)" 0; "~*(?:\b)gopher(?:\b|)" 0;
"~*(?:\b)heritrix(?:\b|)" 0; "~*(?:\b)heritrix(?:\b|)" 0;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
DigitalPebble BDFetch
Lftp Hloader
LinkWalker MIDown tool
Nutch Musobot
Sucker Navroad
TeleportPro RankActive
TheNomad RebelMouse
Turnitin Sottopop
URLy Warning adscanner
clark-crawler satoristudio.net

180
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
360Spider 404enemy
ALittle Client AIBOT
ASPSeek Abonti
Aspiegel AfD-Verbotsverfahren
BDCbot Atomseobot
AwarioSmartBot
BDFetch
BLEXBot BLEXBot
BackDoorBot
Backlink-Ceck Backlink-Ceck
Bigfoot Badass
Blackboard
Blow Blow
Bullseye BuiltWith
BunnySlippers CATExplorador
ChinaClaw CensysInspect
Citoid CheTeam
Claritybot CheeseBot
CyotekWebCopy Cocolyzebot
DIIbot Copyscape
DSearch Crawling at Home Project
Crescent
Curious
Deusu
Digincore Digincore
Download Wonder DomainCrawler
FDM FHscan
Freeuploader FemtosearchBot
GrapeFX FrontPage
HTMLparser FyberSpider
Heritrix G-i-g-a-b-o-t
GPTBot
GetWeb
GridBot
HTTrack
Humanlinks Humanlinks
HybridBot IRLbot
Iblog Iblog
Intelliseek JOC Web Spider
Internet Ninja JustView
Jetty Jyxobot
Lanshanbot LNSpiderguy
LieBaoFast LWP::Simple
Lightspeedsystems Ltx71
MJ12bot Masscan
MSIECrawler Morfeus Fucking Scanner
Mass Downloader Mr.4x3
Mata Hari Net Vampire
Meanpathbot Nikto
Mediatoolkitbot NimbleCrawler
Mojeek Offline Explorer
MolokaiBot
Nessus
NetAnts
Nettrack
NextGenSearchBot
Nibbler
Ninja
OpenLinkProfiler
OrangeSpider OrangeSpider
Page Analyzer OutclicksBot
PeoplePal Pandalytics
Petalbot Pi-Monster
Picscout ProWebWalker
ProPowerBot Pu_iN
Reaper PxBroker
SalesIntelligent PyCurl
ScanAlert Rankivabot
SearchmetricsBot RankurBot
SentiBot SMTBot
Seomoz ScoutJet
SiteCheckerBotCrawler Scrapy
SeobilityBot
SiteExplorer SiteExplorer
SiteLockSpider Sosospider
SmartDownload SpaceBison
SocialRankIOBot Spinn3r
Spammen SuperBot
Spider_Bot T8Abot
Sqworm Thumbor
Steeler TinyTestBot
Swiftbot Web Fuck
T0PHackTeam WebCopier
Telesoft WebLeacher
Telesphoreo WebSauger
Toweyabot WebZIP
Turingos Whack
VeriCiteCrawler Whatweb
Virusdie Widow
WBSearchBot Wotbox
WWW-Collector-E Xaldon WebSpider
WWWOFFLE
Web Auto
WebStripper
WebSucker
Whacker
Xenu Xenu
Zermelo YoudaoBot
Zeus Zeus
cognitiveseo anthropic-ai
dataforseobot archive.org_bot
evc-batch instabid
imagesift.com ips-agent
iubenda-radar isitwp.com
linkfluence lwp-request
magpie-crawler omgili
mediawords s1z.ru
page scorer satoristudio.net
probe-image-size seocompany.store
ripz sogouspider
sexsearcher trendiction.de
zgrab webgains-bot
webpros.com

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
carina-sy.de bookmark4you.com.biz
compliance-george.top cashonads.com
divci-hry.info compliance-barak.xyz
eloconcream.blogspot.com gg-arena.ru
keywordsking.com mindbox.co.za
kupit-adenu.ru ok-ua.info
manuscript.su slowmac.tech
onlywoman.org trafficjunky.net
szamponrevita.pl uhdtv.website
urccvfmc.bloger.index.hr urengoy.pro

196
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
1millionusd.xyz 4replicawatch.net
aa81bf391151884adfa3dd677e41f94be1.com 66cpwgln.space
actulite.com aaa38852e886ac4af1a3cff9b47cab6272.com
afftrack001.com academiacsmendoza.org
alibestsale.com ads-cool.pro
arcteryxsale.online adviceforum.info
autoua.top androids-store.com
ayakino.net arkkivoltti.net
babieca.com asociatia-tipografilor-transilvania.ro
beachpics.com azazu.ru
bear.gotcher.us bausparen.bz.it
belstaffstore.online bonkers.name
bestmouthwash.club bonzbuddy.com
bitcoinpile.com buzz-porno.info
bocoarchives.org centrumcoachingu.com
brainboostingsupplements.org cialischsrx.com
brandbucket.com clean-start.net
buigas.00it.com compliance-george.top
chcu.net compliance-irvin.xyz
cheapmoncler.win compliance-willy.top
chinese-amezon.com custom-product-labels.com
dealitright.click data1.scopich.com
diegolopezcastan.com dealighted.com
drunkmoms.net domaincdn.xyz
elektrozigaretten2.yn.lt drpornogratisx.xxx
ezigarettekaufen1.hpage.com ekto.ee
f07.de eluxer.net
farmprofi.net epngo.bz
fickluder69.com essayservicewriting.org
formseo.com eyessurgery.ru
freejabs.com fefo.gdn
fugarif.ga filmania-x.ru
gallery.rennlist.com freefoto.ca
girls-ufa.ru gazporno.com
handicapbathtubarea.com ge0ip.net
highland-homes.com globalscam.ga
histock.info growboxbl.ru
hostcritique.com gunsvicceadadebt.tk
hostingclub.lk hdseriale.pl
humanorightswatch.org housediz.com
ikearugs.xyz housemilan.ru
ikritikimou.gr humbmosquina.tk
intimshop-fantasy.ru ilmexico.com
jav-fetish.site images-free.net
keenoutlet.online imagez.co
kustanay.kz imediadesk.com
law-check-nine.xyz intim-uslugi.info
lion.playtap.us japan-bearings.ru
liveinternet.ro jerseywholesalechinabiz.com
localmatchbook.com katadhin.co
lonely-mature.com kchaxton.com
marketingvici.com keywordglobal.co.za
keywordsuggests.com
lflash.ru
maximilitary.ru maximilitary.ru
mixed-wrestling.ru mesbuta.info
mlf.hordo.win moroccosurfadventures.com
mobplayer.ru mrcsa.com.au
moncleronline.xyz mrlmedia.net
muabancantho.info myfreemp3.eu
mypornfree.ru myfreetutorials.com
na-telefon.biz najaden.no
notfastfood.ru no-rx.info
novodigs.com ownshop.cf
nuit-artisanale.com pandarastore.top
nuker.com parlament.biz
oxford-book.com.ua phormchina.com
parajumpersstore.online piccshare.com
pathwhelp.org portadd.men
physfunc.ru privatefx-in.ru
pojdelo.weebly.com pussysaga.com
produkto.net remont-ustanovka-tehniki.ru
prohoster.info robot-forex.biz
redirectme.net rus-pornuha.com
russia-tao.ru ruscoininvest.company
samolet.fr safe-app.net
sexspornotub.com seozoom.it
sexuria.net shareyards.com
sharpchallenge.com shemalegalls.blogporn.in
soft-terminal.ru sim-service.net
special-porn.com snip.to
studworks.org spacash.com
sugarkun.com staff.prairiesouth.ca
summerlinhomes411.info strag-invest.ru
teenforporn.com tako3.com
top-l2.com thebluffs.com
trahic.ru tizanidinehcl4mgisitanarcotic.blogspot.com
url-extractor.xyz tpu.ru
treasuretrack-a.akamaihd.net
tubeoffline.com
v24s.net
vbikse.com vbikse.com
viandpet.com villacoloniale.com
vikistars.com vote-up.ru
vpnhowto.info wholesalecheapjerseysfree.com
waterpurifier.club womensplay.net
wineitudes.wordpress.com www.timer4web.com
wormix-cheats.ru x-true.info
www.ehscloud.cn xn--80aafb2a.xn--p1ai
xxx-treker.ru zmoda.hostreo.com
xxxguitars.com
xxxkaz.org
zed21.net
zeroredirect8.com
zryydi.com

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.48 0; 10.1.0.64 0;
127.0.0.1 0; 127.0.0.1 0;

8
dev-tools/tmp/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:18 UTC 2024 ### Updated: Sat Oct 26 05:47:30 UTC 2024
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -1233,6 +1233,8 @@ User-agent: eCatch
Disallow:/ Disallow:/
User-agent: evc-batch User-agent: evc-batch
Disallow:/ Disallow:/
User-agent: everyfeed-spider
Disallow:/
User-agent: facebookscraper User-agent: facebookscraper
Disallow:/ Disallow:/
User-agent: gopher User-agent: gopher

8
robots.txt/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2024.10.4835 ### Version: V4.2024.10.4836
### Updated: Fri Oct 25 22:01:18 UTC 2024 ### Updated: Sat Oct 26 05:47:30 UTC 2024
### Bad Bot Count: 667 ### Bad Bot Count: 668
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -1233,6 +1233,8 @@ User-agent: eCatch
Disallow:/ Disallow:/
User-agent: evc-batch User-agent: evc-batch
Disallow:/ Disallow:/
User-agent: everyfeed-spider
Disallow:/
User-agent: facebookscraper User-agent: facebookscraper
Disallow:/ Disallow:/
User-agent: gopher User-agent: gopher