vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-02 18:50:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2023.08.3919 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2023-08-08 09:03:16 +00:00
parent 230ed56a35
commit 3fd459c56a
37 changed files with 316 additions and 262 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2023.08.3918 #### Version: V4.2023.08.3919
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [642](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [645](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2023.08.3918 #### Version: V4.2023.08.3919
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [642](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [645](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -14,9 +14,9 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2023.08.3918 #### Version: V4.2023.08.3919
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [642](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [645](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

6
_generator_lists/bad-user-agents.list
View file

@ -1,6 +1,3 @@
GPTBot
openai.com
openai
01h4x.com 01h4x.com
360Spider 360Spider
404checker 404checker
@ -162,6 +159,7 @@ Fuzz
FyberSpider FyberSpider
Fyrebot Fyrebot
G-i-g-a-b-o-t G-i-g-a-b-o-t
GPTBot
GT::WWW GT::WWW
GalaxyBot GalaxyBot
Genieo Genieo
@ -607,6 +605,8 @@ mediawords
muhstik-scan muhstik-scan
netEstate\ NE\ Crawler netEstate\ NE\ Crawler
oBot oBot
openai
openai.com
page\ scorer page\ scorer
pcBrowser pcBrowser
plumanalytics plumanalytics

4
_google_analytics_ghost_spam/README.md
View file

@ -3,9 +3,9 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2023.08.3918 #### Version: V4.2023.08.3919
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [642](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [645](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

9
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:24 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

9
conf.d/globalblacklist-testing-version.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2023.08.3918 ### Version: V3.2023.08.3919
### Updated: Mon Aug 7 22:01:25 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

9
conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:24 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

3
dev-tools/_robots_input/robots-input.txt
View file

@ -159,6 +159,7 @@ Fuzz
FyberSpider FyberSpider
Fyrebot Fyrebot
G-i-g-a-b-o-t G-i-g-a-b-o-t
GPTBot
GT::WWW GT::WWW
GalaxyBot GalaxyBot
Genieo Genieo
@ -604,6 +605,8 @@ mediawords
muhstik-scan muhstik-scan
netEstate NE Crawler netEstate NE Crawler
oBot oBot
openai
openai.com
page scorer page scorer
pcBrowser pcBrowser
plumanalytics plumanalytics

2
dev-tools/buildnumber
View file

@ -1 +1 @@
3918 3919

9
dev-tools/globalblacklist-testing.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2023.08.3918 ### Version: V3.2023.08.3919
### Updated: Mon Aug 7 22:01:25 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

9
dev-tools/globalblacklist.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:24 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
138.91.87.24 0; 104.210.132.191 0;

9
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:24 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.121 1; 10.1.0.65 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.121 0; 10.1.0.65 0;
127.0.0.1 0; 127.0.0.1 0;

9
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:24 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.121 1; 10.1.0.65 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.121 0; 10.1.0.65 0;
127.0.0.1 0; 127.0.0.1 0;

9
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:24 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.121 1; 10.1.0.65 1;
127.0.0.1 1; 127.0.0.1 1;

3
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -220,6 +220,7 @@
"~*(?:\b)FyberSpider(?:\b|)" 0; "~*(?:\b)FyberSpider(?:\b|)" 0;
"~*(?:\b)Fyrebot(?:\b|)" 0; "~*(?:\b)Fyrebot(?:\b|)" 0;
"~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0; "~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0;
"~*(?:\b)GPTBot(?:\b|)" 0;
"~*(?:\b)GT::WWW(?:\b|)" 0; "~*(?:\b)GT::WWW(?:\b|)" 0;
"~*(?:\b)GalaxyBot(?:\b|)" 0; "~*(?:\b)GalaxyBot(?:\b|)" 0;
"~*(?:\b)Genieo(?:\b|)" 0; "~*(?:\b)Genieo(?:\b|)" 0;
@ -665,6 +666,8 @@
"~*(?:\b)muhstik-scan(?:\b|)" 0; "~*(?:\b)muhstik-scan(?:\b|)" 0;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0; "~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0;
"~*(?:\b)oBot(?:\b|)" 0; "~*(?:\b)oBot(?:\b|)" 0;
"~*(?:\b)openai(?:\b|)" 0;
"~*(?:\b)openai.com(?:\b|)" 0;
"~*(?:\b)page\ scorer(?:\b|)" 0; "~*(?:\b)page\ scorer(?:\b|)" 0;
"~*(?:\b)pcBrowser(?:\b|)" 0; "~*(?:\b)pcBrowser(?:\b|)" 0;
"~*(?:\b)plumanalytics(?:\b|)" 0; "~*(?:\b)plumanalytics(?:\b|)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.121 0; 10.1.0.65 0;
127.0.0.1 0; 127.0.0.1 0;

9
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:24 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
138.91.87.24 0; 104.210.132.191 0;

9
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:24 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
138.91.87.24 0; 104.210.132.191 0;

9
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:24 UTC 2023 ### Updated: Tue Aug 8 08:59:23 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -308,6 +308,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)FyberSpider(?:\b)" 3; "~*(?:\b)FyberSpider(?:\b)" 3;
"~*(?:\b)Fyrebot(?:\b)" 3; "~*(?:\b)Fyrebot(?:\b)" 3;
"~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3; "~*(?:\b)G-i-g-a-b-o-t(?:\b)" 3;
"~*(?:\b)GPTBot(?:\b)" 3;
"~*(?:\b)GT::WWW(?:\b)" 3; "~*(?:\b)GT::WWW(?:\b)" 3;
"~*(?:\b)GalaxyBot(?:\b)" 3; "~*(?:\b)GalaxyBot(?:\b)" 3;
"~*(?:\b)Genieo(?:\b)" 3; "~*(?:\b)Genieo(?:\b)" 3;
@ -753,6 +754,8 @@ map $http_user_agent $bad_bot {
"~*(?:\b)muhstik-scan(?:\b)" 3; "~*(?:\b)muhstik-scan(?:\b)" 3;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3; "~*(?:\b)netEstate\ NE\ Crawler(?:\b)" 3;
"~*(?:\b)oBot(?:\b)" 3; "~*(?:\b)oBot(?:\b)" 3;
"~*(?:\b)openai(?:\b)" 3;
"~*(?:\b)openai.com(?:\b)" 3;
"~*(?:\b)page\ scorer(?:\b)" 3; "~*(?:\b)page\ scorer(?:\b)" 3;
"~*(?:\b)pcBrowser(?:\b)" 3; "~*(?:\b)pcBrowser(?:\b)" 3;
"~*(?:\b)plumanalytics(?:\b)" 3; "~*(?:\b)plumanalytics(?:\b)" 3;

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.121 1; 10.1.0.65 1;
127.0.0.1 1; 127.0.0.1 1;

3
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -220,6 +220,7 @@
"~*(?:\b)FyberSpider(?:\b|)" 0; "~*(?:\b)FyberSpider(?:\b|)" 0;
"~*(?:\b)Fyrebot(?:\b|)" 0; "~*(?:\b)Fyrebot(?:\b|)" 0;
"~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0; "~*(?:\b)G-i-g-a-b-o-t(?:\b|)" 0;
"~*(?:\b)GPTBot(?:\b|)" 0;
"~*(?:\b)GT::WWW(?:\b|)" 0; "~*(?:\b)GT::WWW(?:\b|)" 0;
"~*(?:\b)GalaxyBot(?:\b|)" 0; "~*(?:\b)GalaxyBot(?:\b|)" 0;
"~*(?:\b)Genieo(?:\b|)" 0; "~*(?:\b)Genieo(?:\b|)" 0;
@ -665,6 +666,8 @@
"~*(?:\b)muhstik-scan(?:\b|)" 0; "~*(?:\b)muhstik-scan(?:\b|)" 0;
"~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0; "~*(?:\b)netEstate\ NE\ Crawler(?:\b|)" 0;
"~*(?:\b)oBot(?:\b|)" 0; "~*(?:\b)oBot(?:\b|)" 0;
"~*(?:\b)openai(?:\b|)" 0;
"~*(?:\b)openai.com(?:\b|)" 0;
"~*(?:\b)page\ scorer(?:\b|)" 0; "~*(?:\b)page\ scorer(?:\b|)" 0;
"~*(?:\b)pcBrowser(?:\b|)" 0; "~*(?:\b)pcBrowser(?:\b|)" 0;
"~*(?:\b)plumanalytics(?:\b|)" 0; "~*(?:\b)plumanalytics(?:\b|)" 0;

18
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Anthill CATExplorador
Calculon InternetSeer
Jorgee Mozlila
MolokaiBot OpenVAS
RocketCrawler RocketCrawler
SlySearch SiteSnagger
Spammen SuperHTTP
WallpapersHD WeSEE
Webalta netEstate NE Crawler
seocompany.store tAkeOut

164
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
01h4x.com ALittle Client
80legs Abonti
AIBOT Aboundex
ASPSeek Aboundexbot
Aipbot AhrefsBot
Alligator AiHitBot
BBBike Asterias
BDFetch AwarioRssBot
AwarioSmartBot
BackDoorBot
BackStreet BackStreet
Buck Bigfoot
BunnySlippers BunnySlippers
BuzzSumo Bytespider
CheeseBot Calculon
ChinaClaw Crescent
Crawling at Home Project Custo
DatabaseDriverMysqli DTS Agent
Disco DataCha0s
DnBCrawler-Analytics DomainAppender
DomainStatsBot Dotbot
Download Wonder EasyDL
Drip
Express WebPictures
Ezooms Ezooms
Fimap G-i-g-a-b-o-t
FlashGet Getintent
Flunky Grabber
Foobot HTTrack
FyberSpider Harvest
HEADMasterSEO InterGET
HTMLparser JOC Web Spider
InfoTekies Jorgee
Intelliseek
Jbrofuzz
JikeSpider
JustView
Jyxobot
Kenjin Spider
Keyword Density
Kozmosbot
Larbin Larbin
LeechFTP LieBaoFast
LexiBot LinkScan
LinkWalker Lmspider
LinksManager MIDown tool
LinqiaMetadataDownloaderBot MarkMonitor
Ltx71
MFC_Tear_Sample
MTRobot
Mass Downloader Mass Downloader
MauiBot Mata Hari
MeanPath Bot Meanpathbot
MolokaiBot Mojeek
Name Intelligence Musobot
NICErsPRO
NearSite NearSite
Nutch NetMechanic
OrangeSpider NetZIP
PageScorer Offline Navigator
Picscout Picscout
PleaseCrawl Pockey
Quick-Crawler ProPowerBot
Probethenet
Pump
RankingBot2
RankurBot
ReGet
Reaper Reaper
Ripper Ripper
Scanbot RocketCrawler
ScrepyBot Rogerbot
Semrush SMTBot
SiteSucker Snapbot
Spanner
Spbot
Sqlmap Sqlmap
Sqlworm Stripper
Sucker
T8Abot T8Abot
TheNomad
Thumbor Thumbor
TightTwatBot Trendiction
Toata True_Robot
TwengaBot TurnitinBot
VCI Vagabondo
Voil Voil
Voltron
WEBDAV WEBDAV
WWW::Mechanize WWW-Collector-E
Web Fetch WWWOFFLE
Web Fuck Wallpapers/3.0
WallpapersHD
Web Auto
Web Sucker
WebCollage WebCollage
WebEnhancer WebFuck
WebReaper WebStripper
Widow Webster
Wonderbot Whatweb
adscanner Woobot
cah.io.community Wprecon
ZoomBot
coccocbot coccocbot
cognitiveseo cognitiveseo
crawler.feedback dataforseobot
gopher ips-agent
lwp-request iubenda-radar
mediawords mediawords
oBot openai.com
siteripz page scorer
ubermetrics-technologies.com pcBrowser
webgains-bot plumanalytics
webprosbot seoscanners
sogouspider
sp_auditbot

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
adloads.net advertex.info
alessandraleone.com bannerswap.com
alice.it cheapjerseysbizwholesale.us
gigixo.com domainanalyzing.xyz
justkillingti.me eyes-on-you.ga
kino2018.club houtings.xyz
mostantikor.ru nagdak.ru
priscilarodrigues.com.br obiavo.net
weburlopener.com volume-pills.biz
xwatt.ru www.kabbalah-red-bracelets.com

198
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
2girls1cup-free.com 123locker.com
3xforum.ro 2728fb936f0.com
a96527gi.beget.tech 40cg.com
aanapa.ru 7zap.com
acheterviagrafr24.com 8lufu.com
acortarurl.es accessoires-mode.in
apibatbrowsecom-a.akamaihd.net actulite.com
apimountainbikei-a.akamaihd.net adidas.frwebs.fr
beatifullandscape.co admanaerofoil.com
beauty-clinic.ru adprotect.net
best-coupon-offer.com aloofly.com
best-way.men alphaforum.ru
bestcalovebracelet.cn amazon-seo-service.com
bingoporno.com app-ready.xyz
blogqpot.com arendatora.ru
briomotor.co autorn.ru
bristolhotel.com azazaporn.com
buehne-fuer-menschenrechte.de bg6s0.com
cfacarrosserie74.com biketank.ga
clips.ua.ac.be brainboostingsupplements.org
commoncrawl.org bravegirlsclub.com
coslab.club brianhenry.co
crystalslot.com casablancamanor.co.za
dataloading.net cbb1smartlist12.click
detki-opt.ru ccjp.eu
domaincheck.io chatroulette.online
e2click.com cmd.kz
elektrischezigarette2.devhub.com continent-e.tv
elektronischezigarettekaufen1.myblog.de crazyprotocol.com
ezigarettenkaufen1.deviantart.com crynet.cc
freemags.cc czeshop.info
freetangodownload.com dipstar.org
geckoandfly.com doyouknowtheword-flummox.ml
goosefishpost.bid dtm-spain.com
hkladys.com elektrischezi.canalblog.com
homemade.gq enskedesquashclub.se
houseofrose.com europeanwatches.ru
i-midias.net.br excitacao.com
inet-traffic.com filmbokep69.com
iptool.xyz findpik.com
iqs.biz.ua forum.poker4life.ru
jackwolfskinoutlet.online forums.toucharcade.com
janettabridal.com handicapbathtubarea.com
katadhin.co hdxnxxtube.mobi
kino-filmi.com humbmosquina.tk
kinozapas.org importchinacoach-teach.com
konoplisemena.com inboxdollars.com
kovesszucs.atw.hu jav-way.site
jennyfire.ru
kellyonline.xyz
kimcurlrvsms.com
kinky-fetishes.com
kinobest.pl
languagecode.com
layola.biz.tc layola.biz.tc
letmacwork.world lineshops.biz
linkpulse.com lyrster.com
magicalfind-a.akamaihd.net machicon-ueno.info
mir-limuzinov.ru magicdiet.gq
mmofreegames.online maxthon.com
mwtpludn.review monclerboots.xyz
najaden.no mydearest.co
netpics.org myhydros.org
new7ob.com nakozhe.com
nextconseil.com naval.jislaaik.com
of-ireland.info pdns.cz
offergroup.info perfection-pleasure.ru
partnersafe.men photo.houseofgaga.ru
pomoc-drogowa.cba.pl picmoonco.pw
porodasobak.net pladform.ru
powitania.pl platezhka.net
printingpeach.com rankexperience.com
pronorm.fr seolab.top
prostitutki-tyumeni.org sexsearch.com
quangcaons.com signx.info
rankingchart.de simple-share-buttons.com
raschtextil.com.ua spy-app.info
reeyanaturopathy.com stromerrealty.com
relayblog.com suhanpacktech.com
risparmiocasa.bz.it superstats.com
ruclicks.com sweepstakes.rewardit.com
sanjosestartups.com taraz.xkaz.org
savetubevideo.com textads.men
sayyoethe.blogspot.co.za thaisamkok.com
searchaddis.com time-japan.ru
sexfreepornoxxx.com toon-families.com
shoppingmiracles.co.uk turbabitload.weebly.com
sotkal.lark.ru urzedowski.eu
sports-supplements.us videojam.tv
sticken.co videositename.com
stillmiracle.com vvpg.ru
teguh.info waysbetter.cn
thenetinfo.com wrz0iuebwhp5fg.freeddns.com
trafficmp.com xcc24.pl
uginekologa.com xfire.com
via-gra.webstarts.com y8games-free.com
viagra.pp.ua your-bearings.com
videochat.life yourmovies.pl
videochat.ph yubikk.info
vipsexfinders.com zscaler.net
wdrake.com
wfb.hatedriveapart.com
wikes.20fr.com
xn--80aaaks3bbhabgbigamdr2h.xn--p1ai
yaderenergy.ru
zzlgxh.com

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.121 0; 10.1.0.65 0;
127.0.0.1 0; 127.0.0.1 0;

12
dev-tools/tmp/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:26 UTC 2023 ### Updated: Tue Aug 8 08:59:24 UTC 2023
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -337,6 +337,8 @@ User-agent: Fyrebot
Disallow:/ Disallow:/
User-agent: G-i-g-a-b-o-t User-agent: G-i-g-a-b-o-t
Disallow:/ Disallow:/
User-agent: GPTBot
Disallow:/
User-agent: GT::WWW User-agent: GT::WWW
Disallow:/ Disallow:/
User-agent: GalaxyBot User-agent: GalaxyBot
@ -1227,6 +1229,10 @@ User-agent: netEstate NE Crawler
Disallow:/ Disallow:/
User-agent: oBot User-agent: oBot
Disallow:/ Disallow:/
User-agent: openai
Disallow:/
User-agent: openai.com
Disallow:/
User-agent: page scorer User-agent: page scorer
Disallow:/ Disallow:/
User-agent: pcBrowser User-agent: pcBrowser

12
robots.txt/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2023.08.3918 ### Version: V4.2023.08.3919
### Updated: Mon Aug 7 22:01:26 UTC 2023 ### Updated: Tue Aug 8 08:59:24 UTC 2023
### Bad Bot Count: 642 ### Bad Bot Count: 645
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -337,6 +337,8 @@ User-agent: Fyrebot
Disallow:/ Disallow:/
User-agent: G-i-g-a-b-o-t User-agent: G-i-g-a-b-o-t
Disallow:/ Disallow:/
User-agent: GPTBot
Disallow:/
User-agent: GT::WWW User-agent: GT::WWW
Disallow:/ Disallow:/
User-agent: GalaxyBot User-agent: GalaxyBot
@ -1227,6 +1229,10 @@ User-agent: netEstate NE Crawler
Disallow:/ Disallow:/
User-agent: oBot User-agent: oBot
Disallow:/ Disallow:/
User-agent: openai
Disallow:/
User-agent: openai.com
Disallow:/
User-agent: page scorer User-agent: page scorer
Disallow:/ Disallow:/
User-agent: pcBrowser User-agent: pcBrowser