vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-02 10:40:36 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2023.10.4059 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2023-10-16 11:30:05 +00:00
parent 4e5307013b
commit fa544659ed
37 changed files with 300 additions and 264 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2023.10.4058 #### Version: V4.2023.10.4059
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [645](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [647](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2023.10.4058 #### Version: V4.2023.10.4059
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [645](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [647](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -14,9 +14,9 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2023.10.4058 #### Version: V4.2023.10.4059
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [645](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [647](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
_generator_lists/bad-user-agents.list
View file

@ -369,6 +369,7 @@ PyCurl
QueryN\ Metasearch QueryN\ Metasearch
Quick-Crawler Quick-Crawler
RSSingBot RSSingBot
Rainbot
RankActive RankActive
RankActiveLinkBot RankActiveLinkBot
RankFlex RankFlex
@ -468,6 +469,7 @@ TheNomad
The\ Intraformant The\ Intraformant
Thumbor Thumbor
TightTwatBot TightTwatBot
TinyTestBot
Titan Titan
Toata Toata
Toweyabot Toweyabot
@ -643,5 +645,3 @@ x22Mozilla
xpymep1.exe xpymep1.exe
zauba.io zauba.io
zgrab zgrab
Rainbot
TinyTestBot

4
_google_analytics_ghost_spam/README.md
View file

@ -3,9 +3,9 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2023.10.4058 #### Version: V4.2023.10.4059
#### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7104](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [645](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [647](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

8
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:34 UTC 2023 ### Updated: Mon Oct 16 11:26:14 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

8
conf.d/globalblacklist-testing-version.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2023.10.4058 ### Version: V3.2023.10.4059
### Updated: Mon Oct 16 10:03:35 UTC 2023 ### Updated: Mon Oct 16 11:26:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

8
conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:34 UTC 2023 ### Updated: Mon Oct 16 11:26:14 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

2
dev-tools/_robots_input/robots-input.txt
View file

@ -369,6 +369,7 @@ PyCurl
QueryN Metasearch QueryN Metasearch
Quick-Crawler Quick-Crawler
RSSingBot RSSingBot
Rainbot
RankActive RankActive
RankActiveLinkBot RankActiveLinkBot
RankFlex RankFlex
@ -468,6 +469,7 @@ TheNomad
The Intraformant The Intraformant
Thumbor Thumbor
TightTwatBot TightTwatBot
TinyTestBot
Titan Titan
Toata Toata
Toweyabot Toweyabot

2
dev-tools/buildnumber
View file

@ -1 +1 @@
4058 4059

8
dev-tools/globalblacklist-testing.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2023.10.4058 ### Version: V3.2023.10.4059
### Updated: Mon Oct 16 10:03:35 UTC 2023 ### Updated: Mon Oct 16 11:26:16 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

8
dev-tools/globalblacklist.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:34 UTC 2023 ### Updated: Mon Oct 16 11:26:14 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
40.84.173.36 0; 20.42.8.219 0;

8
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:34 UTC 2023 ### Updated: Mon Oct 16 11:26:14 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.231 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.231 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

8
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:34 UTC 2023 ### Updated: Mon Oct 16 11:26:14 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.231 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.231 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

8
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:34 UTC 2023 ### Updated: Mon Oct 16 11:26:14 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.231 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -430,6 +430,7 @@
"~*(?:\b)QueryN\ Metasearch(?:\b|)" 0; "~*(?:\b)QueryN\ Metasearch(?:\b|)" 0;
"~*(?:\b)Quick-Crawler(?:\b|)" 0; "~*(?:\b)Quick-Crawler(?:\b|)" 0;
"~*(?:\b)RSSingBot(?:\b|)" 0; "~*(?:\b)RSSingBot(?:\b|)" 0;
"~*(?:\b)Rainbot(?:\b|)" 0;
"~*(?:\b)RankActive(?:\b|)" 0; "~*(?:\b)RankActive(?:\b|)" 0;
"~*(?:\b)RankActiveLinkBot(?:\b|)" 0; "~*(?:\b)RankActiveLinkBot(?:\b|)" 0;
"~*(?:\b)RankFlex(?:\b|)" 0; "~*(?:\b)RankFlex(?:\b|)" 0;
@ -529,6 +530,7 @@
"~*(?:\b)The\ Intraformant(?:\b|)" 0; "~*(?:\b)The\ Intraformant(?:\b|)" 0;
"~*(?:\b)Thumbor(?:\b|)" 0; "~*(?:\b)Thumbor(?:\b|)" 0;
"~*(?:\b)TightTwatBot(?:\b|)" 0; "~*(?:\b)TightTwatBot(?:\b|)" 0;
"~*(?:\b)TinyTestBot(?:\b|)" 0;
"~*(?:\b)Titan(?:\b|)" 0; "~*(?:\b)Titan(?:\b|)" 0;
"~*(?:\b)Toata(?:\b|)" 0; "~*(?:\b)Toata(?:\b|)" 0;
"~*(?:\b)Toweyabot(?:\b|)" 0; "~*(?:\b)Toweyabot(?:\b|)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.231 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

8
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:34 UTC 2023 ### Updated: Mon Oct 16 11:26:14 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
40.84.173.36 0; 20.42.8.219 0;

8
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:34 UTC 2023 ### Updated: Mon Oct 16 11:26:14 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
40.84.173.36 0; 20.42.8.219 0;

8
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:34 UTC 2023 ### Updated: Mon Oct 16 11:26:14 UTC 2023
### Bad Referrer Count: 7104 ### Bad Referrer Count: 7104
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -518,6 +518,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)QueryN\ Metasearch(?:\b)" 3; "~*(?:\b)QueryN\ Metasearch(?:\b)" 3;
"~*(?:\b)Quick-Crawler(?:\b)" 3; "~*(?:\b)Quick-Crawler(?:\b)" 3;
"~*(?:\b)RSSingBot(?:\b)" 3; "~*(?:\b)RSSingBot(?:\b)" 3;
"~*(?:\b)Rainbot(?:\b)" 3;
"~*(?:\b)RankActive(?:\b)" 3; "~*(?:\b)RankActive(?:\b)" 3;
"~*(?:\b)RankActiveLinkBot(?:\b)" 3; "~*(?:\b)RankActiveLinkBot(?:\b)" 3;
"~*(?:\b)RankFlex(?:\b)" 3; "~*(?:\b)RankFlex(?:\b)" 3;
@ -617,6 +618,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)The\ Intraformant(?:\b)" 3; "~*(?:\b)The\ Intraformant(?:\b)" 3;
"~*(?:\b)Thumbor(?:\b)" 3; "~*(?:\b)Thumbor(?:\b)" 3;
"~*(?:\b)TightTwatBot(?:\b)" 3; "~*(?:\b)TightTwatBot(?:\b)" 3;
"~*(?:\b)TinyTestBot(?:\b)" 3;
"~*(?:\b)Titan(?:\b)" 3; "~*(?:\b)Titan(?:\b)" 3;
"~*(?:\b)Toata(?:\b)" 3; "~*(?:\b)Toata(?:\b)" 3;
"~*(?:\b)Toweyabot(?:\b)" 3; "~*(?:\b)Toweyabot(?:\b)" 3;

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.231 1; 10.1.0.16 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -430,6 +430,7 @@
"~*(?:\b)QueryN\ Metasearch(?:\b|)" 0; "~*(?:\b)QueryN\ Metasearch(?:\b|)" 0;
"~*(?:\b)Quick-Crawler(?:\b|)" 0; "~*(?:\b)Quick-Crawler(?:\b|)" 0;
"~*(?:\b)RSSingBot(?:\b|)" 0; "~*(?:\b)RSSingBot(?:\b|)" 0;
"~*(?:\b)Rainbot(?:\b|)" 0;
"~*(?:\b)RankActive(?:\b|)" 0; "~*(?:\b)RankActive(?:\b|)" 0;
"~*(?:\b)RankActiveLinkBot(?:\b|)" 0; "~*(?:\b)RankActiveLinkBot(?:\b|)" 0;
"~*(?:\b)RankFlex(?:\b|)" 0; "~*(?:\b)RankFlex(?:\b|)" 0;
@ -529,6 +530,7 @@
"~*(?:\b)The\ Intraformant(?:\b|)" 0; "~*(?:\b)The\ Intraformant(?:\b|)" 0;
"~*(?:\b)Thumbor(?:\b|)" 0; "~*(?:\b)Thumbor(?:\b|)" 0;
"~*(?:\b)TightTwatBot(?:\b|)" 0; "~*(?:\b)TightTwatBot(?:\b|)" 0;
"~*(?:\b)TinyTestBot(?:\b|)" 0;
"~*(?:\b)Titan(?:\b|)" 0; "~*(?:\b)Titan(?:\b|)" 0;
"~*(?:\b)Toata(?:\b|)" 0; "~*(?:\b)Toata(?:\b|)" 0;
"~*(?:\b)Toweyabot(?:\b|)" 0; "~*(?:\b)Toweyabot(?:\b|)" 0;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Bitacle BBBike
Demon Blackboard
Jorgee Citoid
LeechFTP Discobot
Microsoft Data Access Kinza
Mojeek Majestic-SEO
SEOlyticsCrawler Mediatoolkitbot
WeSEE Twice
Zade backlink-check
demandbase-bot sp_auditbot

170
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
80legs ALittle Client
ASPSeek Aboundexbot
AdsTxtCrawlerTP AfD-Verbotsverfahren
Alexibot Alexibot
Asterias Anarchy
Boardreader Ankit
Bolt AwarioRssBot
BotALot Badass
Brandwatch Barkrowler
CheTeam BetaBot
Cocolyzebot Bitacle
Demon Blow
Digincore CopyRightCheck
Curious
DTS Agent
Deusu
Devil
DigitalPebble DigitalPebble
Discobot DittoSpyder
Dispatch DnBCrawler-Analytics
DnyzBot DomCopBot
DomainAppender DomainStatsBot
Download Wonder
EMail Wolf
EasyDL
Ecxi Ecxi
Firefox/7.0 Exabot
Foobot Ezooms
Go!Zilla FemtosearchBot
Go-Ahead-Got-It Fimap
GrapeshotCrawler GT::WWW
HEADMasterSEO GalaxyBot
HTMLparser Grabber
HTTP::Lite Haansoft
Havij Harvest
Iblog InfoNaviRobot
IndeedBot
InternetSeer
Iskanie Iskanie
JustView JikeSpider
Keybot Translation-Search-Machine Kinza
Leap Lanshanbot
LeechGet Libwhisker
LexiBot Litemage_walker
LinkWalker Ltx71
LinqiaMetadataDownloaderBot MQQBrowser
LinqiaScrapeBot Mail.RU_Bot
Lmspider Microsoft Data Access
MIDown tool Moblie Safari
MSFrontPage NPbot
MSIECrawler Nameprotect
Mister PiX Net Vampire
MolokaiBot NetMechanic
Musobot Netvibes
Needle Offline Explorer
Netcraft Openvas
Nibbler OrangeBot
Page Analyzer Picscout
Papa Foto PictureFinder
Piepmatz Pimonster
ProPowerBot Psbot
RankActive RankActive
RankingBot
RankingBot2
ReGet ReGet
SBIder SBIder
SeekportBot SMTBot
SemanticJuice ScanAlert
SentiBot Scrapy
SearchmetricsBot
Seekport
SemrushBot
Seomoz Seomoz
Siteimprove Siphon
Sogou web spider SiteCheckerBotCrawler
Sottopop Spanner
Steeler Spinn3r
SuperBot T0PHackTeam
Teleport Teleport
Telesoft TightTwatBot
TheNomad Toata
Trendictionbot TwengaBot
Twice VCI
Vacuum VelenPublicWebCrawler
Vagabondo VidibleScraper
WBSearchBot Virusdie
Web Auto WWW::Mechanize
WebAuto WebsiteExtractor
WebEnhancer Whatweb
WebSucker Woobot
Website Quester
Webster
WinHTTrack
Wotbox
Xaldon WebSpider
Xenu Xenu
Zitebot
ZoominfoBot
adscanner adscanner
arquivo.pt autoemailspider
eCatch heritrix
facebookscraper isitwp.com
lwp-request openai.com
meanpathbot page scorer
pcBrowser
probe-image-size
seoscanners seoscanners
seostar seostar
trendiction.com
trendiction.de
voyagerx.com voyagerx.com
zauba.io
zgrab zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
101raccoon.ru 5000-cotydzien.com
chatroulette.video buehne-fuer-menschenrechte.de
cialischmrx.com culturevie.info
event-tracking.com dorratex.tn
gardene.ru go2album.com
glopages.ru howmuchdoestizanidinecost.blogspot.com
letolove.ru ipostroika.ru
lomb.co medizinreisen.de
renecaovilla.online puserving.com
snjatie-geroinovoy-lomki.ru serw.clicksor.com

196
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,100 +1,100 @@
allboard.xobor.de 45en.ru
alloysteel.ru 999webdesign.com
arate.ru adloads.com
avon-ukraine.com aramaicmedia.org
backlinkwatch.com aversis.net
bildsuche.ru aviav.co
bio.trade-jp.net bannerbank.ru
blog.koorg.ru beclean-nn.ru
camel-beach.com bidbuy.co.kr
cartierlove.xyz bonzibuddi.com
cezartabac.ro cialischmrx.com
compliance-alex.xyz commoncrawl.org
compliance-brian.xyz cornomase.win
contextualyield.com corta.co
cordstrap.cc creatives.adbetclickin.pink
culmimg.pw dev.citetick.com
customerguru.in donna7753191.ru
div.as elvenmachine.com
dolohen.com englishgamer.com
dvervmoskvu.ru erolate.com
e-commerce-seo.com erotik0049.com
earncash.com.ua etotupo.ru
evogarage.com etur.ru
executehosting.com exdocsfiles.com
expolicenciaslatam.co ezigarettenshop1.yolasite.com
ezofest.sk fba-mexico.com
f5mtrack.com filesdatabase.net
flashbannernow.com footbalive.org
fordsonmajbor.cf freshsuperbloop.com
forexgb.ru freza-sverlo.ru
forms-mtm.ru frvo.alptandem.ru
gallery.rennlist.com ftns.ru
googst2.ru fyxabomiw.ru
goroda-vsego-mira.ru gearsadspromo.club
gribokstop.com getmiro.com
ihtec2019.org grtyi.com
ilo134uloh.com hdpixent.com
imperiafilm.ru hobbyhuren-datenbank.com
infospot.pt houseofrose.com
innodgfdriverhm.aircus.com ideibiznesa2015.ru
inome.com.ua ilte.info
iskalko.ru infoupdate.org
itbc.kiev.ua intervsem.ru
jikoman.info ispac.org
kaac.ru jobmarket.com.ua
kinostorm.net kinoflux.net
kvartira-sutochno.com kinozapas.org
landmania.ru krasnodar-avtolombards.ru
letsart.ru landscaping.center
limads.men lexixxx.com
lion.playtap.us
locustdesign.co
mafcards.ru
medicinacom.ru
meetlocalchicks.com meetlocalchicks.com
mesbuta.info mega-bony-2017.pl
metallosajding.ru methodsmarketing.com
minet.club mielec.pl
my-floor.in.ua mojpregled.com
naturtreenspicerx.pw moncleroutletonline.win
no-fuel.org money-every-day.com
noscrapleftbehind.co mycouponizemac.com
onclkads.com myra.top
pcgroup.com.uy onlainbesplatno.ru
pharmacyincity.com pendelprognos.se
play.leadzupc.com photostudiolightings.com
popup-fdm.xyz pobeiranie.pl
pozdrawleniya.ru porndairy.in
producm.ru pornokajf.com
prospekt-st.ru promodj.com
purplesphere.in
rada.ru
razamicroelectronics.com razamicroelectronics.com
redhotfreebies.co.uk rebuildermedical.com
rep-am.com redirectingat.com
replicalouboutin.xyz referencemoi.com
ringtonepartner.com resant.ru
sale-japan.com revistaindustria.com
seogadget.ru sbf441.com
sicfor.bcu.cc scanner-jess.top
soft-program.com scanner-willy.top
stretchingabuckblog.com sexblog.pw
supervesti.ru sexsaoy.com
thecoral.com.br sms2x2.ru
thefds.net solinf.co
thegameriders.com speedup-my.site
tizanidinedrugtest.blogspot.com spylog.com
tovaroboom.vast.ru stauga.altervista.org
tsyndicate.com sureone.pro
txxx.com symbaloo.com
videochat.cafe taxi-v-eisk.ru
vip-dom.in topflownews.com
vkontarkte.com v720hd.ru
vykup-avto-krasnodar.ru videochat.ph
vzubah.com videochat.tv.br
watchmygf.net viteonlusarezzo.it
websitevaluebot.com vzlomfb.com
womama.ru webradiology.ru
xcombear.ru www.souvenirua.com
xn----btbdvdh4aafrfciljm6k.xn--p1ai xn--80ajbshivpvn2i.xn--p1ai
xxxtubesafari.com yurgorod.ru
yes-com.com zooggames.com
zaobao.com.sg
zx6.ru

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.231 0; 10.1.0.16 0;
127.0.0.1 0; 127.0.0.1 0;

10
dev-tools/tmp/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:36 UTC 2023 ### Updated: Mon Oct 16 11:26:16 UTC 2023
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -757,6 +757,8 @@ User-agent: Quick-Crawler
Disallow:/ Disallow:/
User-agent: RSSingBot User-agent: RSSingBot
Disallow:/ Disallow:/
User-agent: Rainbot
Disallow:/
User-agent: RankActive User-agent: RankActive
Disallow:/ Disallow:/
User-agent: RankActiveLinkBot User-agent: RankActiveLinkBot
@ -955,6 +957,8 @@ User-agent: Thumbor
Disallow:/ Disallow:/
User-agent: TightTwatBot User-agent: TightTwatBot
Disallow:/ Disallow:/
User-agent: TinyTestBot
Disallow:/
User-agent: Titan User-agent: Titan
Disallow:/ Disallow:/
User-agent: Toata User-agent: Toata

10
robots.txt/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2023.10.4058 ### Version: V4.2023.10.4059
### Updated: Mon Oct 16 10:03:36 UTC 2023 ### Updated: Mon Oct 16 11:26:16 UTC 2023
### Bad Bot Count: 645 ### Bad Bot Count: 647
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -757,6 +757,8 @@ User-agent: Quick-Crawler
Disallow:/ Disallow:/
User-agent: RSSingBot User-agent: RSSingBot
Disallow:/ Disallow:/
User-agent: Rainbot
Disallow:/
User-agent: RankActive User-agent: RankActive
Disallow:/ Disallow:/
User-agent: RankActiveLinkBot User-agent: RankActiveLinkBot
@ -955,6 +957,8 @@ User-agent: Thumbor
Disallow:/ Disallow:/
User-agent: TightTwatBot User-agent: TightTwatBot
Disallow:/ Disallow:/
User-agent: TinyTestBot
Disallow:/
User-agent: Titan User-agent: Titan
Disallow:/ Disallow:/
User-agent: Toata User-agent: Toata