vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-05 04:00:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.01.2841 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-01-09 07:06:17 +00:00
parent 0c4751ba90
commit 777b44bba4
53 changed files with 1170 additions and 1125 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.01.2840 #### Version: V4.2022.01.2841
#### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [632](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,9 +4,9 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.01.2840 #### Version: V4.2022.01.2841
#### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [632](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -13,9 +13,9 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.01.2840 #### Version: V4.2022.01.2841
#### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [632](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

2
_generator_lists/bad-user-agents.list
View file

@ -1,4 +1,3 @@
Seekport
01h4x.com 01h4x.com
360Spider 360Spider
404checker 404checker
@ -400,6 +399,7 @@ ScreenerBot
ScrepyBot ScrepyBot
Searchestate Searchestate
SearchmetricsBot SearchmetricsBot
Seekport
SemanticJuice SemanticJuice
Semrush Semrush
SemrushBot SemrushBot

6
_generator_lists/good-user-agents.list
View file

@ -1,7 +1,5 @@
AdsBot-Google AdsBot-Google
Applebot Applebot
adidxbot
aolbuild
DoCoMo DoCoMo
Feedfetcher-Google Feedfetcher-Google
Google-HTTP-Java-Client Google-HTTP-Java-Client
@ -21,6 +19,9 @@ Slackbot
Slackbot-LinkExpanding Slackbot-LinkExpanding
TwitterBot TwitterBot
Wordpress Wordpress
adidxbot
aolbuild
bing
bingbot bingbot
bingpreview bingpreview
developers.facebook.com developers.facebook.com
@ -33,4 +34,3 @@ msnbot-media
slurp slurp
teoma teoma
yahoo yahoo
bing

4
_google_analytics_ghost_spam/README.md
View file

@ -3,9 +3,9 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.01.2840 #### Version: V4.2022.01.2841
#### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [632](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

9
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:51 UTC 2022 ### Updated: Sun Jan 9 07:00:22 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

9
conf.d/globalblacklist-testing-version.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2840 ### Version: V3.2022.01.2841
### Updated: Sun Jan 9 06:33:52 UTC 2022 ### Updated: Sun Jan 9 07:00:23 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

9
conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:51 UTC 2022 ### Updated: Sun Jan 9 07:00:22 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

1
dev-tools/_robots_input/robots-input.txt
View file

@ -399,6 +399,7 @@ ScreenerBot
ScrepyBot ScrepyBot
Searchestate Searchestate
SearchmetricsBot SearchmetricsBot
Seekport
SemanticJuice SemanticJuice
Semrush Semrush
SemrushBot SemrushBot

9
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2840 ### Version: V3.2022.01.2841
### Updated: Sun Jan 9 06:33:52 UTC 2022 ### Updated: Sun Jan 9 07:00:23 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 1; 10.1.0.222 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 0; 10.1.0.222 0;
127.0.0.1 0; 127.0.0.1 0;

9
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2840 ### Version: V3.2022.01.2841
### Updated: Sun Jan 9 06:33:52 UTC 2022 ### Updated: Sun Jan 9 07:00:23 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 1; 10.1.0.222 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 0; 10.1.0.222 0;
127.0.0.1 0; 127.0.0.1 0;

9
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2840 ### Version: V3.2022.01.2841
### Updated: Sun Jan 9 06:33:52 UTC 2022 ### Updated: Sun Jan 9 07:00:23 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 1; 10.1.0.222 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -460,6 +460,7 @@
"~*(?:\b)ScrepyBot(?:\b|)" 0; "~*(?:\b)ScrepyBot(?:\b|)" 0;
"~*(?:\b)Searchestate(?:\b|)" 0; "~*(?:\b)Searchestate(?:\b|)" 0;
"~*(?:\b)SearchmetricsBot(?:\b|)" 0; "~*(?:\b)SearchmetricsBot(?:\b|)" 0;
"~*(?:\b)Seekport(?:\b|)" 0;
"~*(?:\b)SemanticJuice(?:\b|)" 0; "~*(?:\b)SemanticJuice(?:\b|)" 0;
"~*(?:\b)Semrush(?:\b|)" 0; "~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0; "~*(?:\b)SemrushBot(?:\b|)" 0;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 0; 10.1.0.222 0;
127.0.0.1 0; 127.0.0.1 0;

9
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2840 ### Version: V3.2022.01.2841
### Updated: Sun Jan 9 06:33:52 UTC 2022 ### Updated: Sun Jan 9 07:00:23 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2840 2841

9
dev-tools/globalblacklist-testing.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2840 ### Version: V3.2022.01.2841
### Updated: Sun Jan 9 06:33:52 UTC 2022 ### Updated: Sun Jan 9 07:00:23 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

9
dev-tools/globalblacklist.template
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:51 UTC 2022 ### Updated: Sun Jan 9 07:00:22 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.112.84.204 0; 20.127.127.68 0;

9
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:51 UTC 2022 ### Updated: Sun Jan 9 07:00:22 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 1; 10.1.0.222 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 0; 10.1.0.222 0;
127.0.0.1 0; 127.0.0.1 0;

9
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:51 UTC 2022 ### Updated: Sun Jan 9 07:00:22 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 1; 10.1.0.222 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 0; 10.1.0.222 0;
127.0.0.1 0; 127.0.0.1 0;

9
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:51 UTC 2022 ### Updated: Sun Jan 9 07:00:22 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 1; 10.1.0.222 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-user-agents.conf
View file

@ -460,6 +460,7 @@
"~*(?:\b)ScrepyBot(?:\b|)" 0; "~*(?:\b)ScrepyBot(?:\b|)" 0;
"~*(?:\b)Searchestate(?:\b|)" 0; "~*(?:\b)Searchestate(?:\b|)" 0;
"~*(?:\b)SearchmetricsBot(?:\b|)" 0; "~*(?:\b)SearchmetricsBot(?:\b|)" 0;
"~*(?:\b)Seekport(?:\b|)" 0;
"~*(?:\b)SemanticJuice(?:\b|)" 0; "~*(?:\b)SemanticJuice(?:\b|)" 0;
"~*(?:\b)Semrush(?:\b|)" 0; "~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0; "~*(?:\b)SemrushBot(?:\b|)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 0; 10.1.0.222 0;
127.0.0.1 0; 127.0.0.1 0;

9
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:51 UTC 2022 ### Updated: Sun Jan 9 07:00:22 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.112.84.204 0; 20.127.127.68 0;

9
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:51 UTC 2022 ### Updated: Sun Jan 9 07:00:22 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
20.112.84.204 0; 20.127.127.68 0;

9
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

@ -4,10 +4,10 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:51 UTC 2022 ### Updated: Sun Jan 9 07:00:22 UTC 2022
### Bad Referrer Count: 7087 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -548,6 +548,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ScrepyBot(?:\b)" 3; "~*(?:\b)ScrepyBot(?:\b)" 3;
"~*(?:\b)Searchestate(?:\b)" 3; "~*(?:\b)Searchestate(?:\b)" 3;
"~*(?:\b)SearchmetricsBot(?:\b)" 3; "~*(?:\b)SearchmetricsBot(?:\b)" 3;
"~*(?:\b)Seekport(?:\b)" 3;
"~*(?:\b)SemanticJuice(?:\b)" 3; "~*(?:\b)SemanticJuice(?:\b)" 3;
"~*(?:\b)Semrush(?:\b)" 3; "~*(?:\b)Semrush(?:\b)" 3;
"~*(?:\b)SemrushBot(?:\b)" 3; "~*(?:\b)SemrushBot(?:\b)" 3;
@ -793,6 +794,7 @@ map $http_user_agent $bad_bot {
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### # START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)AdsBot-Google(?:\b)" 0; "~*(?:\b)AdsBot-Google(?:\b)" 0;
"~*(?:\b)Applebot(?:\b)" 0;
"~*(?:\b)DoCoMo(?:\b)" 0; "~*(?:\b)DoCoMo(?:\b)" 0;
"~*(?:\b)Feedfetcher-Google(?:\b)" 0; "~*(?:\b)Feedfetcher-Google(?:\b)" 0;
"~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0; "~*(?:\b)Google-HTTP-Java-Client(?:\b)" 0;
@ -814,6 +816,7 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wordpress(?:\b)" 0; "~*(?:\b)Wordpress(?:\b)" 0;
"~*(?:\b)adidxbot(?:\b)" 0; "~*(?:\b)adidxbot(?:\b)" 0;
"~*(?:\b)aolbuild(?:\b)" 0; "~*(?:\b)aolbuild(?:\b)" 0;
"~*(?:\b)bing(?:\b)" 0;
"~*(?:\b)bingbot(?:\b)" 0; "~*(?:\b)bingbot(?:\b)" 0;
"~*(?:\b)bingpreview(?:\b)" 0; "~*(?:\b)bingpreview(?:\b)" 0;
"~*(?:\b)developers.facebook.com(?:\b)" 0; "~*(?:\b)developers.facebook.com(?:\b)" 0;

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 1; 10.1.0.222 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test_units/blacklist-user-agents.conf
View file

@ -460,6 +460,7 @@
"~*(?:\b)ScrepyBot(?:\b|)" 0; "~*(?:\b)ScrepyBot(?:\b|)" 0;
"~*(?:\b)Searchestate(?:\b|)" 0; "~*(?:\b)Searchestate(?:\b|)" 0;
"~*(?:\b)SearchmetricsBot(?:\b|)" 0; "~*(?:\b)SearchmetricsBot(?:\b|)" 0;
"~*(?:\b)Seekport(?:\b|)" 0;
"~*(?:\b)SemanticJuice(?:\b|)" 0; "~*(?:\b)SemanticJuice(?:\b|)" 0;
"~*(?:\b)Semrush(?:\b|)" 0; "~*(?:\b)Semrush(?:\b|)" 0;
"~*(?:\b)SemrushBot(?:\b|)" 0; "~*(?:\b)SemrushBot(?:\b|)" 0;

2
dev-tools/test_units/good-bots-for-test.list
View file

@ -1,4 +1,5 @@
AdsBot-Google AdsBot-Google
Applebot
DoCoMo DoCoMo
Feedfetcher-Google Feedfetcher-Google
Google-HTTP-Java-Client Google-HTTP-Java-Client
@ -20,6 +21,7 @@ TwitterBot
Wordpress Wordpress
adidxbot adidxbot
aolbuild aolbuild
bing
bingbot bingbot
bingpreview bingpreview
developers.facebook.com developers.facebook.com

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
CSHttp Alexibot
CherryPicker BetaBot
Dispatch Crawling at Home Project
Larbin Genieo
Scrapy GetRight
Sitebeam Grabber
Sucuri Image Sucker
WEBDAV Iria
Web Fuck Rankivabot
WebReaper Re-re

306
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
80legs 360Spider
Abonti 404checker
Aboundexbot ADmantX
AIBOT
ALittle Client
Aboundex
Acunetix
AhrefsBot
AiHitBot AiHitBot
Aipbot AllSubmitter
Alexibot
AlphaBot
Anarchy99 Anarchy99
Ankit
Anthill
Asterias
AwarioRssBot AwarioRssBot
AwarioSmartBot AwarioSmartBot
BBBike BackStreet
BackDoorBot Backlink-Ceck
BackWeb
BacklinkCrawler BacklinkCrawler
Badass Badass
Bandit
Battleztar Bazinga
Bigfoot
Bitacle
Blackboard Blackboard
BlowFish Blow
Boardreader Boardreader
Brandprotect Brandwatch
Bullseye Buck
BunnySlippers
BuzzSumo
CATExplorador CATExplorador
Calculon CCBot
CSHttp
Cegbfeieh
ChinaClaw
Chlooe
Citoid Citoid
Claritybot Claritybot
Cocolyzebot Cliqzbot
Cogentbot Cogentbot
Cosmos Copier
Craftbot CrazyWebCrawler
Crawling at Home Project CrunchBot
Crescent
Curious Curious
Custo DataCha0s
CyotekWebCopy Demon
DSearch Devil
DTS Agent Dirbuster
Disco Disco
Dispatch DittoSpyder
DnBCrawler-Analytics DnBCrawler-Analytics
DnyzBot DnyzBot
DomainAppender
DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
DomainStatsBot Domains Project
ECCP/1.0 Dotbot
EMail Wolf
EasyDL
Ebingbong Ebingbong
Exabot Ecxi
EirGrabber
EroCrawler
Express WebPictures
ExtLinksBot
Extractor
ExtractorPro
EyeNetIE
Ezooms Ezooms
FDM FDM
FHscan FHscan
Firefox/7.0 FemtosearchBot
Fimap
FlashGet FlashGet
Foobot G-i-g-a-b-o-t
Fuzz
FyberSpider
GT::WWW GT::WWW
Genieo Getintent
GetRight
GetWeb
Gotit
GrabNet
Grabber Grabber
Grafula
GrapeFX GrapeFX
GridBot HaosouSpider
HTTrack Harvest
Havij Havij
Heritrix Heritrix
Hloader Hloader
HybridBot
IDBTE4M
IDBot IDBot
IlseBot Image Fetch
Indy Library IndeedBot
InfoNaviRobot
InfoTekies InfoTekies
InterGET InterGET
Internet Ninja Iria
InternetSeer Iskanie
IstellaBot IstellaBot
JamesBOT
Jbrofuzz
JennyBot
JetCar
JikeSpider
Joomla
Jorgee Jorgee
Kenjin Spider
Keyword Density Keyword Density
Kinza
LNSpiderguy LNSpiderguy
Lanshanbot Lanshanbot
Larbin Larbin
LeechFTP
LeechGet LeechGet
LexiBot
Lftp
Libwhisker Libwhisker
LieBaoFast Linkdexbot
Lightspeedsystems LinkextractorPro
LinksManager LinqiaScrapeBot
LinqiaRSSBot Ltx71
Lipperhey
MFC_Tear_Sample
MIDown tool MIDown tool
MSIECrawler MQQBrowser
MTRobot Mag-Net
Magnet
Mail.RU_Bot Mail.RU_Bot
Majestic-SEO
Majestic12
MarkMonitor
MarkWatch MarkWatch
Mass Downloader Mass Downloader
Mata Hari
Mb2345Browser Mb2345Browser
MeanPath Bot
Meanpathbot Meanpathbot
Mediatoolkitbot Mojolicious
Metauri NICErsPRO
MicroMessenger Navroad
Microsoft Data Access
Mojeek
Morfeus Fucking Scanner
Mozlila
Mr.4x3
Msrabot
Nessus Nessus
NetSpider NetSpider
NetZIP NetZIP
Netcraft
Netvibes Netvibes
NextGenSearchBot Nikto
Nibbler Nutch
NimbleCrawler
Ninja
Octopus Octopus
Offline Explorer
Offline Navigator Offline Navigator
OrangeBot OpenLinkProfiler
OpenVAS
Openfind
OrangeSpider
PHPCrawl PHPCrawl
PageThing.com PageAnalyzer
PageScorer
Panscient Panscient
Papa Foto Pavuk
PeoplePal Petalbot
Pi-Monster Picscout
Pimonster PictureFinder
PleaseCrawl Piepmatz
Pockey ProPowerBot
Probethenet ProWebWalker
Pu_iN Psbot
QueryN Metasearch PyCurl
Quick-Crawler
RSSingBot
RankActive RankActive
RankFlex RankFlex
RankingBot RankingBot
RankurBot Rankivabot
RealDownload Reaper
Recorder Recorder
RedesScrapy RepoMonkey
Rogerbot SEOkicks
SBIder SISTRIX
SMTBot SalesIntelligent
ScanAlert
Scanbot
Scrapy Scrapy
Searchestate SearchmetricsBot
SemanticJuice Seomoz
SemrushBot Shodan
SeobilityBot
Siphon
Site Sucker Site Sucker
SiteCheckerBotCrawler SiteExplorer
SiteLockSpider
SiteSnagger
SiteSucker SiteSucker
Sitebeam
Siteimprove
Sitevigil Sitevigil
SmartDownload Snoopy
SocialRankIOBot Sosospider
Sogou web spider Spinn3r
SpankBot
Spbot
SputnikBot
Sqlmap Sqlmap
Sucker Sucuri
SuperBot
SuperHTTP SuperHTTP
SurveyBot SurveyBot
Suzuran Suzuran
T0PHackTeam TeleportPro
Telesoft
Telesphorep Telesphorep
Thumbor Thumbor
Titan Titan
Toata Toata
Toweyabot Toweyabot
Tracemyfile
Trendiction Trendiction
True_Robot
Turingos Turingos
TurnitinBot URLy Warning
TwengaBot URLy.Warning
Twice
Upflow Upflow
VelenPublicWebCrawler VB Project
Vacuum
VeriCiteCrawler
VidibleScraper VidibleScraper
VoidEYE Voil
WEBDAV WEBDAV
WWW-Collector-E WPScan
WWWOFFLE WWW::Mechanize
WallpapersHD WallpapersHD
WeSEE
Web Auto Web Auto
Web Collage Web Fuck
Web Sauger Web Sauger
WebAuto WebAuto
WebBandit WebBandit
WebGo IS WebCollage
WebImageCollector WebCopier
WebEnhancer
WebFetch
WebLeacher WebLeacher
WebPix WebPix
WebReaper WebReaper
WebSauger WebStripper
WebSucker
WebWhacker
WebZIP WebZIP
WinHTTrack Webshag
Woobot Webster
Xaldon_WebSpider Whatweb
Wotbox
Xenu Xenu
YoudaoBot
Zade
Zauba
Zermelo Zermelo
Zeus
Zitebot
ZoominfoBot
ZumBot ZumBot
ZyBorg ZyBorg
arquivo.pt adscanner
autoemailspider archive.org_bot
cah.io.community arquivo-web-crawler
clark-crawler crawler.feedback
cognitiveseo
crawler4j crawler4j
heritrix demandbase-bot
domainsproject.org
evc-batch
facebookscraper
instabid
internetVista monitor internetVista monitor
ips-agent ips-agent
lwp-request
meanpathbot
netEstate NE Crawler netEstate NE Crawler
oBot oBot
page scorer plumanalytics
pcBrowser
polaris version
scalaj-http
seocompany.store seocompany.store
serpstatbot seoscanners
sitechecker.pro siteripz
sogouspider spyfu
sysscan tAkeOut
trendiction.com
trendiction.de trendiction.de
webgains-bot ubermetrics-technologies.com
webmeup-crawler
x09Mozilla x09Mozilla
zauba.io
zgrab zgrab

308
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
404enemy 01h4x.com
80legs ADmantX
AIBOT ALittle Client
Aboundex Abonti
Aboundexbot
Acunetix Acunetix
AfD-Verbotsverfahren AhrefsBot
Alexibot Aipbot
AllSubmitter
Alligator Alligator
Anarchie AlphaBot
AwarioRssBot Apexoo
BBBike Aspiegel
Attach
BDFetch BDFetch
BackDoorBot BLEXBot
Backlink-Ceck Backlink-Ceck
BacklinkCrawler Battleztar Bazinga
Badass BetaBot
BatchFTP Bitacle
Black Hole Black Hole
BlackWidow Blow
Blackboard
Bolt Bolt
BotALot
Brandwatch
Buck Buck
Buddy
BuiltBotTough
BunnySlippers
CATExplorador CATExplorador
CCBot
CODE87
CSHttp CSHttp
Calculon Calculon
CheTeam CazoodleBot
Claritybot Cegbfeieh
CensysInspect
CheeseBot
CherryPicker
Cloud mapping Cloud mapping
Cocolyzebot Cogentbot
Collector Copyscape
CopyRightCheck
Cosmos
Crawling at Home Project Crawling at Home Project
CrazyWebCrawler
CrunchBot
Curious Curious
CyotekWebCopy DIIbot
DTS Agent DSearch
Deusu DatabaseDriverMysqli
Devil Demon
Discoverybot Digincore
DittoSpyder Dirbuster
DnBCrawler-Analytics
DnyzBot
DomCopBot
DomainAppender DomainAppender
Dragonfly DomainCrawler
Drip DomainSigmaCrawler
DomainStatsBot
Dotbot
EMail Siphon
EMail Wolf EMail Wolf
EasyDL EroCrawler
Exabot Extractor
Ezooms ExtractorPro
Extreme Picture Finder
EyeNetIE
FemtosearchBot FemtosearchBot
Fimap
FlashGet FlashGet
Flunky Foobot
Fuzz
G-i-g-a-b-o-t G-i-g-a-b-o-t
GT::WWW GT::WWW
GalaxyBot GalaxyBot
Genieo
GetRight
GetWeb
Go!Zilla
GoZilla GoZilla
Gotit GrapeshotCrawler
GrabNet HTMLparser
Grafula
HMView
HTTP::Lite HTTP::Lite
Haansoft
HaosouSpider
Harvest
Humanlinks
HybridBot HybridBot
IDBTE4M
IDBot
IRLbot IRLbot
Iblog
Id-search
IlseBot
Image Fetch Image Fetch
Image Sucker Image Sucker
InfoNaviRobot Indy Library
Intelliseek
Internet Ninja Internet Ninja
Iria Iria
IstellaBot
JOC Web Spider JOC Web Spider
JamesBOT Jbrofuzz
JennyBot
JetCar
Jetty Jetty
JikeSpider
Joomla Joomla
Jyxobot Jorgee
Keyword Density
Kozmosbot
LNSpiderguy LNSpiderguy
LWP::Simple LWP::Simple
Larbin Larbin
LeechFTP
LexiBot
Lftp Lftp
LibWeb LibWeb
Libwhisker Libwhisker
Lightspeedsystems Lightspeedsystems
Likse
Linkdexbot
LinqiaMetadataDownloaderBot
Lipperhey Lipperhey
Litemage_walker Ltx71
Lmspider
MFC_Tear_Sample
MIDown tool
MIIxpc
MJ12bot
MQQBrowser MQQBrowser
MSFrontPage MSFrontPage
MSIECrawler MSIECrawler
Mag-Net MTRobot
Magnet
Mail.RU_Bot Mail.RU_Bot
Majestic SEO Majestic-SEO
Majestic12 MarkMonitor
MarkWatch
Mass Downloader
MauiBot
MeanPath Bot MeanPath Bot
Meanpathbot Meanpathbot
Metauri MicroMessenger
Microsoft Data Access Microsoft URL Control
Minefield Mr.4x3
Mister PiX Needle
Moblie Safari
Morfeus Fucking Scanner
Mozlila
Msrabot
Nameprotect
Navroad
NetAnts NetAnts
NetSpider NetSpider
NetZIP NetZIP
Netcraft
Netvibes Netvibes
Ninja NimbleCrawler
Not Not
Nuclei
Octopus
Offline Explorer Offline Explorer
Offline Navigator OnCrawl
OpenLinkProfiler Openfind
OutfoxBot OrangeBot
PECL::HTTP
PHPCrawl PHPCrawl
POE-Component-Client-HTTP Page Analyzer
PageAnalyzer PageAnalyzer
PageScorer
Pandalytics
Panscient Panscient
PeoplePal PeoplePal
Petalbot
Pi-Monster Pi-Monster
Piepmatz PictureFinder
Pimonster Pimonster
PleaseCrawl
Pockey
ProPowerBot
ProWebWalker ProWebWalker
PxBroker Psbot
Pu_iN
PyCurl PyCurl
Quick-Crawler QueryN Metasearch
RankingBot RSSingBot
Re-re
ReGet ReGet
RealDownload RealDownload
SEOkicks-Robot Reaper
RebelMouse
Rogerbot
SEOstats
SISTRIX SISTRIX
SMTBot SalesIntelligent
ScanAlert ScanAlert
ScoutJet Scanbot
Scrapy
Screaming
ScreenerBot ScreenerBot
ScrepyBot Seekport
Searchestate SentiBot
Semrush SeoSiteCheckup
SemrushBot Shodan
SiteCheckerBotCrawler Site Sucker
SiteSnagger SiteLockSpider
Sitebeam
Siteimprove Siteimprove
Sitevigil Snake
SmartDownload Sociscraper
Snoopy
Sottopop Sottopop
SpaceBison
Spanner
Spinn3r Spinn3r
Sqlworm
Steeler
Stripper Stripper
Sucuri
SuperHTTP
Surfbot Surfbot
Swiftbot Suzuran
T0PHackTeam Telesphoreo
Teleport TheNomad
TeleportPro Thumbor
Telesphorep
Toweyabot Toweyabot
Turingos Tracemyfile
TurnitinBot Trendiction
Trendictionbot
TwengaBot TwengaBot
Typhoeus Typhoeus
URLy Warning
V-BOT
VCI VCI
Vacuum
Vagabondo Vagabondo
VeriCiteCrawler Voltron
VidibleScraper
WASALive-Bot WASALive-Bot
WEBDAV
WISENutbot WISENutbot
WWW-Collector-E
WWW-Mechanize WWW-Mechanize
WWW::Mechanize WWW::Mechanize
WWWOFFLE
Wallpapers Wallpapers
Wallpapers/3.0 Wallpapers/3.0
WeSEE Web Auto
Web Enhancer Web Collage
Web Fuck
Web Pix Web Pix
Web Sauger Web Sauger
WebAuto Web Sucker
WebBandit WebLeacher
WebEnhancer WebPix
WebFuck WebReaper
WebImageCollector WebWhacker
WebSauger Webshag
WebSucker
WebmasterWorldForumBot
WebsiteExtractor WebsiteExtractor
WebsiteQuester
Webster Webster
Whack WinHTTrack
Who.is Bot Xaldon WebSpider
Widow Xaldon_WebSpider
Woobot Xenu
YoudaoBot YoudaoBot
Zade Zade
Zauba
Zeus Zeus
Zitebot Zitebot
ZmEu ZyBorg
ZoomBot
ZoominfoBot
ZumBot
adscanner adscanner
backlink-check
cah.io.community cah.io.community
coccocbot-web coccocbot-web
cognitiveseo cognitiveseo
crawler.feedback crawler4j
domainsproject.org
eCatch eCatch
evc-batch
gopher
internetVista monitor internetVista monitor
ips-agent
isitwp.com
magpie-crawler
meanpathbot meanpathbot
muhstik-scan plumanalytics
netEstate NE Crawler satoristudio.net
oBot scalaj-http
pcBrowser seocompany.store
polaris version sogouspider
probe-image-size spyfu
ripz
seoscanners
seostar
siteripz
tAkeOut tAkeOut
x09Mozilla trendiction.com
trendiction.de
webpros.com
zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
1millionusd.xyz 28n2gl3wfyb0.ru
cartierbracelet.xyz cindymatches.com
gigixo.com dekorkeramik.ru
i-service.kz panamaforbeginners.com
konoplisemena.com pogodnyyeavarii.gq
m0r0zk0-krava.ru sluganarodu.ru
medmajor.ru vladhistory.com
play.leadzupc.com watracker.net
swinger-mobil.net xn--80ak6aa92e.com
touchmods.fr xrus.org

936
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
007angels.com
0912701309f8ce.com
11235813.webzdarma.cz 11235813.webzdarma.cz
4kepics.com 1314dh.com
4replicawatch.net 178evakuator178.ru
4webmasters.org 1qingdao.com
6hopping.com 24x7-server-support.site
76brighton.co.uk 2rich4bitches.com
absolute-s.ru 40cg.com
absugars.com 8jn.dating-app.ru
academiacsmendoza.org 91abcw.com
actulite.com 991mostfm.co.id
adultfullhd.com 999webdesign.com
aero2.ru a96527gi.beget.tech
aadroid.net
acortarurl.es
ad-words.ru
adsland.men
affordablewebsitesandmobileapps.com
agapovdg.ru
aksonural.ru aksonural.ru
akusherok.ru akuhni.by
aliexpresscashback.club alert-fjg.xyz
allnews24.in alienwheel.es
allbizne.co.ua
allsilver925.co.il allsilver925.co.il
alyeskaresort.com am-se.com
amanda-porn.ga ameblo.jp
amigobulls.com
androids-store.com androids-store.com
anonymousfox.co animaltoplist.com
apishops.ru animenime.ru
apiallgeniusinfo-a.akamaihd.net
apioutoboxnet-a.akamaihd.net
appartement-stumm.at appartement-stumm.at
apple.com-webbrowsing-security.review
apple.com-webbrowsing-security.science apple.com-webbrowsing-security.science
arraty.altervista.org applicationg29.com
ausergrubhof.info appmsr.org
autobudpostach.club arcadeplayhouse.com
autodo.info arkartex.ru
autoplate.info ascat.porn
aviav.eu autoloans.com
autonew.biz
aversis.net
avon-novosib.ru
avtolombard-krasnodar.com
azazaporn.com
azazu.ru
azte.ch azte.ch
b00kmarks.com backlinkwatch.com
bala.getenjoyment.net baltgem.com
bangla-chat-uk-paralud.ga banan.tv
barnfurnituremart.com banksrf.ru
beauty-things.com bannerswap.com
beclean-nn.ru batanga.net
bedandbreakfast.com battle.net
best-games.today battlecarnival.su
bestempresas.es bausparen.bz.it
bibys.com beatifullandscape.co
bildsuche.ru best-coupon-offer.com
bio-japan.net betune.onlinewebshop.net
bjgugu.net.ua bif-ru.info
blobar.org bio-market.kz
blog.koreadaily.com blackcurranthumidifiers.site
blogig.org blog.f00kclan.de
blpmovies.com bloggen.be
bmwhighperformers.com blogqpot.com
bobba.dzaba.com bnt-team.com
bodybuilding-shop.biz bonanza-fortune.men
bolitgorlo.net brateg.xyz
brk-rti.ru btvn.ru
builtwith.com buyantiviralwp.com
buketeg.xyz buydissertation.net
buntube.net buyviagraa.blogspot.com
burger-imperia.com cablecar.us
burn-fat.ga cah.io.community
buy-cheap-online.info calc-for-credit.ru
buytizanidineonline.blogspot.com candlewooddentalcentre.com.au
bycontext.com canzoni.ru
c1.onedmp.com capture-room.com
cacheimages.com cardsharp1.ru
calcularpagerank.com carmuffler.net
camdolls.com carstrends2015.com
cartierbracelet.xyz cartierring.xyz
cartierlovestore.com cashkitten-a.akamaihd.net
cbox.ws cashonads.com
cdncash.com catalogs-parts.com
chatroulette.si cayado.snn.gr
chee-by.biz cezartabac.ro
cigarpass.com cfsrating.sonicwall.com
cinemaflix.website cheapmoncler.pw
cjs.com.ru checktext.org
clicksor.com chocolateslim-u-srbiji.com
clips.ua.ac.be christianlouboutinoutlet.win
co.lumb.co cindymatches.com
clevernt.com
click2pawn.com
clipartnew.com
clknsee.com
clksite.com
clothesforcash.com
codysbbq.com
comeondog.info comeondog.info
comissionka.net compliance-alexa.xyz
compliance-elena.xyz compliance-don.top
conboy.us compliance-margo.top
connectionstrenth.com cool-wedding.net
consorzioilmosaico.org corta.co
contextualyield.com crackzplanet.com
copro.pw creativehutindia.com
craftburg.ru cvety24.by
creams.makeforum.eu dandiyabeats.in
creatives.adbetclickin.pink darmebel.com.ua
czat.wp.pl datadepths.com
date-withme.com daymusam.com
dayibiao.com db.speedup.gdn
ddsoldes.fr degerlund.net
dear-diary.ru dekorkeramik.ru
dgfitness.co delitime.info
diesel-parts28.ru deluxedumps.com
digest-project.ru derevesendeco.com
dikx.gdn designdevise.com
dimkino.ru detskie-konstruktory.ru
dipstar.org dev-seo.blog
directrev.com diarioaconcagua.com
doktoronline.no dickssportinggoods.com
dokumentalkino.net dikqlyremy.info
domain-submit.info dispo.de
domainsphoto.com disruptingdinnerparties.com
domcran.net divci-hry.info
donvito.unas.cz dmmspy.com
dood.live dodlive.mil
downloaddy.net dojki365.online
dpihatinh.gov.vn domainsatcost.com
draniki.org drupa.com
droidlook.net dvd-famille.com
drugspowerstore.com
drugstoreforyou.com
dscaas.website
duawitchrarato.tk
dyt.net dyt.net
ec-file.info e-buyeasy.com
ecommerce-seo.com earncash.com.ua
edwinkonijn.com.au ecommerce-seo.org
elektrischezigarette2.onsugar.com ednorxmed.com
elektronischezigarette2.mex.tl educatemetv.com
energydiet-info.ru ee77ee.com
enews.tech elektrischezigarette2.devhub.com
eralph.tk elektrozigarette1.dreamwidth.org
erot.co elektrozigarettekaufen2.blogse.nl
erotag.com elite-sex-finders.com
es5.com elmacho.xyz
everflora.ru elvenar.com
exhibitionplus.eu en.altezza.travel
extstat.com enbersoft.com
ezigarettenshop2.postbit.com energy-ua.com
f1nder.org eorogo.top
failingmarriege.blogspot.com escort-russian.com
falopicm.pw eu-cookie-law.info
fashionavenuegame.com event-tracking.com
fealq.com everypony.ru
filesdatabase.net expertblog.info
filesmonster.porn eyes-on-you.ga
filkhbr.com ezigarettekaufen2.blox.pl
filmci.pro ezigarettenshop1.yolasite.com
ezrvrentals.com
f0815.de
face.hostingx.eu
familienzahnaerzte.com
fantasticpixcool.com
feeriaclub.ru
ferretsoft.com
ff30236ddef1465f88547e760973d70a.com
filmania-x.ru
filmfanatic.com
financehint.eu
finansov.info finansov.info
findpics.pw finstroy.net
finteks.ru fishtauto.ru
fix-website-errors.com floridahuntingfishingadventures.com
forms-mtm.ru foodcrafts.website
foodgid.net
forums.d2jsp.org forums.d2jsp.org
free-gluten.ru freenode.info
freemags.cc freshdz.com
front.ru froggytube.com
fungamelands.com fullfileaccess.com
galaxy-family.ru fx-brokers-review.com
gallerylisting.com g7m.pl
gamingspark.com gambarkatabaru.com
geoads.com game300.ru
gfaq.ru gateway.zscalerone.net
gg.zzyjxs.com gaz-voshod.ru
gifspics.com gazoblok.net.ua
gigapeta.com ge0ip.org
gok-kasten.net gearcraft.us
goldadpremium.com genericviagrasildenafiled.net
goldandcard.ru germes-trans.com
glcomputers.ru
goldpanningtools.com
golfresa.lucania.se
googleseo.com.tr googleseo.com.tr
green-tea.tv gowreckdiving.com
growboxbl.ru gratuitbaise.com
h2monline.com greatzip.com
hd720kino.ru groupmoney.ru
helicalpile.us gsasearchengineranker.top
hermesreplica.win gta-top.ru
heroz.fr hasfun.com
hostnow.men havepussy.com
hotchatdate.com hdfreeporno.net
hmmm.cz
hobbyhuren-datenbank.com
home-task.com
homeandhealth.ru
homelygarden.com
hop.clickbank.net
hotdl.in hotdl.in
hothot.ru hotelcrocenzi.sm
hspline.com hoverboard360.de
hyhj.info
hystersister.com
i-hobot.ru
iblogpress.xyz iblogpress.xyz
ifmo.ru ideashome.id
images.gyffu.com igrovyeavtomaty777.ru
in-tandem.co iinstalll-fii1leis.jus0wil.pp.ua
indiakino.net ilmen.net
indianmedicaltourismshop.com imagez.co
insomniagamingfestival.com imgarit.pw
interfucks.net incep.imagine-ex.co
iptool.xyz incest-ru.com
ivanstroi.ru infowarcraft.ru
innoslicon.com
intimshop-fantasy.ru
ipchicken.com
isistaylorporn.info
jacago.com
jackwolfskinoutlet.online
jagg.info jagg.info
jeremyeaton.co jbl-charge.info
jerseysbizwholesalecheap.com jennyfire.ru
jewelryandfiligree.com jerseyschinabizwholesale.us
justkillingti.me jerseywholesaleelitestore.com
jobmarket.com.ua
jose.mulinohouse.co
jyrxd.com jyrxd.com
kanimage.com kalandranis.gr
karpun-iris.ru karadene.com
karting196.ru kareliatobacco.ru
kiwe-analytics.com katadhin.co
konteiner24.com kazan.zrus.org
krafte.ru keki.info
kupit-adenu.ru kerch.site
kerei.ru
keyword-suggestions.com
kfon.eu
kino-key.info
kleine-titten.biz
kmgamex.cf
kndxbkdx.bloger.index.hr
kometars.xyz
konica.kz
krasivye-devushki.net
kumuk.info
kvartira-sutochno.com kvartira-sutochno.com
la-fa.ru larose.jb2c.me
lapolis.it layola.biz.tc
law-check-nine.xyz lernur.net
law-check-seven.xyz levaquin750mg.blogspot.com
lazy-z.com limads.men
lechenie-gemorroya.com limtu.ifmo.ru
lechtaczka.net
lesbian.xyz
lignofix.ua
link.ac
linkrr.com linkrr.com
lion.misba.us livejasmin.com
liupis.com livesex-amateure.info
livingcanarias.com lockerz.com
llastbuy.ru locustdesign.co
lomb.co lulea-auktionsverk.se
lovi-moment.com.ua macdamaged.space
luckyshop.net.ua makemoneyonline.com
m-google.xyz
m.b00kmarks.com
magento-crew.net
makedo.ru
malls.com malls.com
marmitaco.cat mantramusic.ru
marwer.info marcogrup.com
masturbate.co.uk masterseek.com
mathgym.com.au mature.free-websites.com
matsdale.com medicovi.com
mbiologi.ru medosmotr-ufa.ru
mebel-vstroika.ru meetingrainstorm.bid
mebelcomplekt.ru mejoresfotos.eu
meccadumps.net member-quiz.com
megakino.net memberty.com
mhi-systems.ru meteocast.net
mikrobiologies.ru meteostate.com
minneapoliscopiers.com midst.eu
misslike.ru mindeyegames.com
mlf.hordo.win minecraft-neo.ru
mnogolok.info mompussy.net
mobplayer.ru monclervests.xyz
mole.pluto.ro moomi-daeri.com
monclerjacketsoutlet.win more-letom.ru
moncleroutletonline.xyz most-kerch.org
montredemarque.nl mouselink.co
moz.com musflashtv.com
mp3ringtone.info musicpro.monster
msk-diplomat.com musicspire.online
muschisexbilder.com muztops.ru
musezone.ru myonigroup.com
musicktab.com myperiod.club
mybackgroundlandscape.blogspot.com na15.ru
mydeathspace.com naturalpharm.com.ua
myftpupload.com nina.az
myyour.eu nrv.co.za
napalm51.nut.cc nsatc.net
naperehresti.info nw-servis.ru
nardulan.com obiavo.ru
new-apps.ru ochistka-stokov.ru
normalegal.ru offer.camp
novodigs.com offside2.5v.pl
npoet.ru oivcvx.website
nudo.ca okout.ru
nzfilecloud.weebly.com old-rock.com
o00.in omgtnc.com
obesidadealgarve.com omoikiri-japan.ru
obiavo.in onefilms.net
offf.info oranga.host.sk
ohmyrings.com orsonet.ru
online-podarki.com paccohichetoti.ml
online-templatestore.com pastaleads.com
onlinefilmz.net pewit.pw
onstrapon.purplesphere.in phpdevops.com
orenburg-gsm.ru picurams.pw
orhonit.com pintattoos.com
paclitor.com pinwallpaper.top
paintingplanet.ru pisanieprac.info
pandarastore.top pix-hd.com
paparazzistudios.com.au pl.aasoldes.fr
parajumpersstore.online placid-rounded-coast.glitch.me
parlament.biz
partner-stop.men
pcboa.se
pensplan.com
pepperstyle.ru
perfection-pleasure.ru
photochki.com
pic2fly.com
pinkduck.ga
pirateday.ru
pistonclasico.com
pk-services.ru
pladform.ru
platezhka.net platezhka.net
play-movie.pl playlott.com
plusnetwork.com pod-muzyku.club
pogodnyyeavarii.gq podshipniki-nsk.ru
pokemongooo.ml polybuild.ru
pornhub-forum.ga portadd.men
pornmania.pl predominant-invent.tk
porno-video-chati.ru privatov-zapisi.ru
pornoklad.net privhosting.com
pornokorol.com
portside.xyz
prefersurvey.net
pretty-mart.com
priceg.com
privetsochi.ru
prize44.com prize44.com
prizefestival.mobi prodess.ru
prod2016.com promalp-universal.ru
profitkode.com prostitutki-yaroslavlya.org
proftests.net proxyradar.com
progress-upakovka.ru push-ad.com
projectforte.ru
prom23.ru
prostitutki-rostova.org
prpops.com
putevka24.ru putevka24.ru
qualitymarketzone.com qcstrtvt.bloger.index.hr
quick-seeker.com qexyfu.bugs3.com
ranksonic.com qitt.ru
razleton.com questionmarque.ch
refererx.com rankia.com
regionshop.biz registry-cleaner.net
remedyotc.com registrysweeper.com
reining.lovasszovetseg.hu
reklama1.ru
remont-mobile-phones.ru
remontvsamare.su
rentalcarnavi.info
resultshub-a.akamaihd.net resultshub-a.akamaihd.net
retailwith.com retreatia.com
reyel1985.webnode.fr revistaindustria.com
rfserial.net
rightenergysolutions.com.au
rimedia.org rimedia.org
ritlweb.com ruclicks.com
rniaeba.ga rvtv.ru
robotixix.com saecsa.co
rocis.site salutmontreal.com
rockma.se santasgift.ml
roma-kukareku.livejournal.com sarf3omlat.com
sa-live.com sbetodiodnye-lampy.ru
sabaapress.com sbtdesign.co.uk
samo-soznanie.ru sbwealthsolutions.ca
sampleletters.net scanhub.ru
saneitconsulting.com scanner-alex.top
saneyes.com scanner-john.top
sanjosestartups.com scrapinghub.com
saugatuck.com scrapy.org
scanner-margo.top sdelatmebel.ru
scenarii-1-sentyabrya.uroki.org.ua searchimage.co
scrnet.biz.ua searchinterneat-a.akamaihd.net
secret.xn--oogle-wmc.com seksotur.ru
semalt.com seonetwizard.com
sendearnings.com serpstat.com
seoboxes.com servisural.ru
seolab.top sexcamamateurchat.com
serptehnika.ru sexvporno.ru
service.adtech.us sharebutton.net
sexpartygirls.net sharpchallenge.com
sfj-ror.no shikiso.info
sharebutton.to ship-marvel.co.ua
sheerseo.com shop.xz618.com
shtaketniki.ru shopfishing.com.ua
shoppingjequiti.com.br
sibtest.ru sibtest.ru
simplepooltips.com sicfor.bcu.cc
simul.co sideeffectsoftizanidine.blogspot.com
sisi-go.ru sildenafilcitratemed.com
skanninge.se simple-share-buttons.com
smart-scripts.com sims-sims.ru
smartpet.ru sitevalued.com
skuteczna-dieta.co.pl
slavic-magic.ru
sledstvie-veli.net
smart-balancewheel.com
snip.tw
snow.nvr163.com snow.nvr163.com
snts.shell-pmr.ru social-fun.ru
socialmonkee.com social-s-ggg.xyz
social-s-iii.xyz
socialmediasuggest.com
socialsignals24.com
soft-terminal.ru
soft1.ru
soheavyblog.com
solinf.co
solmarket.by
soochi.co
space-worry.ml
space2019.top space2019.top
sportbetfair.com squidoo.com
sps-shop.com stroymonolit.su
spy-app.info stroyplus.ru
spymac.net stuff-about-money.com
stariy-baku.com
starpages.net
strangeduckfilms.com
strfls.com
stroy-portal22.ru
studentguide.ru
subj.ukr-lit.com subj.ukr-lit.com
supers.com.ua suchenindeutschland.com
suralink.com sukirgenk.dvrlists.com
sureone.pro survival.betteroffers.review
surflinksmedical.com svbur.ru
suture.co t-bygg.com
svtrd.com
swsociety.se
tacbelarus.ru
tagil.zrus.org
talant-factory.ru
tamada69.com tamada69.com
taximytishi.ru technika-remont.ru
tdsing.ru teenbbw.yopoint.in
tehngr.ru theautoprofit.ml
terrafootwear.us theheroes.ru
thaismartloan.com timecrimea.ru
the-torrent-tracker.blogspot.com tizanidinehcl4mgtabinfo.blogspot.com
the-usa-games.blogspot.com tizanidinehclsideeffects.blogspot.com
thebestweightlosspills.ovh tizanidineinfo.blogspot.com
thebluenoodle.com tizanidinerecreationaluse.blogspot.com
thegameriders.com tizanidinevssoma.blogspot.com
thepantonpractice.co.uk top-l2.com
timeallnews.ru topflownews.com
timhost.ru torrentdownloadhub.com
tizanidineandcipro.blogspot.com tourismvictoria.com
tizanidinehcl2mgtablet.blogspot.com traffictrade.life
tizanidinehcl4mgtablet.blogspot.com traffique.net
tizanidineonline.blogspot.com traffpartners.com
tizanidineoral.blogspot.com tramadolandtizanidine.blogspot.com
tizanidinewithdrawal.blogspot.com tryrating.com
toondinsey.com tubo360.com
top10registrycleaners.com ukkelberg.no
totu.info umnovocaminho.com
toy-shop.top unrealcommander.com
tozup.com upproar.com
tpu.ru uroffer.link
tracfone.com vabasa.inwtrade.com
trackmedia101.com
trafficfactory.biz
trafficgenius.xyz
trafficmania.com
traxdom.ru
truemfilelj.gq
trustl.life
tuberkulezanet.ru
tupper-shop.ru
twincitiescarservice.com
uchetunet.su
uginekologa.com
unblocksit.es
undergroundcityphoto.com
universals.com.ua
uprour.com
uptime-delta.net
uptime-gamma.net
urccvfmc.bloger.index.hr
urengoy.pro
ustion.ru
uvozdeckych.info
v-doc.co
valmetrundan.se
vandrie-ict.nl
veerotech.com veerotech.com
vegascosmetics.ru vegan-foods.us
vesnatehno.com velen.io
videochat.ph venta-prom.ru
videochat.world ventelnos.com
virtuagirl.com video-camer.com
vseigru.one video-chat.love
vzlomsn.org vip-parfumeria.ru
vipsexfinders.com
vkmusics.ru
voloslove.ru
voluumtrk.com
voronezh.xrus.org
w3data.co w3data.co
wait3sec.org wallpaperaccess.com
wanker.us walpaperlist.com
webalan.ru waterefficiency.co
webcamdevochka.com website-analyzer.info
weclipart.com websites-reviews.com
weightatraining.com wejdz-tu.pl
westum.se wellcome2slovenia.ru
wildnatureimages.com westvilletowingservices.co.za
wjgony.com wholesalejerseychinashop.com
worldhistory.biz wholinkstome.com
worldmusicfests.com writersgroup580.web.fc2.com
worldoffiles.ru wsgames.ru
www.labves.ru x-diesel.com
www.wohnkabinen-shop.de xclicks.net
wygraj-teraz.com xgftnlrt.bloger.index.hr
x-diesel.org xitjw.info
xaijo.com xjlottery.com
xbaboon.com xn-----8kcatubaocd1bneepefojs1h2e.xn--p1ai
xblog.in xn----8sbarihbihxpxqgaf0g1e.xn--80adxhks
xblognetwork.com xn--lifehacer-1rb.com
xlolitka.com xtubeporno.net
xmlinde.com xxxdatinglocal.us
xn----7sbahjd3btneuw1joc.xn--p1ai
xn--80aafb2a.xn--p1ai
xn--80aeahghtf8ac5i.xn--p1ai
xn--hxazdsfy.blogspot.com
xvideospornoru.com
yebocasino.co.za yebocasino.co.za
yottos.com yellowstonesafaritours.com
yourmovies.pl yoopsie.com
youtube-downloader.savetubevideo.com youraticles.pl
yun56.co yourporngay.com
za-music.mymobiplanet.com youtoner.it
zahvat.ru ypmuseum.ru
zaloadi.ru zdesoboi.com
zdesformula.ru zigarettenonlinekaufen1.bloog.pl
zebramart.ru zrelaya.pw
zigarettenonlinekaufen2.bloog.pl
znakomstva-piter78.ru
zoogdiesney.com
zoominfo.com
zscaler.net
zvetki.ru
zzlgxh.com

474
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
2daytrendingnews.com 1flag.co.za
2rich4bitches.com 2000k.ru
40cg.com 300richmond.co.nz
51unlim.ru 45en.ru
5ws.dating-app.ru 4webmasters.org
9med.net aac52006ec82a24e08b665f4db2b5013f7.com
aanapa.ru abwa.tk
academia-nsk.org actionnooz.com
adcash.com adelly.bg
adclickservice.com agardomains.com
adf.ly albuteroli.com
ads-seo.men alienwheel.es
adultfriendfinder.com alif-ba-ta.com
affordablewebsitesandmobileapps.com amateurgalls.com
albamargroup.com anal-acrobats.hol.es
ald2014.org anticrawler.org
animal36.com apiholdingmypage-a.akamaihd.net
appsaurus.com apimegabrowsebiz-a.akamaihd.net
aruplighting.com approved.su
avkzarabotok.com arewater.com
avtocredit-legko.ru arius.tech
azartmsl.com artpicso.com
bambi.ck.ua ausergrubhof.info
bestssaker.com becuo.com
beta.hotkeys.com best-seo-offer.com
blackcurranthumidifiers.site bestwebsiteawards.com
blpmovies.com bet365.com
bizcheapjerseyswholesalechina.com
blog.axant.it
bloggerads.net
boanonihaca.tk boanonihaca.tk
boltalko.xyz boc.kz
bonzibuddi.com braindaily.xyz
breget74.ru braip.com.br
burkesales.com bwlx.prepedu.cn
buy-cheap-pills-order-online.com carmuffler.net
californianews.cf casinos4dummies.co.uk
cannibalcheerleader.com cheapmoncler.win
carloans.com chiblackhawks-jerseys.com
cars-modification.net cialisndbrx.com
cartierlove2u.com circlesl.com
cenokos.ru clickhype.com
chatroulette.online com-supportcenter.website
chorus.terakeet.com
citizenclsdriveri7.pen.io
cityadspix.com
claytransformations.info
communisave.co.za communisave.co.za
community.allhiphop.com compliance-alexa.top
concordexoticrentals.com compliance-don.xyz
cordstrap.cc contentlook.co
cxpromote.com crackguru.tk
dafatiri.com customcollegeessays.net
dailyfinancefix.com customerpromos-a.akamaihd.net
daisye.top
danashop.ru
demian.kz demian.kz
divatraffic.com doctissimo.fr
domaincdn.xyz e-collantes.com
driving.kiev.ua e90post.com
dtm-spain.com earn-from-articles.com
dvrlists.com editmedios.com
egvar.net ekto.ee
elektronischezigarettekaufen1.myblog.de el-nation.com
emediate.eu elektrischezigaretten1.blogse.nl
envaseslotusama.com elmifarhangi.com
eorogo.top eropho.net
evening-dating-club.info erotik0049.com
expolicenciaslatam.co european-torches.ru
extremepornos.net euroskat.ru
eyemagination.com excitacao.com
failingmarriege.blogspot.com factorynightclub.com
fetishinside.com faz99.com
fetlifeblog.com feeriaclub.ru
fil.ru financeloan.us
firma-legion.ru findthe.pet
firstdrugmall.ru florida-tourism.net
forexgb.ru fm-upgrade.ru
forodvd.com footbalive.org
forsex.info free-floating-buttons.com
free-traffic.xyz fz139.ttk.ru
freewebs.com gallery.rennlist.com
fungirlsgames.net game-top.su
gambarxkata.co gaming-journal.com
gamezblox.com ge0ip.com
gay-site.store geilehausfrauen.net
gelezki.com given2.com
generic-pills-online.com gk-atlant.info
get-your-social-buttons.info globalscam.ga
getoutofdebtfree.org gok-kasten.net
ggiaro.com goodnightjournal.com
gilsonchiro.xyz gosreg.amchs.ru
girlspicsa.com green-tea.tv
goblacked.com growshop.es
golden-praga.ru gsasearchengineranker.space
googleseo.com.tr gunsvicceadadebt.tk
grtyi.com guruofcasino.com
handsandlegs.ru hermesbelts.xyz
homelygarden.com hesteel.pl
hotcore.info home-task.com
hotgirlhdwallpaper.com hosting-tracker.com
housemilan.ru hothor.se
hspline.com
huimin764128.com huimin764128.com
i-midias.net.br ifirestarter.ru
iamsport.org iframe-toloka.com
imguramx.pw ihtec2019.org
immobiliaremassaro.com ilmexico.com
importchinacoach-teach.com import-sales.com
inbabes.sexushost.com increasewwwtraffic.info
incest-ru.com informatiecentro.be
infazavr.ru irunfar.com
inmate-locator.us jose.mulinohouse.co
investsuccess.org katadhin.co
ispac.org katushka.net
itsdp3.com kiinomaniak.pl
japan-bearings.ru kiskinhouse.com
jerseywholesalebizchina.com kvartir-remont.biz
jerseywholesaleelitestore.com lacave.ntic.fr
joy-penguin.com lanadelreyfans.us
kashubadesign.ru lapitec.eu
kazinogames.lv laudit.ru
ketoanhanoi.info lawyers.cafe
keywordsdoctor.com leadwayau.com
keywordsuggest.org letolove.ru
kino-rf.ru lexixxx.com
kinotorka.ru livingroomdecor.info
lawyersinfo.org magicdiet.gq
liveinternet.ro mb140.ru
lostaruban.com medicovi.com
lottospring.com medkritika.ru
magnetic-bracelets.ru misandesign.se
mbiologi.ru missclub.info
medanestesia.ru mlvc4zzw.space
microstatic.pl modenamebel.ru
mini.7zap.com moneyviking-a.akamaihd.net
misswell.net mosrif.ru
mixed-wrestling.ru mrpornogratis.xxx
mobile-appster.ru mydirtystuff.com
moneyteam24.com myhydros.org
movies-in-theaters.net mzdish.site
mp3ritm.top naturalpharm.com.ua
musicstock.me netvouz.com
mycaf.it nicefloor.co.uk
mylida.org nikhilbahl.com
myplaycity.com nisuturnetdgu.tk
nakrutka.cc nodup.ru
net-radar.com norththeface.store
niroo.info novgorod.xrus.org
nofreezingmac.click okeinfo.online
o-dachnik.ru
ochistka-stokov.ru
okonich.com.ua
okuos.com
oops-cinema.ru
orenburg-gsm.ru
orhonit.com orhonit.com
orion-v.com parfusale.se
pekori.to patol01.pw
poiskzakona.ru pdns.download
poptool.net perfectpracticeweb.com
porn-w.org photo.houseofgaga.ru
pornosexrolik.com picphotos.net
portadd.men piulatte.cz
portal-eu.ru pixelrz.com
priscilarodrigues.com.br pk-pomosch.ru
profit-opportunity.com plastweb.ru
profitkode.com play-mp3.com
progress-upakovka.ru plohaya-kreditnaya-istoriya.ru
prokotov.com pochtovyi-index.ru
proposal-engine.com porno-chaman.info
radiodigital.co poshiv-chehol.ru
rebelmouse.com prizeestates.cricket
rfd-split.hr projectforte.ru
ratemodels.net
regionshop.biz
registrationdomainsite.com
reklamuss.ru
remybutler.fr
ribieiendom.no ribieiendom.no
ring4rhino.com ringporno.com
rockprogblog.com royal-investments.net
rybalka-opt.ru ruex.org.ua
rutor.vip
salmonfishingsacramentoriver.com
samoiedo.it samoiedo.it
savingsslider-a.akamaihd.net sbornik-zakonov.ru
seo-traffic-ranking.info scanner-elena.top
seonetwizard.com scanner-ivan.top
seozoom.it sdsjweb.com
servicecenter.co.ua searchimpression.com
servisural.ru semprofile.com
sexstream.pl serpstat.com
sibdevice.ru sexvporno.ru
site-auditor.online sexy-screen-savers.com
sizeplus.work sim-service.net
sotechco.co simplesite.com
spymac.net skanninge.se
sribno.net smartshoppymac.com
stop-gepatit.te.ua snjatie-geroinovoy-lomki.ru
sumo.com snts.shell-pmr.ru
supervesti.ru soblaznu.net
svarbit.com social-button.xyz
symphonyintegratedhealthcare.com socialmadesimple.com
tagil.zrus.org softonicads.com
taraz.xkaz.org spylog.com
taxi-v-eisk.ru spywarebegone.com
tehngr.ru squidoo.com
teslathemes.com studiofmp.com
tizanidinehcl4mgtabinfo.blogspot.com success-seo.com
tizanidinemusclerelaxant.blogspot.com summerlinhomes411.info
tour-line.net sygraem.com
triplepanda.xyz tarad.com
taranerymagesswa.blogspot.com
tattomedia.com
theguardlan.com
thenews-today.info
threecolumnblogger.com
tivolibasket.it
topclickguru.com
topdownloads.ru
tripper.de
trubywriting.com trubywriting.com
tupper-shop.ru trustaffs.com
u17795.netangels.ru tuckermktg.com
uchil.net turbabitload.weebly.com
turbo-suslik.org
ugogo.info
uhodzalijami.ru uhodzalijami.ru
ukrtvory.in.ua umityangin.net
umekana.ru urlcut.ru
uptimebot.net vapomnoncri.tk
utrolive.ru viagroid.ru
valmetrundan.se video-chat.love
vapsy.com videochat.cafe
varbergsvind.se vip-parfumeria.ru
venerologiya.com virus-respirators.com
vereo.eu viveresaniesnelli.it
via-energy-acquistare.com voloslove.ru
vigrx-original.ru volume-pills.biz
vitoriacabos.com voronezh.xrus.org
voluumtrk.com
vriel.batcave.net
vykup-avto-krasnodar.ru
wardreapptokone.tk wardreapptokone.tk
warezkeeper.com waterpurifier.club
webix.biz
webix.me
website-analyzer.info
website-speed-check.site
website-speed-up.site website-speed-up.site
wesharepics.info
westen-z.life
whengirlsgowild.com whengirlsgowild.com
winterclassichockeyjerseys.com wholesalenfljerseys.us.com
wiosenny-bon-1500.pl wingsofrefuge.net
wladimirpayen.com woman-tampon.ru
wleuaprpxuvr.ga womensterritory.ru
wordpresscore.com worst-sites.online
xclicks.net wstroika.ru
xn----7sbifcamovvfggw9d.xn--p1ai x-lime.com
xn----8sblgmbj1a1bk8l.xn----161-4vemb6cjl7anbaea3afninj.xn--p1ai xkaz.org
xn----itbkqkfiq.xn--p1ai yhit.press
xn--80aagddcgkbcqbad7amllnejg6dya.xn--p1ai youdao.com
yaderenergy.ru yourtemplatefinder.com
yoursearch.me z-master.ru
youtubologia.it zhacker.net
zastroyka.org znaniyapolza.ru
zigarettenonlinekaufen2.drupalgardens.com zophim.me
zona-aqua.ru
zoompegs.com
zoosexart.com
zs2vm.top zs2vm.top

14
dev-tools/test_units/ratelimittest-master.txt
View file

@ -1,21 +1,11 @@
<!DOCTYPE html> <html>
<html>
<head>
<title>Welcome to NginX</title>
</head>
<body>
<div align=center>
<h1>Welcome to the NginX Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><html>
<head><title>503 Service Temporarily Unavailable</title></head> <head><title>503 Service Temporarily Unavailable</title></head>
<body> <body>
<center><h1>503 Service Temporarily Unavailable</h1></center> <center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center> <hr><center>nginx/1.18.0 (Ubuntu)</center>
</body> </body>
</html> </html>
<html> ml><html>
<head><title>503 Service Temporarily Unavailable</title></head> <head><title>503 Service Temporarily Unavailable</title></head>
<body> <body>
<center><h1>503 Service Temporarily Unavailable</h1></center> <center><h1>503 Service Temporarily Unavailable</h1></center>

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.69 0; 10.1.0.222 0;
127.0.0.1 0; 127.0.0.1 0;

8
dev-tools/tmp/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:53 UTC 2022 ### Updated: Sun Jan 9 07:00:24 UTC 2022
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -817,6 +817,8 @@ User-agent: Searchestate
Disallow:/ Disallow:/
User-agent: SearchmetricsBot User-agent: SearchmetricsBot
Disallow:/ Disallow:/
User-agent: Seekport
Disallow:/
User-agent: SemanticJuice User-agent: SemanticJuice
Disallow:/ Disallow:/
User-agent: Semrush User-agent: Semrush

8
robots.txt/robots.txt
View file

@ -6,9 +6,9 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.01.2840 ### Version: V4.2022.01.2841
### Updated: Sun Jan 9 06:33:53 UTC 2022 ### Updated: Sun Jan 9 07:00:24 UTC 2022
### Bad Bot Count: 631 ### Bad Bot Count: 632
################################################### ###################################################
### Version Information ## ### Version Information ##
@ -817,6 +817,8 @@ User-agent: Searchestate
Disallow:/ Disallow:/
User-agent: SearchmetricsBot User-agent: SearchmetricsBot
Disallow:/ Disallow:/
User-agent: Seekport
Disallow:/
User-agent: SemanticJuice User-agent: SemanticJuice
Disallow:/ Disallow:/
User-agent: Semrush User-agent: Semrush