vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-05 20:20:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2021.08.2527 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2021-08-15 05:24:10 +00:00
parent 06ec913cfc
commit cf5abb1836
53 changed files with 1104 additions and 1125 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,8 +4,8 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.08.2526 #### Version: V4.2021.08.2527
#### Bad Referrers Blocked: [7080](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,8 +4,8 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2021.08.2526 #### Version: V4.2021.08.2527
#### Bad Referrers Blocked: [7080](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -11,8 +11,8 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2021.08.2526 #### Version: V4.2021.08.2527
#### Bad Referrers Blocked: [7080](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
_google_analytics_ghost_spam/README.md
View file

@ -3,8 +3,8 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2021.08.2526 #### Version: V4.2021.08.2527
#### Bad Referrers Blocked: [7080](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7079](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [623](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

2
_google_analytics_ghost_spam/google-exclude-01.txt
View file

File diff suppressed because one or more lines are too long

1
_google_webmaster_disavow_links/google-disavow.txt
View file

@ -345,7 +345,6 @@ domain:alibestsale.com
domain:alice.it domain:alice.it
domain:alienwheel.es domain:alienwheel.es
domain:alienwheels.de domain:alienwheels.de
domain:aliexpress.com
domain:aliexpresscashback.club domain:aliexpresscashback.club
domain:alif-ba-ta.com domain:alif-ba-ta.com
domain:alive-ua.com domain:alive-ua.com

7
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:20 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

7
conf.d/globalblacklist-testing-version.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2021.08.2526 ### Version: V3.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:21 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

7
conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:20 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

7
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2021.08.2526 ### Version: V3.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:21 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 1; 10.1.0.142 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 0; 10.1.0.142 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2021.08.2526 ### Version: V3.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:21 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 1; 10.1.0.142 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 0; 10.1.0.142 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2021.08.2526 ### Version: V3.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:21 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 1; 10.1.0.142 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/beta_conf_files_whitelist/bots.d/custom-bad-referrers.conf
View file

@ -392,7 +392,6 @@
"~*(?:\b)alice\.it(?:\b)" 0; "~*(?:\b)alice\.it(?:\b)" 0;
"~*(?:\b)alienwheel\.es(?:\b)" 0; "~*(?:\b)alienwheel\.es(?:\b)" 0;
"~*(?:\b)alienwheels\.de(?:\b)" 0; "~*(?:\b)alienwheels\.de(?:\b)" 0;
"~*(?:\b)aliexpress\.com(?:\b)" 0;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 0; "~*(?:\b)aliexpresscashback\.club(?:\b)" 0;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 0; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 0;
"~*(?:\b)alive\-ua\.com(?:\b)" 0; "~*(?:\b)alive\-ua\.com(?:\b)" 0;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 0; 10.1.0.142 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2021.08.2526 ### Version: V3.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:21 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2526 2527

7
dev-tools/globalblacklist-testing.template
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2021.08.2526 ### Version: V3.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:21 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

7
dev-tools/globalblacklist.template
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:20 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

1
dev-tools/referrers-regex-format-whitelist-test.txt
View file

@ -345,7 +345,6 @@
"~*(?:\b)alice\.it(?:\b)" 0; "~*(?:\b)alice\.it(?:\b)" 0;
"~*(?:\b)alienwheel\.es(?:\b)" 0; "~*(?:\b)alienwheel\.es(?:\b)" 0;
"~*(?:\b)alienwheels\.de(?:\b)" 0; "~*(?:\b)alienwheels\.de(?:\b)" 0;
"~*(?:\b)aliexpress\.com(?:\b)" 0;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 0; "~*(?:\b)aliexpresscashback\.club(?:\b)" 0;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 0; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 0;
"~*(?:\b)alive\-ua\.com(?:\b)" 0; "~*(?:\b)alive\-ua\.com(?:\b)" 0;

1
dev-tools/referrers-regex-format.txt
View file

@ -345,7 +345,6 @@
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.250.69.103 0; 52.183.89.129 0;

7
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:20 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 1; 10.1.0.142 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 0; 10.1.0.142 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:20 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 1; 10.1.0.142 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 0; 10.1.0.142 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:20 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 1; 10.1.0.142 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test1_conf_files_whitelist/bots.d/custom-bad-referrers.conf
View file

@ -392,7 +392,6 @@
"~*(?:\b)alice\.it(?:\b)" 0; "~*(?:\b)alice\.it(?:\b)" 0;
"~*(?:\b)alienwheel\.es(?:\b)" 0; "~*(?:\b)alienwheel\.es(?:\b)" 0;
"~*(?:\b)alienwheels\.de(?:\b)" 0; "~*(?:\b)alienwheels\.de(?:\b)" 0;
"~*(?:\b)aliexpress\.com(?:\b)" 0;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 0; "~*(?:\b)aliexpresscashback\.club(?:\b)" 0;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 0; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 0;
"~*(?:\b)alive\-ua\.com(?:\b)" 0; "~*(?:\b)alive\-ua\.com(?:\b)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 0; 10.1.0.142 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:20 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.250.69.103 0; 52.183.89.129 0;

7
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:20 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
52.250.69.103 0; 52.183.89.129 0;

7
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:27 UTC 2021 ### Updated: Sun Aug 15 05:18:20 UTC 2021
### Bad Referrer Count: 7080 ### Bad Referrer Count: 7079
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -1397,7 +1397,6 @@ map $http_referer $bad_referer {
"~*(?:\b)alice\.it(?:\b)" 1; "~*(?:\b)alice\.it(?:\b)" 1;
"~*(?:\b)alienwheel\.es(?:\b)" 1; "~*(?:\b)alienwheel\.es(?:\b)" 1;
"~*(?:\b)alienwheels\.de(?:\b)" 1; "~*(?:\b)alienwheels\.de(?:\b)" 1;
"~*(?:\b)aliexpress\.com(?:\b)" 1;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 1; "~*(?:\b)aliexpresscashback\.club(?:\b)" 1;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 1;
"~*(?:\b)alive\-ua\.com(?:\b)" 1; "~*(?:\b)alive\-ua\.com(?:\b)" 1;

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 1; 10.1.0.142 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test_units/custom-bad-referrers.conf
View file

@ -392,7 +392,6 @@
"~*(?:\b)alice\.it(?:\b)" 0; "~*(?:\b)alice\.it(?:\b)" 0;
"~*(?:\b)alienwheel\.es(?:\b)" 0; "~*(?:\b)alienwheel\.es(?:\b)" 0;
"~*(?:\b)alienwheels\.de(?:\b)" 0; "~*(?:\b)alienwheels\.de(?:\b)" 0;
"~*(?:\b)aliexpress\.com(?:\b)" 0;
"~*(?:\b)aliexpresscashback\.club(?:\b)" 0; "~*(?:\b)aliexpresscashback\.club(?:\b)" 0;
"~*(?:\b)alif\-ba\-ta\.com(?:\b)" 0; "~*(?:\b)alif\-ba\-ta\.com(?:\b)" 0;
"~*(?:\b)alive\-ua\.com(?:\b)" 0; "~*(?:\b)alive\-ua\.com(?:\b)" 0;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
BetaBot BDCbot
CCBot BDFetch
IndeedBot BuzzSumo
Lftp GermCrawler
Lipperhey Gigablast
Shodan Kinza
Sottopop NetZIP
WebWhacker SpaceBison
Zitebot TwengaBot
polaris version WWW-Collector-E

290
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 360Spider
404enemy
80legs 80legs
ADmantX AIBOT
Abonti Abonti
Aboundex AiHitBot
Aboundexbot Alexibot
Acunetix AllSubmitter
AfD-Verbotsverfahren Alligator
Aipbot
Anarchie
Anarchy Anarchy
Anarchy99
Anthill
Asterias
AwarioRssBot
AwarioSmartBot
BDCbot
BDFetch
BLEXBot BLEXBot
BackDoorBot BackDoorBot
BackStreet BackWeb
Backlink-Ceck Backlink-Ceck
Badass
BatchFTP
Battleztar Bazinga
Bigfoot Bigfoot
Blackboard
Blow Blow
Brandprotect Brandprotect
Brandwatch Buck
Buddy
BunnySlippers BunnySlippers
BuzzSumo
CCBot CCBot
Calculon CSHttp
CazoodleBot CazoodleBot
Cegbfeieh
CensysInspect CensysInspect
CheeseBot
CherryPicker CherryPicker
Chlooe Cloud mapping
Cocolyzebot
Cogentbot
Collector
Copier
CopyRightCheck CopyRightCheck
Copyscape
Craftbot Craftbot
Crawling at Home Project Crawling at Home Project
CrazyWebCrawler CrazyWebCrawler
DBLBot CrunchBot
DataCha0s DIIbot
Demon DSearch
Devil Deusu
Digincore Digincore
Discobot Discobot
DnyzBot
DomCopBot
DomainAppender DomainAppender
DomainCrawler
DomainStatsBot
Domains Project Domains Project
Dotbot Dotbot
Download Wonder Dragonfly
Drip
ECCP/1.0 ECCP/1.0
EMail Siphon EMail Siphon
EMail Wolf EMail Wolf
EasyDL
Ebingbong Ebingbong
Ecxi EirGrabber
Exabot Express WebPictures
Extreme Picture Finder Extreme Picture Finder
EyeNetIE FDM
FemtosearchBot FlashGet
Flunky
Foobot Foobot
FyberSpider Fuzz
G-i-g-a-b-o-t G-i-g-a-b-o-t
GT::WWW Genieo
GetRight GetWeb
Gigabot
Go-Ahead-Got-It Go-Ahead-Got-It
GoZilla
GrabNet GrabNet
Grafula
GrapeFX GrapeFX
GrapeshotCrawler
HTMLparser
HTTP::Lite HTTP::Lite
HTTrack
Haansoft Haansoft
HaosouSpider
Harvest
Havij Havij
Hloader HybridBot
IDBTE4M IDBTE4M
Iblog IDBot
Id-search Id-search
IlseBot IlseBot
Image Fetch Image Fetch
IndeedBot
InfoNaviRobot InfoNaviRobot
InfoTekies
InterGET
Internet Ninja Internet Ninja
InternetSeer InternetSeer
JennyBot Iria
IstellaBot
JamesBOT
JetCar JetCar
Jetty
Joomla Joomla
Jyxobot Jyxobot
Kenjin Spider Kenjin Spider
Keyword Density
Kozmosbot
LWP::Simple LWP::Simple
Lanshanbot
Larbin Larbin
Lftp LibWeb
Libwhisker LieBaoFast
Lightspeedsystems Likse
LinkWalker Linkbot
Linkdexbot Linkdexbot
LinkpadBot LinqiaMetadataDownloaderBot
LinqiaScrapeBot LinqiaRSSBot
Litemage_walker Lmspider
MIDown tool
MIIxpc MIIxpc
MJ12bot MJ12bot
MQQBrowser MQQBrowser
MSFrontPage
MSIECrawler
Mag-Net Mag-Net
Majestic SEO Magnet
Majestic-SEO
Majestic12
MarkMonitor MarkMonitor
Masscan MauiBot
Mediatoolkitbot Mb2345Browser
Metauri NPbot
MicroMessenger Navroad
Microsoft Data Access
Moblie Safari
Mojeek
Morfeus Fucking Scanner
Mr.4x3
Musobot
Net Vampire
NetAnts
NetMechanic NetMechanic
NetSpider NextGenSearchBot
NetZIP
Netcraft
Nibbler Nibbler
Octopus Niki-bot
Offline Navigator Nikto
OnCrawl Nuclei
Nutch
OpenVAS OpenVAS
OrangeBot PageGrabber
OrangeSpider PageThing.com
OutclicksBot
PECL::HTTP
POE-Component-Client-HTTP
PageAnalyzer
Pandalytics Pandalytics
Pavuk Panscient
Pi-Monster Papa Foto
Petalbot
Picscout Picscout
PictureFinder PictureFinder
Piepmatz Piepmatz
Pixray PleaseCrawl
ProPowerBot ProWebWalker
Probethenet
Pu_iN Pu_iN
Pump PxBroker
PyCurl PyCurl
RankActiveLinkBot
RankFlex
RankingBot2 RankingBot2
RankurBot RankurBot
ReGet Re-re
RebelMouse RebelMouse
Recorder
RedesScrapy RedesScrapy
RepoMonkey
Ripper
RocketCrawler RocketCrawler
SBIder
SEOkicks-Robot SEOkicks-Robot
SEOlyticsCrawler SEOlyticsCrawler
Scanbot SMTBot
SearchmetricsBot SalesIntelligent
Semrush ScanAlert
SeoSiteCheckup Scrapy
Screaming
ScrepyBot
SentiBot
SeobilityBot SeobilityBot
Shodan Siphon
Site Sucker Site Sucker
SiteCheckerBotCrawler
SiteSnagger
SiteSucker SiteSucker
Sitebeam Sitevigil
Siteimprove
Snoopy
SocialRankIOBot
Sociscraper Sociscraper
Sogou web spider Sogou web spider
Sosospider
SpaceBison
Spammen Spammen
Spanner Spanner
Spbot Spbot
SputnikBot Spinn3r
Steeler Sqlmap
Stripper
Sucker
SuperBot
SuperHTTP
Suzuran Suzuran
Swiftbot
Szukacz Szukacz
T0PHackTeam
T8Abot T8Abot
TeleportPro
The Intraformant The Intraformant
Thumbor TheNomad
TightTwatBot TightTwatBot
Toata Toweyabot
True_Robot Tracemyfile
Trendiction
Trendictionbot
Turingos Turingos
Turnitin Turnitin
Twice URLy.Warning
UnisterBot
Upflow Upflow
Vacuum VCI
VeriCiteCrawler VidibleScraper
Voil
Voltron Voltron
WBSearchBot WBSearchBot
Web Sauger WEBDAV
WWW-Collector-E
WWW::Mechanize
WallpapersHD
WeSEE
Web Enhancer
Web Fetch
Web Pix
WebBandit WebBandit
WebCollage WebCollage
WebCopier WebFetch
WebLeacher WebImageCollector
WebSauger WebPix
WebWhacker
Webalta
Webshag Webshag
Website Quester Website Quester
Webster WebsiteQuester
Who.is Bot Whack
WiseGuys Robot Widow
Woobot Wonderbot
Wotbox Wotbox
Wprecon Wprecon
YoudaoBot Zermelo
Zauba
Zeus Zeus
ZyBorg Zitebot
arquivo.pt ZoomBot
check1.exe ZumBot
clark-crawler autoemailspider
cah.io.community
coccocbot-web coccocbot-web
com.plumanalytics crawler.feedback
crawler4j demandbase-bot
domainsproject.org
evc-batch evc-batch
internetVista monitor gopher
instabid
lwp-trivial lwp-trivial
meanpathbot
mediawords mediawords
netEstate NE Crawler muhstik-scan
plumanalytics pcBrowser
probe-image-size polaris version
s1z.ru ripz
scan.lol scalaj-http
seobility seobility
seoscanners
seostar
sitechecker.pro sitechecker.pro
sogouspider
sp_auditbot
spyfu
sysscan sysscan
tAkeOut
trendiction.de trendiction.de
ubermetrics-technologies.com
voyagerx.com
webmeup-crawler webmeup-crawler
xpymep1.exe x22Mozilla
zauba.io zgrab

300
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
360Spider 01h4x.com
404enemy 404enemy
AIBOT Aboundex
Aboundexbot Aboundexbot
Aipbot AhrefsBot
AllSubmitter AllSubmitter
Alligator Alligator
Anarchie AlphaBot
Anarchy Ankit
Anarchy99
Anthill Anthill
Aspiegel Aspiegel
BDCbot BBBike
BLEXBot
BackDoorBot
BackStreet
BackWeb BackWeb
Badass Backlink-Ceck
Battleztar Bazinga
BetaBot BetaBot
Bigfoot Bigfoot
Bitacle
Black Hole
BlackWidow BlackWidow
Bolt BuiltBotTough
BotALot BuiltWith
Brandprotect
Brandwatch
Buck
Buddy
CATExplorador CATExplorador
CCBot CODE87
Calculon
CazoodleBot CazoodleBot
CensysInspect Cegbfeieh
CherryPicker CheTeam
CheeseBot
ChinaClaw ChinaClaw
Claritybot Chlooe
Cloud mapping Cocolyzebot
Copier Collector
CopyRightCheck Cosmos
Craftbot
CrunchBot CrunchBot
Curious DSearch
Custo DTS Agent
DBLBot
DIIbot
DataCha0s DataCha0s
DatabaseDriverMysqli Deusu
Demon Digincore
DigitalPebble
Dirbuster Dirbuster
Discobot Discobot
DomCopBot DomCopBot
DomainAppender DomainCrawler
DomainStatsBot DomainStatsBot
Download Wonder Domains Project
Dragonfly EMail Wolf
EasyDL EirGrabber
Express WebPictures Express WebPictures
ExtLinksBot
ExtractorPro ExtractorPro
EyeNetIE Ezooms
FHscan
FemtosearchBot FemtosearchBot
Firefox/7.0 FlashGet
Flunky FyberSpider
Fyrebot Fyrebot
GalaxyBot GT::WWW
GetWeb GetRight
Gigablast Getintent
Go!Zilla Go!Zilla
Go-Ahead-Got-It GoZilla
Gotit
GrabNet
Grabber
Grafula
HEADMasterSEO HEADMasterSEO
HMView HMView
HTTP::Lite HTMLparser
HTTrack
Haansoft
HaosouSpider HaosouSpider
Heritrix Havij
Id-search Humanlinks
Image Sucker HybridBot
InfoTekies IRLbot
Iblog
IndeedBot
InfoNaviRobot
Internet Ninja Internet Ninja
Iria Iskanie
JamesBOT JOC Web Spider
Jbrofuzz Jbrofuzz
JennyBot JennyBot
JikeSpider JetCar
Joomla
Jorgee Jorgee
Kenjin Spider JustView
Kinza
LNSpiderguy LNSpiderguy
Lanshanbot
LeechFTP
LeechGet LeechGet
LexiBot
Lftp Lftp
LibWeb
Libwhisker
Lightspeedsystems
LinkWalker LinkWalker
Linkbot Linkbot
LinkpadBot
LinksManager
LinqiaMetadataDownloaderBot
LinqiaRSSBot LinqiaRSSBot
LinqiaScrapeBot MJ12bot
Lmspider MSFrontPage
Ltx71 Mag-Net
MFC_Tear_Sample
MIDown tool
MQQBrowser
MTRobot
Magnet Magnet
Majestic-SEO
Majestic12
MarkMonitor
MarkWatch MarkWatch
Mass Downloader
MauiBot MauiBot
Mb2345Browser Mb2345Browser
MeanPath Bot MeanPath Bot
MicroMessenger Microsoft Data Access
Microsoft URL Control
Mister PiX
Mojeek Mojeek
NICErsPRO Mojolicious
NearSite Mozlila
Needle Mr.4x3
Musobot
Nessus Nessus
Net Vampire NetAnts
NetMechanic NetSpider
NextGenSearchBot Netcraft
Netvibes
Nibbler
Niki-bot
Nikto Nikto
NimbleCrawler NimbleCrawler
Nimbostratus
Ninja Ninja
Octopus Nmap
Nutch
Offline Navigator Offline Navigator
Openvas OpenLinkProfiler
Openfind
OrangeBot OrangeBot
OutfoxBot OrangeSpider
PHPCrawl PECL::HTTP
PeoplePal POE-Component-Client-HTTP
Piepmatz PageAnalyzer
Pockey PageScorer
Probethenet PageThing.com
Pu_iN Picsearch
Pixray
PleaseCrawl
ProPowerBot
ProWebWalker
Psbot
Pump Pump
QueryN Metasearch QueryN Metasearch
Quick-Crawler Quick-Crawler
RSSingBot RankFlex
RankActiveLinkBot
RankingBot RankingBot
Rankivabot Rankivabot
RankurBot
RealDownload RealDownload
RebelMouse
RedesScrapy RedesScrapy
Ripper Ripper
Rogerbot Rogerbot
SBIder SEOkicks-Robot
SEOkicks
SEOlyticsCrawler
SEOprofiler
SEOstats SEOstats
SISTRIX SMTBot
ScoutJet ScanAlert
Scrapy
Screaming Screaming
ScrepyBot ScrepyBot
Searchestate SentiBot
SearchmetricsBot SeoSiteCheckup
Semrush SeobilityBot
SemrushBot
Seomoz
Shodan Shodan
Siphon
SiteExplorer
Site Sucker Site Sucker
Siteimprove SiteCheckerBotCrawler
SlySearch Sitebeam
SocialRankIOBot SmartDownload
Sociscraper Sogou web spider
Sottopop Sosospider
Spinn3r Spanner
Spbot
Sqlmap
Sqlworm
Sqworm Sqworm
Steeler Stripper
SuperBot SuperBot
SurveyBot Surfbot
Swiftbot
T0PHackTeam T0PHackTeam
Telesoft
Telesphoreo Telesphoreo
TheNomad The Intraformant
Thumbor Thumbor
TightTwatBot TightTwatBot
Toata Tracemyfile
Trendictionbot
True_Robot True_Robot
Turingos Turingos
Turnitin
TurnitinBot TurnitinBot
TwengaBot TwengaBot
URLy Warning
URLy.Warning URLy.Warning
UnisterBot Upflow
VB Project
VCI
Vacuum Vacuum
VeriCiteCrawler VeriCiteCrawler
VoidEYE VidibleScraper
Voil
WASALive-Bot WASALive-Bot
WISENutbot WWW-Collector-E
Wallpapers WWW::Mechanize
WWWOFFLE
WeSEE
Web Collage Web Collage
WebCollage
WebCopier
Web Enhancer Web Enhancer
WebGo IS Web Fuck
WebLeacher
Web Pix Web Pix
WebReaper Web Sauger
WebStripper
Web Sucker Web Sucker
WebCopier
WebFetch
WebGo IS
WebStripper
WebWhacker
Webalta Webalta
WebmasterWorldForumBot
WebsiteExtractor WebsiteExtractor
WebsiteQuester
Whack Whack
Widow Whacker
WinHTTrack Wonderbot
Wotbox Xaldon WebSpider
Xaldon_WebSpider Zade
Xenu Zermelo
Zeus
ZoomBot ZoomBot
ZumBot ZumBot
ZyBorg ZyBorg
arquivo.pt arquivo.pt
backlink-check backlink-check
clark-crawler
coccocbot-web
cognitiveseo
com.plumanalytics
crawl.sogou.com
crawler4j
demandbase-bot demandbase-bot
domainsproject.org evc-batch
eCatch
facebookscraper
gopher gopher
heritrix
instabid
internetVista monitor internetVista monitor
lwp-trivial
meanpathbot meanpathbot
mediawords
netEstate NE Crawler netEstate NE Crawler
page scorer pcBrowser
plumanalytics plumanalytics
polaris version probe-image-size
satoristudio.net
scan.lol scan.lol
sexsearcher
sogouspider
sp_auditbot
sysscan
tAkeOut tAkeOut
trendiction.de ubermetrics-technologies.com
voyagerx.com voyagerx.com
webmeup-crawler webmeup-crawler
webpros.com zauba.io
x22Mozilla
xpymep1.exe
zgrab zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
agreda.pluto.ro api.stathat.com
bag77.ru bidbuy.co.kr
collegeessay19.blogspot.ru blavia.00author.com
elektronischezigarettekaufen1.beeplog.com ezigarettenshop2.postbit.com
ooo-gotovie.ru loginduepunti.it
pourvous.info mistr-x.org
webix.biz muznachas-service.ru
wovis.site nfljerseysforsalewholesaler.com
x-lime.com qwarckoine.com
xn----7sbbahaq9bb5afgiqfliv4m.xn--p1ai swingerseiten.com

936
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
000free.us 0lovespells0.blogspot.com
0n-line.tv 1008.su
1001watch.com.ua 101billion.com
100dollars-seo.com 1688.com
101raccoon.ru 2345.com
12masterov.com 404.mn
1bet.com 5elementov.ru
24-ak.ru 66cpwgln.space
4replicawatch.net 777-club.ru
5kstore.com 7search.com
aa08daf7e13b6345e09e92f771507fa5f4.com 98oi.ru
aa14ab57a3339c4064bd9ae6fad7495b5f.com abbanreddy.com
aaa38852e886ac4af1a3cff9b47cab6272.com abiente.ru
acgs.tk abovetherivernc.com
actulite.com abtasty.com
adinterax.com acortarurl.es
adloads.com adcash.com
adverster.com addray.pro
advertisingtag.net addtoadd.men
advocatemsk.ru adpremium.org
aerogo.com adtech.us
afftrack001.com adviceforum.info
ageofclones.com
ahrefs.com
aibolita.com aibolita.com
akama.com alert.scansafe.net
alfabot.xyz alice.it
all-streaming-media.com alienwheels.de
alphaforum.ru alif-ba-ta.com
alpinism.ru allboard.xobor.de
alvtank.se allnews24.in
ameblo.jp analnoeporno.tv
analytics-ads.xyz
anapa-inns.ru
android-style.com android-style.com
android-vsem.org animal-drawings.com
animenime.ru anniemation.com
apartamentwroclaw.eu applyneedy.xyz
apiappenableinfo-a.akamaihd.net arabsexxxtube.com
apioasisspacenet-a.akamaihd.net arcadeplayhouse.com
apisecretsaucebi-a.akamaihd.net arenanews.com.ua
apple.com-webbrowsing-security.review arkkivoltti.net
arendovalka.xyz
aruplighting.com
asiavirtualsolutions.com
asrv-a.akamoihd.net asrv-a.akamoihd.net
autobrennero.it asscenihotosidea.blogspot.com
avirasecureserver.com atmovs.com
atout-energie-69.com
automobile-spec.com
aviav.ru
avto-oligarh.ru
awency.com awency.com
azazu.ru backiee.com
azinoofficial777.ru bannerbank.ru
backlink4u.net basedecor.ru
baikaleminer.com beachpics.com
banan.tv best-businessman.ru
bannertesting.com bestbrainprod.win
bashtime.ru bestsexyblog.com
basisches-wasser.net bestwrinklecreamnow.com
beatifullandscape.co
beclean-nn.ru
bedcapdealers.com
beetpics.pw
besthomemadepornsites.com
bestwebsitesawards.com
beztuberkuleza.ru beztuberkuleza.ru
bharatdefencekavach.com bigcareer.info
bikini-image.com
bimatoprost-careprost.com.ua bimatoprost-careprost.com.ua
bingo8888.com bonvillan.com
bird1.ru bonzbuddy.com
bitporno.sx brandehk.dk
bjanshee.ru brk-rti.ru
blackhatworth.com bumble.cheapwebsitehoster.com
blackplanet.com buy-forum.ru
bloggers.nl buybest1.biz
boazpower.com bycontext.com
book-bets.com cactussoft.biz
bradleylive.xyz californianews.cf
brainboostingsupplements.org candcstuccoandstone.com
braindaily.xyz candlehandmade.com
brainsandeggs.blogspot.com
braip.com.br
bronzeaid-a.akamaihd.net
business-made-fun.com
butstrap.space
buynorxx.com
buytizanidineonlinenoprescription.blogspot.com
buzzurl.jp
callmd5map.com
canrioloadm.gq canrioloadm.gq
cardiosport.com.ua cartierreplica.xyz
carrauterie.be cayado.snn.gr
carson.getenjoyment.net celejihad.info
chatroulette.world centraletermice.us
cheapestjerseys-wholesale.com cfacarrosserie74.com
cheatcode-lita12.rhcloud.com cfcl.co.uk
check-host.net chatroulette.si
cheapmoncler.pw
check-this-out-now.online
chiblackhawks-jerseys.com chiblackhawks-jerseys.com
cialischsrx.com chinajerseyswholesalecoupons.com
citizenclsdriveri7.pen.io christianlouboutinsaleonline.us
cjmc.info ci.ua
cl.s7.exct.net ciceros.co
claytransformations.info cigarpass.com
clickaider.com citetick.com
clickprotects.com classiquebijoux.ru
clicksotrk.com clean-virus-mac.com
clippingphotoindia.com clickbanksites.info
co.lumb.co
coaching-netz.info
coinsspb.com
comissionka.net
commoncrawl.org
community.allhiphop.com
compliance-alex.xyz
compliance-alexa.xyz
compliance-checker-7.info compliance-checker-7.info
compliance-checker.info compliance-george.xyz
compliance-don.top compliance-jack.top
compliance-elena.top compliance-margo.top
compliance-ivan.top contextualyield.com
compliance-jane.top covadhosting.biz
connectingsingles.com craftinsta.ru
coolgamechannel.com
coolwallpapers-hd.com
copyrightclaims.org
crazyprotocol.com crazyprotocol.com
crest-poloski.ru cubs-jerseys.us
cutalltheshit.com cyclobenzaprinevstizanidine.blogspot.com
czeshop.info dairyindia.in
dating-app.ru db.speedup.gdn
deadlinkchecker.com decorazilla.com
deluxedumps.com dengi-pod-zalog-nedvizhimosti.ru
destinationrealestate.com dentalpearls.com.au
dev-seo.blog deshevo-nedorogo.ru
dev33.dioniqlabb.se design-lands.ru
devochki.top detskie-zabavi.ru
dinkolove.ya.ru dev78.dioniqlabb.se
discountbarbour.online dickssportinggoods.com
diusyawiga.tk
dividendo.ru
djonwatch.ru
dnepr-avtospar.com.ua
docstoc.com docstoc.com
dogoimage.com
domznaniy.ru domznaniy.ru
download-walpaperhd.blogspot.com dotmass.top
downloaddy.net
downloader12.ru
dprtb.com dprtb.com
dscaas.website drunkmoms.net
eandsgallery.com dstroy.su
eaplay.ru dtnlyss.com
ecommerce-seo.org duawitchrarato.tk
ege-russian.ru dumpsmania24.com
election.interferencer.ru editors.choice6912650.hulfingtonpost.com
eleimgo.pw edwinkonijn.com.au
elektronischezigarettekaufen1.beeplog.com efkt.jp
elexies.info ekosmetyki.net.pl
elvel.com.ua elektrozigarette2.webs.com
ero-advertising.com elektrozigaretten1.tumblr.com
erotikchat-24.com eng-lyrics.com
erotiktreff24.info englishgamer.com
espaceinventoristes.com eonpal.com
eugenevaultstorage.com erotik0049.com
ewebarticle.info errorfixing.space
excitacion.info europeanwatches.ru
exmasters.com excitacao.com
exoclick.com
extlabs.io
eyessurgery.ru
eywords-monitoring-your-success.com eywords-monitoring-your-success.com
ezigarettekaufen1.hpage.com
f0815.de f0815.de
fanpagerobot.com fillmewithhappiness.com
fashion-mk.net filmetricsasia.com
fashionavenuegame.com find1friend.com
fastfixing.tech finemanteam.com
feeriaclub.ru fiuxy.com
fikasound.tk flooringinstallation-edmonton.com
filefilter.weebly.com fok.nl
finuse.com fordsonmajbor.cf
flytourisme.org formulaf1results.blogspot.com
forpackningsutveckling.se fortunejack.com
forum20.smailik.org forums.d2jsp.org
foxjuegos.com forums.toucharcade.com
free-laptop-reward.com frbizlist.com
free-fbook-traffic.com
free-stock-illustration.com free-stock-illustration.com
fuckmill.com
fugarif.ga
fungirlsgames.net
fusoradio.info
fyl.com.ru fyl.com.ru
fyxabomiw.ru fym.com.ru
galaxy-family.ru gamedayhouse.com
gallictures.com gamesprite.me
ge0ip.net gamevalue7.weebly.com
gemara.com geckoandfly.com
getresponse.com genericviagrasildenafiled.net
gidonline.one gerhardhealer.com
gilsonchiro.xyz getlaid-xxxhookupdirect.com
globatur.ru getlamborghini.ga
gne8.com gigixo.com
goodhumor24.com girls-ufa.ru
googglet.com goldandcard.ru
gotorussia.com googlemare.com
gsasearchengineranker.site gopixdatabase.com
guardlink.org gq-catalog.gq
guarrasdelporno.xxx greatfind-a.akamaihd.net
hd-film.pl gribokstop.com
hdapp1008-a.akamaihd.net gsasearchengineranker.pw
hdpixion.com gsmlab.pl
health-medical-portal.info gwagka.com
herehloadibs.cf hildinghr.se
histock.info hitmuzik.ru
homedo.fabpage.com home.myplaycity.com
homelygarden.com hostingclub.lk
hotblognetwork.com ictizanidinehcl4mg.blogspot.com
hotcore.info ie.57883.net
hoverboard360.at iframe-toloka.com
howlongdoestizanidinestayinyoursystem.blogspot.com igrovyeavtomaty777.ru
ido3.com iideaidekonkatu.info
ilovevitaly.com ikritikimou.gr
images-free.net imabase.com
immobiliaremassaro.com imagerydatabase.com
import-sales.com imfamous.info
inbabes.sexushost.com imgarit.pw
incomekey.net imguramx.pw
islandminingsupply.wordpress.com imitex-plus.ru
itis4you.com imperiafilm.ru
iwantedmoney.com incanto.in.ua
j33x.com infektsii.com
janettabridal.com inzn.ru
javcoast.com ipchicken.com
javlibrary.cc irkutsk.zrus.org
irunfar.com
iwantmyfreecash.com
izhevsk.zrus.org
jamiembrown.com
japan-bearings.ru
jasonpartington.com
javrip.net
javspace.net javspace.net
jestr.org javstock.com
jmat.cn jo24news.com
johannesburgsingles.co.za
journeydownthescale.info journeydownthescale.info
keywords-monitoring-success.com jpcycles.com
keywords-monitoring-your-success.com justbcause.com
kidskunst.info kanimage.com
kino-rating.ru katadhin.co
kiwi237au.tk kazinogames.lv
klosetkitten.com kenaba.su
koleso24.com.ua kevblog.top
komputers-best.ru keywordhut.com
konteiner24.com kfon.eu
kovesszucs.atw.hu kidd.reunionwatch.com
krasivoe-hd.com kino-rf.ru
kreativperlen.ch klikbonus.com
kvartiry-remont.ucoz.ru knogg.net
kwzf.net kollesa.ru
labafydjxa.su komputernaya-pomosh-moscow.ru
larger.io koral.se
lennatin.info krasnodar.ru
letslowbefast.site krasula.pp.ua
letsrepair.in kurwa.win
kutikomi.net
ladsup.com
languagecode.com
lapolis.it
ledis.top
legionalpha.com
lezbiyanki.net lezbiyanki.net
lida-ru.com linkbolic.com
lider-zhaluzi.kiev.ua lirunet.ru
life.biz.ua llastbuy.ru
link.ac locustdesign.co
linkbux.ru love-baby.cz
liver-chintai.org m1media.net
loftdigital.eu macdamaged.space
lol-smurfs.com magento-crew.net
lolnex.us mamasuper.prom.ua
lostaruban.com manualterap.roleforum.ru
lutherstable.org manve.info
luxmagazine.cf mapquestz.us
macnewtech.com martlinker.com
mainhp.net mearns-tractors.co.uk
make-money-online.com meddesk.ru
makenahartlin.com medictube.ru
manuscript.su memberty.com
manyresultshub-a.akamaihd.net menetie.ru
maridan.com.ua mesto-x.com
marsgatan.com meteostate.com
marwer.info methodsmarketing.com
masterseek.com mex-annushka.ru
mazda-roadsters.com micasainvest.com
meccadumps.net mikrobiologies.ru
medicinacom.ru minet.club
meds-online24.com mini-modus.ru
meetingrainstorm.bid minyetki.ru
michellblog.online mixtapetorrent.com
mielec.pl mlvc4zzw.space
mirmedinfo.ru mnogabukaff.net
mistr-x.org mnogolok.info
mobile.ok.ru mobioffertrck.com
mompussy.net modnie-futbolki.net
moneyviking-a.akamaihd.net monetizationking.net
morepoweronmac.trade money-every-day.com
moscow-clining.ru morlat.altervista.org
moscow.xrus.org moscow.xrus.org
mosdverka.ru mosdverka.ru
moskva.nodup.ru motherboard.vice.com
mountainstream.ms mrinsidesales.com
mozello.ru muizre.ru
mrpornogratis.xxx music.utrolive.ru
music7s.me musicvidz.ru
mwtpludn.review muz-baza.net
my-aladin.com muz-tracker.net
mydownlodablefiles.com my-cash-bot.co
narkologiya-orel.ru myseoconsultant.com
nationalbreakdown.com myyour.eu
naturtreenspicerx.pw na15.ru
netvouz.com narkologiya-belgorod.ru
naughtyconnect.com
needtosellmyhousefast.com
networkad.net
nhl17coinsforps3.gratisblog.biz nhl17coinsforps3.gratisblog.biz
nikkiewart.ru nicovideo.jp
obiavo.ru nodding-passion.tk
onclickpredictiv.com normalegal.ru
onefilms.net november-lax.com
online-hd.pl nrv.co.za
onlyporno.ru o.light.d0t.ru
onstrapon.purplesphere.in oakleyglassesonline.us
ontargetseo.us obiavo.in
ooo-gotovie.ru offer.wpsecurity.website
openmultipleurl.com offers.bycontext.com
oren-cats.ru office2web.com
oyster-green.com olympescort.com
paclitor.com online7777.com
panicatack.com oops-cinema.ru
paparazzistudios.com.au openlibrary.org
par-fallen.ga optom-deshevo.ru
paramountmarble.co.uk oralsexfilme.net
partner-trustworthy.men orel-reshka.net
pateaswing.com owohho.com
patterntrader-en.com paccohichetoti.ml
pavlodar.xkaz.org paperwritingservice17.blogspot.ru
peekyou.com paydayonlinecom.com
photokitchendesign.com petsblogroll.com
picphotos.net pharmacyincity.com
photosaga.info
pictures-and-images.com pictures-and-images.com
picturesmania.com piski.top
piratecams.com pk-pomosch.ru
pix24x7.com poem-paying.gq
pixell.club pokemon-go-play.online
pizdeishn.com poptool.net
pizdopletka.club popup-jdh.xyz
pl-top.pl pornodojd.ru
pl.id-forex.com pornofeuer.com
pogruztehnik.ru pornotubs.com
porndairy.in pornpost.in
postclass.com primedice.com
pr0fit-b0x.com princevc.com
priora-2.com privetsochi.ru
privatbank46.ru prizefestival.mobi
privatov-zapisi.ru prizesbook.online
pro-poly.ru pro-okis.ru
pron.pro prospekt-st.ru
prostitutki-novgoroda.org prostitutki-kharkova.org
pwwysydh.com pshare.biz
pyrodesigns.com.au ptr.ruvds.com
pussyspace.net
qor360.com
quangcaons.com quangcaons.com
qwarckoine.com quebec-bin.com
ragecash.com queerspace.com
rangjued.com quick-seeker.com
rankexperience.com quickcashlimited.com
realting-moscow.ru r-control.ru
redirlock.com randki-sex.com
reeyanaturopathy.com rankia.com
remont-komputerov-notebook.ru reining.lovasszovetseg.hu
replica-watch.ru reklama-i-rabota.ru
resellerclub.com remontbiz.ru
rethinkwasteni.info remontgruzovik.ru
renecaovillasale.online
risparmiocasa.bz.it risparmiocasa.bz.it
rockma.se rocis.site
roma-kukareku.livejournal.com roleforum.ru
rossanasaavedra.net royalads.net
royalvegascasino.com royalvegascasino.com
runstocks.com s-forum.biz
rusoft-zone.ru s-iwantyou.com
s8-nowy-wygraj.comli.com samo-soznanie.ru
sabaapress.com sarafangel.ru
sad-torg.com.ua savefrom.com
sajatvelemeny.com saveriopiazza.it
satoristudio.net scanner-don.top
saugatuck.com scanner-jane.top
searchaddis.com schlampen-treffen.com
searchencrypt.com sdelatmebel.ru
see-your-website-here.com seasonvar.ru
seminarygeorgia59.ga seeingmeerkat.com
seminarykansas904.ml seo-traffic-ranking.info
seo-tools-optimizing.com seocdvig.ru
seoprofiler.com seorank.info
service.adtech.fr sexflirtbook.com
sex-foto.pw sexystrippe.info
sexcamamateurchat.com sharebutton.to
sexvideo-sex.com shopwme.ru
shell-pmr.ru sigmund-freud.co.uk
shikiso.info sitevaluation.org
shillyourcoins.com sklad-24.ru
shisha-swag.de smartadserver.com
shoesonlinebuy.xyz snip.tw
shtora66.ru soblaznu.net
siteheart.net social-s-ggg.xyz
sitevaluation.com socialseet.ru
sitiz.club sogimlecal.tk
sixcooler.de soietvousmaime.fr
skladvaz.ru sonnikforme.ru
skyway24.ru space4update.pw
slim.sellany.ru spravka130.ru
slimcdn.com sps-shop.com
smart-balancewheel.com spywarebegone.com
snsdeainavi.info squidoo.com
sobecjvuwa.com.ru stats-public.grammarly.io
socialbutton.xyz stefanbakosab.se
sofit-dmd.ru sterva.cc
solution4u.com stockquotes.wooeb.com
songoo.wz.cz studiofaca.com
space-worry.ml styro.ru
spasswelt.xyz summerlinhomes411.info
special-porn.com sundrugstore.com
spin2016.cf superstats.com
srecorder.com taximytishi.ru
stanthonyscatholicchurch.org telefonsexsofort.tk
stardevine.com telesvoboda.ru
statistici.ro templates.franklinfire.co
staynplay.net textads.men
steelmaster.lv theheroes.ru
stickers-market.ru tilido.com
strigkaomsk.ru tizanidine4mguses.blogspot.com
strongholdsb.ru tizanidineandcipro.blogspot.com
strv.se tizanidineonline.blogspot.com
studentguide.ru tizanidinepain.blogspot.com
sureone.pro tizanidinestreetvalue.blogspot.com
svensk-poesi.com tizanidinewithdrawalsymptoms.blogspot.com
swagbucks.com tnaionline.org
sweepstakes.rewardit.com tnctrx.com
taraz.xkaz.org todohr.com
tecnoteakviareggio.it top-deal.com.pl
tedxrj.com topsiteminecraft.com
teenforporn.com tourcroatia.co.uk
tetracsaudi.com tracklead.net
the-trader.net tracksurf.daooda.com
the-usa-games.blogspot.com traffixer.com
theallgirlarcade.com trailer.cinemaflix.website
thegioixekhach.com triplepanda.xyz
theprofitsmaker.net truemfilelj.gq
timecrimea.ru trumpetedextremes.com
tizanidinehydrochloride4mgstreetvalue.blogspot.com twu.com.ua
tmm-kurs.ru u-cheats.ru
topsy.com uchetunet.su
torture.ml
tour-line.net
toursmaps.com
tpu.ru
trafaret74.ru
triumf-realty.ru
trkdf.com
trubywriting.com
tx41tclega.ru
ukrobstep.com
ultimatesetnewfreeallsoftupgradesystems.pw
universals.com.ua
unrealcommander.com
ural-buldozer.ru ural-buldozer.ru
urll.eu usdx.us
us-america.ru usiad.net
uytmaster.ru vacances-voyages.info
vadimkravtcov.ru vapomnoncri.tk
vapsy.com vedomstvo.net
vbtracker.net versaut.xxx-cam.webcam
vellings.info
veopornogratis.xxx
vertaform.com vertaform.com
vesnatehno.com
vezuviy.su vezuviy.su
viandpet.com via-energy-cumpara.com
video-chat.love via-gra.webstarts.com
videochat.life viagroid.ru
villacoloniale.com videochat.bz
vkgaleria.com vids18.site
vkonche.com viel.su
vkontaktemusic.ru vitoriacabos.com
vladhistory.com vk-mus.ru
volgograd.xrus.org vltai.com
vozbujdenie.com vroze.com
wait3sec.org vseigru.one
webix.biz vseigry.fun
weboptimizes.com vsexkontakte.net
wedding0venues.tk vzlomfb.com
whatsappdownload10.com vzlomtw.com
wholesalejerseys-cheapest.com warning.or.kr
wingsoffury2.com webalan.ru
winterclassichockeyjerseys.com webcamtalk.net
weblo.com
weburlopener.com
weprik.ru
wesharepics.info
wholesalejerseyscheapjerseys.us.com
winwotgold.pl
withstandingheartwarming.com withstandingheartwarming.com
woman-tampon.ru wosik-dach.service-for-web.de
womensterritory.ru wowcasinoonline.ooo
word-vorlagen.net www.get-free-traffic-now.com
worldinternetauthority.com xn----ctbbcjd3dbsehgi.xn--p1ai
worldoffiles.ru xn--80aaagvmjabrs1aoc9luc.xn--p1ai
wormix-cheats.ru xn--80ak6aa92e.com
x-diesel.com xn--d1aifoe0a9a.top
xingzi-vision.com xn--e1afanlbnfckd7c3d.xn--p1ai
xmlinde.com xnxx699.com
xn--80aaafbn2bc2ahdfrfkln6l.xn--p1ai xvideospornoru.com
xn--oogle-wmc.com
xolodremont.ru
xxx-cam.webcam xxx-cam.webcam
xxxhdvideo.site xzlive.com
xz618.com yaderenergy.ru
yaminecraft.ru youjizz.vc
yottos.com your-bearings.com
yougetsignal.com youresponsive.com
youjizz.com yourporn.com
yourothersite.com youthreaders.com
youtuhe.com zakazvzloma.com
yubikk.info zapatosenventa.info
zarabotat-na-sajte.ru zeikopay.com
zastroyka.org zhacker.net
zeroredirect5.com zkjovpdgxivg.ga
zlatnajesen.com znakom.sibtest.ru
znaturaloriginal.com zog.link
zolotoy-lis.ru zs2vm.top
zoomovies.org
zoosexart.com
zophim.me

486
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
00webcams.com 11pikachu.ru
03e.info 1stat.ru
3-letter-domains.net 4pp13.com
72-news.com 8kisses.eu
76brighton.co.uk accessoires-mode.in
7a2rnuey1tw9ar.ru adman.se
aa625d84f1587749c1ab011d6f269f7d64.com admitad.com
adnotbad.com adult3dgames.com
ads-seo.men adultwebhosting.info
adsfresh.men agapovdg.ru
adviceforum.info akusherok.ru
advokateg.xyz albumsuper.info
adzpower.com all4wap.ru
allforminecraft.ru alldownload.pw
allornamenti.com analnoeporno.tv
amateurmatch.com apelsinnik.website
an-donut.com apibatbrowsecom-a.akamaihd.net
anapa-inns.ru apifasterlightin-a.akamaihd.net
anonymousfox.co apiitsthirteende-a.akamaihd.net
anticrawler.org appsecurityr.com
appfastplay.com arabsexxxtube.com
atout-energie-69.com arabseyes.com
auto4style.ru artpicso.com
avek.ru asscenihotosidea.blogspot.co.za
avkzarabotok.info astana.xxxkaz.org
awaybird.ru atmovs.com
ayerbo.xhost.ro autonew.biz
balans.shahterworld.org aversis.net
balla-boo.se aviav.ru.com
bannertesting.com azbukafree.com
best-way.men batanga.net
bestmarriages.com bayareaaeroclub.org
bestofupload.info beachpics.com
bioscorp.ru becuo.com
blobar.org belstaffstore.online
blog.koorg.ru best-seo-solution.com
bobba.dzaba.com bestwaystogetpaid.us
bradleylive.xyz bgviagrachrx.com
brains2.biz bio-market.kz
biznesluxe.ru
blog.remote-computer.de
blogig.org
brainsandeggs.blogspot.com brainsandeggs.blogspot.com
breastaugmentation.co.za bristolhostel.com
brimstonehillfortress.org bumskontakte.org
browsepulse-a.akamaihd.net burger-imperia.com
buigas.00it.com buyhoverboard.com
buzzonclick.com californianews.cf
cartierlovebracelet.xyz cartierbracelet.xyz
cerev.info cartierlove2u.xyz
chatroulette.online cartierlovestore.xyz
cienum.fr ccbill.com
clicksor.com cdncash.org
clksite.com cialisovercounteratwalmartusa.com
club-lukojl.ru citetick.com
co.lumb.co clknsee.com
codq.info cometorussia.net
coldfilm.ru compliance-viktor.top
compliance-checker-7.info crackzplanet.com
compliance-donald.xyz
confib.ifmo.ru
copro.pw
cscwtalkto.site
custodieva.ru custodieva.ru
customsua.com.ua ddospanel.com
dailyrank.net delitime.info
danceuniverse.ru dieswaene.com
de.zapmeta.com digilander.libero.it
decorationspcs.com dignitasdata.se
diminishedvalueoforegon.com dikx.gdn
dividendo.ru domaineaneblanc.com
dogbreedspicture.net domashneeruporno.com
dpihatinh.gov.vn donvito.unas.cz
eaplay.ru downloadmefiranaratb1972.xpg.com.br
edudocs.net draniki.org
elvenar.com drev.biz
en.home-task.com drunkenstepfather.com
essayassist.com e-collantes.com
euromasterclass.ru education-cz.ru
exlarseva.webblog.es efkt.jp
f012.de elektrozigarette2.wordpressy.pl
facebook-mobile.xyz eloxal.ru
ff30236ddef1465f88547e760973d70a.com engines-usa.com
fickkontakte.org esopini.com
filmbokep69.com eyemagination.com
findanysex.com ezigarettenkaufen1.pagina.gr
flowersbazar.com fashion-mk.net
forex.osobye.ru figuringmoneyout.com
freewlan.info foodcrafts.website
froggytube.com formularz-konkurs.tk
fuel-gas.com foto-basa.com
futbolkisales.ru free411games.com
fx-brokers-review.com frustrated-favorable.gq
gagrasector.ru fun2cell.net
galaxy-family.ru gamesprite.me
gambarkataku.co gfaq.ru
gamedayhouse.com girlporn.ru
gdzkurokam.ru gok-kasten.net
getlamborghini.ga googlefeud.com
getmiro.com gsasearchengineranker.site
goo.ne.jp gtopstats.com
gta-club.ru guigyverpo.cf
haveinc.xyz gunsvicceadadebt.tk
hesteel.pl hanink.biz.ly
hitmuzik.ru hdwalls.xyz
hptwaakw.blog.fc2.com healbio.ru
ilikevitaly.com hulfingtonpost.com
imagez.co iinstalll-fii1leis.jus0wil.pp.ua
imitex-plus.ru iklysha.ml
infazavr.ru imgarit.pw
infoupdate.org imicrovision.com
innoslicon.com indianmedicaltourismshop.com
interior-stickers.ru inome.com.ua
investingclub.ru io9.com
javspace.net iskalko.ru
kinobaks.com ispaniya-costa-blanca.ru
kit-opt.ru istock-mebel.ru
kochanelli.com itrevolution.cf
kraljeva-sutjeska.com kinosed.net
klikbonus.com
la-fa.ru
laapp.com
lanadelreyfans.us
landreferat.ru landreferat.ru
larger.io law-check-nine.xyz
lexiangwan.com lawyers.cafe
lidoradio.com lesbian.xyz
likesdesign.com lexaprogeneric.link
linkbolic.com live-sexcam.tk
lisque.batcave.net lonfon.xyz
littlesexdolls.com make-money-online.com
livejasmin.com manipulyator-peterburg.ru
lomza.info manyresultshub-a.akamaihd.net
losangeles-ads.com marinetraffic.com
lostfilm-online.ru marketingvici.com
machicon-akihabara.info mbiologi.ru
mailemedicinals.com
matchpal-a.akamaihd.net
mature.free-websites.com
mcadamssupplyco.com
mebelcomplekt.ru
meccadumps.net meccadumps.net
metallosajding.ru medispainstitute.com.au
miniads.ca medizinreisen.de
mmstat.com medmajor.ru
mobile-appster.ru megagrabber.ru
mug-na-chas-moscow.ru mjchamonix.org
muzaporn.com mobplayer.net
my-bc.ru moetomnenie.com
mypets.by moneyzzz.ru
naturalshair.site mrvideospornogratis.xxx
netvouz.com namenectar.com
nhl09.ru newstraveller.ru
nomuos.it novosti-hi-tech.ru
novodigs.com
nsatc.net
nufaq.com nufaq.com
o-o-8-o-o.com odoratus.net
odywpjtw.bloger.index.hr omsk.xrus.org
onblastblog.online onlineporno.site
onlinefilmz.net optibuymac.com
ontargetseo.us pagesense.com
orhonit.com pcimforum.com
pacificair.com picsforkeywordsuggestion.com
partybunny.ru pinup-casino1.ru
perfectpracticeweb.com pisanieprac.info
pflexads.com piski.top
picsfair.com pl.aasoldes.fr
pk-pomosch.ru poligon.com
platezhka.net popunder.ru
popup.matchmaker.com popup-hgd.xyz
porngalleries.top pornosee.info
pp-budpostach.com.ua pornosexrolik.com
prchecker.info preparevideosafesystem4unow.space
privat-girl.net printie.com
prostitutki-kharkova.org prlog.ru
pussyfleet.com prodess.ru
rarbg.to putevka24.ru
reimageplus.com questionmarque.ch
relatodelpresente.com.ar quickcashlimited.com
resant.ru rabot.host.sk
resellerclub.com rankscanner.com
retailwith.com raschtextil.com.ua
rfserial.net realnye-otzyvy.info
ric.info reklama1.ru
ruscopybook.com remontbiz.ru
rutor.group risparmiocasa.bz.it
salmonfishingsacramentoriver.com rosbalt.com.ua
samo-soznanie.ru royalads.net
scanner-jack.top rp9.ru
scanner-mark.top ruclicks.com
scansafe.net saitevpatorii.com
search.alot.com saletool.ru
search.pch.com salut-camp.ru
searchwebknow-a.akamaihd.net savefrom.com
semxiu.com scanner-viktor.top
serdcenebolit.com seminarykansas904.ml
serviporno.com servethis.com
sexystrippe.info sexad.net
shillyourcoins.com sexkvartal.com
shop-electron.ru sexyebonyteen.com
signal03.ru shoppingjequiti.com.br
silverage.ru simple-image.com.ua
simplesite.com simplesite.com
sixcooler.de skladvaz.ru
slonechka.ru skypasss.com
smartbalanceworld.com slowmacfaster.trade
soblaznu.net sofit-dmd.ru
sockshares.tv softonicads.com
solution4u.com sonata-arctica.wz.cz
soochi.co sovetogorod.ru
stairliftsarea.com start.myplaycity.com
startufa.ru steamoff.net
statistici.ro stickers-market.ru
stevemonsen.com stopnarco.ru
strongholdsb.ru strag-invest.ru
sugarlyflex.pw superstarfloraluk.com
sumo.com svbur.ru
tandvardshuset.net tacbibirfa.tk
taraz.xkaz.org teenfuck.tv
teesdaleflyballclub.co.uk the-usa-games.blogspot.com
tetracsaudi.com thedownloadfreeonlinegames.blogspot.com
thejournal.ru theheroes.ru
ticketsys.inetwd.com thesmartsearch.net
tonerbox.kz tivolibasket.it
top10registrycleaners.com tizanidinedosageforsleep.blogspot.com
traderzplanet.in tizanidinefibromyalgia.blogspot.com
triplepanda.xyz tizanidinehydrochloride2mg.blogspot.com
tizanidineinteractions.blogspot.com
tizanidinestreetvalue.blogspot.com
tobeyouday.win
tovaroboom.vast.ru
track-rankings.online
tracksz.co
trahvid.com
traxdom.ru
trion.od.ua
tripper.de tripper.de
ttsq.fr tuberkulezanet.ru
turist-strani.ru ua.tc
turizmus.us umnovocaminho.com
ultimatesetnewfreeallsoftupgradesystems.pw validccseller.com
unrealcommander.biz valoresito.com
v-doc.co vapeface.club
vesnatehno.com visitcambridge.org
video-chat.love vroze.com
watch-movies.ru wieseversa.no
wdrake.com winterclassichockeyjerseys.com
webadvance.club woweb.com.ua
webextract.profound.net wpthemedetector.co.uk
westen-v.life www.pinnacle-bets.com
www.888.com xfluro.com
xclicks.net xn--e1aggki3c.xn--80adxhks
xn------7cdbapdecfd4ak1bn0amjffj7afu3y.xn--p1ai xn--h1ahbi.com.ua
xn--80aaafbn2bc2ahdfrfkln6l.xn--p1ai xuki.us
xtrafficplus.com xvideospornoru.com
youdao.com xxxmania.top
yoursearch.me xxxtubesafari.com
zaloadi.ru yes-com.com
zapatosenventa.info yourdesires.ru
zarabiaj-dzis.pl zacreditom.ru
zastenchivosti.net
zawyna.ua
zigzog.ru
zirondelli.it
zvooq.eu

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.0.71 0; 10.1.0.142 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:28 UTC 2021 ### Updated: Sun Aug 15 05:18:22 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2021.08.2526 ### Version: V4.2021.08.2527
### Updated: Sat Aug 14 22:01:28 UTC 2021 ### Updated: Sun Aug 15 05:18:22 UTC 2021
### Bad Bot Count: 623 ### Bad Bot Count: 623
################################################### ###################################################
### Version Information ## ### Version Information ##