vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-03 11:10:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2022.01.2831 [ci skip]

Browse source
This commit is contained in:
mitchellkrogza 2022-01-05 07:03:44 +00:00
parent 42c46ecaea
commit 1e25176f1c
53 changed files with 1142 additions and 1121 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
AUTO-CONFIGURATION.md
View file

@ -4,8 +4,8 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.01.2830 #### Version: V4.2022.01.2831
#### Bad Referrers Blocked: [7086](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
MANUAL-CONFIGURATION.md
View file

@ -4,8 +4,8 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation: ### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________ _______________
#### Version: V4.2022.01.2830 #### Version: V4.2022.01.2831
#### Bad Referrers Blocked: [7086](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

4
README.md
View file

@ -13,8 +13,8 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots) ##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic even Fake Googlebots from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________ _______________
#### Version: V4.2022.01.2830 #### Version: V4.2022.01.2831
#### Bad Referrers Blocked: [7086](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

2
_generator_lists/bad-referrers.list
View file

@ -1,4 +1,3 @@
scanhub.ru
000free.us 000free.us
007angels.com 007angels.com
00author.com 00author.com
@ -5171,6 +5170,7 @@ sbtdesign.co.uk
sbwealthsolutions.ca sbwealthsolutions.ca
sc-specialhost.com sc-specialhost.com
scalerite.co.za scalerite.co.za
scanhub.ru
scanmarine.info scanmarine.info
scanmyphones.com scanmyphones.com
scanner-alex.top scanner-alex.top

4
_google_analytics_ghost_spam/README.md
View file

@ -3,8 +3,8 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM # EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________ _______________
#### Version: V4.2022.01.2830 #### Version: V4.2022.01.2831
#### Bad Referrers Blocked: [7086](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list) #### Bad Referrers Blocked: [7087](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-referrers.list)
#### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list) #### Bad User-Agents (Bots) Blocked: [631](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)
#### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list) #### Fake Googlebots Blocked: [217](https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
____________________ ____________________

2
_google_analytics_ghost_spam/google-exclude-04.txt
View file

File diff suppressed because one or more lines are too long

1
_google_webmaster_disavow_links/google-disavow.txt
View file

@ -5170,6 +5170,7 @@ domain:sbtdesign.co.uk
domain:sbwealthsolutions.ca domain:sbwealthsolutions.ca
domain:sc-specialhost.com domain:sc-specialhost.com
domain:scalerite.co.za domain:scalerite.co.za
domain:scanhub.ru
domain:scanmarine.info domain:scanmarine.info
domain:scanmyphones.com domain:scanmyphones.com
domain:scanner-alex.top domain:scanner-alex.top

7
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:29 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

7
conf.d/globalblacklist-testing-version.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2830 ### Version: V3.2022.01.2831
### Updated: Tue Jan 4 22:01:30 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

7
conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:29 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

7
dev-tools/beta_conf_files/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2830 ### Version: V3.2022.01.2831
### Updated: Tue Jan 4 22:01:30 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 1; 10.1.0.73 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 0; 10.1.0.73 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/beta_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2830 ### Version: V3.2022.01.2831
### Updated: Tue Jan 4 22:01:30 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 1; 10.1.0.73 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/beta_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 0; 10.1.0.73 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/beta_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2830 ### Version: V3.2022.01.2831
### Updated: Tue Jan 4 22:01:30 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/beta_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 1; 10.1.0.73 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/beta_conf_files_whitelist/bots.d/custom-bad-referrers.conf
View file

@ -5217,6 +5217,7 @@
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 0; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 0;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 0; "~*(?:\b)sc\-specialhost\.com(?:\b)" 0;
"~*(?:\b)scalerite\.co\.za(?:\b)" 0; "~*(?:\b)scalerite\.co\.za(?:\b)" 0;
"~*(?:\b)scanhub\.ru(?:\b)" 0;
"~*(?:\b)scanmarine\.info(?:\b)" 0; "~*(?:\b)scanmarine\.info(?:\b)" 0;
"~*(?:\b)scanmyphones\.com(?:\b)" 0; "~*(?:\b)scanmyphones\.com(?:\b)" 0;
"~*(?:\b)scanner\-alex\.top(?:\b)" 0; "~*(?:\b)scanner\-alex\.top(?:\b)" 0;

2
dev-tools/beta_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 0; 10.1.0.73 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/beta_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2830 ### Version: V3.2022.01.2831
### Updated: Tue Jan 4 22:01:30 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/buildnumber
View file

@ -1 +1 @@
2830 2831

7
dev-tools/globalblacklist-testing.template
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V3.2022.01.2830 ### Version: V3.2022.01.2831
### Updated: Tue Jan 4 22:01:30 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

7
dev-tools/globalblacklist.template
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:29 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

1
dev-tools/referrers-regex-format-whitelist-test.txt
View file

@ -5170,6 +5170,7 @@
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 0; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 0;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 0; "~*(?:\b)sc\-specialhost\.com(?:\b)" 0;
"~*(?:\b)scalerite\.co\.za(?:\b)" 0; "~*(?:\b)scalerite\.co\.za(?:\b)" 0;
"~*(?:\b)scanhub\.ru(?:\b)" 0;
"~*(?:\b)scanmarine\.info(?:\b)" 0; "~*(?:\b)scanmarine\.info(?:\b)" 0;
"~*(?:\b)scanmyphones\.com(?:\b)" 0; "~*(?:\b)scanmyphones\.com(?:\b)" 0;
"~*(?:\b)scanner\-alex\.top(?:\b)" 0; "~*(?:\b)scanner\-alex\.top(?:\b)" 0;

1
dev-tools/referrers-regex-format.txt
View file

@ -5170,6 +5170,7 @@
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/test1_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
13.82.134.136 0; 20.185.50.223 0;

7
dev-tools/test1_conf_files/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:29 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 1; 10.1.0.73 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 0; 10.1.0.73 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:29 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 1; 10.1.0.73 1;
127.0.0.1 1; 127.0.0.1 1;

2
dev-tools/test1_conf_files_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 0; 10.1.0.73 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:29 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/test1_conf_files_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 1; 10.1.0.73 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test1_conf_files_whitelist/bots.d/custom-bad-referrers.conf
View file

@ -5217,6 +5217,7 @@
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 0; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 0;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 0; "~*(?:\b)sc\-specialhost\.com(?:\b)" 0;
"~*(?:\b)scalerite\.co\.za(?:\b)" 0; "~*(?:\b)scalerite\.co\.za(?:\b)" 0;
"~*(?:\b)scanhub\.ru(?:\b)" 0;
"~*(?:\b)scanmarine\.info(?:\b)" 0; "~*(?:\b)scanmarine\.info(?:\b)" 0;
"~*(?:\b)scanmyphones\.com(?:\b)" 0; "~*(?:\b)scanmyphones\.com(?:\b)" 0;
"~*(?:\b)scanner\-alex\.top(?:\b)" 0; "~*(?:\b)scanner\-alex\.top(?:\b)" 0;

2
dev-tools/test1_conf_files_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 0; 10.1.0.73 0;
127.0.0.1 0; 127.0.0.1 0;

7
dev-tools/test1_conf_files_whitelist/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:29 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/test2_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
13.82.134.136 0; 20.185.50.223 0;

7
dev-tools/test2_conf_files/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:29 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/test3_conf_files/bots.d/whitelist-ips.conf
View file

@ -45,4 +45,4 @@
# - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions. # - Whitelisting IP's and RANGES here ONLY affects the IP blocking functions.
# - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks. # - This file will NOT allow your own IP to bypass bad User-Agent or Referrer String checks.
# - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST # - To bypass everything for a certain IP see notes in blockbots.conf on SUPER WHITELIST
13.82.134.136 0; 20.185.50.223 0;

7
dev-tools/test3_conf_files/conf.d/globalblacklist.conf
View file

@ -4,9 +4,9 @@
### VERSION INFORMATION # ### VERSION INFORMATION #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:29 UTC 2022 ### Updated: Wed Jan 5 06:57:51 UTC 2022
### Bad Referrer Count: 7086 ### Bad Referrer Count: 7087
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### VERSION INFORMATION ## ### VERSION INFORMATION ##
@ -6230,6 +6230,7 @@ map $http_referer $bad_referer {
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 1;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 1; "~*(?:\b)sc\-specialhost\.com(?:\b)" 1;
"~*(?:\b)scalerite\.co\.za(?:\b)" 1; "~*(?:\b)scalerite\.co\.za(?:\b)" 1;
"~*(?:\b)scanhub\.ru(?:\b)" 1;
"~*(?:\b)scanmarine\.info(?:\b)" 1; "~*(?:\b)scanmarine\.info(?:\b)" 1;
"~*(?:\b)scanmyphones\.com(?:\b)" 1; "~*(?:\b)scanmyphones\.com(?:\b)" 1;
"~*(?:\b)scanner\-alex\.top(?:\b)" 1; "~*(?:\b)scanner\-alex\.top(?:\b)" 1;

2
dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 1; 10.1.0.73 1;
127.0.0.1 1; 127.0.0.1 1;

1
dev-tools/test_units/custom-bad-referrers.conf
View file

@ -5217,6 +5217,7 @@
"~*(?:\b)sbwealthsolutions\.ca(?:\b)" 0; "~*(?:\b)sbwealthsolutions\.ca(?:\b)" 0;
"~*(?:\b)sc\-specialhost\.com(?:\b)" 0; "~*(?:\b)sc\-specialhost\.com(?:\b)" 0;
"~*(?:\b)scalerite\.co\.za(?:\b)" 0; "~*(?:\b)scalerite\.co\.za(?:\b)" 0;
"~*(?:\b)scanhub\.ru(?:\b)" 0;
"~*(?:\b)scanmarine\.info(?:\b)" 0; "~*(?:\b)scanmarine\.info(?:\b)" 0;
"~*(?:\b)scanmyphones\.com(?:\b)" 0; "~*(?:\b)scanmyphones\.com(?:\b)" 0;
"~*(?:\b)scanner\-alex\.top(?:\b)" 0; "~*(?:\b)scanner\-alex\.top(?:\b)" 0;

20
dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,10 +1,10 @@
Claritybot Discoverybot
Gotit EyeNetIE
Grabber Fimap
Haansoft NimbleCrawler
Netcraft Openvas
OnCrawl OrangeSpider
RedesScrapy Pixray
SEOprofiler RSSingBot
Web Fuck Stripper
WebReaper WebFetch

310
dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
01h4x.com 01h4x.com
ADmantX 404checker
ALittle Client AIBOT
ASPSeek AhrefsBot
Abonti Alexibot
Aboundex Alligator
Acunetix
Anarchie Anarchie
Anarchy Anarchy99
Ankit Anthill
Aspiegel AwarioRssBot
Asterias
BBBike BBBike
BDCbot
BDFetch
BackDoorBot
BackStreet BackStreet
Backlink-Ceck BackWeb
BacklinkCrawler
Bandit Bandit
Barkrowler Barkrowler
BatchFTP
BetaBot BetaBot
Bigfoot Blackboard
BlackWidow
Blow Blow
BlowFish Brandwatch
BotALot Buck
Buddy
BuiltBotTough BuiltBotTough
Bullseye BuiltWith
BunnySlippers BunnySlippers
CATExplorador CCBot
Calculon CODE87
CazoodleBot CSHttp
CheTeam CensysInspect
ChinaClaw CherryPicker
Chlooe
Citoid
Claritybot Claritybot
Cocolyzebot Cocolyzebot
CopyRightCheck CopyRightCheck
CyotekWebCopy Cosmos
Custo
DIIbot
DTS Agent DTS Agent
Demon Deusu
Devil Devil
Digincore Digincore
DigitalPebble
Dirbuster Dirbuster
DnyzBot Discobot
Dispatch
DittoSpyder
DnBCrawler-Analytics
DomCopBot DomCopBot
DomainSigmaCrawler DomainCrawler
Domains Project Dotbot
ECCP/1.0 Download Wonder
Dragonfly
Drip
EMail Siphon
EMail Wolf EMail Wolf
Ebingbong
Ecxi Ecxi
EroCrawler
Express WebPictures
ExtLinksBot ExtLinksBot
Extractor Extractor
FDM EyeNetIE
FemtosearchBot FemtosearchBot
Fimap
Firefox/7.0 Firefox/7.0
FlashGet Flunky
Foobot
Freeuploader Freeuploader
Fyrebot
G-i-g-a-b-o-t
Genieo Genieo
GermCrawler
GetRight GetRight
GetWeb
Getintent
Gigabot Gigabot
Go!Zilla Go!Zilla
Grabber Go-Ahead-Got-It
GrapeFX
GrapeshotCrawler GrapeshotCrawler
HEADMasterSEO GridBot
HMView HTTrack
HTMLparser
HTTP::Lite
Haansoft Haansoft
IlseBot Harvest
Image Sucker Heritrix
InfoNaviRobot Humanlinks
HybridBot
IDBTE4M
IRLbot
Id-search
IndeedBot
Indy Library
InfoTekies InfoTekies
InternetSeer IstellaBot
JOC Web Spider JOC Web Spider
Jbrofuzz JamesBOT
JetCar JetCar
Jetty
JikeSpider JikeSpider
Jorgee Joomla
JustView
Keyword Density
LWP::Simple LWP::Simple
Larbin Leap
LeechFTP LibWeb
LexiBot LinkpadBot
Lftp
Libwhisker
LieBaoFast
LinkScan
Linkbot
LinksManager
LinqiaScrapeBot
Lipperhey Lipperhey
Lmspider Lmspider
Ltx71 Ltx71
MIIxpc MIDown tool
MQQBrowser
MSIECrawler MSIECrawler
MTRobot
Mag-Net Mag-Net
Majestic-SEO MarkMonitor
MarkWatch MarkWatch
Mass Downloader Masscan
MauiBot MauiBot
MegaIndex.ru Mb2345Browser
MicroMessenger MeanPath Bot
Metauri
Microsoft Data Access Microsoft Data Access
Morfeus Fucking Scanner Minefield
Mister PiX
Mojolicious
Msrabot Msrabot
NICErsPRO NICErsPRO
Nameprotect NPbot
Navroad Name Intelligence
Needle Needle
Nessus Nessus
NetMechanic Net Vampire
NetSpider NetLyzer
NetZIP
Netcraft
Nettrack
NextGenSearchBot NextGenSearchBot
Nibbler Niki-bot
NimbleCrawler
Nmap
Not
Nutch
Octopus Octopus
Offline Explorer Offline Explorer
OrangeBot OpenLinkProfiler
OutclicksBot OrangeSpider
OutfoxBot Page Analyzer
PECL::HTTP
PHPCrawl
POE-Component-Client-HTTP
PageAnalyzer PageAnalyzer
PageScorer PageScorer
Pandalytics
Panscient
Papa Foto Papa Foto
PeoplePal Piepmatz
Petalbot Probethenet
Pi-Monster QueryN Metasearch
Picscout RankActive
Picsearch RankActiveLinkBot
Pimonster RankingBot2
PleaseCrawl
Pockey
ProPowerBot
ProWebWalker
Psbot
Pump
PyCurl
Quick-Crawler
RankurBot RankurBot
Re-re RebelMouse
ReGet
Recorder
Rogerbot
SBIder SBIder
SEOkicks SEOkicks
SISTRIX SalesIntelligent
Scrapy Scanbot
ScreenerBot ScreenerBot
ScrepyBot Searchestate
SearchmetricsBot SemanticJuice
Semrush
SentiBot SentiBot
SeobilityBot SeobilityBot
SiteExplorer Seomoz
SiteLockSpider Siphon
SiteSucker Site Sucker
Sitebeam SiteSnagger
Siteimprove Sitevigil
SmartDownload
Snapbot
Sogou web spider
Sosospider Sosospider
Sottopop Spanner
SpankBot SputnikBot
Spbot
Sqlmap Sqlmap
Sqlworm
Stripper Stripper
Sucker
SuperBot SuperBot
Surfbot SuperHTTP
SurveyBot
Swiftbot
Szukacz Szukacz
T0PHackTeam T0PHackTeam
Teleport T8Abot
Trendictionbot Telesoft
TurnitinBot Thumbor
Twice Toweyabot
Upflow Tracemyfile
VCI Turnitin
Vacuum TwengaBot
VelenPublicWebCrawler V-BOT
Vagabondo
Virusdie Virusdie
VoidEYE VoidEYE
Voil Voil
Voltron Voltron
WASALive-Bot
WBSearchBot WBSearchBot
WEBDAV WEBDAV
WWW-Mechanize
Wallpapers
WallpapersHD WallpapersHD
WeSEE WeSEE
Web Enhancer
Web Fuck
Web Sauger
Web Sucker
WebBandit
WebCollage
WebCopier WebCopier
WebEnhancer WebLeacher
WebFetch WebSauger
WebImageCollector WebmasterWorldForumBot
WebStripper Webshag
WebSucker WebsiteExtractor
WebWhacker Webster
Webalta
Website Quester
Whacker Whacker
Widow Widow
Wotbox
Wprecon Wprecon
Xenu
Zade Zade
Zermelo
Zeus Zeus
ZmEu ZmEu
ZoomBot
ZoominfoBot ZoominfoBot
adscanner ZumBot
archive.org_bot ZyBorg
autoemailspider autoemailspider
backlink-check
cah.io.community cah.io.community
check1.exe check1.exe
clark-crawler clark-crawler
cognitiveseo
com.plumanalytics com.plumanalytics
crawler4j crawl.sogou.com
dataforseo.com evc-batch
demandbase-bot facebookscraper
domainsproject.org gopher
instabid instabid
ips-agent
lwp-request lwp-request
lwp-trivial muhstik-scan
netEstate NE Crawler
oBot oBot
pcBrowser page scorer
plumanalytics
satoristudio.net satoristudio.net
scan.lol scalaj-http
seocompany.store seoscanners
serpstatbot seostar
sitechecker.pro
sogouspider
spyfu spyfu
sysscan
tAkeOut
trendiction.com trendiction.com
trendiction.de
voyagerx.com voyagerx.com
webgains-bot
webmeup-crawler
webprosbot webprosbot
x22Mozilla x09Mozilla
zauba.io zgrab

314
dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
360Spider 01h4x.com
404checker 80legs
404enemy ADmantX
AIBOT
ALittle Client
ASPSeek
Abonti Abonti
Aboundexbot Aboundex
Acunetix
AfD-Verbotsverfahren AfD-Verbotsverfahren
AhrefsBot
AiHitBot AiHitBot
Aipbot Anarchie
Alexibot Anarchy
AllSubmitter
Anarchy99
Anthill Anthill
Apexoo Attach
BDFetch AwarioRssBot
BLEXBot BackDoorBot
BacklinkCrawler Backlink-Ceck
Barkrowler Barkrowler
BatchFTP Bitacle
Battleztar Bazinga Black Hole
Bigfoot BlackWidow
Blackboard Blackboard
Blow Brandprotect
BlowFish Buck
Brandwatch
BuiltWith
BunnySlippers
BuzzSumo
CATExplorador
CCBot CCBot
CazoodleBot CODE87
CheTeam CheTeam
CheeseBot
Chlooe Chlooe
Claritybot Citoid
Cloud mapping Cloud mapping
Cocolyzebot Cocolyzebot
Copier Copier
CopyRightCheck Copyscape
Cosmos
Crawling at Home Project Crawling at Home Project
CrazyWebCrawler
Crescent Crescent
CrunchBot
Curious Curious
Custo Custo
CyotekWebCopy CyotekWebCopy
DBLBot DBLBot
DIIbot
DSearch DSearch
Demon Discobot
Digincore Discoverybot
Dirbuster
DittoSpyder DittoSpyder
DomCopBot DnyzBot
DomainAppender DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
Dotbot
Drip
ECCP/1.0
EMail Siphon EMail Siphon
EMail Wolf
Ecxi
EirGrabber EirGrabber
Express WebPictures EroCrawler
Extractor Extractor
ExtractorPro EyeNetIE
Extreme Picture Finder Ezooms
FDM FDM
Fimap FemtosearchBot
Firefox/7.0 FlashGet
Foobot
FrontPage FrontPage
Fuzz
FyberSpider
G-i-g-a-b-o-t
GT::WWW GT::WWW
GalaxyBot
Genieo
GermCrawler
GetWeb GetWeb
Getintent Getintent
Go!Zilla Gigabot
Go-Ahead-Got-It
GoZilla
Gotit Gotit
Grabber
Grafula Grafula
GrapeFX
GrapeshotCrawler GrapeshotCrawler
GridBot
HEADMasterSEO
HMView HMView
Harvest HTMLparser
Heritrix
Humanlinks Humanlinks
HybridBot IDBot
IRLbot IlseBot
Iblog Image Sucker
IndeedBot InfoTekies
InfoNaviRobot
Intelliseek Intelliseek
InternetSeer Internet Ninja
Iria JOC Web Spider
IstellaBot
JamesBOT JamesBOT
JennyBot
Jetty
Joomla Joomla
Jorgee JustView
Keyword Density Kenjin Spider
Kinza
Kozmosbot Kozmosbot
LNSpiderguy LNSpiderguy
Larbin Lanshanbot
Lftp LeechGet
Libwhisker
LieBaoFast
Lightspeedsystems
LinkScan LinkScan
LinkWalker Linkbot
LinkextractorPro Linkdexbot
LinqiaRSSBot LinqiaRSSBot
Lipperhey LinqiaScrapeBot
Lipperhey Spider Lipperhey Spider
MIDown tool Ltx71
MFC_Tear_Sample
MIIxpc MIIxpc
MSFrontPage MQQBrowser
MSIECrawler MSIECrawler
MTRobot Majestic SEO
Mail.RU_Bot Majestic12
Majestic-SEO MarkMonitor
MarkWatch Masscan
MauiBot Mata Hari
MeanPath Bot MeanPath Bot
Mediatoolkitbot Microsoft Data Access
Metauri Morfeus Fucking Scanner
MicroMessenger Mozlila
Mister PiX Mr.4x3
Mojolicious NPbot
Musobot Name Intelligence
NetSpider Nameprotect
Nettrack Nessus
NetMechanic
NetZIP
Netcraft
NextGenSearchBot NextGenSearchBot
NimbleCrawler Nimbostratus
Ninja
Offline Explorer Offline Explorer
Offline Navigator Offline Navigator
Openvas OnCrawl
PECL::HTTP OpenLinkProfiler
Page Analyzer Openfind
PageAnalyzer OrangeSpider
OutclicksBot
POE-Component-Client-HTTP
PageThing.com
Pandalytics Pandalytics
Pavuk
PeoplePal PeoplePal
PictureFinder Petalbot
Pu_iN Piepmatz
RSSingBot Pimonster
RankActive PleaseCrawl
RankActiveLinkBot ProPowerBot
ProWebWalker
Psbot
PyCurl
RankFlex RankFlex
RankingBot2 RankingBot2
RankurBot Rankivabot
Re-re
RealDownload RealDownload
Reaper RocketCrawler
RebelMouse
Recorder
RepoMonkey
Ripper
SBIder SBIder
SEOkicks SEOprofiler
SEOkicks-Robot SalesIntelligent
SEOstats
SISTRIX
SMTBot
ScanAlert ScanAlert
ScreenerBot
ScrepyBot ScrepyBot
SeobilityBot Searchestate
Semrush
SemrushBot
SentiBot
Seomoz Seomoz
SiteExplorer SiteLockSpider
SiteSucker
Siteimprove
Sitevigil
Snoopy Snoopy
SocialRankIOBot
Sociscraper Sociscraper
SpaceBison Sogou web spider
Sottopop
Spammen Spammen
Spinn3r Spinn3r
Sqlmap Sqworm
SuperHTTP Stripper
Sucker
Sucuri
Surfbot
Szukacz Szukacz
T8Abot
TeleportPro
Telesoft
Telesphoreo Telesphoreo
The Intraformant
TheNomad TheNomad
TightTwatBot Thumbor
Toata Toata
Tracemyfile True_Robot
Trendictionbot URLy Warning
Twice URLy.Warning
VB Project UnisterBot
V-BOT
VCI
Vagabondo
VelenPublicWebCrawler
VeriCiteCrawler VeriCiteCrawler
VidibleScraper VidibleScraper
VoidEYE
Voil
WBSearchBot
WISENutbot WISENutbot
WPScan WPScan
WWW::Mechanize WWW-Collector-E
Wallpapers WWW-Mechanize
WWWOFFLE
Wallpapers/3.0
Web Auto Web Auto
Web Enhancer
Web Fetch Web Fetch
Web Fuck Web Fuck
WebBandit Web Sauger
WebCollage
WebCopier
WebEnhancer WebEnhancer
WebFetch
WebFuck WebFuck
WebGo IS
WebImageCollector WebImageCollector
WebPix WebSucker
WebReaper
WebSauger
WebWhacker WebWhacker
WebmasterWorldForumBot WebmasterWorldForumBot
Webshag Webshag
Website Quester Website Quester
Webster Whack
Whatweb Whatweb
Who.is Bot
WinHTTrack WinHTTrack
WiseGuys Robot Xaldon WebSpider
Wotbox
Xaldon_WebSpider
Xenu Xenu
YoudaoBot YoudaoBot
Zade Zeus
Zauba ZmEu
Zermelo ZoomBot
Zitebot ZoominfoBot
ZumBot
ZyBorg
adscanner
arquivo.pt
autoemailspider
backlink-check backlink-check
cah.io.community cah.io.community
crawl.sogou.com crawl.sogou.com
crawler.feedback crawler.feedback
crawler4j gopher
demandbase-bot heritrix
evc-batch ips-agent
instabid
meanpathbot meanpathbot
seocompany.store mediawords
page scorer
probe-image-size
ripz
satoristudio.net
seoscanners seoscanners
sitechecker.pro seostar
siteripz sogouspider
sp_auditbot
spyfu spyfu
trendiction.com sysscan
tAkeOut
ubermetrics-technologies.com
voyagerx.com voyagerx.com
webgains-bot
webmeup-crawler
webpros.com webpros.com
webprosbot x09Mozilla
x22Mozilla
zgrab zgrab

20
dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,10 +1,10 @@
anonymizeme.pro affiliate-fr.com
bestempresas.es apidigidocketnet-a.akamaihd.net
fireads.men ccjp.eu
musicstock.me gamebackyard.com
orion-v.com gateway.zscalertwo.net
seinterface.com gratuitbaise.com
stroydetali.ru lostfilm-online.ru
tizanidineandgabapentin.blogspot.com prod2016.com
trafficstars.com remontbiz.ru
vtc.pw stuff-about-money.com

934
dev-tools/test_units/random-referrers-for-test.list
View file

@ -1,500 +1,500 @@
1qingdao.com 2drittel.de
2345.com 4senses.co
2kata.ru 4webmasters.org
3weekdiet.com 51.la
7minuteworkout.com 5kstore.com
a.frcls.fr 999webdesign.com
aa9bd78f328a6a41279d0fad0a88df1901.com aa9d046aab36af4ff182f097f840430d51.com
adbetclickin.pink acmebtn.ml
adman.gr acunetix-referrer.com
admanaerofoil.com adf.ly
adsloads.com adserver-e7.com
adultwebhosting.info adventureparkcostarica.com
advancedmassagebysara.com agecheckadult.com
affiliate-fr.com akama.com
ahmedabadwebs.com akusherok.ru
aihelen.net allfinweb.com
airlandsea.info amazon-seo-service.com
airmaxshoes-2016.com amung.us
aksonural.ru amyfoxfitness.com
akvamaster.dp.ua animebox.com.ua
alif-ba-ta.com anonymizeme.pro
alkoravto.ru apishops.ru
allornamenti.com appearance-cool.com
alta-realestate.com aproposde.com
am-se.com arcteryxsale.online
amoi.tn
ampower.me
anal-acrobats.com
analytics-ads.xyz
android-style.com
apidiamondatanet-a.akamaihd.net
apiwebwebgetcom-a.akamaihd.net
apparel-offer.com
appmsr.org
arcteryxstore.online arcteryxstore.online
artdeko.info ardimobileinfo.ml
artel116.ru arenanews.com.ua
asrv-a.akamoihd.net arendas.net
arraty.altervista.org
artdestshop.eu
asrvvv-a.akamaihd.net
atlasvkusov.ru
auspolice.com
autotrends.today
aviva-limoux.com
avtovolop.ru
ayerbo.xhost.ro
ayodhya.co ayodhya.co
b-styles.xyz azartmix.com
babosas.com azbaseimages.net
baixar-musicas-gratis.com balitouroffice.com
balois.worldbreak.com battleforupdating.site
banksrf.ru belinka.com.ua
bedcapdealers.com best-seo-solution.com
bestdraws.com besthoro.ru
bet-prognoz.com bestmarriages.com
bezcmexa.ru bestofpicture.com
bigames.online besttorrentknifta.weebly.com
biketank.ga bioca.org
bizlist.com.de biographiya.com
boanonihaca.tk bkgr.se
bookmaker-bet.com bloggers.nl
boole.onlinewebshop.net bobba.dzaba.com
bracketsmackdown.com bongacams.com
bonvillan.com
bradleylive.xyz
brainboostingsupplements.org brainboostingsupplements.org
braindaily.xyz brianhenry.co
budilneg.xyz bristolhotel.com.ua
buigas.00it.com buyantiviralwp.com
buy-forum.ru buyessayonline19.blogspot.ru
buy-meds24.com buynorxx.com
buytizanidineonlinenoprescription.blogspot.com
bwlx.prepedu.cn bwlx.prepedu.cn
californianews.cf cablecar.us
canoncdriverq3.pen.io candcstuccoandstone.com
carsdined.org carsdined.org
cartierjusteunclou.xyz
cartierlove.xyz
cartierring.xyz
casinorewards.info casinorewards.info
catalogs-parts.com cbb1smartlist12.click
ccjp.eu ccbill.com
cdnanalytics.xyz
cellfun.mobi cellfun.mobi
chatroulette.world cenokos.ru
cheapjerseysa.com centre-indigo.org.ua
check-host.net cerev.info
chelyabinsk.xrus.org chatroulette.life
children-learningreading.info cherubinimobili.it
chimiver.info cjs.com.ru
choosecuisine.com clickzzs.nl
ci.ua coaching-netz.info
cialisovercounteratwalmartusa.com cocyq.inwtrade.com
circlesl.com collegeessay19.blogspot.ru
citizenclsdriveri7.pen.io collegerentals.com
cleanersoft.com comissionka.net
clickhype.com commoncrawl.org
clicksor.net compliance-checker-7.info
cmd.kz compliance-donald.xyz
coderstate.com compliance-jess.top
cometorussia.net compliance-mary.top
compliance-ivan.top compliance-nelson.top
compliance-walter.top construmac.com.mx
cosmetic.donna7753191.ru cookingmeat.ru
countercrazy.com coolingoods.com
cracksplanet.com cosmeticswomens-womensports.rhcloud.com
crafthubs.com counterbot.com
crd.clan.su
creativehutindia.com creativehutindia.com
credit-respect.ru culmimg.pw
crynet.cc customboxes4less.com
cryptoswap.biz czeshop.info
cs-passion.pl daydream-studio.ru
cscwtalkto.site deda-moroza-zakaz.ru
customcatchcan.com depositfiles-porn.ga
customerpromos-a.akamaihd.net deutschehobbyhuren.net
dailyrank.net dieswaene.com
daptravel.com
davebestdeals.com
daymusam.com
dealwifi.com
decorazilla.com
dekoration.us
demian.kz
derevesendeco.com
deshevo-nedorogo.ru
dev.citetick.com
digest-project.ru digest-project.ru
digilander.libero.it discovertreasurenow.com
dildofotzen.net dmmspy.com
dinkolove.ya.ru dneprsvet.com.ua
disruptingdinnerparties.com doeco.ru
docstoc.com dokfilms.net
documentsite.net
dotmass.top dotmass.top
download-of-the-warez.blogspot.com downvids.net
drhomes.biz dscaas.website
druzhbany.ru dumuelave.xyz
dverimegapolis.ru dvervmoskvu.ru
e-avon.ru e-c.al
e-kwiaciarz.pl ekobata.ru
e2click.com elektrischezigarette1.onsugar.com
ecommerce-seo.org elektronischezi.livejournal.com
econom.co elexies.info
ed-shop01.ru elidelcream.weebly.com
ekn-art.se
elite-sex-finders.com
elvenmachine.com
emailaccountlogin.co
en.home-task.com
engines-usa.com engines-usa.com
euroskat.ru englishgamer.com
ezigarettekaufen.myblog.de enhand.se
ezigaretteshop2.vefblog.net escortplius.com
fakehandbags.xyz etm-consult.de
falcon-images.blogspot.com evening-dating-club.info
fatmaelgarny.com exonline.info
fertilitetsradgivningen.se falopicm.pw
fettefrauen.net familyphysician.ru
findanysex.com faptitans.com
florida-tourism.net fba-mexico.com
flowwwers.com fickkontaktehobbyhuren.com
forpackningsutveckling.se fickluder69.com
fortunejack.com filerockstar298.weebly.com
forum.tvmir.org filkhbr.com
foto-sisek.porngalleries.top filmbokep69.com
freenode.info filmi-v.online
froggytube.com financeloan.us
fugarif.ga find1friend.com
fun-mobi.pl findpics.pw
fun2cell.net findsexguide.com
finstroy.net
fishingwholesale.us
flipper.top
forex-procto.ru
forsex.info
forum.doctissimo.fr
freshmac.space
fsakhalin.ru
ftns.ru
fuckingawesome.com
funnypica.com funnypica.com
gael-s.ru fxtips.ru
gamedayhouse.com galleryawesome.com
gamewrath.com gazoblok.net.ua
gaytube.com gdebestkupit.ru
gcup.ru gearcraft.us
gemara.com gerhardhealer.com
get-your-social-buttons.info
getmyads24.com getmyads24.com
getoutofdebtfree.org girlspicsa.com
girlsatgames.ru gloverid.site
glogow.pl gojiberriess.apishops.ru
golaya.pw
gorabagrata.ru gorabagrata.ru
gosarhivrt.ru goroda-vsego-mira.ru
greendream.com.ua gq-catalog.gq
grizzlysgrill.com grtyi.com
gsasearchenginerankerdiscount.com gsbs.com.ua
gyffu.com gsou.cf
gymi.name guardlink.org
hahashka.ru guigyverpo.cf
hazardky.net gwebtools.com.br
hebr.myddns-flir.com handicapbathtubarea.com
happy.new.yeartwit.com
hauleddes.com
hdapp1008-a.akamaihd.net
hdseriale.pl
healbio.ru
healing-dysplasia.ru
healmytrauma.info
heartofbeijing.blogspot.com
helpmymacfaster.trade helpmymacfaster.trade
hidemyass.com hermesbirkinhandbagoutlets.com
highstairs-a.akamaihd.net hetmanship.xyz
high-speed1.net
hit-kino.ru
hitcpm.com hitcpm.com
hmmm.cz hollywoodactress.info
hoholikik.club homemade.gq
hopeonthestreet.co.uk hop.clickbank.net
hostcritique.com hosting-tracker.com
hotcore.info hoterika.com
hotdl.in hotkeys.com
housemilan.ru hustoon.over-blog.com
hoverboard360.es ilbe.club
hulfingtonpost.com ilmen.net
hunthillfarmtrust.org iloveitaly.ru
ideibiznesa2015.ru imagefinder.site
iframe-toloka.com img.wallpaperstock.net
ihc2015.info imgarit.pw
iklysha.ml imitex-plus.ru
impisr.ru informatiecentro.be
importchinacoach-teach.com
inboxdollars.com
infobabki.ru
infogame.name
infospot.pt
insurple.com insurple.com
interferencer.ru itrevolution.cf
invest-pamm.ru izismile.com
iplusbit.blogspot.co.za javcoast.com
iqbazar.ru jbl-charge.info
iqs.biz.ua jerseywholesalebizchina.com
irkutsk.zrus.org jerseywholesaleelitestore.com
ispac.org jimmychoosale.online
istizanidineacontrolledsubstance.blogspot.com jolic2.com
izhevsk.xrus.org justbcause.com
johannesburgsingles.co.za karaganda.xkaz.org
jose.mulinohouse.co keki.info
justprofit.xyz ketrzyn.pl
kalb.ru keywordspay.com
kchaxton.com koleso24.com.ua
kerei.ru kollekcioner.ru
keywordhouse.com konteiner24.com
kicknights.gq koral.se
kinofak.net kosynka-games.ru
kiwe-analytics.com kreativperlen.ch
knigonosha.net kupit-adenu.ru
knowyournextmove.com kursy-ege.ru
konpax.com kvartira-sutochno.com
kostenlos-sexvideos.com labelwater.se
kumuk.info labvis.host.sk
la-fa.ru lancheck.net
lacasamorett.com lapitec.eu
landscaping.center
layola.biz.tc
letmacworkfaster.site letmacworkfaster.site
levaquin750mg.blogspot.com letto.by
life-instyle.com lion.playtap.us
lifebyleese.com liupis.com
lincolntheatre.com longadventure.com
lmrauction.com longlifelomilomi.info
locustdesign.co
lolnex.us
lostfilm-online.ru
lottospring.com
louboutinbooties.xyz louboutinbooties.xyz
lsex.xyz louboutinreplica.pw
luchshie-topcasino.ru lovi-moment.com.ua
luckyshop.net.ua lowephotos.info
lunamedia.co m5home.ru
macfix.life mac-shield.com
macrotek.ru magiadinamibia.blogspot.com
manve.info malwareremovalcenter.com
marketingvici.com mansparskats.com
marketland.ml marwer.info
maslenka.kz
massage-info.nl
matpre.top
mediawhirl.net mediawhirl.net
medkritika.ru megakino.net
meds-online24.com mensandals.xyz
megavolt.net.ua mhi-systems.ru
michaelkorsoutlet.store micasainvest.com
mikrobiologies.ru minegam.com
mini-modus.ru mini.7zap.com
minyetki.ru mirzonru.net
mirmedinfo.ru modnie-futbolki.net
mitrasound.ru
mnogabukaff.net
mnogolok.info
mobplayer.ru
moesonce.com moesonce.com
moneyteam24.com mojpreskumanie.com
monthlywinners.com monarhs.info
montredemarque.nl
moscow-clining.ru
mosdverka.ru
mototsikl.org mototsikl.org
mrinsidesales.com mowser.com
mturkcontent.com mp3ritm.top
muizre.ru mrbitsandbytes.com
muschisexbilder.com mrcsa.com.au
musezone.su musicktab.com
musicdaddy.net musicstock.me
musicvidz.ru musirc.com
mydownlodablefiles.com myghillie.info
myftpupload.com myonigroup.com
myindospace.com mypets.by
myliveblog.ru napalm51.nut.cc
myonlinepayday.co naphukete.ru
neodownload.webcam narkologiya-orel.ru
nero-us.com naturalpharm.com.ua
netanalytics.xyz naturalshair.site
newhairstylesformen2014.com neks.info
nextlnk12.com nhl17coinsforps3.gratisblog.biz
nikhilbahl.com nikitsyringedrivelg.pen.io
niki-mlt.ru nisuturnetdgu.tk
njpalletremoval.com nrv.co.za
no-fuel.org ntic.fr
northfacestore.online nullrefer.com
noscrapleftbehind.co obiavo.kz
nottyu.xyz okuos.com
novodigs.com online-hit.info
nowtorrents.com onlinemegax.com
npoet.ru onlinetvseries.me
o-o-8-o-o.ru optitrade24.com
onclkads.com oren-cats.ru
one.net.in ossmalta.com
openmediasoft.com palocco.it
oranga.host.sk partner-host.men
orhonit.com pertlocogasilk.tk
osb.se11.ru petrovka-online.com
pandroid.co pic2fly.com
partybunny.ru pinapchik.com
pateaswing.com pinkduck.ga
pdn-4.com pinstake.com
peugeot-club.org piuminiita.com
pewit.pw pix24x7.com
phormchina.com pizdopletka.club
phuketscreen.com pk-services.ru
pio.polytopesexempt.com pl.id-forex.com
pivka.xyz pliks.pl
pixelrz.com plohaya-kreditnaya-istoriya.ru
pk-pomosch.ru polytopesexempt.com
pochemychka.net popads.net
pochtovyi-index.ru popup-fdm.xyz
pons-presse.com pornhub-forum.ga
pornoblood.com porno-chaman.info
pornogad.com porno-play.net
pornohub.me pornosok.ru
pornoreino.com pornotubs.com
pornotubexxx.name porto.abuilder.net
pornoxxx.com.mx pozvonim.com
predominant-invent.tk princeadvantagesales.com
prize44.com priora-2.com
professionalwritingservices15.blogspot.ru prizefestival.mobi
profit-opportunity.com pro-tec.kz
profitfx.online profitkode.com
promoheads.com promover.org
pronekut.com proxyelite.biz
pronorm.fr pst2017.onlinewebshop.net
prosperent.com pts163.ru
pukaporn.com pulse33.ru
qitt.ru purplesphere.in
quangcaons.com pussysaga.com
quit-smoking.ga pussyspace.net
raavidesigns.com qor360.com
radiodigital.co quickloanbank.com
rangoman.date raisedseo.com
redirect.trafficreceiver.club rankexperience.com
registratciya-v-moskve.ru ranksays.com
remontbiz.ru rapevideosmovies.com
reversing.cc rarbg.to
ruspdd.com razamicroelectronics.com
ryetaw.com razleton.com
saddiechoua.com rcpmda.ikan1080.xyz
sashagreyblog.ga remmling.de
scanmyphones.com remote-dba.de
rentalcarnavi.info
rimedia.org
ritlweb.com
rniaeba.ga
rocketchange.ru
rp9.ru
rusprostitute.com
rutor.group
sale-japan.com
saneitconsulting.com
sanjosestartups.com
savingsslider-a.akamaihd.net
sayyoethe.blogspot.co.za
scanner-don.top
scanner-elena.top
scanner-jane.top scanner-jane.top
scanner-josh.top school-diplomat.ru
scanner-julianna.top scripted.com
schlampen-treffen.com sdi-pme.com
scmor.ilxc.cc searchencrypt.com
scrapinghub.com security60-e.com
searchaddis.com seinterface.com
searchlock.com seksvideoonlain.com
seocdvig.ru sel-hoz.com
sex-znakomstva.online seoanalyses.com
sexvideo-sex.com seoholding.com
sexvporno.ru seoprofiler.com
share-buttons-for-free.com serialsway.ucoz.ru
sharebutton.org servingnotice.com
sexreliz.com
sexsearch.com
sharpchallenge.com
shemalegalls.blogporn.in
shiksabd.com
shitting.pro
shivafurnishings.com
shop-electron.ru
shopvilleroyboch.com.ua shopvilleroyboch.com.ua
sibdevice.ru shtaketniki.kz
signal03.ru signoredom.com
site-analyzer.com silvermature.net
site-auditor.online simple-image.com.ua
sitopreferito.it simplesite.com
slowmac.tech sinel.info
sms2x2.ru site-speed-checker.site
snowplanes.com siteaero.com
soblaznu.net smartadserver.com
soda.media snjatie-geroinovoy-lomki.ru
snow.nvr163.com
social-buttons.xyz
social-fun.ru
sofit-dmd.ru
soft-program.com soft-program.com
softlinesolutions.me softomix.com
softxaker.ru solitaire-game.ru
solicita.info sophang8.com
sotechco.co spacash.com
space4update.pw spb.ru
spasswelt.xyz spymac.net
spywarespy.com sr-rekneskap.no
srecorder.com stefanbakosab.se
ssconstruction.co
sstroy44.ru
stair.registrydomainservices.com
statoutlook.info
staynplay.net
sterva.cc
stopnarco.ru
stretchingabuckblog.com
strongholdsb.ru
suggest-keywords.com
superfish.com superfish.com
superinterstitial.com superiends.org
svarbit.com superoboi.com.ua
svetlotorg.ru support.nopeas.sk
szucs.ru svarkagid.com
taboola.com swinginwithme.ru
tacbelarus.ru symbaloo.com
tattomedia.com symphonyintegratedhealthcare.com
teletype.in takeflyte.com
terraclicks.com taqywu51.soup.io
the-usa-games.blogspot.com tastyfoodideas.com
theguardlan.com tattooha.com
tedxrj.com
teenforporn.com
teslathemes.com
thedownloadfreeonlinegames.blogspot.com
thefarmergame.com
thegolfclub.info
thelottosecrets.com thelottosecrets.com
themeforest.net thfox.com
thexart.club thiegs.reco.ws
thruport.com
ticketsys.inetwd.com
timeallnews.ru
timer4web.com timer4web.com
titan-cloud.life
tizanidinedosageforsleep.blogspot.com
tizanidinehcl2mgsideeffects.blogspot.com tizanidinehcl2mgsideeffects.blogspot.com
tizanidinehcl4mgisitanarcotic.blogspot.com top10registrycleaners.com
tizanidinehclsideeffects.blogspot.com torrentred.games
tizanidineorflexeril.blogspot.com torture.ml
tizanidineshowupondrugtest.blogspot.com tovaroboom.vast.ru
tizanidinevssoma.blogspot.com tracxn.com
tizanidinezanaflex.blogspot.com tradedeals.biz
tnaionline.org traderzplanet.in
topbestgames.com tradgardspartner.se
tracksurf.daooda.com traffictrade.life
trafaret74.ru triumf-realty.ru
trafficfactory.biz truck-land.life
trafficmania.com turist-strani.ru
trafficmonetize.org turn-up-life.life
trafficstars.com uggbootsoutletsale.us
trainoffend.ml ukrup.com
trion.od.ua umg-stroy.ru
trucri.me unacittaconte.org
trumpetedextremes.com undergroundcityphoto.com
trustedhealthtips.com unitexindia.com
tubeoffline.com urengoy.pro
tvorozhnaja-zapekanka-recept.ru url-extractor.xyz
unpredictable.ga us-america.ru
uptime-as.net uytmaster.ru
ural-buldozer.ru v24s.net
usacasino.com veles.shop
v-doc.co veopornogratis.xxx
v720hd.ru videtubs.pl
valsalud.com vielporno.net
vapomnoncri.tk vip-file.com
varbergsvind.se vip.51.la
vbabule.net viteonlusarezzo.it
vekzdorov.ru vksex.ru
viandpet.com voloslove.ru
videochaty.ru voluumtracker1.com
vilingstore.net vvpg.ru
vitalads.net wallpapersist.com
vitanail.ru wardreapptokone.tk
viveresaniesnelli.it wechatdownload10.com
vizag.kharkov.ua wejdz-tu.pl
vladimir.xrus.org whereverdesperate.gq
vladimir.zrus.org wikes.20fr.com
vodaodessa.com worldinternetauthority.com
volgograd.xrus.org worldis.me
voltrknc1.com wwwadultcheck.com
vremya.eu xboxster.ru
vtcdns.com xgftnlrt.bloger.index.hr
vzube.com xn--80aaafbn2bc2ahdfrfkln6l.xn--p1ai
watch-movies.ru xn--80atua3d.xn--p1ai
watchmyfb.pl xn--b1adccaf1bzj.xn--p1ai
wdrake.com xn--l1aengat.xn--p1ai
webenlace.com.ar xxxdatinglocal.us
weboptimizes.com xxxhdvideo.site
welck.octopis.com xxxsiterips.xyz
weprik.ru y8games-free.com
wildcattube.com
wildnatureimages.com
wingsofrefuge.net
wnhjavlhezp.gq
wordpress-crew.net
wpsecurity.website
xep.info
xmnb.net
xn--24-glceagatoq7c2a6ioc.xn--p1ai
xn--80aeahghtf8ac5i.xn--p1ai
xn--e1afanlbnfckd7c3d.xn--p1ai
xtubeporno.net
yaderenergy.ru
yadro.ru
yellowfootprints.com yellowfootprints.com
yellowstonevisitortours.com youjizz.vc
youporn-ru.com
yourtemplatefinder.com
youtube-downloader.savetubevideo.com
yuarra.pluto.ro
yugk.net
zastenchivosti.net
zatjmuzu.info zatjmuzu.info
zebradudka.com zebradudka.com
zeg-distribution.com znaturaloriginal.com
zeleznobeton.ru zone-kev717.info
zigarettenonlinekaufen.tumblr.com zoogdinsney.com
zigarettenonlinekaufen2.drupalgardens.com zophim.me
zionstar.net zrus.org
zmoda.hostreo.com
zog.link
zoominfo.com
zscalertwo.net
zvooq.eu zvooq.eu
zvuker.net

488
dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
007angels.com 00webcams.com
1001desert.com 1008.su
273-fz.ru 1qingdao.com
5tey7463.weebly.com 24x7-server-support.site
7zap.com 2nt.ru
abbanreddy.com 5000-cotydzien.com
activecampaign.dreamhosters.com 8gold.com
adf.ly addbin.men
adimmix.com admatic.com.tr
adinterax.com agysya.ru
affordablewebsitesandmobileapps.com aibolita.com
afftrack001.com aihelen.net
alf-img.com alloysteel.ru
amoi.tn amazingninja.com
anaseracresar.tk animalia-life.club
apelsinnik.website anonymous-redirect.com
arabseyes.com apartmentratings.com
arenda-avtoprokat-krasnodar.ru apimountainbikei-a.akamaihd.net
arewater.com apioutoboxnet-a.akamaihd.net
astrochicks.com
auspolice.com auspolice.com
autobudpostach.club bala.getenjoyment.net
autonew.biz beclean-nn.ru
babieca.com belinka.com.ua
babyguns.ru bestfortraders.com
backuperwebcam.weebly.com bestofpicture.com
bannerads.de bestprofits-there.com
best-mam.ru bestwebsiteawards.com
bestssaker.com bet-prognoz.com
bget.ru biocn.dx.am
bimatoprost-careprost.com.ua blockworld.ru
bioca.org bookmark4you.com.biz
bizcheapjerseyswholesalechina.com bradleylive.xyz
bizlist.com.de buybest1.biz
blagovest-med.ru buyessayonline19.blogspot.ru
blogrankers.com buzz-porno.info
boeuklad.com canyougethighofftizanidine.blogspot.com
bracketsmackdown.com carapics.com
brendbutik.ru cartierreplica.top
briomotor.co cartierwatch.xyz
brk-rti.ru
bugof.gq
bugshoot.cn
bupropion-sr-150-mg.us
businesn.men
callejondelpozo.es
candiceloves.us
caveavins.fr caveavins.fr
cementaresearch.se ccjp.eu
cercacamion.it cdncash.com
cheapmoncler.win cheatcode-lita12.rhcloud.com
choosecuisine.com checkpagerank.net
ci.ua clksite.com
citetick.com companies-catalog.com
clickaider.com compliance-checker-7.info
collegerentals.com compliance-george.top
cornerstone-countertops.com coprofam.org
crackzplanet.com cosmetic.donna7753191.ru
creativehutindia.com cxpromote.com
creditwell.ru darkbooks.org
datsun-do.com day-news.info
davebestdeals.com ddospanel.com
dcj-nn.ru deluxedumps.com
den-noch24.ru derevesendeco.com
dengi-pod-zalog-nedvizhimosti.ru dinosaurus.site
detskie-zabavi.ru dnsrsearch.com
dosugrostov.site dojki365.online
ednorxmed.com domainsatcost.com
elektrischezigaretten2.beeplog.com domainsphoto.com
elektrozigarette1.dreamwidth.org doyouknowtheword-flummox.ml
elektrozigaretten1.tumblr.com drillsaw.ru
embedle.com dtnlyss.com
en.home-task.com dvrlists.com
eropho.com educatemetv.com
etm-consult.de egovaleo.it
europages.com.ru elektronischezigarette2.mex.tl
euroskat.ru elektrozigarette2.wordpressy.pl
ezigarettenshop1.yolasite.com eropornosex.ru
farm26.ru ertelecom.ru
etenininrade.ga
exci.ru
expdom.com
factorynightclub.com
falopicm.pw
favorcosmetics.com
fba-mexico.com
feeriaclub.ru feeriaclub.ru
fefo.gdn felizporno.com
fickkontakte.org fenoyl.batcave.net
filmbokep69.com ferieboligkbh.dk
financehint.eu ferrotodo.com
foto-telok.net financeloan.us
freetangodownload.com freeuploader.ml
freevpn.space frequiry.com
freshmac.space fuckingawesome.com
ftns.ru funcrushgames.com
fun-mobi.pl
gamezblox.com gamezblox.com
gidonline.one gheus.altervista.org
glicol.kz globalscam.ga
greenshop.su glogow.pl
hentaiheroes.com goatse.ru
hermesbracelets.xyz gogps.me
high-speed1.net gotcher.us
house.sieraddns.com gpirate.com
hubbble.com greatzip.com
huhn.altervista.org grizzlysgrill.com
hulfingtonpost.com guitar-master.org
i4track.net habermetre.com
igithab.com hanwei.us
iklysha.ml hitmuzik.ru
ikritikimou.gr hollywoodactress.info
imagerydatabase.com hornymatches.com
imediadesk.com howtoclean.club
ilmen.net
images-graphics-pics.com
importchinacoach-teach.com
incep.imagine-ex.co
incest-ru.com
indiasourcemart.in indiasourcemart.in
infokonkurs.ru infobabki.ru
insta-add.pro infobanks.ru
intermesh.net interfucks.net
iqoption.pro iopeninghours.co.uk
javxxx18.com javlibrary.cc
keywordblocks.com jillepille.com
ku6.com jjbabskoe.ru
labplus.ru jongose.ninja
law-check-two.xyz jyvopys.com
letmacworkfaster.world kamorel.com
libertybilisim.com karaganda.xkaz.org
lifehacĸer.com kidd.reunionwatch.com
liveinternet.ro kiwi237au.tk
lonely-mature.com kliksaya.com
lonfon.xyz klumba55.ru
lost-alpha.ru kongoultry.net
lrac.info kosynka-games.ru
luckyshop.net.ua krasnodar-avtolombards.ru
mebelcomplekt.ru laptop-4-less.com
mil-stak.com lenvred.org
mir-limuzinov.ru life.biz.ua
missis.top magazintiande.ru
moinozhki.com mantramusic.ru
mostantikor.ru mexicotravelnet.com
myblogregistercm.tk monclerboots.xyz
myftpupload.com morf.snn.gr
na-telefon.biz moviemail-online.co.uk
nakrutka.cc moyakuhnia.ru
naughtyconnect.com musicdaddy.net
navalwiki.info mystats.xyz
negociosdasha.com naturalbreakthroughsresearch.com
nhl17coinsforps3.gratisblog.biz newstraveller.ru
nisuturnetdgu.tk nfljerseys.online
noumeda.com nicefloor.co.uk
old-rock.com nikitabuch.com
oneclickfiles.com nrv.co.za
oneshotdate.com obiavo.uz
open-odyssey.org offer.camp
osnova3.ru offers.bycontext.com
paintingplanet.ru ons-add.men
park.above.com ooo-gotovie.ru
pb-dv.ru ozoz.it
pics-group.com patol01.pw
pictures-and-images.com paydayonlinecom.com
plastjulgranar.se pcboa.se
pochemychka.net perm-profnastil.ru
pomoc-drogowa.cba.pl pewit.pw
pornoblood.com phpdevops.com
pornophoto.xyz pic-re.blogspot.com
pornoted.com picsfair.com
pornotubs.com picturesfrom.com
pozvonim.com pistonclasico.com
privatefx-in.ru pkr1hand.com
privhosting.com play.leadzupc.com
prokotov.com pochtovyi-index.ru
pssucai.info pod-muzyku.club
rabot.host.sk polcin.de
rfd-split.hr porno-video-chati.ru
risparmiocasa.bz.it pornodojd.ru
rusprostitute.com pornosee.info
sammlungfotos.online printdirectforless.com
scanner-irvin.top profitwithalex.info
seo-2-0.com promalp-universal.ru
seogadget.ru prostitutki-astana.org
seokicks.de pussysaga.com
serptehnika.ru qld10000.net
service-core.ru qwertty.net
sex-dating.co redhotfreebies.co.uk
sexsaoy.com rn-to-bsn.com
shlyahten.ru roznica.com.ua
shmetall.com.ua russia-tao.ru
shoppytoolmac.com sa-live.com
si-unique.com sajatvelemeny.com
sigmund-freud.co.uk sbt-aqua.ru
sirpornogratis.xxx searchimage.co
slowmac.tech searchimpression.com
slujbauborki.ru seinterface.com
smichovbike.cz seoexperimenty.ru
smstraf.ru sexsearch.com
snowplanes.com share-buttons-for-free.com
snworks.com shitmovs.com
softonicads.com sildenafil-tadalafil.info
spravka130.ru simpoed.ufop.br
spywareit.com slim.sellany.ru
spywarenuker.com snip.to
stairliftstrue.com snow.nvr163.com
staticfs.host social-s-ggg.xyz
steamoff.net social-s-hhh.xyz
stop-gepatit.te.ua social-vestnik.ru
stretchmate.net sogimlecal.tk
support.nopeas.sk spb.ru
surcentro.com spy-sts.com
t-bygg.com staff.prairiesouth.ca
taranerymagesswa.blogspot.com streetfire.net
teenbbw.yopoint.in sudexpert66.ru
telefonsexi.com sukirgenk.dvrlists.com
tfxiq.com tattoo33.ru
thaismartloan.com telefonsexsofort.tk
thin.me.pn terraclicks.com
tizanidinehcl2mg.blogspot.com teslathemes.com
tizanidinehcl4mgtabinfo.blogspot.com tiandeural.ru
tizanidinesleepaid.blogspot.com tivolibasket.it
tomatis.gospartner.com tizanidine6mg.blogspot.com
torrents-tracker.com tizanidinedosage.blogspot.com
tracklead.net tizanidinehclsideeffects.blogspot.com
trackmedia101.com tizanidinehydrochloride4mgstreetvalue.blogspot.com
try-rx.com torrent-to-magnet.com
tt-ipd.info toursmaps.com
tuckermktg.com tovaroboom.vast.ru
tula.mdverey.ru trafficshaper.com
uggbootsoutletsale.us turn-up-life.life
url-extractor.xyz tvorozhnaja-zapekanka-recept.ru
utrolive.ru ublaze.ru
vabasa.inwtrade.com unacittaconte.org
vesnatehno.com unece.org
via-energy-order.com unisexjewelry.org
visa-china.ru vancleefreplica.pw
vkonche.com vesnatehno.ru
voluumtracker1.com viagraneggrx.com
warning.or.kr videochat.life
watchdogs-2.ru vipsiterip.org
web-analytics.date vkontaktemusic.ru
weburlopener.com vtc.pw
weburok.com vzube.com
wetgames.ru wardreapptokone.tk
whatsupinfoley.com watchinf.com
whengirlsgowild.com watchmygf.net
womensplay.net wingsoffury2.com
worldlovers.ru wovis.site
wormix-cheats.ru www.wohnkabinen-shop.de
www.kabbalah-red-bracelets.com xgftnlrt.bloger.index.hr
xlolitka.com xn----itbkqkfiq.xn--p1ai
xn----7sbaphztdjeboffeiof6c.xn--p1ai
xn----8sblgmbj1a1bk8l.xn----161-4vemb6cjl7anbaea3afninj.xn--p1ai
xsfetish.org xsfetish.org
xsion.net xxxkaz.org
xwatt.ru youbloodyripper.com
youporn-ru.com zaim-pod-zalog-krasnodar.ru
youraticles.pl zocaparj.kz
yourserverisdown.com zoodrawings.com
zootoplist.com
zscaler.net
zvezdagedon.ru

2
dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
10.1.1.9 0; 10.1.0.73 0;
127.0.0.1 0; 127.0.0.1 0;

4
dev-tools/tmp/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:30 UTC 2022 ### Updated: Wed Jan 5 06:57:52 UTC 2022
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### Version Information ## ### Version Information ##

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information # ### Version Information #
################################################### ###################################################
### Version: V4.2022.01.2830 ### Version: V4.2022.01.2831
### Updated: Tue Jan 4 22:01:30 UTC 2022 ### Updated: Wed Jan 5 06:57:52 UTC 2022
### Bad Bot Count: 631 ### Bad Bot Count: 631
################################################### ###################################################
### Version Information ## ### Version Information ##