vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-02 18:50:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2019.06.1645 [ci skip]

Browse source
This commit is contained in:
Travis 2019-06-27 13:28:17 +02:00
parent b3601d427e
commit 8ff3b4c1fa
34 changed files with 2737 additions and 2377 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.dev-tools/_conf_files_test1/bots.d/whitelist-ips.conf
View file

@ -33,4 +33,4 @@
# 111.111.111.111 0;
35.193.7.13 0;
35.202.145.110 0;

4
.dev-tools/_conf_files_test1/conf.d/globalblacklist.conf
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1644
### Updated: Thu Jun 27 13:12:21 SAST 2019
### Version: V4.2019.06.1645
### Updated: Thu Jun 27 13:24:55 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

2
.dev-tools/_conf_files_testing_changes/bots.d/whitelist-ips.conf
View file

@ -33,4 +33,4 @@
# 111.111.111.111 0;
35.193.7.13 0;
35.202.145.110 0;

4
.dev-tools/_conf_files_testing_changes/conf.d/globalblacklist.conf
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V3.2019.06.1644
### Updated: Thu Jun 27 13:12:23 SAST 2019
### Version: V3.2019.06.1645
### Updated: Thu Jun 27 13:24:56 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

2
.dev-tools/_conf_files_testing_changes_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.85.2 1;
104.154.120.187 1;
127.0.0.1 1;

2
.dev-tools/_conf_files_testing_changes_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,4 +1,4 @@
35.192.85.2 0;
104.154.120.187 0;
127.0.0.1 1;
127.0.0.1 1;
127.0.0.1 1;

643
.dev-tools/_conf_files_testing_changes_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -1,16 +1,21 @@
### **********************************
### THE ULTIMATE NGINX BAD BOT BLOCKER
### **********************************
### -----------------------------------------------------------
### THE NGINX ULTIMATE BAD BOT, BAD IP AND BAD REFERRER BLOCKER
### -----------------------------------------------------------
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1643
### Updated: Thu Jun 27 12:59:42 SAST 2019
### Version: V3.2019.06.1645
### Updated: Thu Jun 27 13:24:56 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################
### VERSION INFORMATION ##
### --------------------------------------------
### HELP SUPPORT THIS PROJECT - Send Me a Coffee
### https://ko-fi.com/mitchellkrog
### --------------------------------------------
##############################################################################
# _ __ _ #
# / |/ /__ _(_)__ __ __ #
@ -28,12 +33,16 @@
### and also has rate limiting functionality for bad bots who you only want to rate limit
### and not actually block out entirely. It is very powerful and also very flexible.
### --------------------------------------------------------------------------
### Created By: https://github.com/mitchellkrogza/
### Repo Url: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
### Copyright Mitchell Krog - <mitchellkrog@gmail.com>
### Contributors: Stuart Cardall - https://github.com/itoffshore
### --------------------------------------------------------------------------
### --------------------------------------------------------------------------
### Tested on: nginx/1.10.3 up to latest Mainstream Version (Ubuntu 16.04)
### --------------------------------------------------------------------------
### This list was developed and is in use on a live Nginx server running some very busy web sites.
### It was built from the ground up using real data from daily logs and is updated almost daily.
@ -41,9 +50,9 @@
### spam referrers, rogue IP address, scanners, scrapers and domain hijacking sites are extensively checked
### before they are added. It is monitored extensively for any false positives.
### *********
### ---------
### Features:
### *********
### ---------
### Clear formatting for Ease of Maintenance.
### Alphabetically ordered lists for Ease of Maintenance.
### Extensive Commenting for Ease of Reference.
@ -54,40 +63,38 @@
### IP range blocking / whitelisting.
### Rate Limiting Functions.
### ************
### ------------
### INSTALLATION
### ************
### ------------
### PLEASE use the install, setup and update scripts provided for you to ease your installation.
### This Auto Installation procedure is documented in the README.md and AUTO-CONFIGURATION.md files.
### Installation, Setup and Update Scripts Contributed by Stuart Cardall - https://github.com/itoffshore
### There are also manual configuration instructions provided for those not wishing to do an auto install.
### ***********************************************
### -----------------------------------------------
### !!!!! PLEASE READ INLINE NOTES ON TESTING !!!!!
### ***********************************************
### -----------------------------------------------
### Note that:
### SETTINGS:
### ---------------------------------------------
### 0 = allowed - no limits
### 1 = allowed or rate limited less restrictive
### 2 = rate limited more
### 3 = block completely
### ---------------------------------------------
### ************************************************************
### ------------------------------------------------------------
### CONTRIBUTING / PULL REQUESTS / ADDING YOUR OWN BAD REFERRERS
### ************************************************************
### ------------------------------------------------------------
### For contributing, corrections or adding bots or referrers to this repo,
### Send a Pull Request (PR) on any of the .list files in the _generator_lists folder
### All Pull Requests will be checked for accuracy before being merged.
# *********************************
# FIRST BLOCK BY USER-AGENT STRINGS
# *********************************
# ***********************
# -----------------------
# !!!!! PLEASE TEST !!!!!
# ***********************
# -----------------------
# ALWAYS test any User-Agent Strings you add here to make sure you have it right
# Use a Chrome Extension called "User-Agent Switcher for Chrome" where you can create your
@ -98,36 +105,46 @@
# curl -I http://www.yourdomain.com -A "GoogleBot" ---- GIVES YOU: HTTP/1.1 200 OK (Meaning web page was served to Client)
# curl -I http://www.yourdomain.com -A "80legs" ---- GIVES YOU: curl: (52) Empty reply from server (Meaning Nginx gave a 444 Dropped Connection)
# In this section we allow/disallow specific User Agents / Bots.
# *********************************************************************
# Now we map all good and bad user agents to a variable called $bad_bot
# *********************************************************************
# =======================
# START BLOCKER FUNCTIONS
# =======================
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# DO NOT EDIT ANYTHING BELOW THIS LINE !!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# =============================
# BEGIN SECTION 1 - USER-AGENTS
# =============================
# ALLOW / BLOCK User Agents / Bots
# -------------------------------------------------------------------
# Map all GOOD and BAD UA (User Agents) to a variable called $bad_bot
# -------------------------------------------------------------------
map $http_user_agent $bad_bot {
default 0;
# ***********************************************
# Include your Own Custom List of Bad User Agents
# ***********************************************
# Use the include file below to further customize your own list of additional user-agents you wish to permanently block
# This include file allows whitelisting and blacklisting of anything specified below it, meaning you can over-ride anything in this blocker to your liking.
# ***********************************************************************************
# -----------------------------------------------------------------------------------
# START CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***********************************************************************************
# -----------------------------------------------------------------------------------
# Include your Own Custom List of Bad User Agents
# Use the include file below to further customize your own list of additional user-agents you wish to permanently block
# This include file allows whitelisting and blacklisting of anything specified below it.
# This include file alows you to over-ride any Bad / Good UA (Bot) declared in this blocker to your liking.
include /etc/nginx/bots.d/blacklist-user-agents.conf;
# *********************************************************************************
# ---------------------------------------------------------------------------------
# END CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *********************************************************************************
# ---------------------------------------------------------------------------------
# *********************************************
# Bad User-Agent Strings That We Block Outright
# *********************************************
# --------------------------------------------------
# BAD UA (User-Agent) Strings That We Block Outright
# --------------------------------------------------
# START BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)360Spider(?:\b|)" 3;
@ -688,9 +705,16 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ZyBorg(?:\b|)" 3;
# END BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# ***********************************************
# Allow Good User-Agent Strings We Know and Trust
# ***********************************************
# --------------------------------------------
# GOOD UA User-Agent Strings We Know and Trust
# --------------------------------------------
# -----------------------------------------------------------------------
# You can over-ride these in /etc/nginx/bots.d/blacklist-user-agents.conf
# by adding the same UA line there and chaning its value of 1
# If you think GoogleBot is bad you would simply add them to
# blacklist-user-agents.conf with a value of 1
# -----------------------------------------------------------------------
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)adidxbot(?:\b|)" 0;
@ -727,12 +751,13 @@ map $http_user_agent $bad_bot {
"~*(?:\b)yahoo(?:\b|)" 0;
# END GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# ***************************************************
# User-Agent Strings Allowed Through but Rate Limited
# ***************************************************
# --------------------------------------------------------
# GOOD UA User-Agent Rate Limiting 1 - Disabled by Default
# --------------------------------------------------------
# Some people block libwww-perl, it used widely in many valid (non rogue) agents
# I allow libwww-perl as I use it for monitoring systems with Munin but it is rate limited
# TO ACTIVATE THIS RATE LIMITING Uncomment these two lines in blockbots.conf
#limit_conn bot1_connlimit 100;
#limit_req zone=bot1_reqlimitip burst=50;
# START ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)jetmon(?:\b|)" 1;
@ -743,9 +768,14 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wget/1.15(?:\b|)" 1;
# END ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# **************************************************************
# Rate Limited User-Agents who get a bit aggressive on bandwidth
# **************************************************************
# -------------------------------------------------------
# GOOD UA User-Agent Rate Limiting 2 - Enabled by Default
# -------------------------------------------------------
# -----------------------------------------------------------------------
# You can over-ride these in /etc/nginx/bots.d/blacklist-user-agents.conf
# by adding the same UA line there and chaning its value of 1
# -----------------------------------------------------------------------
# START LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)Alexa(?:\b|)" 2;
@ -768,163 +798,186 @@ map $http_user_agent $bad_bot {
# END LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
}
# *****************************************
# SECOND BLOCK BY REFERRER STRINGS AND URLS
# *****************************************
# ===========================
# END SECTION 1 - USER-AGENTS
# ===========================
# *****************
# =======================================
# BEGIN SECTION 2 - REFERRERS AND DOMAINS
# =======================================
# ----------------
# PLEASE TEST !!!!
# *****************
# ----------------
# ------------------------------------------------------------------------------------------------------------------------------
# ALWAYS test referrers that you add. This is done manually as follows
# ------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://anything.adcash.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://www.goodwebsite.com/not-adcash --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://www.betterwebsite.com/not/adcash --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://www.google.com --- GIVES YOU: full html output of the web page
# curl -I http://www.yourdomain.com -e http://www.microsoft.com --- GIVES YOU: full html output of the web page
# ------------------------------------------------------------------------------------------------------------------------------
# Because of case-insensitive matching any combination of capitilization in the names will all produce a positive hit
# make sure you always test thoroughly and monitor logs. This section below also does NOT check for a preceding www.
# and it also does not care if the referrer request was sent with http https or even ftp.
# ------------------------------------------------------------------------------------------------------------------------------
# ***********************************************************************
# Now we map all bad referrer words below to a variable called $bad_words
# ***********************************************************************
# ----------------------------------------------------------------
# Map all BAD referrer words below to a variable called $bad_words
# ----------------------------------------------------------------
# --------------------------------
# START Bad Referrer Word Scanning
# --------------------------------
map $http_referer $bad_words {
default 0;
# **************************
# Bad Referrer Word Scanning
# **************************
# -------------------------------------------------------------------------------------------
# These are Words and Terms often found tagged onto domains or within url query strings.
# Create and Customize Your Own Bad Referrer Words Here using the new Include File Method
# New Method Uses the include file below so that when pulling future updates your
# customized list of bad referrer words are automatically now included for you
# Read Comments inside bad-referrer-words.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# BE VERY CAREFUL using the bad-referrer-words.conf file - please read the comment and
# BE VERY CAREFUL using this bad-referrer-words.conf file - please read the comments and
# examples inside the include file for detailed explanations into how seriously this can
# affect your own site from serving assets or other innocent sites from accessing your site
# For safety sake the whitelist-domains.conf file is also loaded here before the bad-referrer-words.conf file.
# For safety sake the whitelist-domains.conf file is also loaded here before the
# bad-referrer-words.conf file is loaded.
# -------------------------------------------------------------------------------------------
# ************************************************************************
# ------------------------------------------------------------------------
# START WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-domains.conf;
# **********************************************************************
# ----------------------------------------------------------------------
# END WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **********************************************************************
# ----------------------------------------------------------------------
# ******************************************************************************
# ------------------------------------------------------------------------------
# START CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ******************************************************************************
# ------------------------------------------------------------------------------
include /etc/nginx/bots.d/bad-referrer-words.conf;
# ****************************************************************************
# ----------------------------------------------------------------------------
# END CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ****************************************************************************
# ----------------------------------------------------------------------------
}
# --------------------------------
# END Bad Referrer Word Scanning
# --------------------------------
# *************************
# Bad Referrer Domain Names
# *************************
# Now a list of bad referrer urls these domains or any combination of them ie .com .net
# will be blocked out. Doesn't matter if the protocol is http, https or even ftp
# ----------------------------------------
# START Good and Bad Referrer Domain Names
# ----------------------------------------
# -------------------------------------------------------------------------------------
# Good and Bad referrer urls Doesn't matter if the protocol is http, https or even ftp
# -------------------------------------------------------------------------------------
# ----------------------
# This section includes:
# **********************
# ----------------------
# --------------------------------------------------------------------------------
# Blocking of SEO company Semalt.com (now merged into this one section)
# MIRAI Botnet Domains Used for Mass Attacks
# Other known bad SEO companies and Ad Hijacking Sites
# Sites linked to malware, adware, clickjacking and ransomware
# Domain names and referrers used in referrer spam and seo hijacking
# Whitelisting of your own GOOD domains / referrers
# Whitelisting of any other GOOD domains / referrers you want explicitly NOT block
# --------------------------------------------------------------------------------
# *****************
# ----------------
# PLEASE TEST !!!!
# *****************
# ----------------
# ------------------------------------------------------------------------------------------------------------------------------------
# ALWAYS test referrers that you add. This is done manually as follows
# ------------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------------
# Because of case-insensitive matching any combination of capitilization will all produce a positive hit - make sure you always test.
# ------------------------------------------------------------------------------------------------------------------------------------
# For Example any of the following variations below of 8gold.com will be detected and blocked
# ------------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://NOT-8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://this.is.not8gOlD.net --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e ftp://8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e ftp://www.weare8gold.NET --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e https://subdomain.8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e https://NOT8GolD.org --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------------
# So if you see a bad referrer from wearegoogle.com and you want to block them just add
# them as "~*wearegoogle.com" don't ever go and do something like "~*google(-|.)" you will
# kill all your SEO in a week.
# ------------------------------------------------------------------------------------------------------------------------------------
# To add your own custom bad referrers use the custom include file
# /etc/nginx/bots.d/custom-bad-referrers.conf
# Or send a Pull Request to add it to the global blacklist for other users.
# In the bad referrers section I also include sites that hotlink images without permission.
# ------------------------------------------------------------------------------------------------------------------------------------
# ***********************************************************************
# Now we map all good & bad referrer urls to variable called #bad_referer
# ***********************************************************************
# --------------------------------------------------------------------
# Map all good & bad referrer DOMAINS to a variable called bad_referer
# --------------------------------------------------------------------
map $http_referer $bad_referer {
hostnames;
default 0;
# *************************************
# GOOD REFERRERS - Spared from Checking
# *************************************
# --------------------------------------------
# GOOD REFERRER DOMAINS - Spared from Checking
# --------------------------------------------
# ---------------------------------------------------------------------------------------
# Add all your own web site domain names and server names in this section
# WHITELIST Your Own Domain Names Here using the new Include File Method
# New Method Uses the include file below so that when pulling future updates your
# whitelisted domain names are automatically now included for you.
# Read Comments inside whitelist-domains.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# ---------------------------------------------------------------------------------------
# ************************************************************************
# ------------------------------------------------------------------------
# START WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-domains.conf;
# **********************************************************************
# ----------------------------------------------------------------------
# END WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **********************************************************************
# ----------------------------------------------------------------------
# ***********************************
# -----------------------------------
# CUSTOM BAD REFERRERS - Add your Own
# ***********************************
# -----------------------------------
# Add any extra bad referrers in the following include file to have them
# permanently included and blocked - avoid duplicates in your custom file
# custom-bad-referrers.conf is BOTH a BLACKLIST AND WHITELIST
# custom-bad-referrers.conf ALLOWS complete over-riding of anything
# If you think google.com is bad you would simply add them to
# custom-bad-referrers.conf with a value of 1
# *************************************************************************
# -------------------------------------------------------------------------
# START CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *************************************************************************
# -------------------------------------------------------------------------
include /etc/nginx/bots.d/custom-bad-referrers.conf;
# ***********************************************************************
# -----------------------------------------------------------------------
# END CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***********************************************************************
# -----------------------------------------------------------------------
# START BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)000free\.us(?:\b|)" 1;
@ -7644,137 +7697,49 @@ map $http_referer $bad_referer {
# END BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ###
}
# =====================================
# END SECTION 2 - REFERRERS AND DOMAINS
# =====================================
# ***********************************************
# WHITELISTING AND BLACKLISTING IP ADDRESS RANGES
# ***********************************************
# ========================================================================
# BEGIN SECTION 3 - WHITELISTING AND BLACKLISTING IP ADDRESSESE AND RANGES
# ========================================================================
# Geo directive to deny and also whitelist certain ip addresses
# --------------------------------------------------------------------------------------
# Map all GOOD and BAD IP Addresses and Ranges to a variable called geo $validate_client
# --------------------------------------------------------------------------------------
geo $validate_client {
# ********************
# First Our Safety Net
# ********************
# Anything not matching our rules is allowed through with default 0;
default 0;
# ***********************************
# Whitelist all your OWN IP addresses
# ***********************************
# ----------------------------------------------
# Whitelist all your OWN IP addresses and Ranges
# ----------------------------------------------
# --------------------------------------------------------------------------------------
# WHITELIST all your own IP addresses using the include file below.
# New Method Uses the include file below so that when pulling future updates your
# whitelisted IP addresses are automatically now included for you.
# Read Comments inside whitelist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# whitelist-ips.conf reigns supreme !!!
# Whatever you add to whitelist-ips.conf will be whitelisted FULL STOP
# Anything blacklisted above this line will be over-ridden by whitelist-ips.conf
# --------------------------------------------------------------------------------------
# **************************************************************************
# --------------------------------------------------------------------------
# START WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **************************************************************************
# --------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# ************************************************************************
# ------------------------------------------------------------------------
# END WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
# ****************
# Google IP Ranges
# ****************
# For Safety Sake all Google's Known IP Ranges are all white listed
# START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
108.177.0.0/17 0;
172.217.0.0/16 0;
173.194.0.0/16 0;
2001:4860:4000::/36 0;
203.208.60.0/24 0;
207.126.144.0/20 0;
209.85.128.0/17 0;
216.239.32.0/19 0;
216.58.192.0/19 0;
2404:6800:4000::/36 0;
2607:f8b0:4000::/36 0;
2800:3f0:4000::/36 0;
2a00:1450:4000::/36 0;
2c0f:fb50:4000::/36 0;
35.192.0.0/12 0;
64.18.0.0/20 0;
64.233.160.0/19 0;
64.68.80.0/21 0;
65.52.0.0/14 0;
66.102.0.0/20 0;
66.249.64.0/19 0;
72.14.192.0/18 0;
74.125.0.0/16 0;
# END GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# **************
# Bing IP Ranges
# **************
# For Safety Sake all Bing's Known IP Ranges are all white listed
# START BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
131.253.21.0/24 0;
131.253.22.0/23 0;
131.253.24.0/21 0;
131.253.24.0/22 0;
131.253.32.0/20 0;
157.54.0.0/15 0;
157.56.0.0/14 0;
157.60.0.0/16 0;
199.30.16.0/24 0;
199.30.27.0/24 0;
207.46.0.0/16 0;
40.112.0.0/13 0;
40.120.0.0/14 0;
40.124.0.0/16 0;
40.125.0.0/17 0;
40.74.0.0/15 0;
40.76.0.0/14 0;
40.80.0.0/12 0;
40.96.0.0/12 0;
# END BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ********************
# Cloudflare IP Ranges
# ********************
# For Safety Sake all Cloudflare's Known IP Ranges are all white listed
# START CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
103.21.244.0/22 0;
103.22.200.0/22 0;
103.31.4.0/22 0;
104.16.0.0/12 0;
108.162.192.0/18 0;
131.0.72.0/22 0;
141.101.64.0/18 0;
162.158.0.0/15 0;
172.64.0.0/13 0;
173.245.48.0/20 0;
188.114.96.0/20 0;
190.93.240.0/20 0;
197.234.240.0/22 0;
198.41.128.0/17 0;
199.27.128.0/21 0;
2400:cb00::/32 0;
2405:8100::/32 0;
2405:b500::/32 0;
2606:4700::/32 0;
2803:f800::/32 0;
2a06:98c0::/29 0;
2c0f:f248::/32 0;
# END CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# *************************
# Wordpress Theme Detectors
# *************************
# -------------------------------------
# BLOCK known Wordpress Theme Detectors
# -------------------------------------
# START WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
@ -7818,10 +7783,11 @@ geo $validate_client {
89.36.223.188 1; #www.cuteseotools.net/wordpress-theme-detector
# END WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
# ****************************************
# NIBBLER - SEO testing and reporting tool
# ****************************************
# ----------------------------------------------
# BLOCK NIBBLER - SEO testing and reporting tool
# ----------------------------------------------
# See - http://nibbler.silktide.com/
# ----------------------------------------------
# START NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
52.201.238.175 1;
@ -7834,33 +7800,10 @@ geo $validate_client {
# END NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
# ****************************
# Known Bad IP's and IP Ranges
# *************************************************
# Blacklist IP addresses and IP Ranges Customizable
# *************************************************
# BLACKLIST all your IP addresses and Ranges using the new include file below.
# New Method Uses the include file below so that when pulling future updates your
# Custom Blacklisted IP addresses are automatically now included for you.
# Read Comments inside blacklist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# ********************************************************************
# START BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ********************************************************************
include /etc/nginx/bots.d/blacklist-ips.conf;
# ******************************************************************
# END BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ******************************************************************
# *********************************************************************************************
# KNOWN BAD IP ADDRESSES
# Source: https://github.com/mitchellkrogza/Suspicious.Snooping.Sniffing.Hacking.IP.Addresses
# A Reliable Daily Updated List Included Below of Known Bad IP Addresses, Snoopers and Sniffers
# *********************************************************************************************
# -----------------------------------------
# BLOCK KNOWN BAD IP ADDRESSES
# Top known bad IP Adresses from abuseIPDB
# -----------------------------------------
# START KNOWN BAD IP ADDRESSES ### DO NOT EDIT THIS LINE AT ALL ###
193.32.163.89 1;
@ -17864,33 +17807,176 @@ geo $validate_client {
46.101.115.65 1;
# END KNOWN BAD IP ADDRESSES ### DO NOT EDIT THIS LINE AT ALL ###
# --------------------------
# WHITELIST Google IP Ranges
# --------------------------
# START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
108.177.0.0/17 0;
172.217.0.0/16 0;
173.194.0.0/16 0;
2001:4860:4000::/36 0;
203.208.60.0/24 0;
207.126.144.0/20 0;
209.85.128.0/17 0;
216.239.32.0/19 0;
216.58.192.0/19 0;
2404:6800:4000::/36 0;
2607:f8b0:4000::/36 0;
2800:3f0:4000::/36 0;
2a00:1450:4000::/36 0;
2c0f:fb50:4000::/36 0;
35.192.0.0/12 0;
64.18.0.0/20 0;
64.233.160.0/19 0;
64.68.80.0/21 0;
65.52.0.0/14 0;
66.102.0.0/20 0;
66.249.64.0/19 0;
72.14.192.0/18 0;
74.125.0.0/16 0;
# END GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ------------------------
# WHITELIST Bing IP Ranges
# ------------------------
# START BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
131.253.21.0/24 0;
131.253.22.0/23 0;
131.253.24.0/21 0;
131.253.24.0/22 0;
131.253.32.0/20 0;
157.54.0.0/15 0;
157.56.0.0/14 0;
157.60.0.0/16 0;
199.30.16.0/24 0;
199.30.27.0/24 0;
207.46.0.0/16 0;
40.112.0.0/13 0;
40.120.0.0/14 0;
40.124.0.0/16 0;
40.125.0.0/17 0;
40.74.0.0/15 0;
40.76.0.0/14 0;
40.80.0.0/12 0;
40.96.0.0/12 0;
# END BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ------------------------------
# WHITELIST Cloudflare IP Ranges
# ------------------------------
# START CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
103.21.244.0/22 0;
103.22.200.0/22 0;
103.31.4.0/22 0;
104.16.0.0/12 0;
108.162.192.0/18 0;
131.0.72.0/22 0;
141.101.64.0/18 0;
162.158.0.0/15 0;
172.64.0.0/13 0;
173.245.48.0/20 0;
188.114.96.0/20 0;
190.93.240.0/20 0;
197.234.240.0/22 0;
198.41.128.0/17 0;
199.27.128.0/21 0;
2400:cb00::/32 0;
2405:8100::/32 0;
2405:b500::/32 0;
2606:4700::/32 0;
2803:f800::/32 0;
2a06:98c0::/29 0;
2c0f:f248::/32 0;
# END CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# -------------------------------------------------
# BLACKLIST IP addresses and IP Ranges Customizable
# -------------------------------------------------
# --------------------------------------------------------------------------------------
# BLACKLIST all your IP addresses and Ranges using the new include file below.
# New Method Uses the include file below so that when pulling future updates your
# Custom Blacklisted IP addresses are automatically now included for you.
# Read Comments inside blacklist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# Anything added to blacklist-ips.conf will over-ride anything whitelisted above
# --------------------------------------------------------------------------------------
# --------------------------------------------------------------------
# START BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# --------------------------------------------------------------------
include /etc/nginx/bots.d/blacklist-ips.conf;
# ------------------------------------------------------------------
# END BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ------------------------------------------------------------------
# ----------------------------------------------
# Whitelist all your OWN IP addresses and Ranges
# ----------------------------------------------
# --------------------------------------------------------------------------------------
# WHITELIST all your own IP addresses using the include file below.
# New Method Uses the include file below so that when pulling future updates your
# whitelisted IP addresses are automatically now included for you.
# Read Comments inside whitelist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# whitelist-ips.conf reigns supreme !!!
# Whatever you add to whitelist-ips.conf will be whitelisted FULL STOP
# Anything blacklisted above this line will be over-ridden by whitelist-ips.conf
# --------------------------------------------------------------------------------------
# --------------------------------------------------------------------------
# START WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# --------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# ------------------------------------------------------------------------
# END WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ------------------------------------------------------------------------
}
# --------------------------------------------------------------------------------------
# WHITELIST your own IPs from the DDOS Filter
# Add your own IP addresses and ranges into the custom include file whitelist-ips.conf
# to spare them from the rate limiting DDOS filter.
# This section includes the same / single whitelist-ips.conf file so you only
# need to edit that include file and have it include here for you too.
# --------------------------------------------------------------------------------------
geo $ratelimited {
default 1;
# ***************************************************************************
# ---------------------------------------------------------------------------
# START WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***************************************************************************
# ---------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# *************************************************************************
# -------------------------------------------------------------------------
# END WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *************************************************************************
# -------------------------------------------------------------------------
}
# *****************************************
# MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# *****************************************
# ======================================================================
# END SECTION 3 - WHITELISTING AND BLACKLISTING IP ADDRESSESE AND RANGES
# ======================================================================
# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================
# --------------------------------------------
# 1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
@ -17898,30 +17984,31 @@ geo $ratelimited {
2 $binary_remote_addr;
}
# ***********************
# SET RATE LIMITING ZONES
# ***********************
# --------------------------
# 2. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# limits for Zone $bad_bot = 1
# Nothing Set - you can set a different zone limiter here if you like
# We issue a 444 response instead to all bad bots.
# Rate limiting will only take effect if on any User-Agents with a value of 2
# limits for Zone $bad_bot = 2
# this rate limiting will only take effect if you change any of the bots and change
# their block value from 1 to 2.
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=6r/m;
### *** MAKE SURE TO ADD to your nginx.conf ***
### server_names_hash_bucket_size 64;
### server_names_hash_max_size 4096;
### limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
### limit_conn_zone $binary_remote_addr zone=addr:50m;
### to allow it to load this large set of domains into memory and to set the rate limiting zones for the DDOS filter.
# ==========================================
# END SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ==========================================
### THE END of the Long and Winding Road
# =====================
# END BLOCKER FUNCTIONS
# =====================
### Also check out my Ultimate Apache Bad Bot Blocker on Github
### https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
### --------------------------------------------
### HELP SUPPORT THIS PROJECT - Send Me a Coffee
### https://ko-fi.com/mitchellkrog
### --------------------------------------------
### FOR APACHE SERVERS
### ---------------------------------------------
### Check out the Ultimate Apache Bad Bot Blocker
### ---------------------------------------------

2
.dev-tools/_conf_files_testing_changes_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.85.2 1;
104.154.120.187 1;
127.0.0.1 1;

2
.dev-tools/_conf_files_testing_changes_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,4 +1,4 @@
35.192.85.2 0;
104.154.120.187 0;
127.0.0.1 1;
127.0.0.1 1;
127.0.0.1 1;

643
.dev-tools/_conf_files_testing_changes_ratelimiting/conf.d/globalblacklist.conf
View file

@ -1,16 +1,21 @@
### **********************************
### THE ULTIMATE NGINX BAD BOT BLOCKER
### **********************************
### -----------------------------------------------------------
### THE NGINX ULTIMATE BAD BOT, BAD IP AND BAD REFERRER BLOCKER
### -----------------------------------------------------------
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1643
### Updated: Thu Jun 27 12:59:42 SAST 2019
### Version: V3.2019.06.1645
### Updated: Thu Jun 27 13:24:56 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################
### VERSION INFORMATION ##
### --------------------------------------------
### HELP SUPPORT THIS PROJECT - Send Me a Coffee
### https://ko-fi.com/mitchellkrog
### --------------------------------------------
##############################################################################
# _ __ _ #
# / |/ /__ _(_)__ __ __ #
@ -28,12 +33,16 @@
### and also has rate limiting functionality for bad bots who you only want to rate limit
### and not actually block out entirely. It is very powerful and also very flexible.
### --------------------------------------------------------------------------
### Created By: https://github.com/mitchellkrogza/
### Repo Url: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
### Copyright Mitchell Krog - <mitchellkrog@gmail.com>
### Contributors: Stuart Cardall - https://github.com/itoffshore
### --------------------------------------------------------------------------
### --------------------------------------------------------------------------
### Tested on: nginx/1.10.3 up to latest Mainstream Version (Ubuntu 16.04)
### --------------------------------------------------------------------------
### This list was developed and is in use on a live Nginx server running some very busy web sites.
### It was built from the ground up using real data from daily logs and is updated almost daily.
@ -41,9 +50,9 @@
### spam referrers, rogue IP address, scanners, scrapers and domain hijacking sites are extensively checked
### before they are added. It is monitored extensively for any false positives.
### *********
### ---------
### Features:
### *********
### ---------
### Clear formatting for Ease of Maintenance.
### Alphabetically ordered lists for Ease of Maintenance.
### Extensive Commenting for Ease of Reference.
@ -54,40 +63,38 @@
### IP range blocking / whitelisting.
### Rate Limiting Functions.
### ************
### ------------
### INSTALLATION
### ************
### ------------
### PLEASE use the install, setup and update scripts provided for you to ease your installation.
### This Auto Installation procedure is documented in the README.md and AUTO-CONFIGURATION.md files.
### Installation, Setup and Update Scripts Contributed by Stuart Cardall - https://github.com/itoffshore
### There are also manual configuration instructions provided for those not wishing to do an auto install.
### ***********************************************
### -----------------------------------------------
### !!!!! PLEASE READ INLINE NOTES ON TESTING !!!!!
### ***********************************************
### -----------------------------------------------
### Note that:
### SETTINGS:
### ---------------------------------------------
### 0 = allowed - no limits
### 1 = allowed or rate limited less restrictive
### 2 = rate limited more
### 3 = block completely
### ---------------------------------------------
### ************************************************************
### ------------------------------------------------------------
### CONTRIBUTING / PULL REQUESTS / ADDING YOUR OWN BAD REFERRERS
### ************************************************************
### ------------------------------------------------------------
### For contributing, corrections or adding bots or referrers to this repo,
### Send a Pull Request (PR) on any of the .list files in the _generator_lists folder
### All Pull Requests will be checked for accuracy before being merged.
# *********************************
# FIRST BLOCK BY USER-AGENT STRINGS
# *********************************
# ***********************
# -----------------------
# !!!!! PLEASE TEST !!!!!
# ***********************
# -----------------------
# ALWAYS test any User-Agent Strings you add here to make sure you have it right
# Use a Chrome Extension called "User-Agent Switcher for Chrome" where you can create your
@ -98,36 +105,46 @@
# curl -I http://www.yourdomain.com -A "GoogleBot" ---- GIVES YOU: HTTP/1.1 200 OK (Meaning web page was served to Client)
# curl -I http://www.yourdomain.com -A "80legs" ---- GIVES YOU: curl: (52) Empty reply from server (Meaning Nginx gave a 444 Dropped Connection)
# In this section we allow/disallow specific User Agents / Bots.
# *********************************************************************
# Now we map all good and bad user agents to a variable called $bad_bot
# *********************************************************************
# =======================
# START BLOCKER FUNCTIONS
# =======================
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# DO NOT EDIT ANYTHING BELOW THIS LINE !!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# =============================
# BEGIN SECTION 1 - USER-AGENTS
# =============================
# ALLOW / BLOCK User Agents / Bots
# -------------------------------------------------------------------
# Map all GOOD and BAD UA (User Agents) to a variable called $bad_bot
# -------------------------------------------------------------------
map $http_user_agent $bad_bot {
default 0;
# ***********************************************
# Include your Own Custom List of Bad User Agents
# ***********************************************
# Use the include file below to further customize your own list of additional user-agents you wish to permanently block
# This include file allows whitelisting and blacklisting of anything specified below it, meaning you can over-ride anything in this blocker to your liking.
# ***********************************************************************************
# -----------------------------------------------------------------------------------
# START CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***********************************************************************************
# -----------------------------------------------------------------------------------
# Include your Own Custom List of Bad User Agents
# Use the include file below to further customize your own list of additional user-agents you wish to permanently block
# This include file allows whitelisting and blacklisting of anything specified below it.
# This include file alows you to over-ride any Bad / Good UA (Bot) declared in this blocker to your liking.
include /etc/nginx/bots.d/blacklist-user-agents.conf;
# *********************************************************************************
# ---------------------------------------------------------------------------------
# END CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *********************************************************************************
# ---------------------------------------------------------------------------------
# *********************************************
# Bad User-Agent Strings That We Block Outright
# *********************************************
# --------------------------------------------------
# BAD UA (User-Agent) Strings That We Block Outright
# --------------------------------------------------
# START BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)360Spider(?:\b|)" 3;
@ -688,9 +705,16 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ZyBorg(?:\b|)" 3;
# END BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# ***********************************************
# Allow Good User-Agent Strings We Know and Trust
# ***********************************************
# --------------------------------------------
# GOOD UA User-Agent Strings We Know and Trust
# --------------------------------------------
# -----------------------------------------------------------------------
# You can over-ride these in /etc/nginx/bots.d/blacklist-user-agents.conf
# by adding the same UA line there and chaning its value of 1
# If you think GoogleBot is bad you would simply add them to
# blacklist-user-agents.conf with a value of 1
# -----------------------------------------------------------------------
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)adidxbot(?:\b|)" 0;
@ -727,12 +751,13 @@ map $http_user_agent $bad_bot {
"~*(?:\b)yahoo(?:\b|)" 0;
# END GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# ***************************************************
# User-Agent Strings Allowed Through but Rate Limited
# ***************************************************
# --------------------------------------------------------
# GOOD UA User-Agent Rate Limiting 1 - Disabled by Default
# --------------------------------------------------------
# Some people block libwww-perl, it used widely in many valid (non rogue) agents
# I allow libwww-perl as I use it for monitoring systems with Munin but it is rate limited
# TO ACTIVATE THIS RATE LIMITING Uncomment these two lines in blockbots.conf
#limit_conn bot1_connlimit 100;
#limit_req zone=bot1_reqlimitip burst=50;
# START ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)jetmon(?:\b|)" 1;
@ -743,9 +768,14 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wget/1.15(?:\b|)" 1;
# END ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# **************************************************************
# Rate Limited User-Agents who get a bit aggressive on bandwidth
# **************************************************************
# -------------------------------------------------------
# GOOD UA User-Agent Rate Limiting 2 - Enabled by Default
# -------------------------------------------------------
# -----------------------------------------------------------------------
# You can over-ride these in /etc/nginx/bots.d/blacklist-user-agents.conf
# by adding the same UA line there and chaning its value of 1
# -----------------------------------------------------------------------
# START LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)Alexa(?:\b|)" 2;
@ -768,163 +798,186 @@ map $http_user_agent $bad_bot {
# END LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
}
# *****************************************
# SECOND BLOCK BY REFERRER STRINGS AND URLS
# *****************************************
# ===========================
# END SECTION 1 - USER-AGENTS
# ===========================
# *****************
# =======================================
# BEGIN SECTION 2 - REFERRERS AND DOMAINS
# =======================================
# ----------------
# PLEASE TEST !!!!
# *****************
# ----------------
# ------------------------------------------------------------------------------------------------------------------------------
# ALWAYS test referrers that you add. This is done manually as follows
# ------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://anything.adcash.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://www.goodwebsite.com/not-adcash --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://www.betterwebsite.com/not/adcash --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://www.google.com --- GIVES YOU: full html output of the web page
# curl -I http://www.yourdomain.com -e http://www.microsoft.com --- GIVES YOU: full html output of the web page
# ------------------------------------------------------------------------------------------------------------------------------
# Because of case-insensitive matching any combination of capitilization in the names will all produce a positive hit
# make sure you always test thoroughly and monitor logs. This section below also does NOT check for a preceding www.
# and it also does not care if the referrer request was sent with http https or even ftp.
# ------------------------------------------------------------------------------------------------------------------------------
# ***********************************************************************
# Now we map all bad referrer words below to a variable called $bad_words
# ***********************************************************************
# ----------------------------------------------------------------
# Map all BAD referrer words below to a variable called $bad_words
# ----------------------------------------------------------------
# --------------------------------
# START Bad Referrer Word Scanning
# --------------------------------
map $http_referer $bad_words {
default 0;
# **************************
# Bad Referrer Word Scanning
# **************************
# -------------------------------------------------------------------------------------------
# These are Words and Terms often found tagged onto domains or within url query strings.
# Create and Customize Your Own Bad Referrer Words Here using the new Include File Method
# New Method Uses the include file below so that when pulling future updates your
# customized list of bad referrer words are automatically now included for you
# Read Comments inside bad-referrer-words.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# BE VERY CAREFUL using the bad-referrer-words.conf file - please read the comment and
# BE VERY CAREFUL using this bad-referrer-words.conf file - please read the comments and
# examples inside the include file for detailed explanations into how seriously this can
# affect your own site from serving assets or other innocent sites from accessing your site
# For safety sake the whitelist-domains.conf file is also loaded here before the bad-referrer-words.conf file.
# For safety sake the whitelist-domains.conf file is also loaded here before the
# bad-referrer-words.conf file is loaded.
# -------------------------------------------------------------------------------------------
# ************************************************************************
# ------------------------------------------------------------------------
# START WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-domains.conf;
# **********************************************************************
# ----------------------------------------------------------------------
# END WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **********************************************************************
# ----------------------------------------------------------------------
# ******************************************************************************
# ------------------------------------------------------------------------------
# START CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ******************************************************************************
# ------------------------------------------------------------------------------
include /etc/nginx/bots.d/bad-referrer-words.conf;
# ****************************************************************************
# ----------------------------------------------------------------------------
# END CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ****************************************************************************
# ----------------------------------------------------------------------------
}
# --------------------------------
# END Bad Referrer Word Scanning
# --------------------------------
# *************************
# Bad Referrer Domain Names
# *************************
# Now a list of bad referrer urls these domains or any combination of them ie .com .net
# will be blocked out. Doesn't matter if the protocol is http, https or even ftp
# ----------------------------------------
# START Good and Bad Referrer Domain Names
# ----------------------------------------
# -------------------------------------------------------------------------------------
# Good and Bad referrer urls Doesn't matter if the protocol is http, https or even ftp
# -------------------------------------------------------------------------------------
# ----------------------
# This section includes:
# **********************
# ----------------------
# --------------------------------------------------------------------------------
# Blocking of SEO company Semalt.com (now merged into this one section)
# MIRAI Botnet Domains Used for Mass Attacks
# Other known bad SEO companies and Ad Hijacking Sites
# Sites linked to malware, adware, clickjacking and ransomware
# Domain names and referrers used in referrer spam and seo hijacking
# Whitelisting of your own GOOD domains / referrers
# Whitelisting of any other GOOD domains / referrers you want explicitly NOT block
# --------------------------------------------------------------------------------
# *****************
# ----------------
# PLEASE TEST !!!!
# *****************
# ----------------
# ------------------------------------------------------------------------------------------------------------------------------------
# ALWAYS test referrers that you add. This is done manually as follows
# ------------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------------
# Because of case-insensitive matching any combination of capitilization will all produce a positive hit - make sure you always test.
# ------------------------------------------------------------------------------------------------------------------------------------
# For Example any of the following variations below of 8gold.com will be detected and blocked
# ------------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://NOT-8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://this.is.not8gOlD.net --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e ftp://8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e ftp://www.weare8gold.NET --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e https://subdomain.8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e https://NOT8GolD.org --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------------
# So if you see a bad referrer from wearegoogle.com and you want to block them just add
# them as "~*wearegoogle.com" don't ever go and do something like "~*google(-|.)" you will
# kill all your SEO in a week.
# ------------------------------------------------------------------------------------------------------------------------------------
# To add your own custom bad referrers use the custom include file
# /etc/nginx/bots.d/custom-bad-referrers.conf
# Or send a Pull Request to add it to the global blacklist for other users.
# In the bad referrers section I also include sites that hotlink images without permission.
# ------------------------------------------------------------------------------------------------------------------------------------
# ***********************************************************************
# Now we map all good & bad referrer urls to variable called #bad_referer
# ***********************************************************************
# --------------------------------------------------------------------
# Map all good & bad referrer DOMAINS to a variable called bad_referer
# --------------------------------------------------------------------
map $http_referer $bad_referer {
hostnames;
default 0;
# *************************************
# GOOD REFERRERS - Spared from Checking
# *************************************
# --------------------------------------------
# GOOD REFERRER DOMAINS - Spared from Checking
# --------------------------------------------
# ---------------------------------------------------------------------------------------
# Add all your own web site domain names and server names in this section
# WHITELIST Your Own Domain Names Here using the new Include File Method
# New Method Uses the include file below so that when pulling future updates your
# whitelisted domain names are automatically now included for you.
# Read Comments inside whitelist-domains.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# ---------------------------------------------------------------------------------------
# ************************************************************************
# ------------------------------------------------------------------------
# START WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-domains.conf;
# **********************************************************************
# ----------------------------------------------------------------------
# END WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **********************************************************************
# ----------------------------------------------------------------------
# ***********************************
# -----------------------------------
# CUSTOM BAD REFERRERS - Add your Own
# ***********************************
# -----------------------------------
# Add any extra bad referrers in the following include file to have them
# permanently included and blocked - avoid duplicates in your custom file
# custom-bad-referrers.conf is BOTH a BLACKLIST AND WHITELIST
# custom-bad-referrers.conf ALLOWS complete over-riding of anything
# If you think google.com is bad you would simply add them to
# custom-bad-referrers.conf with a value of 1
# *************************************************************************
# -------------------------------------------------------------------------
# START CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *************************************************************************
# -------------------------------------------------------------------------
include /etc/nginx/bots.d/custom-bad-referrers.conf;
# ***********************************************************************
# -----------------------------------------------------------------------
# END CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***********************************************************************
# -----------------------------------------------------------------------
# START BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)000free\.us(?:\b|)" 1;
@ -7644,137 +7697,49 @@ map $http_referer $bad_referer {
# END BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ###
}
# =====================================
# END SECTION 2 - REFERRERS AND DOMAINS
# =====================================
# ***********************************************
# WHITELISTING AND BLACKLISTING IP ADDRESS RANGES
# ***********************************************
# ========================================================================
# BEGIN SECTION 3 - WHITELISTING AND BLACKLISTING IP ADDRESSESE AND RANGES
# ========================================================================
# Geo directive to deny and also whitelist certain ip addresses
# --------------------------------------------------------------------------------------
# Map all GOOD and BAD IP Addresses and Ranges to a variable called geo $validate_client
# --------------------------------------------------------------------------------------
geo $validate_client {
# ********************
# First Our Safety Net
# ********************
# Anything not matching our rules is allowed through with default 0;
default 0;
# ***********************************
# Whitelist all your OWN IP addresses
# ***********************************
# ----------------------------------------------
# Whitelist all your OWN IP addresses and Ranges
# ----------------------------------------------
# --------------------------------------------------------------------------------------
# WHITELIST all your own IP addresses using the include file below.
# New Method Uses the include file below so that when pulling future updates your
# whitelisted IP addresses are automatically now included for you.
# Read Comments inside whitelist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# whitelist-ips.conf reigns supreme !!!
# Whatever you add to whitelist-ips.conf will be whitelisted FULL STOP
# Anything blacklisted above this line will be over-ridden by whitelist-ips.conf
# --------------------------------------------------------------------------------------
# **************************************************************************
# --------------------------------------------------------------------------
# START WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **************************************************************************
# --------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# ************************************************************************
# ------------------------------------------------------------------------
# END WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
# ****************
# Google IP Ranges
# ****************
# For Safety Sake all Google's Known IP Ranges are all white listed
# START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
108.177.0.0/17 0;
172.217.0.0/16 0;
173.194.0.0/16 0;
2001:4860:4000::/36 0;
203.208.60.0/24 0;
207.126.144.0/20 0;
209.85.128.0/17 0;
216.239.32.0/19 0;
216.58.192.0/19 0;
2404:6800:4000::/36 0;
2607:f8b0:4000::/36 0;
2800:3f0:4000::/36 0;
2a00:1450:4000::/36 0;
2c0f:fb50:4000::/36 0;
35.192.0.0/12 0;
64.18.0.0/20 0;
64.233.160.0/19 0;
64.68.80.0/21 0;
65.52.0.0/14 0;
66.102.0.0/20 0;
66.249.64.0/19 0;
72.14.192.0/18 0;
74.125.0.0/16 0;
# END GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# **************
# Bing IP Ranges
# **************
# For Safety Sake all Bing's Known IP Ranges are all white listed
# START BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
131.253.21.0/24 0;
131.253.22.0/23 0;
131.253.24.0/21 0;
131.253.24.0/22 0;
131.253.32.0/20 0;
157.54.0.0/15 0;
157.56.0.0/14 0;
157.60.0.0/16 0;
199.30.16.0/24 0;
199.30.27.0/24 0;
207.46.0.0/16 0;
40.112.0.0/13 0;
40.120.0.0/14 0;
40.124.0.0/16 0;
40.125.0.0/17 0;
40.74.0.0/15 0;
40.76.0.0/14 0;
40.80.0.0/12 0;
40.96.0.0/12 0;
# END BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ********************
# Cloudflare IP Ranges
# ********************
# For Safety Sake all Cloudflare's Known IP Ranges are all white listed
# START CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
103.21.244.0/22 0;
103.22.200.0/22 0;
103.31.4.0/22 0;
104.16.0.0/12 0;
108.162.192.0/18 0;
131.0.72.0/22 0;
141.101.64.0/18 0;
162.158.0.0/15 0;
172.64.0.0/13 0;
173.245.48.0/20 0;
188.114.96.0/20 0;
190.93.240.0/20 0;
197.234.240.0/22 0;
198.41.128.0/17 0;
199.27.128.0/21 0;
2400:cb00::/32 0;
2405:8100::/32 0;
2405:b500::/32 0;
2606:4700::/32 0;
2803:f800::/32 0;
2a06:98c0::/29 0;
2c0f:f248::/32 0;
# END CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# *************************
# Wordpress Theme Detectors
# *************************
# -------------------------------------
# BLOCK known Wordpress Theme Detectors
# -------------------------------------
# START WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
@ -7818,10 +7783,11 @@ geo $validate_client {
89.36.223.188 1; #www.cuteseotools.net/wordpress-theme-detector
# END WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
# ****************************************
# NIBBLER - SEO testing and reporting tool
# ****************************************
# ----------------------------------------------
# BLOCK NIBBLER - SEO testing and reporting tool
# ----------------------------------------------
# See - http://nibbler.silktide.com/
# ----------------------------------------------
# START NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
52.201.238.175 1;
@ -7834,33 +7800,10 @@ geo $validate_client {
# END NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
# ****************************
# Known Bad IP's and IP Ranges
# *************************************************
# Blacklist IP addresses and IP Ranges Customizable
# *************************************************
# BLACKLIST all your IP addresses and Ranges using the new include file below.
# New Method Uses the include file below so that when pulling future updates your
# Custom Blacklisted IP addresses are automatically now included for you.
# Read Comments inside blacklist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# ********************************************************************
# START BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ********************************************************************
include /etc/nginx/bots.d/blacklist-ips.conf;
# ******************************************************************
# END BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ******************************************************************
# *********************************************************************************************
# KNOWN BAD IP ADDRESSES
# Source: https://github.com/mitchellkrogza/Suspicious.Snooping.Sniffing.Hacking.IP.Addresses
# A Reliable Daily Updated List Included Below of Known Bad IP Addresses, Snoopers and Sniffers
# *********************************************************************************************
# -----------------------------------------
# BLOCK KNOWN BAD IP ADDRESSES
# Top known bad IP Adresses from abuseIPDB
# -----------------------------------------
# START KNOWN BAD IP ADDRESSES ### DO NOT EDIT THIS LINE AT ALL ###
193.32.163.89 1;
@ -17864,33 +17807,176 @@ geo $validate_client {
46.101.115.65 1;
# END KNOWN BAD IP ADDRESSES ### DO NOT EDIT THIS LINE AT ALL ###
# --------------------------
# WHITELIST Google IP Ranges
# --------------------------
# START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
108.177.0.0/17 0;
172.217.0.0/16 0;
173.194.0.0/16 0;
2001:4860:4000::/36 0;
203.208.60.0/24 0;
207.126.144.0/20 0;
209.85.128.0/17 0;
216.239.32.0/19 0;
216.58.192.0/19 0;
2404:6800:4000::/36 0;
2607:f8b0:4000::/36 0;
2800:3f0:4000::/36 0;
2a00:1450:4000::/36 0;
2c0f:fb50:4000::/36 0;
35.192.0.0/12 0;
64.18.0.0/20 0;
64.233.160.0/19 0;
64.68.80.0/21 0;
65.52.0.0/14 0;
66.102.0.0/20 0;
66.249.64.0/19 0;
72.14.192.0/18 0;
74.125.0.0/16 0;
# END GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ------------------------
# WHITELIST Bing IP Ranges
# ------------------------
# START BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
131.253.21.0/24 0;
131.253.22.0/23 0;
131.253.24.0/21 0;
131.253.24.0/22 0;
131.253.32.0/20 0;
157.54.0.0/15 0;
157.56.0.0/14 0;
157.60.0.0/16 0;
199.30.16.0/24 0;
199.30.27.0/24 0;
207.46.0.0/16 0;
40.112.0.0/13 0;
40.120.0.0/14 0;
40.124.0.0/16 0;
40.125.0.0/17 0;
40.74.0.0/15 0;
40.76.0.0/14 0;
40.80.0.0/12 0;
40.96.0.0/12 0;
# END BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ------------------------------
# WHITELIST Cloudflare IP Ranges
# ------------------------------
# START CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
103.21.244.0/22 0;
103.22.200.0/22 0;
103.31.4.0/22 0;
104.16.0.0/12 0;
108.162.192.0/18 0;
131.0.72.0/22 0;
141.101.64.0/18 0;
162.158.0.0/15 0;
172.64.0.0/13 0;
173.245.48.0/20 0;
188.114.96.0/20 0;
190.93.240.0/20 0;
197.234.240.0/22 0;
198.41.128.0/17 0;
199.27.128.0/21 0;
2400:cb00::/32 0;
2405:8100::/32 0;
2405:b500::/32 0;
2606:4700::/32 0;
2803:f800::/32 0;
2a06:98c0::/29 0;
2c0f:f248::/32 0;
# END CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# -------------------------------------------------
# BLACKLIST IP addresses and IP Ranges Customizable
# -------------------------------------------------
# --------------------------------------------------------------------------------------
# BLACKLIST all your IP addresses and Ranges using the new include file below.
# New Method Uses the include file below so that when pulling future updates your
# Custom Blacklisted IP addresses are automatically now included for you.
# Read Comments inside blacklist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# Anything added to blacklist-ips.conf will over-ride anything whitelisted above
# --------------------------------------------------------------------------------------
# --------------------------------------------------------------------
# START BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# --------------------------------------------------------------------
include /etc/nginx/bots.d/blacklist-ips.conf;
# ------------------------------------------------------------------
# END BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ------------------------------------------------------------------
# ----------------------------------------------
# Whitelist all your OWN IP addresses and Ranges
# ----------------------------------------------
# --------------------------------------------------------------------------------------
# WHITELIST all your own IP addresses using the include file below.
# New Method Uses the include file below so that when pulling future updates your
# whitelisted IP addresses are automatically now included for you.
# Read Comments inside whitelist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# whitelist-ips.conf reigns supreme !!!
# Whatever you add to whitelist-ips.conf will be whitelisted FULL STOP
# Anything blacklisted above this line will be over-ridden by whitelist-ips.conf
# --------------------------------------------------------------------------------------
# --------------------------------------------------------------------------
# START WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# --------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# ------------------------------------------------------------------------
# END WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ------------------------------------------------------------------------
}
# --------------------------------------------------------------------------------------
# WHITELIST your own IPs from the DDOS Filter
# Add your own IP addresses and ranges into the custom include file whitelist-ips.conf
# to spare them from the rate limiting DDOS filter.
# This section includes the same / single whitelist-ips.conf file so you only
# need to edit that include file and have it include here for you too.
# --------------------------------------------------------------------------------------
geo $ratelimited {
default 1;
# ***************************************************************************
# ---------------------------------------------------------------------------
# START WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***************************************************************************
# ---------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# *************************************************************************
# -------------------------------------------------------------------------
# END WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *************************************************************************
# -------------------------------------------------------------------------
}
# *****************************************
# MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# *****************************************
# ======================================================================
# END SECTION 3 - WHITELISTING AND BLACKLISTING IP ADDRESSESE AND RANGES
# ======================================================================
# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================
# --------------------------------------------
# 1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
@ -17898,30 +17984,31 @@ geo $ratelimited {
2 $binary_remote_addr;
}
# ***********************
# SET RATE LIMITING ZONES
# ***********************
# --------------------------
# 2. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# limits for Zone $bad_bot = 1
# Nothing Set - you can set a different zone limiter here if you like
# We issue a 444 response instead to all bad bots.
# Rate limiting will only take effect if on any User-Agents with a value of 2
# limits for Zone $bad_bot = 2
# this rate limiting will only take effect if you change any of the bots and change
# their block value from 1 to 2.
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=6r/m;
### *** MAKE SURE TO ADD to your nginx.conf ***
### server_names_hash_bucket_size 64;
### server_names_hash_max_size 4096;
### limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
### limit_conn_zone $binary_remote_addr zone=addr:50m;
### to allow it to load this large set of domains into memory and to set the rate limiting zones for the DDOS filter.
# ==========================================
# END SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ==========================================
### THE END of the Long and Winding Road
# =====================
# END BLOCKER FUNCTIONS
# =====================
### Also check out my Ultimate Apache Bad Bot Blocker on Github
### https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
### --------------------------------------------
### HELP SUPPORT THIS PROJECT - Send Me a Coffee
### https://ko-fi.com/mitchellkrog
### --------------------------------------------
### FOR APACHE SERVERS
### ---------------------------------------------
### Check out the Ultimate Apache Bad Bot Blocker
### ---------------------------------------------

2
.dev-tools/_conf_files_testing_changes_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.85.2 1;
104.154.120.187 1;
127.0.0.1 1;

2
.dev-tools/_conf_files_testing_changes_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,4 +1,4 @@
35.192.85.2 0;
104.154.120.187 0;
127.0.0.1 1;
127.0.0.1 1;
127.0.0.1 1;

643
.dev-tools/_conf_files_testing_changes_whitelist/conf.d/globalblacklist.conf
View file

@ -1,16 +1,21 @@
### **********************************
### THE ULTIMATE NGINX BAD BOT BLOCKER
### **********************************
### -----------------------------------------------------------
### THE NGINX ULTIMATE BAD BOT, BAD IP AND BAD REFERRER BLOCKER
### -----------------------------------------------------------
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1643
### Updated: Thu Jun 27 12:59:42 SAST 2019
### Version: V3.2019.06.1645
### Updated: Thu Jun 27 13:24:56 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################
### VERSION INFORMATION ##
### --------------------------------------------
### HELP SUPPORT THIS PROJECT - Send Me a Coffee
### https://ko-fi.com/mitchellkrog
### --------------------------------------------
##############################################################################
# _ __ _ #
# / |/ /__ _(_)__ __ __ #
@ -28,12 +33,16 @@
### and also has rate limiting functionality for bad bots who you only want to rate limit
### and not actually block out entirely. It is very powerful and also very flexible.
### --------------------------------------------------------------------------
### Created By: https://github.com/mitchellkrogza/
### Repo Url: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
### Copyright Mitchell Krog - <mitchellkrog@gmail.com>
### Contributors: Stuart Cardall - https://github.com/itoffshore
### --------------------------------------------------------------------------
### --------------------------------------------------------------------------
### Tested on: nginx/1.10.3 up to latest Mainstream Version (Ubuntu 16.04)
### --------------------------------------------------------------------------
### This list was developed and is in use on a live Nginx server running some very busy web sites.
### It was built from the ground up using real data from daily logs and is updated almost daily.
@ -41,9 +50,9 @@
### spam referrers, rogue IP address, scanners, scrapers and domain hijacking sites are extensively checked
### before they are added. It is monitored extensively for any false positives.
### *********
### ---------
### Features:
### *********
### ---------
### Clear formatting for Ease of Maintenance.
### Alphabetically ordered lists for Ease of Maintenance.
### Extensive Commenting for Ease of Reference.
@ -54,40 +63,38 @@
### IP range blocking / whitelisting.
### Rate Limiting Functions.
### ************
### ------------
### INSTALLATION
### ************
### ------------
### PLEASE use the install, setup and update scripts provided for you to ease your installation.
### This Auto Installation procedure is documented in the README.md and AUTO-CONFIGURATION.md files.
### Installation, Setup and Update Scripts Contributed by Stuart Cardall - https://github.com/itoffshore
### There are also manual configuration instructions provided for those not wishing to do an auto install.
### ***********************************************
### -----------------------------------------------
### !!!!! PLEASE READ INLINE NOTES ON TESTING !!!!!
### ***********************************************
### -----------------------------------------------
### Note that:
### SETTINGS:
### ---------------------------------------------
### 0 = allowed - no limits
### 1 = allowed or rate limited less restrictive
### 2 = rate limited more
### 3 = block completely
### ---------------------------------------------
### ************************************************************
### ------------------------------------------------------------
### CONTRIBUTING / PULL REQUESTS / ADDING YOUR OWN BAD REFERRERS
### ************************************************************
### ------------------------------------------------------------
### For contributing, corrections or adding bots or referrers to this repo,
### Send a Pull Request (PR) on any of the .list files in the _generator_lists folder
### All Pull Requests will be checked for accuracy before being merged.
# *********************************
# FIRST BLOCK BY USER-AGENT STRINGS
# *********************************
# ***********************
# -----------------------
# !!!!! PLEASE TEST !!!!!
# ***********************
# -----------------------
# ALWAYS test any User-Agent Strings you add here to make sure you have it right
# Use a Chrome Extension called "User-Agent Switcher for Chrome" where you can create your
@ -98,36 +105,46 @@
# curl -I http://www.yourdomain.com -A "GoogleBot" ---- GIVES YOU: HTTP/1.1 200 OK (Meaning web page was served to Client)
# curl -I http://www.yourdomain.com -A "80legs" ---- GIVES YOU: curl: (52) Empty reply from server (Meaning Nginx gave a 444 Dropped Connection)
# In this section we allow/disallow specific User Agents / Bots.
# *********************************************************************
# Now we map all good and bad user agents to a variable called $bad_bot
# *********************************************************************
# =======================
# START BLOCKER FUNCTIONS
# =======================
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# DO NOT EDIT ANYTHING BELOW THIS LINE !!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# =============================
# BEGIN SECTION 1 - USER-AGENTS
# =============================
# ALLOW / BLOCK User Agents / Bots
# -------------------------------------------------------------------
# Map all GOOD and BAD UA (User Agents) to a variable called $bad_bot
# -------------------------------------------------------------------
map $http_user_agent $bad_bot {
default 0;
# ***********************************************
# Include your Own Custom List of Bad User Agents
# ***********************************************
# Use the include file below to further customize your own list of additional user-agents you wish to permanently block
# This include file allows whitelisting and blacklisting of anything specified below it, meaning you can over-ride anything in this blocker to your liking.
# ***********************************************************************************
# -----------------------------------------------------------------------------------
# START CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***********************************************************************************
# -----------------------------------------------------------------------------------
# Include your Own Custom List of Bad User Agents
# Use the include file below to further customize your own list of additional user-agents you wish to permanently block
# This include file allows whitelisting and blacklisting of anything specified below it.
# This include file alows you to over-ride any Bad / Good UA (Bot) declared in this blocker to your liking.
include /etc/nginx/bots.d/blacklist-user-agents.conf;
# *********************************************************************************
# ---------------------------------------------------------------------------------
# END CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *********************************************************************************
# ---------------------------------------------------------------------------------
# *********************************************
# Bad User-Agent Strings That We Block Outright
# *********************************************
# --------------------------------------------------
# BAD UA (User-Agent) Strings That We Block Outright
# --------------------------------------------------
# START BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)360Spider(?:\b|)" 3;
@ -688,9 +705,16 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ZyBorg(?:\b|)" 3;
# END BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# ***********************************************
# Allow Good User-Agent Strings We Know and Trust
# ***********************************************
# --------------------------------------------
# GOOD UA User-Agent Strings We Know and Trust
# --------------------------------------------
# -----------------------------------------------------------------------
# You can over-ride these in /etc/nginx/bots.d/blacklist-user-agents.conf
# by adding the same UA line there and chaning its value of 1
# If you think GoogleBot is bad you would simply add them to
# blacklist-user-agents.conf with a value of 1
# -----------------------------------------------------------------------
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)adidxbot(?:\b|)" 0;
@ -727,12 +751,13 @@ map $http_user_agent $bad_bot {
"~*(?:\b)yahoo(?:\b|)" 0;
# END GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# ***************************************************
# User-Agent Strings Allowed Through but Rate Limited
# ***************************************************
# --------------------------------------------------------
# GOOD UA User-Agent Rate Limiting 1 - Disabled by Default
# --------------------------------------------------------
# Some people block libwww-perl, it used widely in many valid (non rogue) agents
# I allow libwww-perl as I use it for monitoring systems with Munin but it is rate limited
# TO ACTIVATE THIS RATE LIMITING Uncomment these two lines in blockbots.conf
#limit_conn bot1_connlimit 100;
#limit_req zone=bot1_reqlimitip burst=50;
# START ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)jetmon(?:\b|)" 1;
@ -743,9 +768,14 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wget/1.15(?:\b|)" 1;
# END ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# **************************************************************
# Rate Limited User-Agents who get a bit aggressive on bandwidth
# **************************************************************
# -------------------------------------------------------
# GOOD UA User-Agent Rate Limiting 2 - Enabled by Default
# -------------------------------------------------------
# -----------------------------------------------------------------------
# You can over-ride these in /etc/nginx/bots.d/blacklist-user-agents.conf
# by adding the same UA line there and chaning its value of 1
# -----------------------------------------------------------------------
# START LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)Alexa(?:\b|)" 2;
@ -768,163 +798,186 @@ map $http_user_agent $bad_bot {
# END LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
}
# *****************************************
# SECOND BLOCK BY REFERRER STRINGS AND URLS
# *****************************************
# ===========================
# END SECTION 1 - USER-AGENTS
# ===========================
# *****************
# =======================================
# BEGIN SECTION 2 - REFERRERS AND DOMAINS
# =======================================
# ----------------
# PLEASE TEST !!!!
# *****************
# ----------------
# ------------------------------------------------------------------------------------------------------------------------------
# ALWAYS test referrers that you add. This is done manually as follows
# ------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://anything.adcash.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://www.goodwebsite.com/not-adcash --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://www.betterwebsite.com/not/adcash --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://www.google.com --- GIVES YOU: full html output of the web page
# curl -I http://www.yourdomain.com -e http://www.microsoft.com --- GIVES YOU: full html output of the web page
# ------------------------------------------------------------------------------------------------------------------------------
# Because of case-insensitive matching any combination of capitilization in the names will all produce a positive hit
# make sure you always test thoroughly and monitor logs. This section below also does NOT check for a preceding www.
# and it also does not care if the referrer request was sent with http https or even ftp.
# ------------------------------------------------------------------------------------------------------------------------------
# ***********************************************************************
# Now we map all bad referrer words below to a variable called $bad_words
# ***********************************************************************
# ----------------------------------------------------------------
# Map all BAD referrer words below to a variable called $bad_words
# ----------------------------------------------------------------
# --------------------------------
# START Bad Referrer Word Scanning
# --------------------------------
map $http_referer $bad_words {
default 0;
# **************************
# Bad Referrer Word Scanning
# **************************
# -------------------------------------------------------------------------------------------
# These are Words and Terms often found tagged onto domains or within url query strings.
# Create and Customize Your Own Bad Referrer Words Here using the new Include File Method
# New Method Uses the include file below so that when pulling future updates your
# customized list of bad referrer words are automatically now included for you
# Read Comments inside bad-referrer-words.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# BE VERY CAREFUL using the bad-referrer-words.conf file - please read the comment and
# BE VERY CAREFUL using this bad-referrer-words.conf file - please read the comments and
# examples inside the include file for detailed explanations into how seriously this can
# affect your own site from serving assets or other innocent sites from accessing your site
# For safety sake the whitelist-domains.conf file is also loaded here before the bad-referrer-words.conf file.
# For safety sake the whitelist-domains.conf file is also loaded here before the
# bad-referrer-words.conf file is loaded.
# -------------------------------------------------------------------------------------------
# ************************************************************************
# ------------------------------------------------------------------------
# START WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-domains.conf;
# **********************************************************************
# ----------------------------------------------------------------------
# END WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **********************************************************************
# ----------------------------------------------------------------------
# ******************************************************************************
# ------------------------------------------------------------------------------
# START CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ******************************************************************************
# ------------------------------------------------------------------------------
include /etc/nginx/bots.d/bad-referrer-words.conf;
# ****************************************************************************
# ----------------------------------------------------------------------------
# END CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ****************************************************************************
# ----------------------------------------------------------------------------
}
# --------------------------------
# END Bad Referrer Word Scanning
# --------------------------------
# *************************
# Bad Referrer Domain Names
# *************************
# Now a list of bad referrer urls these domains or any combination of them ie .com .net
# will be blocked out. Doesn't matter if the protocol is http, https or even ftp
# ----------------------------------------
# START Good and Bad Referrer Domain Names
# ----------------------------------------
# -------------------------------------------------------------------------------------
# Good and Bad referrer urls Doesn't matter if the protocol is http, https or even ftp
# -------------------------------------------------------------------------------------
# ----------------------
# This section includes:
# **********************
# ----------------------
# --------------------------------------------------------------------------------
# Blocking of SEO company Semalt.com (now merged into this one section)
# MIRAI Botnet Domains Used for Mass Attacks
# Other known bad SEO companies and Ad Hijacking Sites
# Sites linked to malware, adware, clickjacking and ransomware
# Domain names and referrers used in referrer spam and seo hijacking
# Whitelisting of your own GOOD domains / referrers
# Whitelisting of any other GOOD domains / referrers you want explicitly NOT block
# --------------------------------------------------------------------------------
# *****************
# ----------------
# PLEASE TEST !!!!
# *****************
# ----------------
# ------------------------------------------------------------------------------------------------------------------------------------
# ALWAYS test referrers that you add. This is done manually as follows
# ------------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------------
# Because of case-insensitive matching any combination of capitilization will all produce a positive hit - make sure you always test.
# ------------------------------------------------------------------------------------------------------------------------------------
# For Example any of the following variations below of 8gold.com will be detected and blocked
# ------------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://NOT-8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://this.is.not8gOlD.net --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e ftp://8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e ftp://www.weare8gold.NET --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e https://subdomain.8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e https://NOT8GolD.org --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------------
# So if you see a bad referrer from wearegoogle.com and you want to block them just add
# them as "~*wearegoogle.com" don't ever go and do something like "~*google(-|.)" you will
# kill all your SEO in a week.
# ------------------------------------------------------------------------------------------------------------------------------------
# To add your own custom bad referrers use the custom include file
# /etc/nginx/bots.d/custom-bad-referrers.conf
# Or send a Pull Request to add it to the global blacklist for other users.
# In the bad referrers section I also include sites that hotlink images without permission.
# ------------------------------------------------------------------------------------------------------------------------------------
# ***********************************************************************
# Now we map all good & bad referrer urls to variable called #bad_referer
# ***********************************************************************
# --------------------------------------------------------------------
# Map all good & bad referrer DOMAINS to a variable called bad_referer
# --------------------------------------------------------------------
map $http_referer $bad_referer {
hostnames;
default 0;
# *************************************
# GOOD REFERRERS - Spared from Checking
# *************************************
# --------------------------------------------
# GOOD REFERRER DOMAINS - Spared from Checking
# --------------------------------------------
# ---------------------------------------------------------------------------------------
# Add all your own web site domain names and server names in this section
# WHITELIST Your Own Domain Names Here using the new Include File Method
# New Method Uses the include file below so that when pulling future updates your
# whitelisted domain names are automatically now included for you.
# Read Comments inside whitelist-domains.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# ---------------------------------------------------------------------------------------
# ************************************************************************
# ------------------------------------------------------------------------
# START WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-domains.conf;
# **********************************************************************
# ----------------------------------------------------------------------
# END WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **********************************************************************
# ----------------------------------------------------------------------
# ***********************************
# -----------------------------------
# CUSTOM BAD REFERRERS - Add your Own
# ***********************************
# -----------------------------------
# Add any extra bad referrers in the following include file to have them
# permanently included and blocked - avoid duplicates in your custom file
# custom-bad-referrers.conf is BOTH a BLACKLIST AND WHITELIST
# custom-bad-referrers.conf ALLOWS complete over-riding of anything
# If you think google.com is bad you would simply add them to
# custom-bad-referrers.conf with a value of 1
# *************************************************************************
# -------------------------------------------------------------------------
# START CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *************************************************************************
# -------------------------------------------------------------------------
include /etc/nginx/bots.d/custom-bad-referrers.conf;
# ***********************************************************************
# -----------------------------------------------------------------------
# END CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***********************************************************************
# -----------------------------------------------------------------------
# START BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)000free\.us(?:\b|)" 1;
@ -7644,137 +7697,49 @@ map $http_referer $bad_referer {
# END BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ###
}
# =====================================
# END SECTION 2 - REFERRERS AND DOMAINS
# =====================================
# ***********************************************
# WHITELISTING AND BLACKLISTING IP ADDRESS RANGES
# ***********************************************
# ========================================================================
# BEGIN SECTION 3 - WHITELISTING AND BLACKLISTING IP ADDRESSESE AND RANGES
# ========================================================================
# Geo directive to deny and also whitelist certain ip addresses
# --------------------------------------------------------------------------------------
# Map all GOOD and BAD IP Addresses and Ranges to a variable called geo $validate_client
# --------------------------------------------------------------------------------------
geo $validate_client {
# ********************
# First Our Safety Net
# ********************
# Anything not matching our rules is allowed through with default 0;
default 0;
# ***********************************
# Whitelist all your OWN IP addresses
# ***********************************
# ----------------------------------------------
# Whitelist all your OWN IP addresses and Ranges
# ----------------------------------------------
# --------------------------------------------------------------------------------------
# WHITELIST all your own IP addresses using the include file below.
# New Method Uses the include file below so that when pulling future updates your
# whitelisted IP addresses are automatically now included for you.
# Read Comments inside whitelist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# whitelist-ips.conf reigns supreme !!!
# Whatever you add to whitelist-ips.conf will be whitelisted FULL STOP
# Anything blacklisted above this line will be over-ridden by whitelist-ips.conf
# --------------------------------------------------------------------------------------
# **************************************************************************
# --------------------------------------------------------------------------
# START WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **************************************************************************
# --------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# ************************************************************************
# ------------------------------------------------------------------------
# END WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
# ****************
# Google IP Ranges
# ****************
# For Safety Sake all Google's Known IP Ranges are all white listed
# START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
108.177.0.0/17 0;
172.217.0.0/16 0;
173.194.0.0/16 0;
2001:4860:4000::/36 0;
203.208.60.0/24 0;
207.126.144.0/20 0;
209.85.128.0/17 0;
216.239.32.0/19 0;
216.58.192.0/19 0;
2404:6800:4000::/36 0;
2607:f8b0:4000::/36 0;
2800:3f0:4000::/36 0;
2a00:1450:4000::/36 0;
2c0f:fb50:4000::/36 0;
35.192.0.0/12 0;
64.18.0.0/20 0;
64.233.160.0/19 0;
64.68.80.0/21 0;
65.52.0.0/14 0;
66.102.0.0/20 0;
66.249.64.0/19 0;
72.14.192.0/18 0;
74.125.0.0/16 0;
# END GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# **************
# Bing IP Ranges
# **************
# For Safety Sake all Bing's Known IP Ranges are all white listed
# START BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
131.253.21.0/24 0;
131.253.22.0/23 0;
131.253.24.0/21 0;
131.253.24.0/22 0;
131.253.32.0/20 0;
157.54.0.0/15 0;
157.56.0.0/14 0;
157.60.0.0/16 0;
199.30.16.0/24 0;
199.30.27.0/24 0;
207.46.0.0/16 0;
40.112.0.0/13 0;
40.120.0.0/14 0;
40.124.0.0/16 0;
40.125.0.0/17 0;
40.74.0.0/15 0;
40.76.0.0/14 0;
40.80.0.0/12 0;
40.96.0.0/12 0;
# END BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ********************
# Cloudflare IP Ranges
# ********************
# For Safety Sake all Cloudflare's Known IP Ranges are all white listed
# START CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
103.21.244.0/22 0;
103.22.200.0/22 0;
103.31.4.0/22 0;
104.16.0.0/12 0;
108.162.192.0/18 0;
131.0.72.0/22 0;
141.101.64.0/18 0;
162.158.0.0/15 0;
172.64.0.0/13 0;
173.245.48.0/20 0;
188.114.96.0/20 0;
190.93.240.0/20 0;
197.234.240.0/22 0;
198.41.128.0/17 0;
199.27.128.0/21 0;
2400:cb00::/32 0;
2405:8100::/32 0;
2405:b500::/32 0;
2606:4700::/32 0;
2803:f800::/32 0;
2a06:98c0::/29 0;
2c0f:f248::/32 0;
# END CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# *************************
# Wordpress Theme Detectors
# *************************
# -------------------------------------
# BLOCK known Wordpress Theme Detectors
# -------------------------------------
# START WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
@ -7818,10 +7783,11 @@ geo $validate_client {
89.36.223.188 1; #www.cuteseotools.net/wordpress-theme-detector
# END WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
# ****************************************
# NIBBLER - SEO testing and reporting tool
# ****************************************
# ----------------------------------------------
# BLOCK NIBBLER - SEO testing and reporting tool
# ----------------------------------------------
# See - http://nibbler.silktide.com/
# ----------------------------------------------
# START NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
52.201.238.175 1;
@ -7834,33 +7800,10 @@ geo $validate_client {
# END NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
# ****************************
# Known Bad IP's and IP Ranges
# *************************************************
# Blacklist IP addresses and IP Ranges Customizable
# *************************************************
# BLACKLIST all your IP addresses and Ranges using the new include file below.
# New Method Uses the include file below so that when pulling future updates your
# Custom Blacklisted IP addresses are automatically now included for you.
# Read Comments inside blacklist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# ********************************************************************
# START BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ********************************************************************
include /etc/nginx/bots.d/blacklist-ips.conf;
# ******************************************************************
# END BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ******************************************************************
# *********************************************************************************************
# KNOWN BAD IP ADDRESSES
# Source: https://github.com/mitchellkrogza/Suspicious.Snooping.Sniffing.Hacking.IP.Addresses
# A Reliable Daily Updated List Included Below of Known Bad IP Addresses, Snoopers and Sniffers
# *********************************************************************************************
# -----------------------------------------
# BLOCK KNOWN BAD IP ADDRESSES
# Top known bad IP Adresses from abuseIPDB
# -----------------------------------------
# START KNOWN BAD IP ADDRESSES ### DO NOT EDIT THIS LINE AT ALL ###
193.32.163.89 1;
@ -17864,33 +17807,176 @@ geo $validate_client {
46.101.115.65 1;
# END KNOWN BAD IP ADDRESSES ### DO NOT EDIT THIS LINE AT ALL ###
# --------------------------
# WHITELIST Google IP Ranges
# --------------------------
# START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
108.177.0.0/17 0;
172.217.0.0/16 0;
173.194.0.0/16 0;
2001:4860:4000::/36 0;
203.208.60.0/24 0;
207.126.144.0/20 0;
209.85.128.0/17 0;
216.239.32.0/19 0;
216.58.192.0/19 0;
2404:6800:4000::/36 0;
2607:f8b0:4000::/36 0;
2800:3f0:4000::/36 0;
2a00:1450:4000::/36 0;
2c0f:fb50:4000::/36 0;
35.192.0.0/12 0;
64.18.0.0/20 0;
64.233.160.0/19 0;
64.68.80.0/21 0;
65.52.0.0/14 0;
66.102.0.0/20 0;
66.249.64.0/19 0;
72.14.192.0/18 0;
74.125.0.0/16 0;
# END GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ------------------------
# WHITELIST Bing IP Ranges
# ------------------------
# START BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
131.253.21.0/24 0;
131.253.22.0/23 0;
131.253.24.0/21 0;
131.253.24.0/22 0;
131.253.32.0/20 0;
157.54.0.0/15 0;
157.56.0.0/14 0;
157.60.0.0/16 0;
199.30.16.0/24 0;
199.30.27.0/24 0;
207.46.0.0/16 0;
40.112.0.0/13 0;
40.120.0.0/14 0;
40.124.0.0/16 0;
40.125.0.0/17 0;
40.74.0.0/15 0;
40.76.0.0/14 0;
40.80.0.0/12 0;
40.96.0.0/12 0;
# END BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ------------------------------
# WHITELIST Cloudflare IP Ranges
# ------------------------------
# START CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
103.21.244.0/22 0;
103.22.200.0/22 0;
103.31.4.0/22 0;
104.16.0.0/12 0;
108.162.192.0/18 0;
131.0.72.0/22 0;
141.101.64.0/18 0;
162.158.0.0/15 0;
172.64.0.0/13 0;
173.245.48.0/20 0;
188.114.96.0/20 0;
190.93.240.0/20 0;
197.234.240.0/22 0;
198.41.128.0/17 0;
199.27.128.0/21 0;
2400:cb00::/32 0;
2405:8100::/32 0;
2405:b500::/32 0;
2606:4700::/32 0;
2803:f800::/32 0;
2a06:98c0::/29 0;
2c0f:f248::/32 0;
# END CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# -------------------------------------------------
# BLACKLIST IP addresses and IP Ranges Customizable
# -------------------------------------------------
# --------------------------------------------------------------------------------------
# BLACKLIST all your IP addresses and Ranges using the new include file below.
# New Method Uses the include file below so that when pulling future updates your
# Custom Blacklisted IP addresses are automatically now included for you.
# Read Comments inside blacklist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# Anything added to blacklist-ips.conf will over-ride anything whitelisted above
# --------------------------------------------------------------------------------------
# --------------------------------------------------------------------
# START BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# --------------------------------------------------------------------
include /etc/nginx/bots.d/blacklist-ips.conf;
# ------------------------------------------------------------------
# END BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ------------------------------------------------------------------
# ----------------------------------------------
# Whitelist all your OWN IP addresses and Ranges
# ----------------------------------------------
# --------------------------------------------------------------------------------------
# WHITELIST all your own IP addresses using the include file below.
# New Method Uses the include file below so that when pulling future updates your
# whitelisted IP addresses are automatically now included for you.
# Read Comments inside whitelist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# whitelist-ips.conf reigns supreme !!!
# Whatever you add to whitelist-ips.conf will be whitelisted FULL STOP
# Anything blacklisted above this line will be over-ridden by whitelist-ips.conf
# --------------------------------------------------------------------------------------
# --------------------------------------------------------------------------
# START WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# --------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# ------------------------------------------------------------------------
# END WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ------------------------------------------------------------------------
}
# --------------------------------------------------------------------------------------
# WHITELIST your own IPs from the DDOS Filter
# Add your own IP addresses and ranges into the custom include file whitelist-ips.conf
# to spare them from the rate limiting DDOS filter.
# This section includes the same / single whitelist-ips.conf file so you only
# need to edit that include file and have it include here for you too.
# --------------------------------------------------------------------------------------
geo $ratelimited {
default 1;
# ***************************************************************************
# ---------------------------------------------------------------------------
# START WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***************************************************************************
# ---------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# *************************************************************************
# -------------------------------------------------------------------------
# END WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *************************************************************************
# -------------------------------------------------------------------------
}
# *****************************************
# MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# *****************************************
# ======================================================================
# END SECTION 3 - WHITELISTING AND BLACKLISTING IP ADDRESSESE AND RANGES
# ======================================================================
# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================
# --------------------------------------------
# 1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
@ -17898,30 +17984,31 @@ geo $ratelimited {
2 $binary_remote_addr;
}
# ***********************
# SET RATE LIMITING ZONES
# ***********************
# --------------------------
# 2. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# limits for Zone $bad_bot = 1
# Nothing Set - you can set a different zone limiter here if you like
# We issue a 444 response instead to all bad bots.
# Rate limiting will only take effect if on any User-Agents with a value of 2
# limits for Zone $bad_bot = 2
# this rate limiting will only take effect if you change any of the bots and change
# their block value from 1 to 2.
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=6r/m;
### *** MAKE SURE TO ADD to your nginx.conf ***
### server_names_hash_bucket_size 64;
### server_names_hash_max_size 4096;
### limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
### limit_conn_zone $binary_remote_addr zone=addr:50m;
### to allow it to load this large set of domains into memory and to set the rate limiting zones for the DDOS filter.
# ==========================================
# END SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ==========================================
### THE END of the Long and Winding Road
# =====================
# END BLOCKER FUNCTIONS
# =====================
### Also check out my Ultimate Apache Bad Bot Blocker on Github
### https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
### --------------------------------------------
### HELP SUPPORT THIS PROJECT - Send Me a Coffee
### https://ko-fi.com/mitchellkrog
### --------------------------------------------
### FOR APACHE SERVERS
### ---------------------------------------------
### Check out the Ultimate Apache Bad Bot Blocker
### ---------------------------------------------

4
.dev-tools/globalblacklist-testing.template
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V3.2019.06.1644
### Updated: Thu Jun 27 13:12:23 SAST 2019
### Version: V3.2019.06.1645
### Updated: Thu Jun 27 13:24:56 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

4
.dev-tools/globalblacklist.template
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1644
### Updated: Thu Jun 27 13:12:21 SAST 2019
### Version: V4.2019.06.1645
### Updated: Thu Jun 27 13:24:55 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

2
.dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.192.85.2 1;
104.154.120.187 1;
127.0.0.1 1;

96
.dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,50 +1,50 @@
AiHitBot
Asterias
backlink-check
BlowFish
CheTeam
CrunchBot
Ebingbong
EyeNetIE
GermCrawler
Go-Ahead-Got-It
Gotit
GT::WWW
Humanlinks
Image Sucker
Indy Library
JennyBot
Larbin
Lmspider
Mass Downloader
netEstate NE Crawler
Nettrack
NetZIP
Nutch
OpenVAS
OutclicksBot
Aboundex
Bandit
CherryPicker
Collector
Craftbot
DomainAppender
EasyDL
EMail Siphon
Fimap
Firefox/7.0
GetWeb
Gigablast
GoZilla
Go!Zilla
HaosouSpider
HTMLparser
InfoNaviRobot
Jbrofuzz
LinkextractorPro
LinqiaMetadataDownloaderBot
LWP::Simple
Masscan
MFC_Tear_Sample
MIDown tool
Ninja
PageAnalyzer
PictureFinder
Pixray
scan.lol
SEOkicks-Robot
SiteSnagger
Site Sucker
Snapbot
Sottopop
Surfbot
Suzuran
Szukacz
Psbot
RankActiveLinkBot
RankFlex
RankingBot2
RankurBot
ScreenerBot
Searchestate
SemrushBot
SISTRIX
SpankBot
T8Abot
Titan
Toweyabot
trendiction.com
Webalta
WebAuto
WebImageCollector
WebmasterWorldForumBot
WebWhacker
WISENutbot
WWW::Mechanize
Zeus
ZmEu
Teleport
Tracemyfile
VB Project
VeriCiteCrawler
Wallpapers/3.0
WebCopier
WEBDAV
WebLeacher
WebsiteQuester
WeSEE
Whatweb
Xaldon_WebSpider
zgrab

276
.dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
ADmantX
404checker
80legs
Abonti
AhrefsBot
AIBOT
AiHitBot
Aipbot
Alexibot
AllSubmitter
AlphaBot
Anarchie
Apexoo
ASPSeek
Asterias
archive.org_bot
Attach
autoemailspider
BacklinkCrawler
BackStreet
BackWeb
Badass
BBBike
Bigfoot
Bandit
BetaBot
Blackboard
Blow
Boardreader
Bolt
Black Hole
BlackWidow
BLEXBot
BlowFish
BotALot
Buddy
BuiltBotTough
BuiltWith
BuzzSumo
CATExplorador
CazoodleBot
CCBot
Cegbfeieh
CherryPicker
CheTeam
ChinaClaw
CheeseBot
Chlooe
Claritybot
Cogentbot
Cliqzbot
cognitiveseo
Collector
com.plumanalytics
Copier
CopyRightCheck
Copyscape
Cosmos
Craftbot
crawl.sogou.com
Crescent
CrunchBot
CSHttp
Curious
DatabaseDriverMysqli
DataCha0s
demandbase-bot
Deusu
Digincore
DigitalPebble
Dirbuster
DIIbot
Disco
Discoverybot
Dispatch
DittoSpyder
DnyzBot
DomainSigmaCrawler
Dragonfly
Drip
DomainStatsBot
Dotbot
DSearch
DTS Agent
EirGrabber
EMail Siphon
EMail Wolf
EroCrawler
evc-batch
Express WebPictures
Extreme Picture Finder
Extractor
ExtractorPro
Ezooms
FDM
FemtosearchBot
FHscan
Fimap
Firefox/7.0
Flunky
FlashGet
Foobot
FyberSpider
Fyrebot
GalaxyBot
Genieo
Getintent
GetRight
GetWeb
Gigablast
Gigabot
G-i-g-a-b-o-t
Gotit
GrabNet
GrapeFX
Grafula
GrapeshotCrawler
GridBot
Haansoft
HEADMasterSEO
Heritrix
HTMLparser
Humanlinks
HybridBot
Iblog
HaosouSpider
Harvest
Hloader
HTTrack
Id-search
IlseBot
Image Fetch
Image Sucker
IndeedBot
Indy Library
InfoNaviRobot
InfoTekies
Intelliseek
Iskanie
JamesBOT
InternetSeer
IRLbot
Jbrofuzz
JikeSpider
JOC Web Spider
Jorgee
JustView
Jyxobot
Keyword Density
Lanshanbot
LexiBot
Lftp
LinkextractorPro
LinkpadBot
Lightspeedsystems
Likse
LinksManager
LinkWalker
LinqiaScrapeBot
LinqiaMetadataDownloaderBot
LinqiaRSSBot
Lipperhey
Lipperhey Spider
Ltx71
lwp-request
LWP::Simple
Mag-Net
Mail.RU_Bot
Majestic12
Majestic SEO
Majestic-SEO
MarkWatch
Masscan
Mass Downloader
Mata Hari
meanpathbot
Mediatoolkitbot
MegaIndex.ru
Metauri
MFC_Tear_Sample
Microsoft Data Access
Microsoft URL Control
MIDown tool
Mister PiX
MJ12bot
Mojeek
Morfeus Fucking Scanner
Mr.4x3
MSFrontPage
MSIECrawler
muhstik-scan
Musobot
Nameprotect
Navroad
Msrabot
MS Web Services Client Protocol
Needle
NetAnts
Netcraft
Nettrack
Nibbler
netEstate NE Crawler
NetLyzer
NetSpider
Netvibes
NICErsPRO
Niki-bot
NimbleCrawler
Nimbostratus
Ninja
Octopus
Offline Explorer
OnCrawl
Openvas
Nutch
oBot
Openfind
OpenVAS
OrangeBot
OrangeSpider
page scorer
OutclicksBot
Page Analyzer
PageGrabber
Panscient
Papa Foto
Pavuk
pcBrowser
PeoplePal
Picsearch
PictureFinder
Pi-Monster
Pimonster
Pixray
PleaseCrawl
plumanalytics
Pockey
POE-Component-Client-HTTP
Probethenet
PxBroker
QueryN Metasearch
Quick-Crawler
RankActiveLinkBot
RankActive
RankFlex
RankingBot
RankingBot2
Rankivabot
RebelMouse
RedesScrapy
Reaper
RepoMonkey
RocketCrawler
SBIder
Ripper
SalesIntelligent
ScanAlert
Scanbot
ScoutJet
SearchmetricsBot
scan.lol
Semrush
SemrushBot
SEOkicks
SEOkicks-Robot
SEOlyticsCrawler
SEOprofiler
Seomoz
seoscanners
SeoSiteCheckup
sexsearcher
SiteExplorer
SEOstats
serpstatbot
Shodan
Siphon
SISTRIX
Siteimprove
SiteSucker
SiteLockSpider
SiteSnagger
Site Sucker
Sitevigil
SlySearch
SMTBot
Snake
Snoopy
SocialRankIOBot
sogouspider
Sociscraper
Sogou web spider
Sosospider
Sottopop
SpaceBison
SpankBot
Spammen
Spanner
SputnikBot
spyfu
Sqlmap
Sqworm
Steeler
Surfbot
Sucker
Sucuri
SurveyBot
Suzuran
Szukacz
T0PHackTeam
Teleport
tAkeOut
Telesphoreo
Telesphorep
The Intraformant
TheNomad
Thumbor
Trendiction
Titan
Toata
Toweyabot
Trendictionbot
trendiction.com
trendiction.de
Turingos
TurnitinBot
TwengaBot
Twice
UnisterBot
URLy.Warning
Typhoeus
Upflow
Vacuum
VB Project
Vagabondo
VCI
VoidEYE
Voltron
VidibleScraper
Virusdie
Voil
Wallpapers/3.0
WallpapersHD
WBSearchBot
Webalta
WebBandit
WebCollage
Web Collage
WebCopier
Web Enhancer
Web Fuck
WebFuck
WebImageCollector
WebLeacher
Web Pix
WebPix
WebReaper
WebSauger
Web Sauger
Webshag
WebsiteQuester
Website Quester
Webster
WebStripper
WebZIP
WeSEE
Whacker
Whatweb
WinHTTrack
WISENutbot
Wprecon
Whack
Wonderbot
Woobot
WPScan
WWW-Collector-E
WWW-Mechanize
WWW::Mechanize
WWWOFFLE
x09Mozilla
x22Mozilla
Xaldon_WebSpider
Zade
Zauba
xpymep1.exe
zauba.io
Zermelo
Zitebot
ZmEu

266
.dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
360Spider
404checker
404enemy
Aboundex
ADmantX
AfD-Verbotsverfahren
AhrefsBot
AIBOT
AiHitBot
Aipbot
Apexoo
archive.org_bot
Attach
Badass
Alexibot
AlphaBot
Anarchie
Asterias
Backlink-Ceck
BacklinkCrawler
BackWeb
Bandit
Barkrowler
Battleztar Bazinga
BBBike
Bitacle
Bigfoot
Black Hole
BlackWidow
BotALot
Brandprotect
BLEXBot
Blow
BlowFish
Bolt
Brandwatch
Buddy
BuiltWith
Bullseye
CCBot
Calculon
Cegbfeieh
CheTeam
CheeseBot
CherryPicker
ChinaClaw
Chlooe
Claritybot
Cloud mapping
coccocbot-web
Cogentbot
cognitiveseo
com.plumanalytics
Collector
Copier
Cosmos
CopyRightCheck
Copyscape
Craftbot
crawler4j
crawl.sogou.com
CrazyWebCrawler
Custo
crawler.feedback
Crescent
DatabaseDriverMysqli
DataCha0s
DBLBot
demandbase-bot
Demon
Deusu
Devil
DigitalPebble
DIIbot
Disco
Discoverybot
Dispatch
DnyzBot
DomainAppender
DomainSigmaCrawler
Dotbot
Drip
EasyDL
Ebingbong
eCatch
ECCP/1.0
EirGrabber
EMail Siphon
EMail Wolf
EroCrawler
ExtLinksBot
Evil
Express WebPictures
Extractor
ExtractorPro
Extreme Picture Finder
EyeNetIE
Ezooms
facebookscraper
FemtosearchBot
FHscan
Firefox/7.0
FlashGet
Foobot
FyberSpider
Genieo
Getintent
Freeuploader
Fyrebot
GalaxyBot
GermCrawler
GetRight
GetWeb
Gotit
Go!Zilla
Grabber
GoZilla
GrabNet
Grafula
GrapeshotCrawler
GridBot
Haansoft
HaosouSpider
Havij
HMView
HTMLparser
HTTP::Lite
HTTrack
Humanlinks
Iblog
IDBot
Id-search
IlseBot
IndeedBot
Indy Library
InfoTekies
instabid
InternetSeer
ips-agent
Intelliseek
Internet Ninja
internetVista monitor
Iria
IRLbot
Iskanie
JamesBOT
JennyBot
Jbrofuzz
JetCar
Jetty
Joomla
Jorgee
JustView
Jyxobot
Kenjin Spider
Larbin
LeechFTP
LexiBot
Lightspeedsystems
LinkpadBot
Keyword Density
Kozmosbot
Libwhisker
Linkdexbot
LinkextractorPro
LinkScan
LinksManager
LinqiaMetadataDownloaderBot
LinqiaRSSBot
LinqiaScrapeBot
Lipperhey
Lipperhey Spider
Litemage_walker
Lmspider
LNSpiderguy
Ltx71
lwp-trivial
Magnet
magpie-crawler
Mail.RU_Bot
Mag-Net
MarkWatch
Masscan
Mass Downloader
Mata Hari
MauiBot
Meanpathbot
MeanPath Bot
Mediatoolkitbot
mediawords
Metauri
Microsoft Data Access
MIDown tool
MIIxpc
MJ12bot
Mojeek
Morfeus Fucking Scanner
Mr.4x3
muhstik-scan
Musobot
Name Intelligence
Nameprotect
Navroad
Needle
Nessus
Netcraft
Nettrack
Net Vampire
Netvibes
Nibbler
NICErsPRO
Niki-bot
Nikto
Ninja
Nutch
Openfind
OpenLinkProfiler
Nmap
Octopus
Openvas
OpenVAS
OutclicksBot
OrangeBot
OrangeSpider
OutfoxBot
PageGrabber
PageScorer
Pandalytics
Panscient
PECL::HTTP
PeoplePal
PageAnalyzer
Page Analyzer
Papa Foto
pcBrowser
PHPCrawl
Picscout
ProPowerBot
Pixray
plumanalytics
Pockey
POE-Component-Client-HTTP
ProWebWalker
Pump
PxBroker
QueryN Metasearch
PyCurl
Quick-Crawler
RankActive
RankFlex
RankingBot2
RankurBot
RealDownload
Reaper
Rankivabot
RebelMouse
Recorder
s1z.ru
RedesScrapy
ReGet
SalesIntelligent
ScanAlert
scan.lol
ScreenerBot
SearchmetricsBot
SBIder
Screaming
Searchestate
SemrushBot
SEOkicks-Robot
SEOstats
SEOkicks
SeoSiteCheckup
Siphon
SISTRIX
SiteExplorer
Sitebeam
Siteimprove
SiteLockSpider
SiteSnagger
SiteSucker
Site Sucker
SlySearch
Snake
Snapbot
Snoopy
SocialRankIOBot
Sociscraper
sogouspider
Sogou web spider
Sottopop
SpaceBison
Spanner
Spammen
sp_auditbot
SputnikBot
Sqlmap
Sqlworm
Sqworm
Steeler
SuperHTTP
Surfbot
SurveyBot
Stripper
Sucker
Swiftbot
T8Abot
tAkeOut
Teleport
Telesphoreo
TeleportPro
Telesphorep
Thumbor
Toweyabot
Tracemyfile
trendiction.de
TurnitinBot
TwengaBot
Twice
The Intraformant
Titan
Trendiction
Trendictionbot
UnisterBot
Upflow
URLy.Warning
URLy Warning
Vacuum
VCI
Vagabondo
VB Project
VeriCiteCrawler
Virusdie
WBSearchBot
Voil
Voltron
Web Auto
WebBandit
WebCollage
Web Collage
WebCopier
WEBDAV
WebEnhancer
WebFetch
Web Enhancer
Web Fetch
WebFuck
WebLeacher
webmeup-crawler
Web Pix
WebReaper
WebSauger
Web Sauger
Webshag
WebsiteExtractor
WebsiteQuester
Website Quester
WebStripper
WebWhacker
WebZIP
Whatweb
WeSEE
Whacker
Who.is Bot
WinHTTrack
WISENutbot
Woobot
Wotbox
WWW-Mechanize
WPScan
WWW::Mechanize
x09Mozilla
x22Mozilla
Xaldon_WebSpider
Xenu
xpymep1.exe
Zauba
zauba.io
Zermelo
YoudaoBot
Zade
Zitebot
ZmEu

198
.dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,100 +1,100 @@
3th.co.in
5i2.net
academia-nsk.org
adtech.de
advancedmassagebysara.com
ahmedabadwebs.com
akama.com
amt-k.ru
amyfoxfitness.com
anime.dougasouko.com
apifasterlightin-a.akamaihd.net
ap.senai.br
asrvrep-a.akamaihd.net
bannerconnect.net
bear.gotcher.us
bet-prognoz.com
blackhatworth.com
blavia.00author.com
bloglag.com
bluesalt.co
briomotor.co
bugof.gq
buntube.net
cementaresearch.se
chatroulette.life
chatseo.com
cowblog.fr
crynet.cc
demenageur.com
divci-hry.info
domain.webkeyit.com
doublepimp.com
ee77ee.com
elektrozigaretten1.postbit.com
eshop.md
exchanges-bet.com
extlabs.io
freshberry.com.ua
googlemare.com
gotcher.us
gpirate.com
hermesreplica.win
hitsbox.info
hol.es
igadgetsworld.com
immobiliaremassaro.com
interfucks.net
irkutsk.zrus.org
kinohall.ru
klikbonus.com
likrot.com
lottospring.com
monarchfind-a.akamaihd.net
montredemarque.nl
mrbitsandbytes.com
mvpicton.co.uk
myhealthcare.com
myxdate.info
nac-bearings.ru
new-post.tk
nsatc.net
offf.info
oneclickfiles.com
partner-host.men
partnerline.men
pastaleads.com
pawli.eu
pochemychka.net
pornoblood.com
pornosee.info
porno-video-chati.ru
remont-comp-pomosh.ru
ring4rhino.com
royalads.net
00it.com
24x7-server-support.site
4replicawatch.net
7makemoneyonline.com
adult3dgames.com
ad-words.ru
arvut.org
axbocz.net
becuo.com
bildsuche.ru
buyessay3.blogspot.ru
buyfriend.ru
club-musics.ru
collegeessay19.blogspot.ru
compliance-donald.xyz
custom-product-labels.com
downloadeer.net
euroskat.ru
everypony.ru
ezigarettenkaufen2.dreamwidth.org
felizporno.com
flipper.top
flowersbazar.com
fullgirl.ru
geoads.com
gloverid.site
gosreg.amchs.ru
gotwebsite1.com
graphid.com
growshop.es
gsasearchengineranker.top
gsmtlf.ru
hao123.com
hostnow.men
ideibiznesa2015.ru
incanto.in.ua
infogame.name
inspiring-desperate.tk
iomoio.net
iscblog.info
istizanidineanopiate.blogspot.com
joingames.org
journeydownthescale.info
juliadiets.com
kadashihotel.com
krasnodar.zrus.org
lapitec.eu
lockerz.com
lovi-moment.com.ua
mesto-x.com
mirtorrent.net
mmostrike.ru
moonci.ru
mtmtv.info
mypornfree.ru
narutonaruto.ru
notasprensa.info
o-o-6-o-o.ru
pattersonsweb.com
pdamods.ru
pflexads.com
pixelrz.com
pony-business.com
potoideas.us
promodj.com
pskcijdc.bloger.index.hr
razorweb-a.akamaihd.net
richinvestmonitor.com
ric.info
rimedia.org
rockprogblog.com
rucrypt.com
sanatorrii.ru
sbricur.com
seojokes.net
socialseet.ru
soc-proof.su
soheavyblog.com
sonata-arctica.wz.cz
soviet-portal.do.am
spb.afora.ru
spidtest.org
stroiminsk.org
tackletarts.co
thefarmergame.com
torrentgamer.net
tracfone.com
trichizobswiv.agddns.net
umityangin.net
urlopener.blogspot.com.au
vertaform.com
vkmusics.ru
webjam.com
whiteproduct.com
xn----itbeirbjbi7bc6bh2d.xn--p1ai
xtube.com
youdao.com
zeroredirect8.com
zverokruh-shop.cz
sharebutton.org
shemalegalls.blogporn.in
siteheart.net
sitevalued.com
skyway24.ru
snow.nvr163.com
sptslmtrafms.com
sugarkun.com
taaaak.com
thebestphotos.eu
thedownloadfreeonlinegames.blogspot.com
topshef.ru
turizmus.us
uggbootsoutletsale.us
unblocksit.es
uptimebot.net
videochat.ph
vzubah.com
winterclassichockeyjerseys.com
winx-play.ru
woman-h.ru
xaijo.com
yellowstonesafaritours.com
zixizop.net.ru
zynax.ua
zzbroya.com.ua

1710
.dev-tools/test_units/random-referrers-for-test.list
View file

File diff suppressed because it is too large Load diff

480
.dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
2020iscoming.info
24videos.tv
256bit.by
7zap.com
adktrailmap.com
ads-cool.pro
adultactioncam.com
advancedsoftwaresupport.com
ad-words.ru
alekseevec.ru
allblogroll.com
aosexkontakte.net
apartmentbay.ru
apibetweenlinesn-a.akamaihd.net
atmagroup.ru
audiofree.ru
auto4style.ru
autochoixspinelli.com
avtovolop.ru
ayakino.net
azadnegar.com
batanga.net
battle.net
benchmarkcommunications.co.uk
berrymall.ru
betonka.pro
bif-ru.info
bigames.online
binaryoptionscops.info
bio-market.kz
blackwitchcraft.ru
bolezniorganov.ru
bongacams.com
bookmaker-bet.com
bosman.pluto.ro
1001watch.com.ua
1688.com
4webmasters.org
8xv8.com
abiente.ru
academiacsmendoza.org
acc.eu.org
actulite.com
adclickthru.net
adsloads.com
adtech.fr
adultfriendfinder.com
adultfullhd.com
aktivator-windows10.blogspot.com
alarmobninsk.ru
alessandraleone.com
all4invest.info
allkrim.com
alpinism.ru
amazingpic.net
ameblo.jp
animal-drawings.com
apartamentwroclaw.eu
api.stathat.com
apparel-offer.com
autobudpostach.club
aviav.ru.com
azazu.ru
babieca.com
babyfactory.fr
bdsmgalls.net
beclean-nn.ru
bedandbreakfast.com
best-deals-products.com
besttorrentknifta.weebly.com
bezlimitko.xyz
binomo.com
bloggers.nl
bobba.dzaba.com
brakehawk.com
brimstonehillfortress.org
bristolhotel.com.ua
budilneg.xyz
buigas.00it.com
buzzonclick.com
calcularpagerank.com
canacopegdl.com
candycrushshop.com
chiblackhawks-jerseys.com
cialis-samples.com
ciproandtizanidine.blogspot.com
clickcash.com
clodo.ru
codysbbq.com
compliance-alexa.xyz
custom-product-labels.com
dailyfinancefix.com
dalavia.ru
dandiyabeats.in
dealwifi.com
defenderxtactical.com
dekoration.us
buketeg.xyz
businesn.men
buy-cheap-pills-order-online.com
candypeople.se
cartierlove2u.xyz
cartujano-pre.de
ccbill.com
chatseo.com
cheap-pills-norx.com
chelnytruck.ru
christianlouboutinshoes.xyz
colehaanoutlet.store
community.allhiphop.com
compliance-barak.top
compliance-barak.xyz
compliance-olga.top
comsysnet.com
crynet.cc
d2jsp.org
data1.scopich.com
dekorkeramik.ru
destinationrealestate.com
detmebel.su
dlya-android.org
dominateforex.ml
dptaughtme.com
elektrischezigarette1.blog.pl
elkacentr.ru
elmifarhangi.com
elvel.com.ua
enternet.ee
estelight.ru
european-torches.ru
dengi-pod-zalog-nedvizhimosti.ru
den-noch24.ru
diusyawiga.tk
dmmspy.com
dnsrsearch.com
docsportal.net
dojki-hd.com
domaineaneblanc.com
donna7753191.ru
dvrlists.com
editors.choice6912650.hulfingtonpost.com
elektronischezigarettekaufen1.myblog.de
encodable.com
engines-usa.com
enhand.se
eralph.tk
evogarage.com
extstat.com
fbdownloader.com
fdzone.org
fickkontaktehobbyhuren.com
fickluder69.com
fix-website-errors.com
formulaantiuban.com
forum-engineering.ru
free-today.com
freza-sverlo.ru
funcrushgames.com
gazoblok.net.ua
geileweiber.tk
gfaq.ru
goroda-vsego-mira.ru
grtyi.com
handsandlegs.ru
havepussy.com
hornymatches.com
hothor.se
houseofgaga.ru
ideoworld.org
igrovyeavtomaty777.ru
ilovevitaly.com
imagerydatabase.com
impotentik.com
instakink.com
int.search.tb.ask.com
it-max.com.ua
jamiembrown.com
jeremyeaton.co
jpcycles.com
juliaworld.net
kanimage.com
karadene.com
keywordsdoctor.com
kidskunst.info
lafriore.ru
landinez.co
larchik.net
laserpen.club
leadwayau.com
lenvred.org
lescinq.com
letslowbefast.today
lflash.ru
lider-zhaluzi.kiev.ua
lignofix.ua
linkpulse.com
linkredirect.biz
linksharingt.com
littleberry.ru
mac-shield.com
master-muznachas.ru
exdocsfiles.com
extremepornos.net
ezigarettekaufen2.yolasite.com
f07.de
filmi-v.online
forum.tvmir.org
freejabs.com
galeon.com
gamblingnerd.com
game-top.su
gamewrath.com
gamezblox.com
gemara.com
generousdeal-a.akamaihd.net
getyourimage.club
glavprofit.ru
gojiberriess.apishops.ru
grizzlysgrill.com
gsasearchengineranker.pw
habermetre.com
hidemyass.com
homemature.net
hoverboard360.es
ilmexico.com
imgarcade.com
inspiring-desperate.tk
instasexyblog.com
investyb.com
invivo.hu
istizanidineanarcoticdrug.blogspot.com
istizanidineanopiate.blogspot.com
ivoiretechnocom.ci
iwantmyfreecash.com
iyasimasennka.com
japfm.com
jav-way.site
jerseychinabizwholesale.us
job.icivil.ir
journalhome.com
kakablog.net
kino-ecran.ru
koleso24.com.ua
komputernaya-pomosh-moscow.ru
krasnodar.zrus.org
krassh.ru
kupiproday.com.ua
kursy-ege.ru
law-check-seven.xyz
letsrepair.in
locksmith.jp
madot.onlinewebshop.net
massage-info.nl
matb3aa.com
matchpal-a.akamaihd.net
m.b00kmarks.com
mbiologi.ru
mesto-x.com
metarip.ru
minegam.com
mojpreskumanie.com
moneymaster.ru
mielec.pl
mir-betting.ru
mmoguider.ru
mnogolok.info
moinozhki.com
monclerboots.xyz
monetizer.com-01.site
money-for-placing-articles.com
montazhnic.ru
moroccosurfadventures.com
myfreecams.com
mylida.org
myonigroup.com
myplaycity.com
nextlnk12.com
nikhilbahl.com
november-lax.com
oconto.ru
olgacvetmet.com
onstrapon.purplesphere.in
orgasmatrix.com
ororodnik.goodbb.ru
osb.se11.ru
paclitor.com
pandarastore.top
pandroid.co
partner-host.men
picsearch.com
pinkduck.ga
playfortuna-play.ru
pochtovyi-index.ru
podshipniki-nsk.ru
pony-business.com
pornoblood.com
moviemail-online.co.uk
music7s.me
mypets.by
narkologiya-belgorod.ru
narosty.com
newstraveller.ru
nfljerseysforsalewholesaler.com
nfljerseys.online
normalegal.ru
notfastfood.ru
novodigs.com
nvssf.com
obnal.org
onlainbesplatno.ru
onlinewritingjobs17.blogspot.ru
orion-code-access.net
osoznanie-narkotikam.net
paleohub.info
pensplan4u.com
photosaga.info
pic2fly.com
piccdata.com
picquery.com
picsforkeywordsuggestion.com
pictures-and-images.net
pierrehardysale.online
pipki.r.acdnpro.com
pornobrazzers.biz
pornofiljmi.com
prchecker.info
potolokelekor.ru
pozdrawleniya.ru
predmety.in.ua
primedice.com
privacyassistant.net
prizesbook.online
profitkode.com
propranolol40mg.blogspot.com
ptr.ruvds.com
pukaporn.com
rasteniya-vs-zombi.ru
realitykings.com
remontvsamare.su
responsive-test.net
rockingclicks.com
sad-torg.com.ua
scanmyphones.com
scanner-margo.top
scanner-mary.top
searchengineranker.email
searchimpression.com
searchmywindow-a.akamaihd.net
seccioncontrabajo.com
security60-e.com
seeresultshub-a.akamaihd.net
seorank.info
serpstat.com
sexblog.pw
sexkontakteao.info
sex-watch.com
sfj-ror.no
shiksabd.com
shoesonlinebuy.cn
shoppingmiracles.co.uk
sideeffectsoftizanidine.blogspot.com
simul.co
sisiynas.ru
site-auditor.online
slowmac.tech
smokewithrabbits.com
snworks.com
socialbookmarksubmission.org
softxaker.ru
souvenir.cc
sticken.co
stretchingabuckblog.com
strigkaomsk.ru
stroicol.net
suchenindeutschland.com
superstats.com
susanholtphotography.com
swinger-mobil.net
swiped.su
push-ad.com
pushdata.sendpulse.com
qitt.ru
qld10000.net
qualitymarketzone.com
quelle.ru
razleton.com
refererx.com
rennlist.com
ritlweb.com
roznica.com.ua
runtnc.net
rvtv.ru
sbprabooks.com
scanmarine.info
scanner-jess.top
scanner-willy.top
scrapinghub.com
scripted.com
seeingmeerkat.com
segol.tv
semaltmedia.com
sexs-foto.com
sfd-chess.ru
shopfishing.com.ua
skylta.com
smart-balancewheel.com
social-s-iii.xyz
sonata-arctica.wz.cz
soundfrost.org
sukirgenk.dvrlists.com
sundrugstore.com
sweepstakes.rewardit.com
syvertsen-da.no
tamada69.com
tfxiq.com
thefarmergame.com
thegamerznetwork.com
thetardistimes.ovh
thexart.club
tizanidineduringpregnancy.blogspot.com
tizanidineformigraines.blogspot.com
tizanidineingredients.blogspot.com
tjkckpytpnje.com
tmearegion26.com
theguardlan.com
tmtrck.com
tootoo.to
topclickguru.com
track-rankings.online
traffixer.com
trailer.cinemaflix.website
tx41tclega.ru
uhdtv.website
ultimatesetnewfreeallsoftupgradesystems.pw
uogonline.com
uprour.com
ussearche.cf
v24s.net
vapomnoncri.tk
videochat.cafe
vintontech.info
vinylvault.co.uk
webenlace.com.ar
webshoppermac.com
website-analytics.online
topshef.ru
tourismvictoria.com
traffmonster.info
tripper.de
turkeyreport.tk
twu.com.ua
uptime.com
urzedowski.eu
veopornogratis.xxx
villakohlanta.nu
vkak.ru
vriel.batcave.net
vsesubwaysurfers.com
wait3sec.org
watchmyfb.pl
waterefficiency.co
weprik.ru
wetgames.ru
wjgony.com
w-journal.ru
wladimirpayen.com
www.888.com
wwwadultcheck.com
wygraj-skiny.win
x-diesel.info
xn----itbkqkfiq.xn--p1ai
xtrafficplus.com
your-bearings.com
youtubedownload.org
zoogdisany.com
zrizvtrnpale.tk
word-vorlagen.net
worldhistory.biz
www.kabbalah-red-bracelets.com
xboxster.ru
x-mix.info
xn----7sbho2agebbhlivy.xn--p1ai
xn--e1aggki3c.xn--80adxhks
xn--h1ahbi.com.ua
yeartwit.com
youtoner.it
zdesoboi.com
zeroredirect5.com
zigzog.ru
zoogdinsney.com

99
.dev-tools/test_units/ratelimittest.txt
View file

@ -0,0 +1,99 @@
<!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html>

2
.dev-tools/test_units/whitelist-ips.conf
View file

@ -1,4 +1,4 @@
35.192.85.2 0;
104.154.120.187 0;
127.0.0.1 1;
127.0.0.1 1;
127.0.0.1 1;

BIN
.latest_release/bots.d.tar.gz
View file

Binary file not shown.

BIN
.latest_release/conf.d.tar.gz
View file

Binary file not shown.

2
AUTO-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________
#### Version: V4.2019.06.1644
#### Version: V4.2019.06.1645
#### Bad Referrer Count: 6713
#### Bad Bot Count: 556
____________________

2
MANUAL-CONFIGURATION.md
View file

@ -4,7 +4,7 @@
### PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY :exclamation:
_______________
#### Version: V4.2019.06.1644
#### Version: V4.2019.06.1645
#### Bad Referrer Count: 6713
#### Bad Bot Count: 556
____________________

2
README.md
View file

@ -11,7 +11,7 @@
##### The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking. Stop and Block all kinds of bad internet traffic from ever reaching your web sites. [PLEASE SEE: Definition of Bad Bots](#define-bad-bots)
_______________
#### Version: V4.2019.06.1644
#### Version: V4.2019.06.1645
#### Bad Referrer Count: 6713
#### Bad Bot Count: 556
____________________

2
_google_analytics_ghost_spam/README.md
View file

@ -3,7 +3,7 @@
# EASY CONFIGURATION INSTRUCTIONS FOR STOPPING GOOGLE ANALYTICS "GHOST" SPAM
_______________
#### Version: V4.2019.06.1644
#### Version: V4.2019.06.1645
#### Bad Referrer Count: 6713
#### Bad Bot Count: 556
____________________

4
_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1644
### Updated: Thu Jun 27 13:12:21 SAST 2019
### Version: V4.2019.06.1645
### Updated: Thu Jun 27 13:24:55 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

4
conf.d/globalblacklist-testing-version.conf
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V3.2019.06.1644
### Updated: Thu Jun 27 13:12:23 SAST 2019
### Version: V3.2019.06.1645
### Updated: Thu Jun 27 13:24:56 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

4
conf.d/globalblacklist.conf
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1644
### Updated: Thu Jun 27 13:12:21 SAST 2019
### Version: V4.2019.06.1645
### Updated: Thu Jun 27 13:24:55 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

4
robots.txt/robots.txt
View file

@ -6,8 +6,8 @@
### Version Information #
###################################################
### Version: V4.2019.06.1644
### Updated: Thu Jun 27 13:12:24 SAST 2019
### Version: V4.2019.06.1645
### Updated: Thu Jun 27 13:24:58 SAST 2019
### Bad Bot Count: 556
###################################################
### Version Information ##