vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-10 15:34:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

V4.2019.06.1634 [ci skip]

Browse source
This commit is contained in:
Travis 2019-06-27 10:52:35 +02:00
parent 6fac493778
commit 8125dcded8
39 changed files with 19498 additions and 2085 deletions
Download patch file Download diff file Expand all files Collapse all files

5
.dev-tools/_conf_files_test1/bots.d/blockbots.conf
View file

@ -38,10 +38,11 @@
# More IP's can be added example > "(127.0.0.1)|(192.168.0.1)|(192.168.1.1)"
# If you even blacklisted 127.0.0.1 or your own IP by giving it a value of 1 in any of the includes, this will over-ride that block.
if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
# UNCOMMENT THE NEXT 4 LINES TO ACTIVATE THE SUPER WHITELIST
#if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
#set $bad_bot '0'; #Uncommenting this line will disable bad_bots functionality for specified IP(s)
#set $validate_client '0'; #Uncommenting this line will disable validate_client ip blocking functionality for specified IP(s)
}
#}
# --------------
# BLOCK BAD BOTS

2
.dev-tools/_conf_files_test1/bots.d/whitelist-ips.conf
View file

@ -33,4 +33,4 @@
# 111.111.111.111 0;
35.192.85.2 0;
34.66.178.120 0;

4
.dev-tools/_conf_files_test1/conf.d/globalblacklist.conf
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1633
### Updated: Thu Jun 27 09:10:20 SAST 2019
### Version: V4.2019.06.1634
### Updated: Thu Jun 27 10:50:24 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

5
.dev-tools/_conf_files_testing_changes/bots.d/blockbots.conf
View file

@ -38,10 +38,11 @@
# More IP's can be added example > "(127.0.0.1)|(192.168.0.1)|(192.168.1.1)"
# If you even blacklisted 127.0.0.1 or your own IP by giving it a value of 1 in any of the includes, this will over-ride that block.
if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
# UNCOMMENT THE NEXT 4 LINES TO ACTIVATE THE SUPER WHITELIST
#if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
#set $bad_bot '0'; #Uncommenting this line will disable bad_bots functionality for specified IP(s)
#set $validate_client '0'; #Uncommenting this line will disable validate_client ip blocking functionality for specified IP(s)
}
#}
# --------------
# BLOCK BAD BOTS

2
.dev-tools/_conf_files_testing_changes/bots.d/whitelist-ips.conf
View file

@ -33,4 +33,4 @@
# 111.111.111.111 0;
35.192.85.2 0;
34.66.178.120 0;

447
.dev-tools/_conf_files_testing_changes/conf.d/globalblacklist.conf
View file

@ -1,22 +1,21 @@
### **********************************
### THE ULTIMATE NGINX BAD BOT BLOCKER
### **********************************
### !!!!!!!!!!!!!
### TEST TEMPLATE
### THIS VERSION IS USED FOR TESTING ANY POTENTIAL BREAKING CHANGES
### !!!!!!!!!!!!!
### -----------------------------------------------------------
### THE NGINX ULTIMATE BAD BOT, BAD IP AND BAD REFERRER BLOCKER
### -----------------------------------------------------------
### VERSION INFORMATION #
###################################################
### Version: V3.2019.06.1633
### Updated: Thu Jun 27 09:10:21 SAST 2019
### Version: V3.2019.06.1634
### Updated: Thu Jun 27 10:50:26 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################
### VERSION INFORMATION ##
### --------------------------------------------
### HELP SUPPORT THIS PROJECT - Send Me a Coffee
### https://ko-fi.com/mitchellkrog
### --------------------------------------------
##############################################################################
# _ __ _ #
# / |/ /__ _(_)__ __ __ #
@ -34,12 +33,16 @@
### and also has rate limiting functionality for bad bots who you only want to rate limit
### and not actually block out entirely. It is very powerful and also very flexible.
### --------------------------------------------------------------------------
### Created By: https://github.com/mitchellkrogza/
### Repo Url: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
### Copyright Mitchell Krog - <mitchellkrog@gmail.com>
### Contributors: Stuart Cardall - https://github.com/itoffshore
### --------------------------------------------------------------------------
### --------------------------------------------------------------------------
### Tested on: nginx/1.10.3 up to latest Mainstream Version (Ubuntu 16.04)
### --------------------------------------------------------------------------
### This list was developed and is in use on a live Nginx server running some very busy web sites.
### It was built from the ground up using real data from daily logs and is updated almost daily.
@ -47,9 +50,9 @@
### spam referrers, rogue IP address, scanners, scrapers and domain hijacking sites are extensively checked
### before they are added. It is monitored extensively for any false positives.
### *********
### ---------
### Features:
### *********
### ---------
### Clear formatting for Ease of Maintenance.
### Alphabetically ordered lists for Ease of Maintenance.
### Extensive Commenting for Ease of Reference.
@ -60,40 +63,38 @@
### IP range blocking / whitelisting.
### Rate Limiting Functions.
### ************
### ------------
### INSTALLATION
### ************
### ------------
### PLEASE use the install, setup and update scripts provided for you to ease your installation.
### This Auto Installation procedure is documented in the README.md and AUTO-CONFIGURATION.md files.
### Installation, Setup and Update Scripts Contributed by Stuart Cardall - https://github.com/itoffshore
### There are also manual configuration instructions provided for those not wishing to do an auto install.
### ***********************************************
### -----------------------------------------------
### !!!!! PLEASE READ INLINE NOTES ON TESTING !!!!!
### ***********************************************
### -----------------------------------------------
### Note that:
### SETTINGS:
### ---------------------------------------------
### 0 = allowed - no limits
### 1 = allowed or rate limited less restrictive
### 2 = rate limited more
### 3 = block completely
### ---------------------------------------------
### ************************************************************
### ------------------------------------------------------------
### CONTRIBUTING / PULL REQUESTS / ADDING YOUR OWN BAD REFERRERS
### ************************************************************
### ------------------------------------------------------------
### For contributing, corrections or adding bots or referrers to this repo,
### Send a Pull Request (PR) on any of the .list files in the _generator_lists folder
### All Pull Requests will be checked for accuracy before being merged.
# *********************************
# FIRST BLOCK BY USER-AGENT STRINGS
# *********************************
# ***********************
# -----------------------
# !!!!! PLEASE TEST !!!!!
# ***********************
# -----------------------
# ALWAYS test any User-Agent Strings you add here to make sure you have it right
# Use a Chrome Extension called "User-Agent Switcher for Chrome" where you can create your
@ -104,36 +105,46 @@
# curl -I http://www.yourdomain.com -A "GoogleBot" ---- GIVES YOU: HTTP/1.1 200 OK (Meaning web page was served to Client)
# curl -I http://www.yourdomain.com -A "80legs" ---- GIVES YOU: curl: (52) Empty reply from server (Meaning Nginx gave a 444 Dropped Connection)
# In this section we allow/disallow specific User Agents / Bots.
# *********************************************************************
# Now we map all good and bad user agents to a variable called $bad_bot
# *********************************************************************
# =======================
# START BLOCKER FUNCTIONS
# =======================
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# DO NOT EDIT ANYTHING BELOW THIS LINE !!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# =============================
# BEGIN SECTION 1 - USER-AGENTS
# =============================
# ALLOW / BLOCK User Agents / Bots
# -------------------------------------------------------------------
# Map all GOOD and BAD UA (User Agents) to a variable called $bad_bot
# -------------------------------------------------------------------
map $http_user_agent $bad_bot {
default 0;
# ***********************************************
# Include your Own Custom List of Bad User Agents
# ***********************************************
# Use the include file below to further customize your own list of additional user-agents you wish to permanently block
# This include file allows whitelisting and blacklisting of anything specified below it, meaning you can over-ride anything in this blocker to your liking.
# ***********************************************************************************
# -----------------------------------------------------------------------------------
# START CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***********************************************************************************
# -----------------------------------------------------------------------------------
# Include your Own Custom List of Bad User Agents
# Use the include file below to further customize your own list of additional user-agents you wish to permanently block
# This include file allows whitelisting and blacklisting of anything specified below it.
# This include file alows you to over-ride any Bad / Good UA (Bot) declared in this blocker to your liking.
include /etc/nginx/bots.d/blacklist-user-agents.conf;
# *********************************************************************************
# ---------------------------------------------------------------------------------
# END CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *********************************************************************************
# ---------------------------------------------------------------------------------
# *********************************************
# Bad User-Agent Strings That We Block Outright
# *********************************************
# --------------------------------------------------
# BAD UA (User-Agent) Strings That We Block Outright
# --------------------------------------------------
# START BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)360Spider(?:\b|)" 3;
@ -694,9 +705,16 @@ map $http_user_agent $bad_bot {
"~*(?:\b)ZyBorg(?:\b|)" 3;
# END BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# ***********************************************
# Allow Good User-Agent Strings We Know and Trust
# ***********************************************
# --------------------------------------------
# GOOD UA User-Agent Strings We Know and Trust
# --------------------------------------------
# -----------------------------------------------------------------------
# You can over-ride these in /etc/nginx/bots.d/blacklist-user-agents.conf
# by adding the same UA line there and chaning its value of 1
# If you think GoogleBot is bad you would simply add them to
# blacklist-user-agents.conf with a value of 1
# -----------------------------------------------------------------------
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)adidxbot(?:\b|)" 0;
@ -733,12 +751,13 @@ map $http_user_agent $bad_bot {
"~*(?:\b)yahoo(?:\b|)" 0;
# END GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# ***************************************************
# User-Agent Strings Allowed Through but Rate Limited
# ***************************************************
# --------------------------------------------------------
# GOOD UA User-Agent Rate Limiting 1 - Disabled by Default
# --------------------------------------------------------
# Some people block libwww-perl, it used widely in many valid (non rogue) agents
# I allow libwww-perl as I use it for monitoring systems with Munin but it is rate limited
# TO ACTIVATE THIS RATE LIMITING Uncomment these two lines in blockbots.conf
#limit_conn bot1_connlimit 100;
#limit_req zone=bot1_reqlimitip burst=50;
# START ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)jetmon(?:\b|)" 1;
@ -749,9 +768,14 @@ map $http_user_agent $bad_bot {
"~*(?:\b)Wget/1.15(?:\b|)" 1;
# END ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
# **************************************************************
# Rate Limited User-Agents who get a bit aggressive on bandwidth
# **************************************************************
# -------------------------------------------------------
# GOOD UA User-Agent Rate Limiting 2 - Enabled by Default
# -------------------------------------------------------
# -----------------------------------------------------------------------
# You can over-ride these in /etc/nginx/bots.d/blacklist-user-agents.conf
# by adding the same UA line there and chaning its value of 1
# -----------------------------------------------------------------------
# START LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)Alexa(?:\b|)" 2;
@ -774,163 +798,186 @@ map $http_user_agent $bad_bot {
# END LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
}
# *****************************************
# SECOND BLOCK BY REFERRER STRINGS AND URLS
# *****************************************
# ===========================
# END SECTION 1 - USER-AGENTS
# ===========================
# *****************
# =======================================
# BEGIN SECTION 2 - REFERRERS AND DOMAINS
# =======================================
# ----------------
# PLEASE TEST !!!!
# *****************
# ----------------
# ------------------------------------------------------------------------------------------------------------------------------
# ALWAYS test referrers that you add. This is done manually as follows
# ------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://anything.adcash.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://www.goodwebsite.com/not-adcash --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://www.betterwebsite.com/not/adcash --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://www.google.com --- GIVES YOU: full html output of the web page
# curl -I http://www.yourdomain.com -e http://www.microsoft.com --- GIVES YOU: full html output of the web page
# ------------------------------------------------------------------------------------------------------------------------------
# Because of case-insensitive matching any combination of capitilization in the names will all produce a positive hit
# make sure you always test thoroughly and monitor logs. This section below also does NOT check for a preceding www.
# and it also does not care if the referrer request was sent with http https or even ftp.
# ------------------------------------------------------------------------------------------------------------------------------
# ***********************************************************************
# Now we map all bad referrer words below to a variable called $bad_words
# ***********************************************************************
# ----------------------------------------------------------------
# Map all BAD referrer words below to a variable called $bad_words
# ----------------------------------------------------------------
# --------------------------------
# START Bad Referrer Word Scanning
# --------------------------------
map $http_referer $bad_words {
default 0;
# **************************
# Bad Referrer Word Scanning
# **************************
# -------------------------------------------------------------------------------------------
# These are Words and Terms often found tagged onto domains or within url query strings.
# Create and Customize Your Own Bad Referrer Words Here using the new Include File Method
# New Method Uses the include file below so that when pulling future updates your
# customized list of bad referrer words are automatically now included for you
# Read Comments inside bad-referrer-words.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# BE VERY CAREFUL using the bad-referrer-words.conf file - please read the comment and
# BE VERY CAREFUL using this bad-referrer-words.conf file - please read the comments and
# examples inside the include file for detailed explanations into how seriously this can
# affect your own site from serving assets or other innocent sites from accessing your site
# For safety sake the whitelist-domains.conf file is also loaded here before the bad-referrer-words.conf file.
# For safety sake the whitelist-domains.conf file is also loaded here before the
# bad-referrer-words.conf file is loaded.
# -------------------------------------------------------------------------------------------
# ************************************************************************
# ------------------------------------------------------------------------
# START WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-domains.conf;
# **********************************************************************
# ----------------------------------------------------------------------
# END WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **********************************************************************
# ----------------------------------------------------------------------
# ******************************************************************************
# ------------------------------------------------------------------------------
# START CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ******************************************************************************
# ------------------------------------------------------------------------------
include /etc/nginx/bots.d/bad-referrer-words.conf;
# ****************************************************************************
# ----------------------------------------------------------------------------
# END CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ****************************************************************************
# ----------------------------------------------------------------------------
}
# --------------------------------
# END Bad Referrer Word Scanning
# --------------------------------
# *************************
# Bad Referrer Domain Names
# *************************
# Now a list of bad referrer urls these domains or any combination of them ie .com .net
# will be blocked out. Doesn't matter if the protocol is http, https or even ftp
# ----------------------------------------
# START Good and Bad Referrer Domain Names
# ----------------------------------------
# -------------------------------------------------------------------------------------
# Good and Bad referrer urls Doesn't matter if the protocol is http, https or even ftp
# -------------------------------------------------------------------------------------
# ----------------------
# This section includes:
# **********************
# ----------------------
# --------------------------------------------------------------------------------
# Blocking of SEO company Semalt.com (now merged into this one section)
# MIRAI Botnet Domains Used for Mass Attacks
# Other known bad SEO companies and Ad Hijacking Sites
# Sites linked to malware, adware, clickjacking and ransomware
# Domain names and referrers used in referrer spam and seo hijacking
# Whitelisting of your own GOOD domains / referrers
# Whitelisting of any other GOOD domains / referrers you want explicitly NOT block
# --------------------------------------------------------------------------------
# *****************
# ----------------
# PLEASE TEST !!!!
# *****************
# ----------------
# ------------------------------------------------------------------------------------------------------------------------------------
# ALWAYS test referrers that you add. This is done manually as follows
# ------------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------------
# Because of case-insensitive matching any combination of capitilization will all produce a positive hit - make sure you always test.
# ------------------------------------------------------------------------------------------------------------------------------------
# For Example any of the following variations below of 8gold.com will be detected and blocked
# ------------------------------------------------------------------------------------------------------------------------------------
# curl -I http://www.yourdomain.com -e http://NOT-8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e http://this.is.not8gOlD.net --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e ftp://8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e ftp://www.weare8gold.NET --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e https://subdomain.8gold.com --- GIVES YOU: curl: (52) Empty reply from server
# curl -I http://www.yourdomain.com -e https://NOT8GolD.org --- GIVES YOU: curl: (52) Empty reply from server
# ------------------------------------------------------------------------------------------------------------------------------------
# So if you see a bad referrer from wearegoogle.com and you want to block them just add
# them as "~*wearegoogle.com" don't ever go and do something like "~*google(-|.)" you will
# kill all your SEO in a week.
# ------------------------------------------------------------------------------------------------------------------------------------
# To add your own custom bad referrers use the custom include file
# /etc/nginx/bots.d/custom-bad-referrers.conf
# Or send a Pull Request to add it to the global blacklist for other users.
# In the bad referrers section I also include sites that hotlink images without permission.
# ------------------------------------------------------------------------------------------------------------------------------------
# ***********************************************************************
# Now we map all good & bad referrer urls to variable called #bad_referer
# ***********************************************************************
# --------------------------------------------------------------------
# Map all good & bad referrer DOMAINS to a variable called bad_referer
# --------------------------------------------------------------------
map $http_referer $bad_referer {
hostnames;
default 0;
# *************************************
# GOOD REFERRERS - Spared from Checking
# *************************************
# --------------------------------------------
# GOOD REFERRER DOMAINS - Spared from Checking
# --------------------------------------------
# ---------------------------------------------------------------------------------------
# Add all your own web site domain names and server names in this section
# WHITELIST Your Own Domain Names Here using the new Include File Method
# New Method Uses the include file below so that when pulling future updates your
# whitelisted domain names are automatically now included for you.
# Read Comments inside whitelist-domains.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# ---------------------------------------------------------------------------------------
# ************************************************************************
# ------------------------------------------------------------------------
# START WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-domains.conf;
# **********************************************************************
# ----------------------------------------------------------------------
# END WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **********************************************************************
# ----------------------------------------------------------------------
# ***********************************
# -----------------------------------
# CUSTOM BAD REFERRERS - Add your Own
# ***********************************
# -----------------------------------
# Add any extra bad referrers in the following include file to have them
# permanently included and blocked - avoid duplicates in your custom file
# custom-bad-referrers.conf is BOTH a BLACKLIST AND WHITELIST
# custom-bad-referrers.conf ALLOWS complete over-riding of anything
# If you think google.com is bad you would simply add them to
# custom-bad-referrers.conf with a value of 1
# *************************************************************************
# -------------------------------------------------------------------------
# START CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *************************************************************************
# -------------------------------------------------------------------------
include /etc/nginx/bots.d/custom-bad-referrers.conf;
# ***********************************************************************
# -----------------------------------------------------------------------
# END CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***********************************************************************
# -----------------------------------------------------------------------
# START BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ###
"~*(?:\b)000free\.us(?:\b|)" 1;
@ -7650,26 +7697,24 @@ map $http_referer $bad_referer {
# END BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ###
}
# =====================================
# END SECTION 2 - REFERRERS AND DOMAINS
# =====================================
# ***********************************************
# WHITELISTING AND BLACKLISTING IP ADDRESS RANGES
# ***********************************************
# ========================================================================
# BEGIN SECTION 3 - WHITELISTING AND BLACKLISTING IP ADDRESSESE AND RANGES
# ========================================================================
# Geo directive to deny and also whitelist certain ip addresses
# --------------------------------------------------------------------------------------
# Map all GOOD and BAD IP Addresses and Ranges to a variable called geo $validate_client
# --------------------------------------------------------------------------------------
geo $validate_client {
# ********************
# First Our Safety Net
# ********************
# Anything not matching our rules is allowed through with default 0;
default 0;
# *************************
# Wordpress Theme Detectors
# *************************
# -------------------------------------
# BLOCK known Wordpress Theme Detectors
# -------------------------------------
# START WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
@ -7713,10 +7758,11 @@ geo $validate_client {
89.36.223.188 1; #www.cuteseotools.net/wordpress-theme-detector
# END WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
# ****************************************
# NIBBLER - SEO testing and reporting tool
# ****************************************
# ----------------------------------------------
# BLOCK NIBBLER - SEO testing and reporting tool
# ----------------------------------------------
# See - http://nibbler.silktide.com/
# ----------------------------------------------
# START NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
52.201.238.175 1;
@ -7728,11 +7774,11 @@ geo $validate_client {
54.242.250.203 1;
# END NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
# *********************************************************************************************
# KNOWN BAD IP ADDRESSES
# Source: https://github.com/mitchellkrogza/Suspicious.Snooping.Sniffing.Hacking.IP.Addresses
# A Reliable Daily Updated List Included Below of Known Bad IP Addresses, Snoopers and Sniffers
# *********************************************************************************************
# -----------------------------------------
# BLOCK KNOWN BAD IP ADDRESSES
# Top known bad IP Adresses from abuseIPDB
# -----------------------------------------
# START KNOWN BAD IP ADDRESSES ### DO NOT EDIT THIS LINE AT ALL ###
193.32.163.89 1;
@ -17736,11 +17782,9 @@ geo $validate_client {
46.101.115.65 1;
# END KNOWN BAD IP ADDRESSES ### DO NOT EDIT THIS LINE AT ALL ###
# ****************
# Google IP Ranges
# ****************
# For Safety Sake all Google's Known IP Ranges are all white listed
# --------------------------
# WHITELIST Google IP Ranges
# --------------------------
# START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
108.177.0.0/17 0;
@ -17768,11 +17812,9 @@ geo $validate_client {
74.125.0.0/16 0;
# END GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# **************
# Bing IP Ranges
# **************
# For Safety Sake all Bing's Known IP Ranges are all white listed
# ------------------------
# WHITELIST Bing IP Ranges
# ------------------------
# START BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
131.253.21.0/24 0;
@ -17796,11 +17838,9 @@ geo $validate_client {
40.96.0.0/12 0;
# END BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# ********************
# Cloudflare IP Ranges
# ********************
# For Safety Sake all Cloudflare's Known IP Ranges are all white listed
# ------------------------------
# WHITELIST Cloudflare IP Ranges
# ------------------------------
# START CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
103.21.244.0/22 0;
@ -17827,75 +17867,91 @@ geo $validate_client {
2c0f:f248::/32 0;
# END CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
# *************************************************
# Blacklist IP addresses and IP Ranges Customizable
# *************************************************
# -------------------------------------------------
# BLACKLIST IP addresses and IP Ranges Customizable
# -------------------------------------------------
# --------------------------------------------------------------------------------------
# BLACKLIST all your IP addresses and Ranges using the new include file below.
# New Method Uses the include file below so that when pulling future updates your
# Custom Blacklisted IP addresses are automatically now included for you.
# Read Comments inside blacklist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# Anything added to blacklist-ips.conf will over-ride anything whitelisted above
# --------------------------------------------------------------------------------------
# ********************************************************************
# --------------------------------------------------------------------
# START BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ********************************************************************
# --------------------------------------------------------------------
include /etc/nginx/bots.d/blacklist-ips.conf;
# ******************************************************************
# ------------------------------------------------------------------
# END BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ******************************************************************
# ------------------------------------------------------------------
# ----------------------------------------------
# Whitelist all your OWN IP addresses and Ranges
# ----------------------------------------------
# ***********************************
# Whitelist all your OWN IP addresses
# ***********************************
# --------------------------------------------------------------------------------------
# WHITELIST all your own IP addresses using the include file below.
# New Method Uses the include file below so that when pulling future updates your
# whitelisted IP addresses are automatically now included for you.
# Read Comments inside whitelist-ips.conf for customization tips.
# Updating the main globalblacklist.conf file will not touch your custom include files
# whitelist-ips.conf reigns supreme !!!
# Whatever you add to whitelist-ips.conf will be whitelisted FULL STOP
# Anything blacklisted above this line will be over-ridden by whitelist-ips.conf
# --------------------------------------------------------------------------------------
# **************************************************************************
# --------------------------------------------------------------------------
# START WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# **************************************************************************
# --------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# ************************************************************************
# ------------------------------------------------------------------------
# END WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ************************************************************************
# ------------------------------------------------------------------------
}
# --------------------------------------------------------------------------------------
# WHITELIST your own IPs from the DDOS Filter
# Add your own IP addresses and ranges into the custom include file whitelist-ips.conf
# to spare them from the rate limiting DDOS filter.
# This section includes the same / single whitelist-ips.conf file so you only
# need to edit that include file and have it include here for you too.
# --------------------------------------------------------------------------------------
geo $ratelimited {
default 1;
# ***************************************************************************
# ---------------------------------------------------------------------------
# START WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# ***************************************************************************
# ---------------------------------------------------------------------------
include /etc/nginx/bots.d/whitelist-ips.conf;
# *************************************************************************
# -------------------------------------------------------------------------
# END WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ###
# *************************************************************************
# -------------------------------------------------------------------------
}
# *****************************************
# MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# *****************************************
# ======================================================================
# END SECTION 3 - WHITELISTING AND BLACKLISTING IP ADDRESSESE AND RANGES
# ======================================================================
# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================
# --------------------------------------------
# 1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
@ -17903,30 +17959,31 @@ geo $ratelimited {
2 $binary_remote_addr;
}
# ***********************
# SET RATE LIMITING ZONES
# ***********************
# --------------------------
# 2. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# limits for Zone $bad_bot = 1
# Nothing Set - you can set a different zone limiter here if you like
# We issue a 444 response instead to all bad bots.
# Rate limiting will only take effect if on any User-Agents with a value of 2
# limits for Zone $bad_bot = 2
# this rate limiting will only take effect if you change any of the bots and change
# their block value from 1 to 2.
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=6r/m burst=1;
### *** MAKE SURE TO ADD to your nginx.conf ***
### server_names_hash_bucket_size 64;
### server_names_hash_max_size 4096;
### limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
### limit_conn_zone $binary_remote_addr zone=addr:50m;
### to allow it to load this large set of domains into memory and to set the rate limiting zones for the DDOS filter.
# ==========================================
# END SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ==========================================
### THE END of the Long and Winding Road
# =====================
# END BLOCKER FUNCTIONS
# =====================
### Also check out my Ultimate Apache Bad Bot Blocker on Github
### https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
### --------------------------------------------
### HELP SUPPORT THIS PROJECT - Send Me a Coffee
### https://ko-fi.com/mitchellkrog
### --------------------------------------------
### FOR APACHE SERVERS
### ---------------------------------------------
### Check out the Ultimate Apache Bad Bot Blocker
### ---------------------------------------------

2
.dev-tools/_conf_files_testing_changes_ip_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.202.145.110 1;
35.192.136.167 1;
127.0.0.1 1;

7
.dev-tools/_conf_files_testing_changes_ip_whitelist/bots.d/blockbots.conf
View file

@ -39,10 +39,11 @@
# More IP's can be added example > "(127.0.0.1)|(192.168.0.1)|(192.168.1.1)"
# If you even blacklisted 127.0.0.1 or your own IP by giving it a value of 1 in any of the includes, this will over-ride that block.
if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
# UNCOMMENT THE NEXT 4 LINES TO ACTIVATE THE SUPER WHITELIST
#if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
#set $bad_bot '0'; #Uncommenting this line will disable bad_bots functionality for specified IP(s)
set $validate_client '0'; #Uncommenting this line will disable validate_client ip blocking functionality for specified IP(s)
}
#set $validate_client '0'; #Uncommenting this line will disable validate_client ip blocking functionality for specified IP(s)
#}
# --------------
# BLOCK BAD BOTS

2
.dev-tools/_conf_files_testing_changes_ip_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
35.202.145.110 0;
35.192.136.167 0;
127.0.0.1 0;

4
.dev-tools/_conf_files_testing_changes_ip_whitelist/conf.d/globalblacklist.conf
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1632
### Updated: Wed Jun 26 17:03:38 SAST 2019
### Version: V4.2019.06.1633
### Updated: Thu Jun 27 09:10:20 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

2
.dev-tools/_conf_files_testing_changes_ratelimiting/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.202.145.110 1;
35.192.136.167 1;
127.0.0.1 1;

7
.dev-tools/_conf_files_testing_changes_ratelimiting/bots.d/blockbots.conf
View file

@ -39,10 +39,11 @@
# More IP's can be added example > "(127.0.0.1)|(192.168.0.1)|(192.168.1.1)"
# If you even blacklisted 127.0.0.1 or your own IP by giving it a value of 1 in any of the includes, this will over-ride that block.
if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
# UNCOMMENT THE NEXT 4 LINES TO ACTIVATE THE SUPER WHITELIST
#if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
#set $bad_bot '0'; #Uncommenting this line will disable bad_bots functionality for specified IP(s)
set $validate_client '0'; #Uncommenting this line will disable validate_client ip blocking functionality for specified IP(s)
}
#set $validate_client '0'; #Uncommenting this line will disable validate_client ip blocking functionality for specified IP(s)
#}
# --------------
# BLOCK BAD BOTS

2
.dev-tools/_conf_files_testing_changes_ratelimiting/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
35.202.145.110 0;
35.192.136.167 0;
127.0.0.1 0;

4
.dev-tools/_conf_files_testing_changes_ratelimiting/conf.d/globalblacklist.conf
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1632
### Updated: Wed Jun 26 17:03:38 SAST 2019
### Version: V4.2019.06.1633
### Updated: Thu Jun 27 09:10:20 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

2
.dev-tools/_conf_files_testing_changes_whitelist/bots.d/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.202.145.110 1;
35.192.136.167 1;
127.0.0.1 1;

7
.dev-tools/_conf_files_testing_changes_whitelist/bots.d/blockbots.conf
View file

@ -39,10 +39,11 @@
# More IP's can be added example > "(127.0.0.1)|(192.168.0.1)|(192.168.1.1)"
# If you even blacklisted 127.0.0.1 or your own IP by giving it a value of 1 in any of the includes, this will over-ride that block.
if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
# UNCOMMENT THE NEXT 4 LINES TO ACTIVATE THE SUPER WHITELIST
#if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
#set $bad_bot '0'; #Uncommenting this line will disable bad_bots functionality for specified IP(s)
set $validate_client '0'; #Uncommenting this line will disable validate_client ip blocking functionality for specified IP(s)
}
#set $validate_client '0'; #Uncommenting this line will disable validate_client ip blocking functionality for specified IP(s)
#}
# --------------
# BLOCK BAD BOTS

2
.dev-tools/_conf_files_testing_changes_whitelist/bots.d/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
35.202.145.110 0;
35.192.136.167 0;
127.0.0.1 0;

4
.dev-tools/_conf_files_testing_changes_whitelist/conf.d/globalblacklist.conf
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1632
### Updated: Wed Jun 26 17:03:38 SAST 2019
### Version: V4.2019.06.1633
### Updated: Thu Jun 27 09:10:20 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

17447
.dev-tools/globalblacklist-testing.template
View file

File diff suppressed because it is too large Load diff

4
.dev-tools/globalblacklist.template
View file

@ -4,8 +4,8 @@
### VERSION INFORMATION #
###################################################
### Version: V4.2019.06.1633
### Updated: Thu Jun 27 09:10:20 SAST 2019
### Version: V4.2019.06.1634
### Updated: Thu Jun 27 10:50:24 SAST 2019
### Bad Referrer Count: 6713
### Bad Bot Count: 556
###################################################

2
.dev-tools/test_units/blacklist-ips.conf
View file

@ -1,2 +1,2 @@
35.202.145.110 1;
35.192.136.167 1;
127.0.0.1 1;

94
.dev-tools/test_units/random-bots-for-test-quick.list
View file

@ -1,50 +1,50 @@
ADmantX
ASPSeek
Attach
backlink-check
CCBot
Dispatch
DomainAppender
Drip
EirGrabber
EroCrawler
GermCrawler
Intelliseek
internetVista monitor
IRLbot
Jetty
JustView
lwp-request
Magnet
Metauri
MSFrontPage
Navroad
Net Vampire
AhrefsBot
Asterias
BackDoorBot
Backlink-Ceck
Battleztar Bazinga
Blow
Brandprotect
com.plumanalytics
Cosmos
DIIbot
DomainStatsBot
ExtLinksBot
Gigablast
HEADMasterSEO
Jorgee
Larbin
Lftp
LinqiaScrapeBot
Lmspider
Microsoft URL Control
Mojeek
Needle
NetMechanic
Nimbostratus
Pimonster
POE-Component-Client-HTTP
PxBroker
Rankivabot
Screaming
SEOlyticsCrawler
SEOprofiler
Siphon
SISTRIX
PageAnalyzer
PageScorer
PECL::HTTP
plumanalytics
ProPowerBot
Psbot
RankFlex
ScanAlert
scan.lol
ScoutJet
Searchestate
SEOkicks
SiteLockSpider
SiteSnagger
Snoopy
Sogou web spider
Spammen
Sqworm
Sucuri
tAkeOut
Thumbor
Toata
Twice
sysscan
Teleport
URLy.Warning
Voil
WebBandit
WebEnhancer
Web Enhancer
WebGo IS
WebPix
WebSauger
WiseGuys Robot
WPScan
VidibleScraper
WebLeacher
WebWhacker
WebZIP
WISENutbot
x22Mozilla
Xaldon WebSpider

254
.dev-tools/test_units/random-bots-for-test.list
View file

@ -1,250 +1,250 @@
360Spider
404enemy
80legs
Abonti
AfD-Verbotsverfahren
404checker
ADmantX
AhrefsBot
AIBOT
Alexibot
AiHitBot
Aipbot
Alligator
AllSubmitter
Apexoo
archive.org_bot
Asterias
Attach
autoemailspider
BacklinkCrawler
Bandit
Barkrowler
Battleztar Bazinga
BackStreet
Badass
BBBike
BDFetch
Bigfoot
Blackboard
Black Hole
BDCbot
BetaBot
Bitacle
BLEXBot
BlowFish
Boardreader
Brandprotect
Brandwatch
Buddy
BuiltBotTough
Bullseye
BunnySlippers
CATExplorador
Calculon
CazoodleBot
CCBot
CheeseBot
CherryPicker
ChinaClaw
Chlooe
Claritybot
Cloud mapping
Cliqzbot
coccocbot-web
Cogentbot
com.plumanalytics
cognitiveseo
Copier
Crescent
CSHttp
Craftbot
crawl.sogou.com
Curious
DBLBot
DatabaseDriverMysqli
demandbase-bot
Devil
Deusu
Digincore
DIIbot
Disco
Discoverybot
Dispatch
DomainAppender
DomainCrawler
Dotbot
DomainStatsBot
Download Wonder
Dragonfly
Drip
DSearch
DTS Agent
EasyDL
Ebingbong
ECCP/1.0
Ecxi
EMail Siphon
EroCrawler
evc-batch
Exabot
ExtLinksBot
Extractor
Extreme Picture Finder
Ezooms
FHscan
Fimap
Firefox/7.0
FlashGet
Flunky
FrontPage
FyberSpider
Genieo
Getintent
GetRight
GetWeb
G-i-g-a-b-o-t
Go-Ahead-Got-It
GoZilla
Grabber
GrabNet
Go!Zilla
GrapeFX
GrapeshotCrawler
Haansoft
Harvest
HaosouSpider
HEADMasterSEO
HMView
HTMLparser
HTTP::Lite
Humanlinks
IDBot
InfoTekies
Id-search
IndeedBot
Indy Library
instabid
internetVista monitor
ips-agent
IRLbot
Iria
Iskanie
IstellaBot
JamesBOT
Jbrofuzz
JetCar
Jetty
JOC Web Spider
Joomla
Jorgee
JustView
Jyxobot
Kenjin Spider
LeechFTP
Kozmosbot
Larbin
LibWeb
Lightspeedsystems
Likse
Linkdexbot
Libwhisker
LinkpadBot
LinkScan
LinksManager
LinkWalker
LinqiaMetadataDownloaderBot
LinqiaScrapeBot
Lipperhey Spider
LWP::Simple
lwp-trivial
Magnet
magpie-crawler
Majestic SEO
MarkMonitor
Mag-Net
Mail.RU_Bot
Majestic12
MarkWatch
Masscan
Mass Downloader
Mata Hari
meanpathbot
MeanPath Bot
MauiBot
Mediatoolkitbot
mediawords
MFC_Tear_Sample
Microsoft Data Access
Microsoft URL Control
MIDown tool
MIIxpc
Morfeus Fucking Scanner
MSFrontPage
MSIECrawler
muhstik-scan
Musobot
Nameprotect
NearSite
Nessus
NetAnts
Netcraft
NetLyzer
NetMechanic
NetSpider
Nettrack
Netvibes
NetZIP
NextGenSearchBot
Niki-bot
NimbleCrawler
Nibbler
NICErsPRO
Nikto
Ninja
NPbot
Octopus
Offline Explorer
Offline Navigator
OnCrawl
Openfind
Openvas
OpenLinkProfiler
OutclicksBot
OutfoxBot
PageAnalyzer
PageScorer
PageGrabber
page scorer
Pandalytics
Panscient
Papa Foto
pcBrowser
PECL::HTTP
PeoplePal
PHPCrawl
Pavuk
Picscout
Picsearch
PictureFinder
Pi-Monster
plumanalytics
Pixray
Pockey
ProPowerBot
ProWebWalker
Psbot
Pump
Quick-Crawler
RankFlex
PyCurl
RankActiveLinkBot
RankingBot2
Rankivabot
RedesScrapy
ReGet
RepoMonkey
RocketCrawler
Rogerbot
s1z.ru
ScanAlert
SalesIntelligent
SBIder
Scanbot
scan.lol
Scrapy
Searchestate
SEOkicks
SEOlyticsCrawler
Seomoz
SeoSiteCheckup
SEOstats
serpstatbot
Shodan
Sitebeam
SiteExplorer
SlySearch
sexsearcher
SISTRIX
Siteimprove
Sitevigil
SmartDownload
SMTBot
Snake
Snoopy
SocialRankIOBot
Sociscraper
sogouspider
Sosospider
Spanner
Spinn3r
Sogou web spider
Sottopop
Spammen
SpankBot
sp_auditbot
spyfu
Sqlmap
Sqlworm
Sqworm
Steeler
Stripper
Sucker
SuperHTTP
Surfbot
Suzuran
Swiftbot
sysscan
Szukacz
T8Abot
T0PHackTeam
tAkeOut
Teleport
TeleportPro
TheNomad
Toata
Toweyabot
Tracemyfile
Trendiction
Trendictionbot
trendiction.com
True_Robot
Turnitin
TurnitinBot
Twice
UnisterBot
Upflow
Typhoeus
URLy Warning
VB Project
VCI
VeriCiteCrawler
VidibleScraper
Virusdie
VoidEYE
Voil
Voltron
WASALive-Bot
WallpapersHD
WBSearchBot
WebAuto
Web Auto
Web Collage
WebBandit
WEBDAV
Web Enhancer
WebFetch
Web Fetch
Web Fuck
WebGo IS
WebLeacher
WebmasterWorldForumBot
webmeup-crawler
WebPix
WebSauger
WebReaper
WebsiteExtractor
WebsiteQuester
Website Quester
Webster
WebSucker
Web Sucker
WebWhacker
WebZIP
WeSEE
Whacker
Who.is Bot
Widow
WinHTTrack
WiseGuys Robot
Wonderbot
WPScan
WWW-Collector-E
WWW-Mechanize
WWWOFFLE
x09Mozilla
x22Mozilla
Xaldon WebSpider
Xaldon_WebSpider
xpymep1.exe
YoudaoBot
Zauba
zauba.io
Zermelo
Zeus
zgrab
ZmEu
ZyBorg

252
.dev-tools/test_units/random-bots-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
404checker
404enemy
Aboundex
360Spider
80legs
Aboundexbot
Acunetix
AiHitBot
Alligator
Anarchie
Apexoo
archive.org_bot
autoemailspider
BackDoorBot
backlink-check
BacklinkCrawler
Battleztar Bazinga
BackStreet
BackWeb
Badass
Bandit
Barkrowler
BDCbot
BDFetch
BetaBot
Bigfoot
Blackboard
Bitacle
Black Hole
BlackWidow
BLEXBot
Blow
Bolt
BotALot
Brandprotect
Boardreader
Brandwatch
Buddy
BuiltWith
BuzzSumo
Calculon
BuiltBotTough
Bullseye
CazoodleBot
CCBot
Cegbfeieh
CheTeam
ChinaClaw
Cliqzbot
Cloud mapping
Cogentbot
com.plumanalytics
CopyRightCheck
Copyscape
Cosmos
Craftbot
crawl.sogou.com
CrazyWebCrawler
Crescent
CrunchBot
Custo
CSHttp
Curious
demandbase-bot
Demon
Devil
Digincore
Disco
DigitalPebble
Discobot
Dispatch
DomainStatsBot
Download Wonder
DittoSpyder
DomainAppender
DomainSigmaCrawler
Dotbot
Dragonfly
DTS Agent
EasyDL
Ebingbong
eCatch
ECCP/1.0
EirGrabber
EMail Siphon
EroCrawler
Evil
Express WebPictures
Exabot
ExtLinksBot
Extreme Picture Finder
facebookscraper
FemtosearchBot
Firefox/7.0
Freeuploader
EyeNetIE
FDM
FHscan
FlashGet
Flunky
FrontPage
FyberSpider
GalaxyBot
Getintent
GetRight
GetWeb
Gigablast
Gigabot
G-i-g-a-b-o-t
Gotit
GoZilla
Go!Zilla
Grabber
Grafula
GrapeshotCrawler
Haansoft
GT::WWW
HaosouSpider
Harvest
Havij
Heritrix
HTMLparser
HTTrack
Hloader
HybridBot
Iblog
IlseBot
Image Fetch
Image Sucker
Indy Library
InfoNaviRobot
InfoTekies
Intelliseek
InterGET
internetVista monitor
Iria
Iskanie
JamesBOT
Jbrofuzz
JennyBot
JetCar
Jetty
Joomla
Kenjin Spider
Kozmosbot
Lanshanbot
Larbin
LeechFTP
LexiBot
Libwhisker
Lightspeedsystems
Likse
LinkScan
LinksManager
LinqiaMetadataDownloaderBot
LinkWalker
LinqiaRSSBot
Magnet
Litemage_walker
LNSpiderguy
lwp-trivial
Mag-Net
Mail.RU_Bot
Majestic SEO
Majestic12
MarkMonitor
MarkWatch
Mass Downloader
Mata Hari
MauiBot
mediawords
meanpathbot
MeanPath Bot
Mediatoolkitbot
MegaIndex.ru
MFC_Tear_Sample
MIIxpc
Microsoft Data Access
Mister PiX
Mojeek
Mr.4x3
MSFrontPage
MSIECrawler
muhstik-scan
Musobot
Name Intelligence
NearSite
netEstate NE Crawler
Needle
Nessus
Netcraft
NetLyzer
NetMechanic
NetSpider
Netvibes
NetZIP
NICErsPRO
NimbleCrawler
NextGenSearchBot
Nimbostratus
Nmap
NPbot
Nutch
Octopus
Offline Explorer
OnCrawl
OpenLinkProfiler
Openvas
OpenVAS
OrangeSpider
OutclicksBot
OrangeBot
OutfoxBot
PageAnalyzer
Page Analyzer
page scorer
Pandalytics
Panscient
Papa Foto
PeoplePal
Picscout
Picsearch
Pimonster
Pi-Monster
ProPowerBot
PleaseCrawl
plumanalytics
Probethenet
ProWebWalker
Psbot
PyCurl
Pump
PxBroker
QueryN Metasearch
RankFlex
RankingBot
RankingBot2
Rankivabot
RankurBot
RealDownload
Reaper
RebelMouse
Ripper
SBIder
RocketCrawler
SalesIntelligent
ScanAlert
Scanbot
scan.lol
Scrapy
Screaming
Searchestate
Semrush
SemrushBot
SEOkicks-Robot
SEOlyticsCrawler
Seomoz
SEOprofiler
seoscanners
SeoSiteCheckup
serpstatbot
Siphon
SISTRIX
sexsearcher
Sitebeam
SiteExplorer
SiteSnagger
Site Sucker
Sitevigil
SlySearch
SmartDownload
SMTBot
Snapbot
SocialRankIOBot
Sociscraper
Spanner
Spbot
Sottopop
SpaceBison
Spinn3r
spyfu
Sqlmap
Stripper
Sucker
Sucuri
SuperHTTP
Surfbot
SurveyBot
Swiftbot
sysscan
Szukacz
Teleport
TeleportPro
TheNomad
Thumbor
T8Abot
Telesoft
Telesphorep
The Intraformant
TightTwatBot
Titan
Toweyabot
Toata
Trendiction
Trendictionbot
trendiction.com
True_Robot
Turingos
TwengaBot
Twice
UnisterBot
Upflow
URLy.Warning
Vagabondo
VB Project
VCI
VidibleScraper
Virusdie
Voil
Wallpapers/3.0
WallpapersHD
WEBDAV
WebFetch
WASALive-Bot
Webalta
WebAuto
WebBandit
WebEnhancer
WebFuck
Web Fuck
WebGo IS
WebImageCollector
WebmasterWorldForumBot
webmeup-crawler
WebPix
Web Pix
WebSauger
WebsiteExtractor
Webshag
WebsiteQuester
Website Quester
Webster
WebSucker
Web Sucker
WebZIP
Whack
Whacker
Whatweb
Who.is Bot
WebWhacker
WeSEE
Widow
WinHTTrack
WiseGuys Robot
WISENutbot
Wonderbot
Woobot
Wprecon
WPScan
WWW-Collector-E
WWWOFFLE
x09Mozilla
WWW-Mechanize
Xaldon WebSpider
Xaldon_WebSpider
xpymep1.exe
YoudaoBot
Zade
zauba.io
Zeus
Zauba
Zermelo
zgrab
Zitebot
ZumBot
ZyBorg

198
.dev-tools/test_units/random-referrers-for-test-quick.list
View file

@ -1,100 +1,100 @@
24videos.tv
7search.com
adbetclickin.pink
adultfullhd.com
alif-ba-ta.com
apiallgeniusinfo-a.akamaihd.net
aproposde.com
atmagroup.ru
autochoixspinelli.com
avon-novosib.ru
azlex.uz
bag77.ru
bard-real.com.ua
blackcurranthumidifiers.site
blog.f00kclan.de
buypuppies.ca
canacopegdl.com
cfacarrosserie74.com
chatroulette.si
chlooe.com
compliance-george.top
compliance-mary.top
counterbot.com
dailyfinancefix.com
delta-line.men
design-lands.ru
devochki.top
drev.biz
drugstoreforyou.com
ekto.ee
eralph.tk
europages.com.ru
freecamdollars.com
freshmac.space
gael-s.ru
gagrasector.ru
ggiaro.com
golfresa.lucania.se
gotorussia.com
gourcy.altervista.org
ico.re
images-graphics-pics.com
imguramx.pw
interior-stickers.ru
iomoio.net
itag.pw
jagg.info
lsex.xyz
lsp-awak-perikanan.com
mediaoffers.click
medicineseasybuy.com
mensandals.xyz
minet.club
mixx.com
montredemarque.nl
myplaycity.com
okmusic.jp
panchro.co.uk
pinstake.com
pliks.pl
00it.com
2kata.ru
4kepics.com
83net.jp
acc.eu.org
afftrack001.com
alltheviews.com
anal-acrobats.hol.es
apple.com-webbrowsing-security.review
asseenontv.ru
atovh.local-finders.com
ausergrubhof.info
beauty-b0x.pl
bestsexyblog.com
bitcoins-live.ru
bkgr.se
blogporn.in
brainboosting.club
brothers-smaller.ru
businesn.men
buy-forum.ru
calstaterealty.us
candcstuccoandstone.com
cartierlovestore.xyz
chelyabinsk.xrus.org
custodieva.ru
damedingel.ya.ru
dangphoto.trade
destinationrealestate.com
dnepr-avtospar.com.ua
doctissimo.fr
dostavka-v-krym.com
easycommerce.cf
eldiariodeguadalajara.com
eropornosex.ru
erotiktreff24.info
fatfasts-4tmz.com
fettefrauen.net
fishingwholesale.us
foxjuegos.com
free-video-tool.com
garciniaxt.us
gazporno.com
globatur.ru
graphicwe.org
gz2.bbsoldes.fr
hamptonoaks.ca
healgastro.com
heatpower.ru
hkladys.com
hotdl.in
kchaxton.com
kosynka-games.ru
letmacworkfaster.site
liupis.com
loadopia.com
lovi-moment.com.ua
make-money-online.com
makemoneyonline.com
manimpotence.com
njkmznnb.ru
nudo.ca
ochistka-stokov.ru
onlineslotmaschine.com
paleohub.info
pk-services.ru
pornogad.com
pornophoto.xyz
pregnant.guru
rabot.host.sk
rangoman.date
ranksonic.org
scanner-john.top
seoheap.com
sergiorossistore.online
sethrollins.net
social-search.me
solicita.info
sprttrack.com
ssconstruction.co
stuff-about-money.com
szamponrevita.pl
tehngr.ru
tizanidine4mg.blogspot.com
tizanidineonline.blogspot.com
toondinsey.com
traffmonster.info
ufa.xrus.org
vancleefreplica.pw
vchulkax.com
viddyoze.com
villacoloniale.com
voloomoney.com
vriel.batcave.net
vucms.com
wallpapers-best.com
warningzscaler.heraeus.com
website-audit.com.ua
westvilletowingservices.co.za
whipme.yopoint.in
writingservices17.blogspot.ru
wwwadultcheck.com
xn----7sbaaabaei0cc8aj5bj0bncejx.xn--p1ai
yellowads.men
youtuhe.com
zocaparj.kz
zrelaya.pw
pornoslive.net
questionmarque.ch
quit-smoking.ga
rankings-analytics.com
reyel1985.webnode.fr
ric.info
samara.rosfirm.ru
serialsway.ucoz.ru
ship-marvel.co.ua
slimcdn.com
social-fun.ru
songplanet.ru
space4updating.win
staff.prairiesouth.ca
stard.shop
sygraem.com
the-usa-games.blogspot.com
token-lab.org
toolsky.com
tuckpointingmasonrysystems.com
uptime-alpha.net
usiad.net
video-production.com
wesharepics.site
whatzmyip.net
wiosenny-bon-1500.pl
yginekologa.com
zeikopay.com
zetmaster.ru
zhacker.net
zixizop.net.ru
zvetki.ru

1740
.dev-tools/test_units/random-referrers-for-test.list
View file

File diff suppressed because it is too large Load diff

484
.dev-tools/test_units/random-referrers-for-whitelist-test.list
View file

@ -1,250 +1,250 @@
1adult.com
1-free-share-buttons.com
28n2gl3wfyb0.ru
2itech.ru
2rich4bitches.com
4inn.ru
51.la
6hopping.com
7xc4n.com
991mostfm.co.id
acgs.tk
acortarurl.es
adbetclickin.pink
akuhni.by
alevinefesleri.com
aliexpresscashback.club
alldubai.biz
allstatesugarbowl.org
amanda-porn.ga
amigobulls.com
anonymous-redirect.com
antons-transporte.de
apiholdingmypage-a.akamaihd.net
apiluckyleapnet-a.akamaihd.net
apiportalmorecom-a.akamaihd.net
apple.com-cleaner.systems
arendas.net
as5000.com
atout-energie-69.com
audiobangout.com
auto4style.ru
avek.ru
baksman.com
big-cash.net
bizfly.info
00webcams.com
100searchengines.com
101flag.ru
11235813.webzdarma.cz
3w24.com
4senses.co
4ureyesonly.com
98oi.ru
abcdeg.xyz
adminshop.com
aibolita.com
aihelen.net
air-edem.ru
airmaxshoes-2016.com
alice.it
allwomen.info
alphacarolinas.com
alt-servis.ru
amyfoxfitness.com
android4fun.org
angry-fermi-7633.arukascloud.io
apidiamondatanet-a.akamaihd.net
apilinkswiftco-a.akamaihd.net
apps-analytics.net
autobudpostach.club
bankofthewext.com
best-way.men
bestwrinklecreamnow.com
bif-ru.info
bigpictures.club
bio-optomarket.ru
bitcoins-live.ru
blagovest-med.ru
bloke.com
boltalko.xyz
b-styles.xyz
bumskontakte.org
cacheimages.com
cam-kontakte.org
cand.jp
canyougethighofftizanidine.blogspot.com
capacitacionyaprendizaje.com
carina-sy.de
carsplate.com
cascadelink.org
cayado.snn.gr
cejewelry.xyz
cerev.info
cheap-pills-norx.com
cheapwebsitehoster.com
checktext.org
chinajerseyswholesalecoupons.com
collegerentals.com
compliance-jessica.top
compliance-josh.top
craftinsta.ru
danschawbel.com
darmebel.com.ua
delfin-aqua.com.ua
dinosaurus.site
discovertreasure-a.akamaihd.net
discovertreasurenow.com
dkvorota.ru
dogsrun.net
domoysshop.ru
dostavka-v-ukrainu.ru
bluesalt.co
boeuklad.com
bombla.org
btc4u.ru
budilneg.xyz
buyessaynow.biz
buyhoverboard.com
buynorxx.com
bzero1jewelry.net
c1.onedmp.com
carapics.com
casablancamanor.co.za
celebsopics.com
centrdebut.ru
cercacamion.it
chatroulette.video
cialiswithoutadoctor.net
clickso.com
clips.ua.ac.be
compliance-andrew.top
compliance-elena.top
contextualyield.com
counciltally.com
cracksplanet.com
crazyboost.pro
creditmoney.com.ua
customerpromos-a.akamaihd.net
cv.wallhade.co
dailyfinancefix.com
darkbooks.org
data1.scopich.com
day-news.info
ddpills.com
deda-moroza-zakaz.ru
delitime.info
detki-opt.ru
detskie-zabavi.ru
detsky-nabytek.info
divatraffic.com
dmmspy.com
dms-sw.ru
dneprsvet.com.ua
downloadmefiranaratb1972.xpg.com.br
droidlook.net
eandsgallery.com
easyshoppermac.com
ebooknovel.club
enge-fotzen.info
erot.co
erotik0049.com
exoclick.com
eyessurgery.ru
fefo.gdn
ferieboligkbh.dk
filmgo.ru
findthe.pet
flashbannernow.com
flauntyoursite.com
florida-tourism.net
forodvd.com
forpackningsutveckling.se
fototravel.eu
free411games.com
freecamdollars.com
gamerscorps.com
getoutofdebtfree.org
global-ics.co.za
gnuetella.com
graphicwe.org
gsasearchengineranker.site
harmonyglen.us
havepussy.com
headpress.ru
hermesreplica.win
hitsbox.info
homeandhealth.ru
hotkeys.com
ido3.com
imabase.com
dpihatinh.gov.vn
draniki.org
drunkmoms.net
dverimegapolis.ru
dynainbox.com
dyt.net
eduserver.net
ee77ee.com
ekosmetyki.net.pl
electrik-avenue.com
emediate.eu
epngo.bz
europeanwatches.ru
everypony.ru
evogarage.com
faz99.com
femmesdenudees.com
fickenbumsen.net
fivedwld.cf
flprog.com
forexgb.ru
forms-mtm.ru
forsex.info
foto-telok.net
gallerylisting.com
gameonasia.com
gameplexcity.com
gamerextra.com
gazoblok.net.ua
globalsurfari.com
gsasearchengineranker.pw
hasshe.com
healmytrauma.info
hermesbirkinhandbagoutlets.com
hermesbracelets.xyz
hetmanship.xyz
hkdiiohi.skyrock.com
hol.es
hopeonthestreet.co.uk
hotshoppymac.com
hully.altervista.org
hunterboots.online
hunthillfarmtrust.org
iamsport.org
ilovevitaly.com
imagerydatabase.com
imgata.com
imperia31.ru
interfucks.net
jav-fetish.com
javxxx18.com
jerseyswholesalechinalimited.com
jobmarket.com.ua
johannesburgsingles.co.za
kakadu-interior.com.ua
kemerovo.zrus.org
indianmedicaltourismshop.com
iphantom.com
javcoast.com
jerseywholesalechinabiz.com
jillepille.com
juliaworld.net
kanimage.com
karachev-city.ru
kichenaid.ru
kochanelli.com
kollesa.ru
konkursowo-24.pl
kosmetyki.tm.pl
kreativperlen.ch
kredit-blog.ru
ku6.com
kvartira-sutochno.com
labelwater.se
lancheck.net
lashstudia.ru
lazymanyoga.com
ledpolice.ru
lignofix.ua
linerdrilling.com
link.ac
livingroomdecor.info
lockerz.com
lsex.xyz
lyrics.home-task.com
machicon-ueno.info
mansiondelrio.co
mediawhirl.net
meduza-consult.ru
megaapteka.ru
mister-shop.com
mncrftpcs.com
mobsfun.net
monarchfind-a.akamaihd.net
moviemail-online.co.uk
mturkcontent.com
myanyone.net
mycaf.it
naval.jislaaik.com
newyorkhotelsmotels.info
nextbackgroundcheck.gq
ngps1.ru
nordstar.pro
normalegal.ru
novatech.vn
nullrefer.com
okmusic.jp
panchro.xyz
pawli.eu
pipki.r.acdnpro.com
pixshark.com
pokemongooo.ml
pornobrazzers.biz
poshiv-chehol.ru
pozdravleniya-c.ru
pricheski-video.com
princevc.com
pronorm.fr
prosmibank.ru
psvita.ru
purchasepillsnorx.com
rankings-analytics.com
recordpage-a.akamaihd.net
reeyanaturopathy.com
remont-fridge-tv.ru
risparmiocasa.bz.it
roma-kukareku.livejournal.com
ryetaw.com
scanner-barak.top
scanner-jessica.top
scanner-mary.top
schoolfiles.net
searchaddis.com
searchinterneat-a.akamaihd.net
seksotur.ru
seo-platform.com
servicecenter.co.ua
s-forum.biz
shitmovs.com
sildenafilcitratemed.com
skladvaz.ru
skyway24.ru
smsactivator.ru
sofit-dmd.ru
space4updating.win
spylog.com
styro.ru
summerlinhomes411.info
svolze.com
swimpool.ca
tasteidea.com
teguh.info
tfxiq.com
kmgamex.cf
liky.co.ua
ljusihus.se
lostaruban.com
m292.info
magnetic-bracelets.ru
matb3aa.com
mediafresh.online
medi-fitt.hu
middlerush-a.akamaihd.net
mikrobiologies.ru
miniads.ca
miniature.io
mir-limuzinov.ru
mistr-x.org
mobile.ok.ru
mojpregled.com
monclercoats.xyz
monitorwebsitespeed.com
myftpupload.com
myiptest.com
nastydollars.com
ok.ru
olympescort.com
onlinewritingjobs17.blogspot.ru
ooo-gotovie.ru
openmultipleurl.com
orel-reshka.net
orenburg-gsm.ru
osoznanie-narkotikam.net
oxford-book.com.ua
pagesense.com
pattersonsweb.com
pcboa.se
pdns.download
photochki.com
pio.polytopesexempt.com
pixell.club
pojdelo.weebly.com
polimga.pw
popander.mobi
pornogad.com
porno-video-chati.ru
praisong.net
proekt-gaz.ru
psoriasis-file.trade
remedyotc.com
remybutler.fr
rewardit.com
rp9.ru
sa-live.com
sbdl.no
scrapinghub.com
seminarygeorgia59.ga
sendearnings.com
seoexperimenty.ru
seorankinglinks.xyz
service.adtech.fr
sextracker.de
soc-econom-problems.ru
socialmonkee.com
social-s-ggg.xyz
social-s-hhh.xyz
soft-terminal.ru
sotkal.lark.ru
space-worry.ml
speechfoodie.com
startufa.ru
sterva.cc
stevemonsen.com
superlist.biz
suralink.com
surffoundation.nl
surintech.ac.th
svnuppsalaorebro.se
sweepstakes.rewardit.com
taxi-v-eisk.ru
tds-advert002.info
telefonsexsofort.tk
templates.franklinfire.co
thaismartloan.com
thebluffs.com
titelhelden.eu
tizanidineduringpregnancy.blogspot.com
tizanidinehcl2mgsideeffects.blogspot.com
toondinsey.com
topmira.com
torrent-newgames.com
toursmaps.com
trafficjunky.com
trafficmania.com
trion.od.ua
truck-land.life
try-rx.com
ttsq.fr
tupper-posuda.ru
twincitiescarservice.com
ugogo.info
umg-stroy.ru
ural-buldozer.ru
viagengrarx.com
videochat.life
thecoolimages.net
tizanidinehydrochloride4mgstreetvalue.blogspot.com
tizanidinerecreationaluse.blogspot.com
tizanidinevssoma.blogspot.com
tootoo.to
toys.erolove.in
tripper.de
tuckpointingmasonrysystems.com
turbo-suslik.org
tvand.ru
tv-spoty.info
twojebook.pl
txxx.com
vacuumscleaner.com
viel.su
vikistars.com
vipms.ru
vipromoffers.com
vpnhowto.info
vpnmouse.com
vykup-avto-krasnodar.ru
visa-pasport.ru
visitcambridge.org
vladhistory.com
vvpg.ru
watchinf.com
watch-movies.ru
website-datenbank.de
wholesalenfljerseys.us.com
womama.ru
workle.website
wpthemedetector.co.uk
webadvance.club
wesharepics.info
wtsindia.in
wttavern.com
x-lime.net
xn--80adaggc5bdhlfamsfdij4p7b.xn--p1ai
xn----itbeirbjbi7bc6bh2d.xn--p1ai
x-porno.video
x-stars.ru
xxxkaz.org
yourserverisdown.com
zapiszto.pl
zarepta.com
zastenchivosti.net
zigarettenonlinekaufen.tumblr.com
zvetki.ru
xn----8sbhefaln6acifdaon5c6f4axh.xn--p1ai
xsion.net
xxx-treker.ru
yougotanewdomain.com
yourads.website
yurgorod.ru
yur-p.ru
zoosexart.com
zvezdagedon.ru

129
.dev-tools/test_units/ratelimittest.txt
View file

@ -1,129 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html><!DOCTYPE html>
<html>
<head>
<title>Welcome to the Nginx</title>
</head>
<body>
<div align=center>
<h1>Welcome to the Nginx Ultimate Bad Bot Blocker</h1>
<p>This is merely used for testing !!!</p>
</body>
</html>

2
.dev-tools/test_units/whitelist-ips.conf
View file

@ -1,2 +1,2 @@
35.202.145.110 0;
35.192.136.167 0;
127.0.0.1 0;