Added Custom Fail2Ban Jail for Dealing with Repeat Offenders at Firewall Level

see the Fail2Ban folder for instructions

Fail2Ban/README.md

@@ -0,0 +1,51 @@
+# Fail2Ban Blacklist for Repeat Offenders of Nginx (action.d)
+
+### Author: Mitchell Krog <mitchellkrog@gmail.com>
+### Version: 1.0
+
+# Add on for Nginx Ultimate Bad Bot blocker
+GitHub: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
+
+
+##### Tested On: Fail2Ban 0.91
+##### Server: Ubuntu 16.04
+##### Firewall: IPTables
+
+### Dependancies: 
+-requires nginxrepeatoffender.conf in /etc/fail2ban/filter.d folder
+-requires nginxrepeatoffender.conf in /etc/fail2ban/action.d folder
+-requires jail settings called [nginxrepeatoffender]
+-requires nginx.repeatoffender file in /etc/fail2ban
+-create with sudo touch /etc/fail2ban/nginx.repeatoffender
+-chmod +x /etc/fail2ban/nginx.repeatoffender
+
+#### Drawbacks: 
+Only works with IPTables
+
+
+#### Based on: 
+The Recidive Jail from Fail2Ban
+
+This custom filter and action for Fail2Ban will monitor your Nginx logs and perma-ban
+any IP address that has generated far too many 444 errors over a 1 week period
+and ban them for 1 day. This works like a charm as an add-on for my Nginx Bad
+Bot Blocker which takes care of generating the 444 errors based on the extensive
+list of Bad Referers, Bots, Scrapers and IP addresses that it covers. This provides short
+block periods of one day which is enough to keep agressive bots from filling up your log files.
+See - https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker for more info on the Nginx Bad Bot Blocker
+
+This custom action requires a custom jail in your jail.local file for Fail2Ban
+
+Your jail file would be configured as follows
+
+```
+[nginxrepeatoffender]
+enabled = true
+logpath = %(nginx_access_log)s
+filter = nginxrepeatoffender
+banaction = nginxrepeatoffender
+bantime  = 86400   ; 1 day
+findtime = 604800   ; 1 week
+maxretry = 20
+```
+

Fail2Ban/action.d/nginxrepeatoffender.conf

@@ -0,0 +1,102 @@
+# /etc/fail2ban/action.d/nginxrepeatoffender.conf
+# Fail2Ban Blacklist for Repeat Offenders of Nginx (action.d)
+#
+# Author: Mitchell Krog <mitchellkrog@gmail.com>
+# Version: 1.0
+#
+# Add on for Nginx Ultimate Bad Bot blocker
+# GitHub: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
+#
+# Tested On: Fail2Ban 0.91
+# Server: Ubuntu 16.04
+# Firewall: IPTables
+#
+# Dependancies: requires nginxrepeatoffender.conf in /etc/fail2ban/filter.d folder
+#               requires jail settings called [nginxrepeatoffender]
+#				requires nginx.repeatoffender file in /etc/fail2ban
+#				create with sudo touch /etc/fail2ban/nginx.repeatoffender
+#				chmod +x /etc/fail2ban/nginx.repeatoffender
+#
+# Drawbacks: Only works with IPTables
+#
+# Based on: The Recidive Jail from Fail2Ban
+#			This custom filter and action will monitor your Nginx logs and perma-ban
+#			any IP address that has generated far too many 444 errors over a 1 week period
+#			and ban them for 1 day. This works like a charm as an add-on for my Nginx Bad
+#			Bot Blocker which takes care of generating the 444 errors based on the extensive
+#			list of Bad Referers, Bots, Scrapers and IP addresses it covers.
+#			See - https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker for more info
+#
+# This custom action requires a custom jail in your 
+# jail.local file for Fail2Ban
+# 
+# Your jail file would be configured as follows
+#
+# [nginxrepeatoffender]
+# enabled = true
+# logpath = %(nginx_access_log)s
+# filter = nginxrepeatoffender
+# banaction = nginxrepeatoffender
+# bantime  = 86400   ; 1 day
+# findtime = 604800   ; 1 week
+# maxretry = 20
+#
+
+[INCLUDES]
+before = iptables-common.conf
+
+
+[Definition]
+# Option:  actionstart
+# Notes.:  command executed once at the start of Fail2Ban.
+# Values:  CMD
+#
+
+actionstart = <iptables> -N f2b-<name>
+              <iptables> -A f2b-<name> -j <returntype>
+              <iptables> -I <chain> -p <protocol> -j f2b-<name>
+              # Sort and Check for Duplicate IPs in our text file and Remove Them
+              sort -u /etc/fail2ban/nginx.repeatoffender -o /etc/fail2ban/nginx.repeatoffender
+              # Persistent banning of IPs reading from our nginx.repeatoffender text file
+              # and adding them to IPTables on our jail startup command
+              cat /etc/fail2ban/nginx.repeatoffender | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done
+
+# Option:  actionstop
+# Notes.:  command executed once at the end of Fail2Ban
+# Values:  CMD
+#
+
+actionstop = <iptables> -D <chain> -p <protocol> -j f2b-<name>
+             <iptables> -F f2b-<name>
+             <iptables> -X f2b-<name>
+
+# Option:  actioncheck
+# Notes.:  command executed once before each actionban command
+# Values:  CMD
+#
+
+actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'
+
+# Option:  actionban
+# Notes.:  command executed when banning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+
+actionban = <iptables> -I f2b-<name> 1 -s <ip> -j DROP
+			# Add the new IP ban to our nginx.repeatoffender file
+            echo '<ip>' >> /etc/fail2ban/nginx.repeatoffender
+
+# Option:  actionunban
+# Notes.:  command executed when unbanning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionunban = <iptables> -D f2b-<name> -s <ip> -j DROP
+			# Remove IP from our nginx.repeatoffender file
+			sed -i -e '/<ip>/d' /etc/fail2ban/nginx.repeatoffender
+
+[Init]
+

Fail2Ban/filter.d/nginxrepeatoffender.conf

@@ -0,0 +1,61 @@
+# /etc/fail2ban/filter.d/nginxrepeatoffender.conf
+# Fail2Ban Blacklist for Repeat Offenders of Nginx (filter.d)
+#
+# Author: Mitchell Krog <mitchellkrog@gmail.com>
+# Version: 1.0
+#
+# Add on for Nginx Ultimate Bad Bot blocker
+# GitHub: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
+#
+# Tested On: Fail2Ban 0.91
+# Server: Ubuntu 16.04
+# Firewall: IPTables
+#
+# Dependancies: requires nginxrepeatoffender.conf in /etc/fail2ban/action.d folder
+#               requires jail settings called [nginxrepeatoffender]
+#				requires nginx.repeatoffender file in /etc/fail2ban
+#				create with sudo touch /etc/fail2ban/nginx.repeatoffender
+#				chmod +x /etc/fail2ban/nginx.repeatoffender
+#
+# Drawbacks: Only works with IPTables
+#
+# Based on: The Recidive Jail from Fail2Ban
+#			This custom filter and action will monitor your Nginx logs and perma-ban
+#			any IP address that has generated far too many 444 errors over a 1 week period
+#			and ban them for 1 day. This works like a charm as an add-on for my Nginx Bad
+#			Bot Blocker which takes care of generating the 444 errors based on the extensive
+#			list of Bad Referers, Bots, Scrapers and IP addresses it covers.
+#			See - https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker for more info
+#
+# This custom action requires a custom jail in your 
+# jail.local file for Fail2Ban
+# 
+# Your jail file would be configured as follows
+#
+# [nginxrepeatoffender]
+# enabled = true
+# logpath = %(nginx_access_log)s
+# filter = nginxrepeatoffender
+# banaction = nginxrepeatoffender
+# bantime  = 86400   ; 1 day
+# findtime = 604800   ; 1 week
+# maxretry = 20
+#
+
+
+[Definition]
+
+_daemon = fail2ban\.actions\s*
+
+# The name of the jail that this filter is used for. In jail.conf, name the 
+# jail using this filter 'nginxrepeatoffender', or change this line!
+_jailname = nginxrepeatoffender
+
+failregex = ^<HOST> -.*GET.*444\s0
+ignoreregex = 
+
+[Init]
+
+journalmatch = _SYSTEMD_UNIT=fail2ban.service PRIORITY=5
+
+# Author: Mitchell Krog 

Fail2Ban/jail.local

@@ -0,0 +1,9 @@
+# Place this at the bottom of your jail.local file adjust ban and find times as your require
+[nginxrepeatoffender]
+enabled = true
+logpath = %(nginx_access_log)s
+filter = nginxrepeatoffender
+banaction = nginxrepeatoffender
+bantime  = 86400   ; 1 day
+findtime = 604800   ; 1 week
+maxretry = 20

README.md

@@ -32,7 +32,7 @@ nginx.conf file.
 -    Link Research and Backlink Testing Tools
 -    Stopping Google Analytics Ghost Spam
 
-(2061 bad referers, bots, seo companies and counting)
+(2065 bad referers, bots, seo companies and counting)
 
 Bots attempt to make themselves look like other software or web sites by disguising their user agent. 
 Their user agent names may look harmless, perfectly legitimate even. 
@@ -249,6 +249,13 @@ Analytics sites for you in 2 easy clicks and it is FREE.
 I have added the creation of a Google Disavow text file called google-disavow.txt. This file can be used in Google's Webmaster
 Tools to block all these domains out as spammy or bad links. Use with caution.
 
+## Blocking Agressive Bots at Firewall Level Using Fail2Ban
+
+I have added a custom Fail2Ban filter and action that I have written which monitors your Nginx logs for bots that generate
+a large number of 444 errors. This custom jail for Fail2Ban will scan logs over a 1 week period and ban the offender for 24 hours.
+It helps a great deal in keeping out some repeat offenders and preventing them from filling up your log files with 444 errors.
+See the Fail2Ban folder for instructions on configuring this great add on for the Nginx Bad Bot Blocker.
+
 
 # IT FORKING WORKS !!!
 ## Just Enjoy now what the Nginx Bad Bot Blocker Can Do For You and Your Web Sites.