vikarti.anatra/nginx-ultimate-bad-bot-blocker
1
0
Fork 0
mirror of https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git synced 2025-09-09 22:05:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Fix Duplicate Cyveillance Ranges Issue.

Browse source
This commit is contained in:
Mitchell Krog 2017-08-21 11:34:24 +02:00
parent ef692ffc17
commit 975ac45840
No known key found for this signature in database
GPG key ID: 2BB9A077C8DD6994
22 changed files with 254 additions and 129 deletions
Download patch file Download diff file Expand all files Collapse all files

36
.dev-tools/prepare-robots-input.sh
View file

@ -4,34 +4,44 @@
# Copyright: Mitchell Krog - https://github.com/mitchellkrogza
# Repo Url: https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites
#########################################################################
# _ __ _ #
# / |/ /__ _(_)__ __ __ #
# / / _ `/ / _ \\ \ / #
# /_/|_/\_, /_/_//_/_\_\ #
# __/___/ __ ___ __ ___ __ __ #
# / _ )___ ____/ / / _ )___ / /_ / _ )/ /__ ____/ /_____ ____ #
# / _ / _ `/ _ / / _ / _ \/ __/ / _ / / _ \/ __/ '_/ -_) __/ #
# /____/\_,_/\_,_/ /____/\___/\__/ /____/_/\___/\__/_/\_\\__/_/ #
#########################################################################
##############################################################################
# _ __ _ #
# / |/ /__ _(_)__ __ __ #
# / / _ `/ / _ \\ \ / #
# /_/|_/\_, /_/_//_/_\_\ #
# __/___/ __ ___ __ ___ __ __ #
# / _ )___ ____/ / / _ )___ / /_ / _ )/ /__ ____/ /_____ ____ #
# / _ / _ `/ _ / / _ / _ \/ __/ / _ / / _ \/ __/ '_/ -_) __/ #
# /____/\_,_/\_,_/ /____/\___/\__/ /____/_/\___/\__/_/\_\\__/_/ #
# #
##############################################################################
# ******************************
# Specify Input and Output Files
# ******************************
#Specify Input and Output File
# ****************************
_input=$TRAVIS_BUILD_DIR/_generator_lists/bad-user-agents.list
_output=$TRAVIS_BUILD_DIR/.dev-tools/_robots_input/robots-input.txt
# *****************
# Truncate our file
# *****************
sudo truncate -s 0 $_output
# ********************************************
# Use sed to strip the \ out of the input file
# ***************************************************
# ********************************************
sed 's/\\ / /g' $_input > $_output
# ***************************************************
# Use cut to strip the domains out of the url strings
# ***************************************************
#cut -d'\' -f3 $_input > $_output
# *************************************
# Sort our output file and remove dupes
# *************************************
sort -u $_output -o $_output