diff --git a/AUTO-CONFIGURATION.md b/AUTO-CONFIGURATION.md index b67c5b2e6..6ca0ba53d 100755 --- a/AUTO-CONFIGURATION.md +++ b/AUTO-CONFIGURATION.md @@ -6,7 +6,7 @@ ### Version Information # ******************************************** -#### Version: V3.2017.07.636 +#### Version: V3.2017.07.637 #### Bad Referrer Count: 5030 #### Bad Bot Count: 479 ******************************************** diff --git a/MANUAL-CONFIGURATION.md b/MANUAL-CONFIGURATION.md index 6e6232a4a..60b49687f 100755 --- a/MANUAL-CONFIGURATION.md +++ b/MANUAL-CONFIGURATION.md @@ -6,7 +6,7 @@ ### Version Information # ******************************************** -#### Version: V3.2017.07.636 +#### Version: V3.2017.07.637 #### Bad Referrer Count: 5030 #### Bad Bot Count: 479 ******************************************** diff --git a/README.md b/README.md index 072d15df7..9abba1b31 100755 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ ### Version Information # ******************************************** -#### Version: V3.2017.07.636 +#### Version: V3.2017.07.637 #### Bad Referrer Count: 5030 #### Bad Bot Count: 479 ******************************************** diff --git a/_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf b/_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf index 615eda5f0..a558c7723 100755 --- a/_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf +++ b/_sample_config_files/Engintron_for_cPanel_WHM_Configuration_Example/etc/nginx/conf.d/globalblacklist.conf @@ -2,30 +2,31 @@ ### THE ULTIMATE NGINX BAD BOT BLOCKER ### ********************************** -### Version Information # +### VERSION INFORMATION # ################################################### ### Version: V3.2017.06.631 ### Updated: Wed Jun 28 16:42:57 SAST 2017 ### Bad Referrer Count: 4947 ### Bad Bot Count: 480 ################################################### -### Version Information ## +### VERSION INFORMATION # ### This file implements a checklist / blacklist for good user agents, bad user agents and -### bad referrers. It also has whitelisting for your own IP's and known good IP Ranges +### bad referrers on Nginx Web Server. It also has whitelisting for your own IP's and known good IP Ranges ### and also has rate limiting functionality for bad bots who you only want to rate limit -### and not actually block out entirely. It is powerful and also flexible. +### and not actually block out entirely. It is very powerful and also very flexible. ### Created By: https://github.com/mitchellkrogza/ ### Repo Url: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker ### Copyright Mitchell Krog - +### Contributors: Stuart Cardall - https://github.com/itoffshore ### Tested on: nginx/1.10.3 (Ubuntu 16.04) ### This list was developed and is in use on a live Nginx server running some very busy web sites. ### It was built from the ground up using real data from daily logs and is updated almost daily. ### It has been extensively tested for false positives and all additions to the lists of bad user agents, -### spam referers, rogue IP address, scanners, scrapers and domain hijacking sites are extensively checked +### spam referrers, rogue IP address, scanners, scrapers and domain hijacking sites are extensively checked ### before they are added. It is monitored extensively for any false positives. ### ********* @@ -35,39 +36,24 @@ ### Alphabetically ordered lists for Ease of Maintenance. ### Extensive Commenting for Ease of Reference. ### Extensive bad_bot list -### Extensive bad_referer list (please excuse the nasty words and domains) +### Extensive bad_referrer list (please excuse the nasty words and domains) ### Simple regex patterns versus complicated messy regex patterns. ### Checks regardless of http / https urls or the lack of any protocol sent. ### IP range blocking / whitelisting. ### Rate Limiting Functions. -### *** PLEASE READ ALL INLINE NOTES ON TESTING !!!! +### ************ +### INSTALLATION +### ************ -### I have this set up as an include in nginx.conf as -### Include /etc/nginx/conf.d/globalblacklist.conf -### This is loaded and available for any vhost to use in its config -### Each vhost then just needs the include file mentioned below for it to take effect. +### PLEASE use the install, setup and update scripts provided for you to ease your installation. +### This Auto Installation procedure is documented in the README.md and AUTO-CONFIGURATION.md files. +### Installation, Setup and Update Scripts Contributed by Stuart Cardall - https://github.com/itoffshore +### There are also manual configuration instructions provided for those not wishing to do an auto install. -### In Most cases your nginx.conf should already have an include statement as follows -### Include /etc/nginx/conf.d/* -### If that is the case then you can ignore the above include statement as Nginx will -### load anything in the conf.d folder and make it available to all sites. - -### All you then need to do is use the include statements below in the server {} block of a vhost file for it to take effect. -# server { -# #Config stuff here -# include /etc/nginx/bots.d/blockbots.conf -# include /etc/nginx/bots.d/ddos.conf -# #Other config stuff here -# } - -### Need I say, please don't just copy and paste this without reviewing what bots and -### referers are being blocked, you may want to exclude certain of them -### Also make SURE to whitelist your own IP's in the geo $bad_referer section. -### Know why you are using this or why you want to use it before you do, the implications -### are quite severe. - -### *** PLEASE READ INLINE NOTES ON TESTING !!!! +### *********************************************** +### !!!!! PLEASE READ INLINE NOTES ON TESTING !!!!! +### *********************************************** ### Note that: ### 0 = allowed - no limits @@ -75,45 +61,36 @@ ### 2 = rate limited more ### 3 = block completely -### NEED I say do a "sudo nginx -t" to test the config is okay after adding these -### and if so then "sudo service nginx reload" for it to take effect. - -### *** MAKE SURE TO ADD to your nginx.conf *** -### server_names_hash_bucket_size 64; -### server_names_hash_max_size 4096; -### limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s; -### limit_conn_zone $binary_remote_addr zone=addr:50m; -### to allow it to load this large set of domains into memory and to set the rate limiting zones for the DDOS filter. - -### ADDING YOUR OWN BAD REFERERS -### Fork your own local copy and then -### Send a Pull Request by following the instructions in the Pull_Requests_Here_Please folder. +### ************************************************************ +### CONTRIBUTING / PULL REQUESTS / ADDING YOUR OWN BAD REFERRERS +### ************************************************************ +### For contributing, corrections or adding bots or referrers to this repo, +### Send a Pull Request (PR) on any of the .list files in the _generator_lists folder +### All Pull Requests will be checked for accuracy before being merged. # ********************************* # FIRST BLOCK BY USER-AGENT STRINGS # ********************************* -# *************** -# PLEASE TEST !!! -# *************** +# *********************** +# !!!!! PLEASE TEST !!!!! +# *********************** # ALWAYS test any User-Agent Strings you add here to make sure you have it right # Use a Chrome Extension called "User-Agent Switcher for Chrome" where you can create your # own custom lists of User-Agents and test them easily against your rules below. -# You can also use Curl to test user-agents as per example below -# curl -I http://www.yourdomain.com -A "GoogleBot" << 200 OK -# curl -I http://www.yourdomain.com -A "80legs" <<< 444 Dropped Connection +# You can also use curl from the command line to test user-agents as per the examples below: -# Here we also allow specific User Agents to come through that we want to allow +# curl -I http://www.yourdomain.com -A "GoogleBot" ---- GIVES YOU: HTTP/1.1 200 OK (Meaning web page was served to Client) +# curl -I http://www.yourdomain.com -A "80legs" ---- GIVES YOU: curl: (52) Empty reply from server (Meaning Nginx gave a 444 Dropped Connection) -# PLEASE NOTE: In all lists below I use Nginx case-insensitive matching ~* -# This means regardless of how you type the word, upper or lowercase or mixed it will -# be detected by Nginx Regex. Some Names are Capitalised simply for Ease of Reading. -# Especially important for both Googlebot and googlebot to be allowed through no? +# In this section we allow/disallow specific User Agents / Bots. +# ********************************************************************* # Now we map all good and bad user agents to a variable called $bad_bot +# ********************************************************************* map $http_user_agent $bad_bot { @@ -122,12 +99,18 @@ map $http_user_agent $bad_bot { # *********************************************** # Include your Own Custom List of Bad User Agents # *********************************************** -# use the include file below to further customize your own list of additional -# user-agents you wish to permanently block -# START BLACKLISTED USER AGENTS ### DO NOT EDIT THIS LINE AT ALL ### +# Use the include file below to further customize your own list of additional user-agents you wish to permanently block + +# *********************************************************************************** +# START CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# *********************************************************************************** + include /etc/nginx/bots.d/blacklist-user-agents.conf; -# END BLACKLISTED USER AGENTS ### DO NOT EDIT THIS LINE AT ALL ### + +# ********************************************************************************* +# END CUSTOM BLACKLISTED USER AGENTS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ********************************************************************************* # *********************************************** @@ -166,9 +149,10 @@ map $http_user_agent $bad_bot { "~yahoo" 0; # END GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ### -# ************************************************** -# User-Agent Strings Allowed Throug but Rate Limited -# ************************************************** +# *************************************************** +# User-Agent Strings Allowed Through but Rate Limited +# *************************************************** + # Some people block libwww-perl, it us widely used in many valid (non rogue) agents # I allow libwww-perl as I use it for monitoring systems with Munin but it is rate limited @@ -208,543 +192,525 @@ map $http_user_agent $bad_bot { # ********************************************* # Bad User-Agent Strings That We Block Outright # ********************************************* + # This includes: # Known Vulnerability Scanners (now merged into one section) # START BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ### - "~*360Spider" 3; - "~*80legs" 3; - "~*Abonti" 3; - "~*Aboundex" 3; - "~*Acunetix" 3; - "~*ADmantX" 3; - "~*AhrefsBot" 3; - "~*AIBOT" 3; - "~*AiHitBot" 3; - "~*Aipbot" 3; - "~*Alexibot" 3; - "~*Alligator" 3; - "~*AllSubmitter" 3; - "~*Anarchie" 3; - "~*Apexoo" 3; - "~*ASPSeek" 3; - "~*Asterias" 3; - "~*Attach" 3; - "~*autoemailspider" 3; - "~*BackDoorBot" 3; - "~*BackStreet" 3; - "~*BackWeb" 3; - "~*Badass" 3; - "~*Bandit" 3; - "~*BatchFTP" 3; - "~*Battleztar\ Bazinga" 3; - "~*BBBike" 3; - "~*BDFetch" 3; - "~*BetaBot" 3; - "~*Bigfoot" 3; - "~*Bitacle" 3; - "~*Blackboard" 3; - "~*Black\ Hole" 3; - "~*BlackWidow" 3; - "~*BLEXBot" 3; - "~*Blow" 3; - "~*BlowFish" 3; - "~*Boardreader" 3; - "~*Bolt" 3; - "~*BotALot" 3; - "~*Brandprotect" 3; - "~*Brandwatch" 3; - "~*Bubing" 3; - "~*Buddy" 3; - "~*BuiltBotTough" 3; - "~*BuiltWith" 3; - "~*Bullseye" 3; - "~*BunnySlippers" 3; - "~*BuzzSumo" 3; - "~*Calculon" 3; - "~*CATExplorador" 3; - "~*CazoodleBot" 3; - "~*CCBot" 3; - "~*Cegbfeieh" 3; - "~*CheeseBot" 3; - "~*CherryPicker" 3; - "~*ChinaClaw" 3; - "~*Chlooe" 3; - "~*Claritybot" 3; - "~*Cliqzbot" 3; - "~*Cogentbot" 3; - "~*Collector" 3; - "~*Copier" 3; - "~*CopyRightCheck" 3; - "~*Copyscape" 3; - "~*Cosmos" 3; - "~*Craftbot" 3; - "~*CrazyWebCrawler" 3; - "~*Crescent" 3; - "~*CSHttp" 3; - "~*Curious" 3; - "~*Custo" 3; - "~*DatabaseDriverMysqli" 3; - "~*DBLBot" 3; - "~*Demon" 3; - "~*Deusu" 3; - "~*Devil" 3; - "~*DIIbot" 3; - "~*Dirbuster" 3; - "~*Disco" 3; - "~*Discobot" 3; - "~*Discoverybot" 3; - "~*DittoSpyder" 3; - "~*DomainAppender" 3; - "~*DomainCrawler" 3; - "~*DomainSigmaCrawler" 3; - "~*Dotbot" 3; - "~*Download\ Demon" 3; - "~*Download\ Devil" 3; - "~*Download\ Wonder" 3; - "~*Dragonfly" 3; - "~*Drip" 3; - "~*DTS\ Agent" 3; - "~*EasyDL" 3; - "~*Ebingbong" 3; - "~*eCatch" 3; - "~*ECCP/1.0" 3; - "~*Ecxi" 3; - "~*EirGrabber" 3; - "~*EMail\ Collector" 3; - "~*EMail\ Extractor" 3; - "~*EMail\ Siphon" 3; - "~*EMail\ Wolf" 3; - "~*EroCrawler" 3; - "~*Evil" 3; - "~*Exabot" 3; - "~*Express\ WebPictures" 3; - "~*Extractor" 3; - "~*ExtractorPro" 3; - "~*EyeNetIE" 3; - "~*Ezooms" 3; - "~*FHscan" 3; - "~*Fimap" 3; - "~*Findxbot" 3; - "~*Firefox/7.0" 3; - "~*FlashGet" 3; - "~*Flunky" 3; - "~*Foobot" 3; - "~*Freeuploader" 3; - "~*FrontPage" 3; - "~*Fyrebot" 3; - "~*GalaxyBot" 3; - "~*Genieo" 3; - "~*Getintent" 3; - "~*GetRight" 3; - "~*GetWeb" 3; - "~*Gigablast" 3; - "~*Gigabot" 3; - "~*Go-Ahead-Got-It" 3; - "~*Gotit" 3; - "~*GoZilla" 3; - "~*Go!Zilla" 3; - "~*Grabber" 3; - "~*GrabNet" 3; - "~*Grafula" 3; - "~*GrapeFX" 3; - "~*GrapeshotCrawler" 3; - "~*GridBot" 3; - "~*GT::WWW" 3; - "~*GuzzleHttp" 3; - "~*HaosouSpider" 3; - "~*Harvest" 3; - "~*Havij" 3; - "~*HEADMasterSEO" 3; - "~*Heritrix" 3; - "~*Hloader" 3; - "~*HMView" 3; - "~*HTMLparser" 3; - "~*HTTP::Lite" 3; - "~*HTTrack" 3; - "~*Humanlinks" 3; - "~*HybridBot" 3; - "~*Iblog" 3; - "~*IDBot" 3; - "~*Id-search" 3; - "~*IlseBot" 3; - "~*Image\ Fetch" 3; - "~*Image\ Stripper" 3; - "~*Image\ Sucker" 3; - "~*Indy\ Library" 3; - "~*InfoNaviRobot" 3; - "~*InfoTekies" 3; - "~*Intelliseek" 3; - "~*InterGET" 3; - "~*Internet\ Ninja" 3; - "~*InternetSeer" 3; - "~*internetVista\ monitor" 3; - "~*Iria" 3; - "~*IRLbot" 3; - "~*Iskanie" 3; - "~*JamesBOT" 3; - "~*Jbrofuzz" 3; - "~*JennyBot" 3; - "~*JetCar" 3; - "~*JikeSpider" 3; - "~*JOC\ Web\ Spider" 3; - "~*Joomla" 3; - "~*JustView" 3; - "~*Jyxobot" 3; - "~*Kenjin\ Spider" 3; - "~*Keyword\ Density" 3; - "~*Lanshanbot" 3; - "~*Larbin" 3; - "~*LeechFTP" 3; - "~*LeechGet" 3; - "~*LexiBot" 3; - "~*Lftp" 3; - "~*LibWeb" 3; - "~*Libwhisker" 3; - "~*Lightspeedsystems" 3; - "~*Likse" 3; - "~*Linkdexbot" 3; - "~*LinkextractorPro" 3; - "~*LinkpadBot" 3; - "~*LinkScan" 3; - "~*LinksManager" 3; - "~*LinkWalker" 3; - "~*LinqiaMetadataDownloaderBot" 3; - "~*LinqiaRSSBot" 3; - "~*LinqiaScrapeBot" 3; - "~*Lipperhey" 3; - "~*Litemage_walker" 3; - "~*Lmspider" 3; - "~*LNSpiderguy" 3; - "~*Ltx71" 3; - "~*lwp-request" 3; - "~*LWP::Simple" 3; - "~*lwp-trivial" 3; - "~*Magnet" 3; - "~*Mag-Net" 3; - "~*magpie-crawler" 3; - "~*Mail.ru" 3; - "~*Majestic12" 3; - "~*MarkMonitor" 3; - "~*MarkWatch" 3; - "~*Masscan" 3; - "~*Mass\ Downloader" 3; - "~*Mata\ Hari" 3; - "~*Meanpathbot" 3; - "~*MegaIndex.ru" 3; - "~*Metauri" 3; - "~*MFC_Tear_Sample" 3; - "~*Microsoft\ Data\ Access" 3; - "~*Microsoft\ URL\ Control" 3; - "~*MIDown\ tool" 3; - "~*MIIxpc" 3; - "~*Mister\ PiX" 3; - "~*MJ12bot" 3; - "~*Mojeek" 3; - "~*MSFrontPage" 3; - "~*MSIE\ 6.0" 3; - "~*MSIECrawler" 3; - "~*Msrabot" 3; - "~*MS\ Web\ Services\ Client\ Protocol" 3; - "~*Musobot" 3; - "~*Name\ Intelligence" 3; - "~*Nameprotect" 3; - "~*Navroad" 3; - "~*NearSite" 3; - "~*Needle" 3; - "~*Nessus" 3; - "~*NetAnts" 3; - "~*Netcraft" 3; - "~*netEstate\ NE\ Crawler" 3; - "~*NetLyzer" 3; - "~*NetMechanic" 3; - "~*NetSpider" 3; - "~*Nettrack" 3; - "~*Net\ Vampire" 3; - "~*NetZIP" 3; - "~*NextGenSearchBot" 3; - "~*Nibbler" 3; - "~*NICErsPRO" 3; - "~*Niki-bot" 3; - "~*Nikto" 3; - "~*NimbleCrawler" 3; - "~*Ninja" 3; - "~*Nmap" 3; - "~*NPbot" 3; - "~*Nutch" 3; - "~*Octopus" 3; - "~*Offline\ Explorer" 3; - "~*Offline\ Navigator" 3; - "~*Openfind" 3; - "~*OpenLinkProfiler" 3; - "~*Openvas" 3; - "~*OrangeBot" 3; - "~*OrangeSpider" 3; - "~*OutfoxBot" 3; - "~*PageAnalyzer" 3; - "~*Page\ Analyzer" 3; - "~*PageGrabber" 3; - "~*Page\ Grabber" 3; - "~*page\ scorer" 3; - "~*PageScorer" 3; - "~*Panscient" 3; - "~*Papa\ Foto" 3; - "~*Pavuk" 3; - "~*pcBrowser" 3; - "~*PECL::HTTP" 3; - "~*PeoplePal" 3; - "~*PHPCrawl" 3; - "~*Picscout" 3; - "~*Picsearch" 3; - "~*Pimonster" 3; - "~*Pi-Monster" 3; - "~*Pixray" 3; - "~*PleaseCrawl" 3; - "~*Pockey" 3; - "~*POE-Component-Client-HTTP" 3; - "~*Probethenet" 3; - "~*ProPowerBot" 3; - "~*ProWebWalker" 3; - "~*Proximic" 3; - "~*Psbot" 3; - "~*Pump" 3; - "~*PyCurl" 3; - "~*QueryN\ Metasearch" 3; - "~*Qwantify" 3; - "~*RankActiveLinkBot" 3; - "~*Rankivabot" 3; - "~*RealDownload" 3; - "~*Reaper" 3; - "~*Recorder" 3; - "~*RedesScrapy" 3; - "~*ReGet" 3; - "~*RepoMonkey" 3; - "~*Ripper" 3; - "~*RocketCrawler" 3; - "~*Rogerbot" 3; - "~*SalesIntelligent" 3; - "~*SBIder" 3; - "~*ScanAlert" 3; - "~*Scanbot" 3; - "~*Scrapy" 3; - "~*Screaming" 3; - "~*Screaming\ Frog\ SEO\ Spider" 3; - "~*ScreenerBot" 3; - "~*Searchestate" 3; - "~*SearchmetricsBot" 3; - "~*Semrush" 3; - "~*SemrushBot" 3; - "~*SEOkicks" 3; - "~*SEOkicks-Robot" 3; - "~*SEOlyticsCrawler" 3; - "~*Seomoz" 3; - "~*SEOprofiler" 3; - "~*SEOstats" 3; - "~*Siphon" 3; - "~*SISTRIX" 3; - "~*SISTRIX\ Crawler" 3; - "~*Sitebeam" 3; - "~*SiteExplorer" 3; - "~*Siteimprove" 3; - "~*SiteLockSpider" 3; - "~*SiteSnagger" 3; - "~*SiteSucker" 3; - "~*Site\ Sucker" 3; - "~*Sitevigil" 3; - "~*Slackbot-LinkExpanding" 3; - "~*SlySearch" 3; - "~*SmartDownload" 3; - "~*Snake" 3; - "~*Snapbot" 3; - "~*Snoopy" 3; - "~*SocialRankIOBot" 3; - "~*Sogou\ web\ spider" 3; - "~*Sosospider" 3; - "~*SpaceBison" 3; - "~*Spammen" 3; - "~*SpankBot" 3; - "~*Spanner" 3; - "~*Spbot" 3; - "~*Spinn3r" 3; - "~*SputnikBot" 3; - "~*Sqlmap" 3; - "~*Sqlworm" 3; - "~*Sqworm" 3; - "~*Steeler" 3; - "~*Stripper" 3; - "~*Sucker" 3; - "~*Sucuri" 3; - "~*SuperBot" 3; - "~*SuperHTTP" 3; - "~*Surfbot" 3; - "~*SurveyBot" 3; - "~*Suzuran" 3; - "~*Swiftbot" 3; - "~*Szukacz" 3; - "~*T0PHackTeam" 3; - "~*T8Abot" 3; - "~*tAkeOut" 3; - "~*Teleport" 3; - "~*TeleportPro" 3; - "~*Telesoft" 3; - "~*Telesphoreo" 3; - "~*Telesphorep" 3; - "~*The\ Intraformant" 3; - "~*TheNomad" 3; - "~*TightTwatBot" 3; - "~*Titan" 3; - "~*Toata" 3; - "~*Toweyabot" 3; - "~*Trendictionbot" 3; - "~*True_Robot" 3; - "~*Turingos" 3; - "~*TurnitinBot" 3; - "~*Turnitin\ Bot" 3; - "~*Turnitin\ Robot" 3; - "~*TwengaBot" 3; - "~*Twice" 3; - "~*Typhoeus" 3; - "~*UnisterBot" 3; - "~*URLy.Warning" 3; - "~*URLy\ Warning" 3; - "~*Vacuum" 3; - "~*Vagabondo" 3; - "~*VB\ Project" 3; - "~*VCI" 3; - "~*VeriCiteCrawler" 3; - "~*VidibleScraper" 3; - "~*VoidEYE" 3; - "~*Voil" 3; - "~*Voltron" 3; - "~*Wallpapers/3.0" 3; - "~*WallpapersHD" 3; - "~*WASALive-Bot" 3; - "~*WBSearchBot" 3; - "~*Webalta" 3; - "~*WebAuto" 3; - "~*Web\ Auto" 3; - "~*WebBandit" 3; - "~*Web\ Bandit" 3; - "~*WebCollage" 3; - "~*Web\ Collage" 3; - "~*WebCopier" 3; - "~*Web\ Copier" 3; - "~*WEBDAV" 3; - "~*WEBDAV\ Client" 3; - "~*WebEnhancer" 3; - "~*Web\ Enhancer" 3; - "~*WebFetch" 3; - "~*Web\ Fetch" 3; - "~*WebFuck" 3; - "~*Web\ Fuck" 3; - "~*WebGo\ IS" 3; - "~*WebImageCollector" 3; - "~*Web\ Image\ Collector" 3; - "~*WebLeacher" 3; - "~*WebmasterWorldForumBot" 3; - "~*webmeup-crawler" 3; - "~*WebPix" 3; - "~*Web\ Pix" 3; - "~*WebReaper" 3; - "~*Web\ Reaper" 3; - "~*WebSauger" 3; - "~*Web\ Sauger" 3; - "~*Webshag" 3; - "~*WebsiteExtractor" 3; - "~*Website\ Extractor" 3; - "~*WebsiteQuester" 3; - "~*Website\ Quester" 3; - "~*Webster" 3; - "~*WebStripper" 3; - "~*Web\ Stripper" 3; - "~*WebSucker" 3; - "~*Web\ Sucker" 3; - "~*WebWhacker" 3; - "~*Web\ Whacker" 3; - "~*WebZIP" 3; - "~*WeSEE" 3; - "~*Whack" 3; - "~*Whacker" 3; - "~*Whatweb" 3; - "~*Widow" 3; - "~*WinHTTrack" 3; - "~*WiseGuys\ Robot" 3; - "~*WISENutbot" 3; - "~*Wonderbot" 3; - "~*Woobot" 3; - "~*Wotbox" 3; - "~*Wprecon" 3; - "~*WPScan" 3; - "~*WWW-Collector-E" 3; - "~*WWW-Mechanize" 3; - "~*WWW::Mechanize" 3; - "~*WWWOFFLE" 3; - "~*x22Mozilla" 3; - "~*Xaldon_WebSpider" 3; - "~*Xaldon\ WebSpider" 3; - "~*Xenu" 3; - "~*YoudaoBot" 3; - "~*Zade" 3; - "~*Zermelo" 3; - "~*Zeus" 3; - "~*Zgrab" 3; - "~*Zitebot" 3; - "~*ZmEu" 3; - "~*ZumBot" 3; - "~*ZyBorg" 3; + "~360Spider" 3; + "~80legs" 3; + "~Abonti" 3; + "~Aboundex" 3; + "~Acunetix" 3; + "~ADmantX" 3; + "~AhrefsBot" 3; + "~AIBOT" 3; + "~AiHitBot" 3; + "~Aipbot" 3; + "~Alexibot" 3; + "~Alligator" 3; + "~AllSubmitter" 3; + "~Anarchie" 3; + "~Apexoo" 3; + "~ASPSeek" 3; + "~Asterias" 3; + "~Attach" 3; + "~autoemailspider" 3; + "~BackDoorBot" 3; + "~BackStreet" 3; + "~BackWeb" 3; + "~Badass" 3; + "~Bandit" 3; + "~BatchFTP" 3; + "~Battleztar\ Bazinga" 3; + "~BBBike" 3; + "~BDFetch" 3; + "~BetaBot" 3; + "~Bigfoot" 3; + "~Bitacle" 3; + "~Blackboard" 3; + "~Black\ Hole" 3; + "~BlackWidow" 3; + "~BLEXBot" 3; + "~Blow" 3; + "~BlowFish" 3; + "~Boardreader" 3; + "~Bolt" 3; + "~BotALot" 3; + "~Brandprotect" 3; + "~Brandwatch" 3; + "~Bubing" 3; + "~Buddy" 3; + "~BuiltBotTough" 3; + "~BuiltWith" 3; + "~Bullseye" 3; + "~BunnySlippers" 3; + "~BuzzSumo" 3; + "~Calculon" 3; + "~CATExplorador" 3; + "~CazoodleBot" 3; + "~CCBot" 3; + "~Cegbfeieh" 3; + "~CheeseBot" 3; + "~CherryPicker" 3; + "~ChinaClaw" 3; + "~Chlooe" 3; + "~Claritybot" 3; + "~Cliqzbot" 3; + "~Cogentbot" 3; + "~Collector" 3; + "~Copier" 3; + "~CopyRightCheck" 3; + "~Copyscape" 3; + "~Cosmos" 3; + "~Craftbot" 3; + "~CrazyWebCrawler" 3; + "~Crescent" 3; + "~CSHttp" 3; + "~Curious" 3; + "~Custo" 3; + "~DatabaseDriverMysqli" 3; + "~DBLBot" 3; + "~Demon" 3; + "~Deusu" 3; + "~Devil" 3; + "~DIIbot" 3; + "~Dirbuster" 3; + "~Disco" 3; + "~Discobot" 3; + "~Discoverybot" 3; + "~DittoSpyder" 3; + "~DomainAppender" 3; + "~DomainCrawler" 3; + "~DomainSigmaCrawler" 3; + "~Dotbot" 3; + "~Download\ Demon" 3; + "~Download\ Devil" 3; + "~Download\ Wonder" 3; + "~Dragonfly" 3; + "~Drip" 3; + "~DTS\ Agent" 3; + "~EasyDL" 3; + "~Ebingbong" 3; + "~eCatch" 3; + "~ECCP/1.0" 3; + "~Ecxi" 3; + "~EirGrabber" 3; + "~EMail\ Collector" 3; + "~EMail\ Extractor" 3; + "~EMail\ Siphon" 3; + "~EMail\ Wolf" 3; + "~EroCrawler" 3; + "~Evil" 3; + "~Exabot" 3; + "~Express\ WebPictures" 3; + "~Extractor" 3; + "~ExtractorPro" 3; + "~EyeNetIE" 3; + "~Ezooms" 3; + "~FHscan" 3; + "~Fimap" 3; + "~Findxbot" 3; + "~Firefox/7.0" 3; + "~FlashGet" 3; + "~Flunky" 3; + "~Foobot" 3; + "~Freeuploader" 3; + "~FrontPage" 3; + "~Fyrebot" 3; + "~GalaxyBot" 3; + "~Genieo" 3; + "~Getintent" 3; + "~GetRight" 3; + "~GetWeb" 3; + "~Gigablast" 3; + "~Gigabot" 3; + "~Go-Ahead-Got-It" 3; + "~Gotit" 3; + "~GoZilla" 3; + "~Go!Zilla" 3; + "~Grabber" 3; + "~GrabNet" 3; + "~Grafula" 3; + "~GrapeFX" 3; + "~GrapeshotCrawler" 3; + "~GridBot" 3; + "~GT::WWW" 3; + "~HaosouSpider" 3; + "~Harvest" 3; + "~Havij" 3; + "~HEADMasterSEO" 3; + "~Heritrix" 3; + "~Hloader" 3; + "~HMView" 3; + "~HTMLparser" 3; + "~HTTP::Lite" 3; + "~HTTrack" 3; + "~Humanlinks" 3; + "~HybridBot" 3; + "~Iblog" 3; + "~IDBot" 3; + "~Id-search" 3; + "~IlseBot" 3; + "~Image\ Fetch" 3; + "~Image\ Stripper" 3; + "~Image\ Sucker" 3; + "~Indy\ Library" 3; + "~InfoNaviRobot" 3; + "~InfoTekies" 3; + "~Intelliseek" 3; + "~InterGET" 3; + "~Internet\ Ninja" 3; + "~InternetSeer" 3; + "~internetVista\ monitor" 3; + "~Iria" 3; + "~IRLbot" 3; + "~Iskanie" 3; + "~JamesBOT" 3; + "~Jbrofuzz" 3; + "~JennyBot" 3; + "~JetCar" 3; + "~JikeSpider" 3; + "~JOC\ Web\ Spider" 3; + "~Joomla" 3; + "~JustView" 3; + "~Jyxobot" 3; + "~Kenjin\ Spider" 3; + "~Keyword\ Density" 3; + "~Lanshanbot" 3; + "~Larbin" 3; + "~LeechFTP" 3; + "~LeechGet" 3; + "~LexiBot" 3; + "~Lftp" 3; + "~LibWeb" 3; + "~Libwhisker" 3; + "~Lightspeedsystems" 3; + "~Likse" 3; + "~Linkdexbot" 3; + "~LinkextractorPro" 3; + "~LinkpadBot" 3; + "~LinkScan" 3; + "~LinksManager" 3; + "~LinkWalker" 3; + "~LinqiaMetadataDownloaderBot" 3; + "~LinqiaRSSBot" 3; + "~LinqiaScrapeBot" 3; + "~Lipperhey" 3; + "~Litemage_walker" 3; + "~Lmspider" 3; + "~LNSpiderguy" 3; + "~Ltx71" 3; + "~lwp-request" 3; + "~LWP::Simple" 3; + "~lwp-trivial" 3; + "~Magnet" 3; + "~Mag-Net" 3; + "~magpie-crawler" 3; + "~Mail.ru" 3; + "~Majestic12" 3; + "~MarkMonitor" 3; + "~MarkWatch" 3; + "~Masscan" 3; + "~Mass\ Downloader" 3; + "~Mata\ Hari" 3; + "~Meanpathbot" 3; + "~MegaIndex.ru" 3; + "~Metauri" 3; + "~MFC_Tear_Sample" 3; + "~Microsoft\ Data\ Access" 3; + "~Microsoft\ URL\ Control" 3; + "~MIDown\ tool" 3; + "~MIIxpc" 3; + "~Mister\ PiX" 3; + "~MJ12bot" 3; + "~Mojeek" 3; + "~MSFrontPage" 3; + "~MSIE\ 6.0" 3; + "~MSIECrawler" 3; + "~Msrabot" 3; + "~MS\ Web\ Services\ Client\ Protocol" 3; + "~Musobot" 3; + "~Name\ Intelligence" 3; + "~Nameprotect" 3; + "~Navroad" 3; + "~NearSite" 3; + "~Needle" 3; + "~Nessus" 3; + "~NetAnts" 3; + "~Netcraft" 3; + "~netEstate\ NE\ Crawler" 3; + "~NetLyzer" 3; + "~NetMechanic" 3; + "~NetSpider" 3; + "~Nettrack" 3; + "~Net\ Vampire" 3; + "~NetZIP" 3; + "~NextGenSearchBot" 3; + "~Nibbler" 3; + "~NICErsPRO" 3; + "~Niki-bot" 3; + "~Nikto" 3; + "~NimbleCrawler" 3; + "~Ninja" 3; + "~Nmap" 3; + "~NPbot" 3; + "~Nutch" 3; + "~Octopus" 3; + "~Offline\ Explorer" 3; + "~Offline\ Navigator" 3; + "~Openfind" 3; + "~OpenLinkProfiler" 3; + "~Openvas" 3; + "~OrangeBot" 3; + "~OrangeSpider" 3; + "~OutfoxBot" 3; + "~PageAnalyzer" 3; + "~Page\ Analyzer" 3; + "~PageGrabber" 3; + "~Page\ Grabber" 3; + "~page\ scorer" 3; + "~PageScorer" 3; + "~Panscient" 3; + "~Papa\ Foto" 3; + "~Pavuk" 3; + "~pcBrowser" 3; + "~PECL::HTTP" 3; + "~PeoplePal" 3; + "~PHPCrawl" 3; + "~Picscout" 3; + "~Picsearch" 3; + "~Pimonster" 3; + "~Pi-Monster" 3; + "~Pixray" 3; + "~PleaseCrawl" 3; + "~Pockey" 3; + "~POE-Component-Client-HTTP" 3; + "~Probethenet" 3; + "~ProPowerBot" 3; + "~ProWebWalker" 3; + "~Proximic" 3; + "~Psbot" 3; + "~Pump" 3; + "~PyCurl" 3; + "~QueryN\ Metasearch" 3; + "~Qwantify" 3; + "~RankActiveLinkBot" 3; + "~Rankivabot" 3; + "~RealDownload" 3; + "~Reaper" 3; + "~Recorder" 3; + "~RedesScrapy" 3; + "~ReGet" 3; + "~RepoMonkey" 3; + "~Ripper" 3; + "~RocketCrawler" 3; + "~Rogerbot" 3; + "~SalesIntelligent" 3; + "~SBIder" 3; + "~ScanAlert" 3; + "~Scanbot" 3; + "~Scrapy" 3; + "~Screaming" 3; + "~Screaming\ Frog\ SEO\ Spider" 3; + "~ScreenerBot" 3; + "~Searchestate" 3; + "~SearchmetricsBot" 3; + "~Semrush" 3; + "~SemrushBot" 3; + "~SEOkicks" 3; + "~SEOkicks-Robot" 3; + "~SEOlyticsCrawler" 3; + "~Seomoz" 3; + "~SEOprofiler" 3; + "~SEOstats" 3; + "~Siphon" 3; + "~SISTRIX" 3; + "~SISTRIX\ Crawler" 3; + "~Sitebeam" 3; + "~SiteExplorer" 3; + "~Siteimprove" 3; + "~SiteLockSpider" 3; + "~SiteSnagger" 3; + "~SiteSucker" 3; + "~Site\ Sucker" 3; + "~Sitevigil" 3; + "~Slackbot-LinkExpanding" 3; + "~SlySearch" 3; + "~SmartDownload" 3; + "~Snake" 3; + "~Snapbot" 3; + "~Snoopy" 3; + "~SocialRankIOBot" 3; + "~Sogou\ web\ spider" 3; + "~Sosospider" 3; + "~SpaceBison" 3; + "~Spammen" 3; + "~SpankBot" 3; + "~Spanner" 3; + "~Spbot" 3; + "~Spinn3r" 3; + "~SputnikBot" 3; + "~Sqlmap" 3; + "~Sqlworm" 3; + "~Sqworm" 3; + "~Steeler" 3; + "~Stripper" 3; + "~Sucker" 3; + "~Sucuri" 3; + "~SuperBot" 3; + "~SuperHTTP" 3; + "~Surfbot" 3; + "~SurveyBot" 3; + "~Suzuran" 3; + "~Swiftbot" 3; + "~Szukacz" 3; + "~T0PHackTeam" 3; + "~T8Abot" 3; + "~tAkeOut" 3; + "~Teleport" 3; + "~TeleportPro" 3; + "~Telesoft" 3; + "~Telesphoreo" 3; + "~Telesphorep" 3; + "~The\ Intraformant" 3; + "~TheNomad" 3; + "~TightTwatBot" 3; + "~Titan" 3; + "~Toata" 3; + "~Toweyabot" 3; + "~Trendictionbot" 3; + "~True_Robot" 3; + "~Turingos" 3; + "~TurnitinBot" 3; + "~Turnitin\ Bot" 3; + "~Turnitin\ Robot" 3; + "~TwengaBot" 3; + "~Twice" 3; + "~Typhoeus" 3; + "~UnisterBot" 3; + "~URLy.Warning" 3; + "~URLy\ Warning" 3; + "~Vacuum" 3; + "~Vagabondo" 3; + "~VB\ Project" 3; + "~VCI" 3; + "~VeriCiteCrawler" 3; + "~VidibleScraper" 3; + "~VoidEYE" 3; + "~Voil" 3; + "~Voltron" 3; + "~Wallpapers/3.0" 3; + "~WallpapersHD" 3; + "~WASALive-Bot" 3; + "~WBSearchBot" 3; + "~Webalta" 3; + "~WebAuto" 3; + "~Web\ Auto" 3; + "~WebBandit" 3; + "~Web\ Bandit" 3; + "~WebCollage" 3; + "~Web\ Collage" 3; + "~WebCopier" 3; + "~Web\ Copier" 3; + "~WEBDAV" 3; + "~WEBDAV\ Client" 3; + "~WebEnhancer" 3; + "~Web\ Enhancer" 3; + "~WebFetch" 3; + "~Web\ Fetch" 3; + "~WebFuck" 3; + "~Web\ Fuck" 3; + "~WebGo\ IS" 3; + "~WebImageCollector" 3; + "~Web\ Image\ Collector" 3; + "~WebLeacher" 3; + "~WebmasterWorldForumBot" 3; + "~webmeup-crawler" 3; + "~WebPix" 3; + "~Web\ Pix" 3; + "~WebReaper" 3; + "~Web\ Reaper" 3; + "~WebSauger" 3; + "~Web\ Sauger" 3; + "~Webshag" 3; + "~WebsiteExtractor" 3; + "~Website\ Extractor" 3; + "~WebsiteQuester" 3; + "~Website\ Quester" 3; + "~Webster" 3; + "~WebStripper" 3; + "~Web\ Stripper" 3; + "~WebSucker" 3; + "~Web\ Sucker" 3; + "~WebWhacker" 3; + "~Web\ Whacker" 3; + "~WebZIP" 3; + "~WeSEE" 3; + "~Whack" 3; + "~Whacker" 3; + "~Whatweb" 3; + "~Widow" 3; + "~WinHTTrack" 3; + "~WiseGuys\ Robot" 3; + "~WISENutbot" 3; + "~Wonderbot" 3; + "~Woobot" 3; + "~Wotbox" 3; + "~Wprecon" 3; + "~WPScan" 3; + "~WWW-Collector-E" 3; + "~WWW-Mechanize" 3; + "~WWW::Mechanize" 3; + "~WWWOFFLE" 3; + "~x22Mozilla" 3; + "~Xaldon_WebSpider" 3; + "~Xaldon\ WebSpider" 3; + "~Xenu" 3; + "~YoudaoBot" 3; + "~Zade" 3; + "~Zermelo" 3; + "~Zeus" 3; + "~Zgrab" 3; + "~Zitebot" 3; + "~ZmEu" 3; + "~ZumBot" 3; + "~ZyBorg" 3; # END BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ### } -# **************************************** -# SECOND BLOCK BY REFERER STRINGS AND URLS -# **************************************** - -# Add here all referrer words and URL's that are to blocked. +# ***************************************** +# SECOND BLOCK BY REFERRER STRINGS AND URLS +# ***************************************** # ***************** # PLEASE TEST !!!! # ***************** -# ALWAYS test referers that you add. This is done manually as follows +# ALWAYS test referrers that you add. This is done manually as follows -# curl -I http://www.yourdomain.com -e http://anything.adcash.com -# curl -I http://www.yourdomain.com -e http://www.goodwebsite.com/not-adcash -# curl -I http://www.yourdomain.com -e http://www.betterwebsite.com/not/adcash +# curl -I http://www.yourdomain.com -e http://anything.adcash.com --- GIVES YOU: curl: (52) Empty reply from server +# curl -I http://www.yourdomain.com -e http://www.goodwebsite.com/not-adcash --- GIVES YOU: curl: (52) Empty reply from server +# curl -I http://www.yourdomain.com -e http://www.betterwebsite.com/not/adcash --- GIVES YOU: curl: (52) Empty reply from server -# This uses curl to send the referer string to your site and you should see an immediate -# 403 Forbidden Error or No Response at all if you use the 444 error like I do. +# curl -I http://www.yourdomain.com -e http://www.google.com --- GIVES YOU: full html output of the web page +# curl -I http://www.yourdomain.com -e http://www.microsoft.com --- GIVES YOU: full html output of the web page -# Because of case-insensitive matching any combination of capitilization in the names -# will all produce a positive hit - make sure you always test thoroughly and monitor logs -# This also does NOT check for a preceding www. nor does it check for it ending in .com -# .net .org or any long string attached at the end. It also does not care if the referer -# was sent with http https or even ftp. +# Because of case-insensitive matching any combination of capitilization in the names will all produce a positive hit +# make sure you always test thoroughly and monitor logs. This section below also does NOT check for a preceding www. +# and it also does not care if the referrer request was sent with http https or even ftp. -# REAL WORLD EXAMPLE -# ******************* -# If you were a photographer like me and say took a photo of a "girl" and you then posted -# a blog showing everyone your new photo and your blog slug / permalink was -# http://www.mysite.com/blog/photo-of-girl/ -# You can go and monitor your logs and you will see lots of 444 from other pages on your -# site that have been clicked on sending that page as a referer so in the example below -# you will generate a 403 error. - -# curl --referer http://www.mysite.com/blog/photo-of-girl/ http://www.mysite.com/ - -# So please be careful with these and think carefully before you add new words. -# Remember we are trying to keep out the general riff-raff not kill your web sites. - -# ********************************************************************** -# Now we map all bad referer words below to a variable called $bad_words -# ********************************************************************** +# *********************************************************************** +# Now we map all bad referrer words below to a variable called $bad_words +# *********************************************************************** map $http_referer $bad_words { default 0; -# ************************* -# Bad Referer Word Scanning -# ************************* +# ************************** +# Bad Referrer Word Scanning +# ************************** # These are Words and Terms often found tagged onto domains or within url query strings. # Create and Customize Your Own Bad Referrer Words Here using the new Include File Method @@ -752,18 +718,27 @@ map $http_referer $bad_words { # customized list of bad referrer words are automatically now included for you # Read Comments inside bad-referrer-words.conf for customization tips. # Updating the main globalblacklist.conf file will not touch your custom include files + # BE VERY CAREFUL using the bad-referrer-words.conf file - please read the comment and + # examples inside the include file for detailed explanations into how seriously this can + # affect your own site from serving assets or other innocent sites from accessing your site + +# ****************************************************************************** +# START CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ****************************************************************************** -# START CUSTOM BAD REFERRER WORDS ### DO NOT EDIT THIS LINE AT ALL ### include /etc/nginx/bots.d/bad-referrer-words.conf; -# END CUSTOM BAD REFERRER WORDS ### DO NOT EDIT THIS LINE AT ALL ### + +# **************************************************************************** +# END CUSTOM BAD REFERRER WORDS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# **************************************************************************** } -# ************************ -# Bad Referer Domain Names -# ************************ +# ************************* +# Bad Referrer Domain Names +# ************************* -# Now a list of bad referer urls these domains or any combination of them ie .com .net +# Now a list of bad referrer urls these domains or any combination of them ie .com .net # will be blocked out. Doesn't matter if the protocol is http, https or even ftp # This section includes: @@ -777,78 +752,86 @@ map $http_referer $bad_words { # PLEASE TEST !!!! # ***************** -# ALWAYS test referers that you add. This is done manually as follows +# ALWAYS test referrers that you add. This is done manually as follows -# curl -I http://www.yourdomain.com -e http://8gold.com +# curl -I http://www.yourdomain.com -e http://8gold.com --- GIVES YOU: curl: (52) Empty reply from server -# This uses curl to send the referer string to your site and you should see an immediate -# 403 Forbidden Error or No Response at all if you use the 444 error like I do. +# Because of case-insensitive matching any combination of capitilization will all produce a positive hit - make sure you always test. -# Because of case-insensitive matching any combination of capitilization -# will all produce a positive hit - make sure you always test. +# For Example any of the following variations below of 8gold.com will be detected and blocked -# curl -I http://www.yourdomain.com -e http://NOT-8gold.com -# curl -I http://www.yourdomain.com -e http://this.is.not8gOlD.net -# curl -I http://www.yourdomain.com -e ftp://8gold.com -# curl -I http://www.yourdomain.com -e ftp://www.weare8gold.NET -# curl -I http://www.yourdomain.com -e https://subdomain.8gold.com -# curl -I http://www.yourdomain.com -e https://NOT8GolD.org +# curl -I http://www.yourdomain.com -e http://NOT-8gold.com --- GIVES YOU: curl: (52) Empty reply from server +# curl -I http://www.yourdomain.com -e http://this.is.not8gOlD.net --- GIVES YOU: curl: (52) Empty reply from server +# curl -I http://www.yourdomain.com -e ftp://8gold.com --- GIVES YOU: curl: (52) Empty reply from server +# curl -I http://www.yourdomain.com -e ftp://www.weare8gold.NET --- GIVES YOU: curl: (52) Empty reply from server +# curl -I http://www.yourdomain.com -e https://subdomain.8gold.com --- GIVES YOU: curl: (52) Empty reply from server +# curl -I http://www.yourdomain.com -e https://NOT8GolD.org --- GIVES YOU: curl: (52) Empty reply from server -# This works exactly like the bad referer word lists above and is very strict !!! -# I have gone for the simple stricter approach which blocks all variants for those -# who just hop out and but another domain name. - -# So if you see a bad referer from wearegoogle.com and you want to block them just add +# So if you see a bad referrer from wearegoogle.com and you want to block them just add # them as "~*wearegoogle.com" don't ever go and do something like "~*google(-|.)" you will -# kill all your SEO in a week. Rather also send a Pull Request by following the instructions -# in the Pull_Requests_Here_Please folder. +# kill all your SEO in a week. -# I also include any sites that hotlink images from my sites into the list below. -# There are hundreds of image stealing sites out there so this list WILL grow now doubt. +# To add your own custom bad referrers use the custom include file +# /etc/nginx/bots.d/custom-bad-referrers.conf +# Or send a Pull Request to add it to the global blacklist for other users. +# In the bad referrers section I also include sites that hotlink images without permission. # *********************************************************************** -# Now we map all good & bad referer urls to variable called #bad_referer +# Now we map all good & bad referrer urls to variable called #bad_referer # *********************************************************************** map $http_referer $bad_referer { hostnames; default 0; -# ************************************ -# GOOD REFERERS - Spared from Checking -# ************************************ +# ************************************* +# GOOD REFERRERS - Spared from Checking +# ************************************* # Add all your own web site domain names and server names in this section -# WHITELIST Your Own Domain Names Here using the Include File Method +# WHITELIST Your Own Domain Names Here using the new Include File Method # New Method Uses the include file below so that when pulling future updates your # whitelisted domain names are automatically now included for you. # Read Comments inside whitelist-domains.conf for customization tips. # Updating the main globalblacklist.conf file will not touch your custom include files -# START WHITELISTED DOMAINS ### DO NOT EDIT THIS LINE AT ALL ### +# ************************************************************************ +# START WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ************************************************************************ + include /etc/nginx/bots.d/whitelist-domains.conf; -# END WHITELISTED DOMAINS ### DO NOT EDIT THIS LINE AT ALL ### -# ******************************************* -# CUSTOM BAD REFERERS - Add your Own -# ******************************************* +# ********************************************************************** +# END WHITELISTED DOMAINS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ********************************************************************** -# Add any extra bad referers in the following include file to have them +# *********************************** +# CUSTOM BAD REFERRERS - Add your Own +# *********************************** + +# Add any extra bad referrers in the following include file to have them # permanently included and blocked - avoid duplicates in your custom file -# START CUSTOM BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ### - include /etc/nginx/bots.d/custom-bad-referrers.conf; -# END CUSTOM BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ### +# ************************************************************************* +# START CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ************************************************************************* -# START BAD REFERERS ### DO NOT EDIT THIS LINE AT ALL ### + include /etc/nginx/bots.d/custom-bad-referrers.conf; + +# *********************************************************************** +# END CUSTOM BAD REFERRERS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# *********************************************************************** + +# START BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ### "~*000free.us" 1; "~*007angels.com" 1; "~*00author.com" 1; "~*00go.com" 1; "~*00it.com" 1; "~*00webcams.com" 1; + "~*01apple.com" 1; "~*0912701309f8ce.com" 1; "~*0c47f8422d3f.com" 1; "~*0n-line.tv" 1; @@ -886,6 +869,7 @@ map $http_referer $bad_referer { "~*202ch.com" 1; "~*20pascals.nl" 1; "~*214jaluwobapef.cf" 1; + "~*21h2o.com" 1; "~*24videos.tv" 1; "~*24x7-server-support.site" 1; "~*256bit.by" 1; @@ -928,6 +912,7 @@ map $http_referer $bad_referer { "~*5forex.ru" 1; "~*5i2.net" 1; "~*5kstore.com" 1; + "~*5tey7463.weebly.com" 1; "~*5u.com" 1; "~*66cpwgln.space" 1; "~*6hopping.com" 1; @@ -990,6 +975,7 @@ map $http_referer $bad_referer { "~*acheterviagrafr24.com" 1; "~*acortarurl.es" 1; "~*actionnooz.com" 1; + "~*activecampaign.dreamhosters.com" 1; "~*activepr.ru" 1; "~*actulite.com" 1; "~*acunetix-referrer.com" 1; @@ -1072,6 +1058,7 @@ map $http_referer $bad_referer { "~*agroeconom.kz" 1; "~*agysya.ru" 1; "~*ahrefs.com" 1; + "~*aihelen.net" 1; "~*aimayangzhi.com" 1; "~*air-edem.ru" 1; "~*airlandsea.info" 1; @@ -1133,6 +1120,8 @@ map $http_referer $bad_referer { "~*alpharma.net" 1; "~*alphavisions.net" 1; "~*alpinism.ru" 1; + "~*altamayoztourism.com" 1; + "~*alta-realestate.com" 1; "~*alveris.ru" 1; "~*alyeskaresort.com" 1; "~*am15.net" 1; @@ -1164,6 +1153,7 @@ map $http_referer $bad_referer { "~*android-systems.ru" 1; "~*android-vsem.org" 1; "~*angigreene.com" 1; + "~*angkortours.vn" 1; "~*animali.deagostinipassion.it" 1; "~*animebox.com.ua" 1; "~*anime.dougasouko.com" 1; @@ -1172,6 +1162,7 @@ map $http_referer $bad_referer { "~*anonymizeme.pro" 1; "~*anticrawler.org" 1; "~*anti-virus-removal.info" 1; + "~*antons-transporte.de" 1; "~*aosexkontakte.net" 1; "~*aosheng-tech.com" 1; "~*apartamentwroclaw.eu" 1; @@ -1201,6 +1192,7 @@ map $http_referer $bad_referer { "~*apiwebwebgetcom-a.akamaihd.net" 1; "~*apiwoodensealcom-a.akamaihd.net" 1; "~*apparel-offer.com" 1; + "~*appartement-stumm.at" 1; "~*apper.de" 1; "~*appfastplay.com" 1; "~*appfixing.space" 1; @@ -1254,6 +1246,7 @@ map $http_referer $bad_referer { "~*asia-forum.ru" 1; "~*asiengirls.net" 1; "~*asmxsatadriverin.aircus.com" 1; + "~*asociatia-tipografilor-transilvania.ro" 1; "~*asophoto.com" 1; "~*asrv-a.akamaihd.net" 1; "~*asrv-a.akamoihd.net" 1; @@ -1454,6 +1447,7 @@ map $http_referer $bad_referer { "~*beztuberkuleza.ru" 1; "~*bfz.biz" 1; "~*bharatdefencekavach.com" 1; + "~*bibys.com" 1; "~*bidbuy.co.kr" 1; "~*bidr.trellian.com" 1; "~*bif-ru.info" 1; @@ -1501,6 +1495,7 @@ map $http_referer $bad_referer { "~*blagovest-med.ru" 1; "~*blavia.00author.com" 1; "~*blockworld.ru" 1; + "~*blog.axant.it" 1; "~*blog.f00kclan.de" 1; "~*bloggedporn.com" 1; "~*bloggen.be" 1; @@ -1555,6 +1550,7 @@ map $http_referer $bad_referer { "~*brainxs.us" 1; "~*braip.com.br" 1; "~*brakehawk.com" 1; + "~*brandehk.dk" 1; "~*brandov.ru" 1; "~*break-the-chains.com" 1; "~*breastaugmentation.co.za" 1; @@ -1581,6 +1577,7 @@ map $http_referer $bad_referer { "~*btvn.ru" 1; "~*buchananshardware.com" 1; "~*budpost.com.ua" 1; + "~*buehne-fuer-menschenrechte.de" 1; "~*bugof.gq" 1; "~*buigas.00it.com" 1; "~*builtwith.com" 1; @@ -1625,6 +1622,7 @@ map $http_referer $bad_referer { "~*bvlgaribracelet.xyz" 1; "~*bvlgariring.xyz" 1; "~*bvlgariwallet.xyz" 1; + "~*bwlx.prepedu.cn" 1; "~*bycontext.com" 1; "~*byme.se" 1; "~*bytimedance.ru" 1; @@ -1646,6 +1644,7 @@ map $http_referer $bad_referer { "~*calvet.altervista.org" 1; "~*camdenmemorials.com" 1; "~*camdolls.com" 1; + "~*camel-beach.com" 1; "~*cam-kontakte.org" 1; "~*candcstuccoandstone.com" 1; "~*candelluxsklep.pl" 1; @@ -1693,6 +1692,7 @@ map $http_referer $bad_referer { "~*cartierring.xyz" 1; "~*cartiertrinity.xyz" 1; "~*cartierwatch.xyz" 1; + "~*cartujano-pre.de" 1; "~*casablancamanor.co.za" 1; "~*cashkitten-a.akamaihd.net" 1; "~*cashonads.com" 1; @@ -1812,9 +1812,11 @@ map $http_referer $bad_referer { "~*clicksor.net" 1; "~*clicksotrk.com" 1; "~*clipartnew.com" 1; + "~*clippingphotoindia.com" 1; "~*clips.ua.ac.be" 1; "~*clksite.com" 1; "~*clmforexeu.com" 1; + "~*clothesforcash.com" 1; "~*cl.s7.exct.net" 1; "~*club-musics.ru" 1; "~*club-samodelkin.ru" 1; @@ -1872,12 +1874,14 @@ map $http_referer $bad_referer { "~*compucelunlock.net" 1; "~*com-secure.download" 1; "~*com-supportcenter.website" 1; + "~*comsysnet.com" 1; "~*concealthyself.com" 1; "~*conciergegroup.org" 1; "~*concordexoticrentals.com" 1; "~*confib.ifmo.ru" 1; "~*connectionstrenth.com" 1; "~*conocer-sanabria.com" 1; + "~*consorzioilmosaico.org" 1; "~*constantaservice.net" 1; "~*construmac.com.mx" 1; "~*contextualyield.com" 1; @@ -1927,6 +1931,7 @@ map $http_referer $bad_referer { "~*crirussian.ru" 1; "~*crystalslot.com" 1; "~*cscwtalkto.site" 1; + "~*csgo4.win" 1; "~*cs-passion.pl" 1; "~*cubook.supernew.org" 1; "~*cubs-jerseys.us" 1; @@ -1990,6 +1995,7 @@ map $http_referer $bad_referer { "~*dear-diary.ru" 1; "~*decenttools.com" 1; "~*decorationspcs.com" 1; + "~*decorazilla.com" 1; "~*deda-moroza-zakaz.ru" 1; "~*degerlund.net" 1; "~*dekoration.us" 1; @@ -2004,6 +2010,7 @@ map $http_referer $bad_referer { "~*den-noch24.ru" 1; "~*dentalpearls.com.au" 1; "~*depositfiles-porn.ga" 1; + "~*derevesendeco.com" 1; "~*descargar-musica-gratis.net" 1; "~*designdevise.com" 1; "~*design-ideas.info" 1; @@ -2015,6 +2022,7 @@ map $http_referer $bad_referer { "~*detskie-konstruktory.ru" 1; "~*detskie-zabavi.ru" 1; "~*deutschehobbyhuren.net" 1; + "~*dev78.dioniqlabb.se" 1; "~*devochki.top" 1; "~*dev-seo.blog" 1; "~*de.zapmeta.com" 1; @@ -2154,9 +2162,11 @@ map $http_referer $bad_referer { "~*edelstahlschornstein-123.de" 1; "~*editmedios.com" 1; "~*ed-shop01.ru" 1; + "~*educatemetv.com" 1; "~*education-cz.ru" 1; "~*ee77ee.com" 1; "~*efkt.jp" 1; + "~*efnor-ac.com" 1; "~*egovaleo.it" 1; "~*egvar.net" 1; "~*ekaterinburg.xrus.org" 1; @@ -2260,8 +2270,10 @@ map $http_referer $bad_referer { "~*esnm.ru" 1; "~*espaceinventoristes.com" 1; "~*essayservicewriting.org" 1; + "~*estelight.ru" 1; "~*este-line.com.ua" 1; "~*estibot.com" 1; + "~*etm-consult.de" 1; "~*eu-cookie-law.blogspot.com" 1; "~*eu-cookie-law.info" 1; "~*eugenevaultstorage.com" 1; @@ -2280,6 +2292,7 @@ map $http_referer $bad_referer { "~*everflora.ru" 1; "~*everypony.ru" 1; "~*evidencecleanergold.com" 1; + "~*evogarage.com" 1; "~*evrotekhservis.ru" 1; "~*excaliburfilms.com" 1; "~*exchangeit.gq" 1; @@ -2331,10 +2344,12 @@ map $http_referer $bad_referer { "~*f5mtrack.com" 1; "~*fable.in.ua" 1; "~*facebook-mobile.xyz" 1; + "~*face.hostingx.eu" 1; "~*factorynightclub.com" 1; "~*failingmarriege.blogspot.com" 1; "~*fakehandbags.xyz" 1; "~*falcoware.com" 1; + "~*familienzahnaerzte.com" 1; "~*family1st.ca" 1; "~*familyholiday.ml" 1; "~*familyphysician.ru" 1; @@ -2375,6 +2390,7 @@ map $http_referer $bad_referer { "~*ferieboligkbh.dk" 1; "~*fermersovet.ru" 1; "~*ferretsoft.com" 1; + "~*ferrotodo.com" 1; "~*fertilitetsradgivningen.se" 1; "~*fetishinside.com" 1; "~*fetroshok.ru" 1; @@ -2435,6 +2451,7 @@ map $http_referer $bad_referer { "~*flooringinstallation-edmonton.com" 1; "~*floridahuntingfishingadventures.com" 1; "~*floridamhca.org" 1; + "~*floridamobilebillboards.com" 1; "~*florida-tourism.net" 1; "~*flowersbazar.com" 1; "~*flowwwers.com" 1; @@ -2475,6 +2492,7 @@ map $http_referer $bad_referer { "~*foto-sisek.porngalleries.top" 1; "~*foto-telok.net" 1; "~*fototravel.eu" 1; + "~*foto-weinberger.at" 1; "~*fotoxxxru.com" 1; "~*fotzen-ficken.com" 1; "~*foxinsocks.ru" 1; @@ -2637,6 +2655,7 @@ map $http_referer $bad_referer { "~*gktt.ru" 1; "~*gkvector.ru" 1; "~*glall.ru" 1; + "~*glasof.es" 1; "~*glass-msk.ru" 1; "~*glastecfilms.com.my" 1; "~*glavprofit.ru" 1; @@ -2938,6 +2957,7 @@ map $http_referer $bad_referer { "~*imfamous.info" 1; "~*imgarcade.com" 1; "~*imgata.com" 1; + "~*i-midias.net.br" 1; "~*iminent.com" 1; "~*imitex-plus.ru" 1; "~*immobiliaremassaro.com" 1; @@ -2945,6 +2965,7 @@ map $http_referer $bad_referer { "~*imperiafilm.ru" 1; "~*impisr.edunsk.ru" 1; "~*impisr.ru" 1; + "~*importchinacoach-teach.com" 1; "~*import-sales.com" 1; "~*impresagaia.it" 1; "~*inbabes.sexushost.com" 1; @@ -2972,8 +2993,10 @@ map $http_referer $bad_referer { "~*insider.pro" 1; "~*insomniagamingfestival.com" 1; "~*insta-add.pro" 1; + "~*integritylandscapeservices.com" 1; "~*intelhdgraphicsgtdrive6w.metroblog.com" 1; "~*intellekt21.ru" 1; + "~*intellektmedia.at" 1; "~*interesnie-faktu.ru" 1; "~*interferencer.ru" 1; "~*interfucks.net" 1; @@ -3012,6 +3035,7 @@ map $http_referer $bad_referer { "~*i-service.kz" 1; "~*isistaylorporn.info" 1; "~*iskalko.ru" 1; + "~*islamtoday.co.za" 1; "~*islandminingsupply.wordpress.com" 1; "~*isoveti.ru" 1; "~*ispaniya-costa-blanca.ru" 1; @@ -3036,6 +3060,7 @@ map $http_referer $bad_referer { "~*iwantmyfreecash.com" 1; "~*iwanttodeliver.com" 1; "~*ix20.ru" 1; + "~*iyasimasennka.com" 1; "~*izhevsk.xrus.org" 1; "~*izhevsk.zrus.org" 1; "~*izismile.com" 1; @@ -3064,10 +3089,13 @@ map $http_referer $bad_referer { "~*jerseywholesalebizchina.com" 1; "~*jerseywholesalechinabiz.com" 1; "~*jerseywholesaleelitestore.com" 1; + "~*jestr.org" 1; "~*jetsli.de" 1; + "~*jewelryandfiligree.com" 1; "~*jillepille.com" 1; "~*jimmychoosale.online" 1; "~*jjbabskoe.ru" 1; + "~*jmat.cn" 1; "~*job.icivil.ir" 1; "~*joessmogtestonly.com" 1; "~*johannesburgsingles.co.za" 1; @@ -3090,6 +3118,7 @@ map $http_referer $bad_referer { "~*kaac.ru" 1; "~*kabbalah-red-bracelets.com" 1; "~*kadashihotel.com" 1; + "~*kaidalibor.de" 1; "~*kalandranis.gr" 1; "~*kalb.ru" 1; "~*kamagragelusa.net" 1; @@ -3118,6 +3147,7 @@ map $http_referer $bad_referer { "~*kerwinandcariza.com" 1; "~*ketoanhanoi.info" 1; "~*ketrzyn.pl" 1; + "~*keyhantercume.com" 1; "~*keywordhut.com" 1; "~*keywordsking.com" 1; "~*keywords-monitoring-success.com" 1; @@ -3158,6 +3188,7 @@ map $http_referer $bad_referer { "~*kleine-titten.biz" 1; "~*kletkimehan.ru" 1; "~*kliksaya.com" 1; + "~*klosetkitten.com" 1; "~*kmd-pto.ru" 1; "~*kmgamex.cf" 1; "~*kndxbkdx.bloger.index.hr" 1; @@ -3200,6 +3231,7 @@ map $http_referer $bad_referer { "~*krasnodar.ru" 1; "~*krasnodar.xrus.org" 1; "~*krasnodar.zrus.org" 1; + "~*kreativperlen.ch" 1; "~*kredit-pod-zalog-krasnodar.ru" 1; "~*kriokomora.info" 1; "~*krynica.info" 1; @@ -3242,6 +3274,7 @@ map $http_referer $bad_referer { "~*laptop-4-less.com" 1; "~*laptoper.net" 1; "~*larchik.net" 1; + "~*larose.jb2c.me" 1; "~*larutti.ru" 1; "~*lasvegaslockandsafe.com" 1; "~*laudit.ru" 1; @@ -3347,6 +3380,7 @@ map $http_referer $bad_referer { "~*longadventure.com" 1; "~*long-beach-air-conditioning.com" 1; "~*longgreen.info" 1; + "~*lookingglassemb.com" 1; "~*losangeles-ads.com" 1; "~*lost-alpha.ru" 1; "~*lostfilm-online.ru" 1; @@ -3362,6 +3396,7 @@ map $http_referer $bad_referer { "~*lowephotos.info" 1; "~*low-format.ru" 1; "~*lsex.xyz" 1; + "~*lsp-awak-perikanan.com" 1; "~*ltvperf.com" 1; "~*lubetube.com" 1; "~*luckyshop.net.ua" 1; @@ -3442,6 +3477,7 @@ map $http_referer $bad_referer { "~*mature.free-websites.com" 1; "~*max-eclat.men" 1; "~*maximilitary.ru" 1; + "~*maximpartnerspr.com" 1; "~*maxthon.com" 1; "~*maxxtor.eu" 1; "~*mazda-roadsters.com" 1; @@ -3449,6 +3485,7 @@ map $http_referer $bad_referer { "~*mcadamssupplyco.com" 1; "~*mcar.in.ua" 1; "~*mcnamaratech.com" 1; + "~*mearns-tractors.co.uk" 1; "~*mebel-alait.ru" 1; "~*mebelcomplekt.ru" 1; "~*mebeldekor.com.ua" 1; @@ -3501,6 +3538,7 @@ map $http_referer $bad_referer { "~*metabar.ru" 1; "~*metalonly.info" 1; "~*metarip.ru" 1; + "~*metascephe.com" 1; "~*metaxalonevstizanidine.blogspot.com" 1; "~*methodsmarketing.com" 1; "~*mex-annushka.ru" 1; @@ -3520,6 +3558,7 @@ map $http_referer $bad_referer { "~*migente.com" 1; "~*mikrobiologies.ru" 1; "~*milblueprint.com" 1; + "~*militarysale.pro" 1; "~*millionare.com" 1; "~*mindbox.co.za" 1; "~*minecraft-neo.ru" 1; @@ -3683,6 +3722,7 @@ map $http_referer $bad_referer { "~*mylesosibirsk.ru" 1; "~*mylida.org" 1; "~*myliveblog.ru" 1; + "~*myonigroup.com" 1; "~*myonlinepayday.co" 1; "~*my.opera.com" 1; "~*myperiod.club" 1; @@ -3723,6 +3763,7 @@ map $http_referer $bad_referer { "~*nbsproject.ru" 1; "~*negociosdasha.com" 1; "~*negral.pluto.ro" 1; + "~*nelc.edu.eg" 1; "~*neobux-bg.info" 1; "~*neodownload.webcam" 1; "~*nerudlogistik.ru" 1; @@ -3893,6 +3934,7 @@ map $http_referer $bad_referer { "~*orgasmus-virtual.com" 1; "~*orhonit.com" 1; "~*origin-my.ru" 1; + "~*orion-v.com" 1; "~*ororodnik.goodbb.ru" 1; "~*orsonet.ru" 1; "~*osb.se11.ru" 1; @@ -3921,6 +3963,7 @@ map $http_referer $bad_referer { "~*pagesense.com" 1; "~*paidonlinesites.com" 1; "~*paleohub.info" 1; + "~*palocco.it" 1; "~*palvira.com.ua" 1; "~*pammik.ru" 1; "~*panamaforbeginners.com" 1; @@ -3943,6 +3986,7 @@ map $http_referer $bad_referer { "~*partner-stop.men" 1; "~*partner-trustworthy.men" 1; "~*partybunny.ru" 1; + "~*pateaswing.com" 1; "~*pathwhelp.org" 1; "~*patol01.pw" 1; "~*pattersonsweb.com" 1; @@ -4119,6 +4163,7 @@ map $http_referer $bad_referer { "~*portadd.men" 1; "~*portal-eu.ru" 1; "~*portnoff.od.ua" 1; + "~*porto.abuilder.net" 1; "~*portside.cc" 1; "~*portside.xyz" 1; "~*poshiv-chehol.ru" 1; @@ -4249,6 +4294,7 @@ map $http_referer $bad_referer { "~*quelle.ru" 1; "~*questionmarque.ch" 1; "~*quickbuck.com" 1; + "~*quickcashlimited.com" 1; "~*quickloanbank.com" 1; "~*quick-offer.com" 1; "~*quick-seeker.com" 1; @@ -4344,6 +4390,7 @@ map $http_referer $bad_referer { "~*retailwith.com" 1; "~*retreatia.com" 1; "~*reversing.cc" 1; + "~*revistaindustria.com" 1; "~*rewardpoll.com" 1; "~*reward-survey.net" 1; "~*reyel1985.webnode.fr" 1; @@ -4351,6 +4398,7 @@ map $http_referer $bad_referer { "~*rfd-split.hr" 1; "~*rff-cfal.info" 1; "~*rialp.getenjoyment.net" 1; + "~*ribieiendom.no" 1; "~*richinvestmonitor.com" 1; "~*ric.info" 1; "~*ricorsogiustizia.org" 1; @@ -4405,6 +4453,7 @@ map $http_referer $bad_referer { "~*rybalka-opt.ru" 1; "~*s8-nowy-wygraj.comli.com" 1; "~*sabaapress.com" 1; + "~*sabizonline.com" 1; "~*sack.net" 1; "~*sad-torg.com.ua" 1; "~*sady-urala.ru" 1; @@ -4413,6 +4462,7 @@ map $http_referer $bad_referer { "~*sakhboard.ru" 1; "~*sale-japan.com" 1; "~*saletool.ru" 1; + "~*salmonfishingsacramentoriver.com" 1; "~*salutmontreal.com" 1; "~*samoiedo.it" 1; "~*samolet.fr" 1; @@ -4509,6 +4559,7 @@ map $http_referer $bad_referer { "~*searchtooknow-a.akamaihd.net" 1; "~*searchwebknow-a.akamaihd.net" 1; "~*seasonvar.ru" 1; + "~*se.bnt-team.com" 1; "~*seccioncontrabajo.com" 1; "~*secretscook.ru" 1; "~*secret.xn--oogle-wmc.com" 1; @@ -4614,6 +4665,7 @@ map $http_referer $bad_referer { "~*shodanhq.com" 1; "~*shoesonlinebuy.cn" 1; "~*shoesonlinebuy.xyz" 1; + "~*shohanb.com" 1; "~*shop.acim.org" 1; "~*shopcheermakeup.info" 1; "~*shop-electron.ru" 1; @@ -4676,6 +4728,7 @@ map $http_referer $bad_referer { "~*sixcooler.de" 1; "~*sizeplus.work" 1; "~*skachat-besplatno-obrazcy.ru" 1; + "~*skanninge.se" 1; "~*skatestick.bid" 1; "~*sk.golden-praga.ru" 1; "~*skincrate.net" 1; @@ -4735,6 +4788,7 @@ map $http_referer $bad_referer { "~*socialbutton.xyz" 1; "~*social-fun.ru" 1; "~*socialmadesimple.com" 1; + "~*socialmediasuggest.com" 1; "~*social-search.me" 1; "~*socialseet.ru" 1; "~*social-s-ggg.xyz" 1; @@ -4910,6 +4964,7 @@ map $http_referer $bad_referer { "~*syvertsen-da.no" 1; "~*szamponrevita.pl" 1; "~*t3chtonic.com" 1; + "~*taaaak.com" 1; "~*tabletkinaodchudzanie.com.pl" 1; "~*taboola.com" 1; "~*tacbelarus.ru" 1; @@ -5118,6 +5173,7 @@ map $http_referer $bad_referer { "~*tracksz.co" 1; "~*tradedeals.biz" 1; "~*traderzplanet.in" 1; + "~*tradgardspartner.se" 1; "~*trafaret74.ru" 1; "~*traffic100.com" 1; "~*traffic2cash.org" 1; @@ -5165,6 +5221,7 @@ map $http_referer $bad_referer { "~*tubeline.biz" 1; "~*tuberkulezanet.ru" 1; "~*tuberkuleznik.ru" 1; + "~*tubo360.com" 1; "~*tuckermktg.com" 1; "~*tuckpointingmasonrysystems.com" 1; "~*tula.howotorg.ru" 1; @@ -5227,6 +5284,7 @@ map $http_referer $bad_referer { "~*umityangin.net" 1; "~*umnovocaminho.com" 1; "~*um-razum.ru" 1; + "~*unacittaconte.org" 1; "~*underthesite.com" 1; "~*uni.me" 1; "~*unimodemhalfduplefw.pen.io" 1; @@ -5274,6 +5332,7 @@ map $http_referer $bad_referer { "~*vadimkravtcov.ru" 1; "~*validdomain.xyz" 1; "~*valkiria-tk.ru" 1; + "~*valmetrundan.se" 1; "~*valoresito.com" 1; "~*valuado.com" 1; "~*valueclick.com" 1; @@ -5509,6 +5568,7 @@ map $http_referer $bad_referer { "~*witherrom55.eklablog.fr" 1; "~*withstandingheartwarming.com" 1; "~*w-journal.ru" 1; + "~*wladimirpayen.com" 1; "~*wma-x.com" 1; "~*wnoz.de" 1; "~*womama.ru" 1; @@ -5552,6 +5612,7 @@ map $http_referer $bad_referer { "~*wyniki-lista.pl" 1; "~*wzgyyq.com" 1; "~*xaijo.com" 1; + "~*xaylapdiendanang.com" 1; "~*xbaboon.com" 1; "~*xblog.in" 1; "~*xboxster.ru" 1; @@ -5663,9 +5724,11 @@ map $http_referer $bad_referer { "~*yeartwit.com" 1; "~*yebocasino.com" 1; "~*yebocasino.co.za" 1; + "~*yellocloud.be" 1; "~*yhit.press" 1; "~*ynymnwbm.bloger.index.hr" 1; "~*yogamatsexpert.com" 1; + "~*yoluxuryevents.com" 1; "~*yoopsie.com" 1; "~*yopoint.in" 1; "~*yorkshireccc.com" 1; @@ -5735,6 +5798,7 @@ map $http_referer $bad_referer { "~*zebradudka.com" 1; "~*zebramart.ru" 1; "~*zed21.net" 1; + "~*zeg-distribution.com" 1; "~*zeleznobeton.ru" 1; "~*zero1.it" 1; "~*zerocash.msk.ru" 1; @@ -5791,8 +5855,7 @@ map $http_referer $bad_referer { "~*zx6.ru" 1; "~*zytpirwai.net" 1; "~*zzbroya.com.ua" 1; -# END BAD REFERERS ### DO NOT EDIT THIS LINE AT ALL ### - +# END BAD REFERRERS ### DO NOT EDIT THIS LINE AT ALL ### } @@ -5822,29 +5885,21 @@ geo $validate_client { # Read Comments inside whitelist-ips.conf for customization tips. # Updating the main globalblacklist.conf file will not touch your custom include files -# START WHITELISTED IP RANGES ### DO NOT EDIT THIS LINE AT ALL ### +# ************************************************************************** +# START WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ************************************************************************** + include /etc/nginx/bots.d/whitelist-ips.conf; -# END WHITELISTED IP RANGES ### DO NOT EDIT THIS LINE AT ALL ### -# *********** -# Google Bots -# *********** +# ************************************************************************ +# END WHITELISTED IP RANGES ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ************************************************************************ -# For Safety Sake Google's Known BOT IP Ranges are all white listed in case you add -# anything lower down that you mistakenly picked up as a bad bot. +# **************** +# Google IP Ranges +# **************** -# UA "AdsBot-Google (+http://www.google.com/adsbot.html)" -# UA "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.3; +http://www.google.com/bot.html)" -# UA "Googlebot-Image/1.0" -# UA "Googlebot/2.1 (+http://www.google.com/bot.html)" -# UA "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" -# UA "Googlebot/Test (+http://www.googlebot.com/bot.html)" -# UA "Googlebot/Test" -# UA "Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)" -# UA "Mediapartners-Google/2.1" -# UA "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -# UA "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" -# UA "gsa-crawler (Enterprise; S4-E9LJ2B82FJJAA; me@mycompany.com)" +# For Safety Sake all Google's Known IP Ranges are all white listed # START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ### 108.177.8.0/21 0; @@ -5871,9 +5926,11 @@ geo $validate_client { 74.125.0.0/16 0; # END GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ### -# ********* -# Bing Bots -# ********* +# ************** +# Bing IP Ranges +# ************** + +# For Safety Sake all Bing's Known IP Ranges are all white listed # START BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ### 131.253.24.0/22 0; @@ -5901,6 +5958,12 @@ geo $validate_client { 65.55.55.0/24 0; # END BING IP RANGES ### DO NOT EDIT THIS LINE AT ALL ### +# ******************** +# Cloudflare IP Ranges +# ******************** + +# For Safety Sake all Cloudflare's Known IP Ranges are all white listed + # START CLOUDFLARE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ### 103.21.244.0/22 0; 103.22.200.0/22 0; @@ -5931,34 +5994,34 @@ geo $validate_client { # ************************* # START WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ### - 104.197.51.76 1; #makeawebsitehub.com/theme-detector/ - 108.167.189.81 1; #whatpress.net - 109.73.225.87 1; #theseotools.net/wp-theme-detector - 13.68.211.181 1; #hackertarget.com - 142.4.218.201 1; #builtwith.com (bilby.builtwith.com / monty.builtwith.com / trends.builtwith.com) - 149.56.33.22 1; #freeonlineseo.org/wordpress-theme-detector - 158.69.187.171 1; #builtwith.com - 158.69.26.58 1; #www.mythemedetector.com/detector - 162.13.185.20 1; #makeawebsitehub.com/theme-detector/ - 173.237.189.235 1; #seo-tools.freedirectorywebsites.com/wordpress-theme-detector - 173.255.210.133 1; #www.scanwp.com - 185.45.14.186 1; #whoiswp.com - 192.163.217.239 1; #scanwp.net - 192.185.4.40 1; #whatwpthemeisthat.com - 192.95.29.139 1; #seotoolstation.com/wordpress-theme-detector - 192.99.17.79 1; #builtwith.com (oscar.builtwith.com) - 198.27.69.229 1; #builtwith.com (fluffy.builtwith.com) - 198.58.124.46 1; #makeawebsitehub.com/theme-detector/ - 199.241.28.124 1; #wordpressthemedetector.org - 212.71.238.108 1; #pentest-tools.com (Vulnerability Scanner) - 37.247.121.179 1; #wpthemedetector.com - 37.60.253.215 1; #wploop.com/wordpress-theme-detector-tool/ - 45.63.68.250 1; #www.cuteseotools.net/wordpress-theme-detector - 45.79.139.191 1; #wprecon.com - 50.116.84.148 1; #detectwptheme.com - 52.87.112.125 1; #whattheme.com - 66.96.183.60 1; #www.callseotools.com/wordpress-theme-detector - 89.36.223.188 1; #www.cuteseotools.net/wordpress-theme-detector + 104.197.51.76 1; #makeawebsitehub.com/theme-detector/ + 108.167.189.81 1; #whatpress.net + 109.73.225.87 1; #theseotools.net/wp-theme-detector + 13.68.211.181 1; #hackertarget.com + 142.4.218.201 1; #builtwith.com (bilby.builtwith.com / monty.builtwith.com / trends.builtwith.com) + 149.56.33.22 1; #freeonlineseo.org/wordpress-theme-detector + 158.69.187.171 1; #builtwith.com + 158.69.26.58 1; #www.mythemedetector.com/detector + 162.13.185.20 1; #makeawebsitehub.com/theme-detector/ + 173.237.189.235 1; #seo-tools.freedirectorywebsites.com/wordpress-theme-detector + 173.255.210.133 1; #www.scanwp.com + 185.45.14.186 1; #whoiswp.com + 192.163.217.239 1; #scanwp.net + 192.185.4.40 1; #whatwpthemeisthat.com + 192.95.29.139 1; #seotoolstation.com/wordpress-theme-detector + 192.99.17.79 1; #builtwith.com (oscar.builtwith.com) + 198.27.69.229 1; #builtwith.com (fluffy.builtwith.com) + 198.58.124.46 1; #makeawebsitehub.com/theme-detector/ + 199.241.28.124 1; #wordpressthemedetector.org + 212.71.238.108 1; #pentest-tools.com (Vulnerability Scanner) + 37.247.121.179 1; #wpthemedetector.com + 37.60.253.215 1; #wploop.com/wordpress-theme-detector-tool/ + 45.63.68.250 1; #www.cuteseotools.net/wordpress-theme-detector + 45.79.139.191 1; #wprecon.com + 50.116.84.148 1; #detectwptheme.com + 52.87.112.125 1; #whattheme.com + 66.96.183.60 1; #www.callseotools.com/wordpress-theme-detector + 89.36.223.188 1; #www.cuteseotools.net/wordpress-theme-detector # END WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ### # **************************************** @@ -5989,24 +6052,36 @@ geo $validate_client { # Read Comments inside blacklist-ips.conf for customization tips. # Updating the main globalblacklist.conf file will not touch your custom include files -# START BLACKLISTED IPS ### DO NOT EDIT THIS LINE AT ALL ### +# ******************************************************************** +# START BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ******************************************************************** + include /etc/nginx/bots.d/blacklist-ips.conf; -# END BLACKLISTED IPS ### DO NOT EDIT THIS LINE AT ALL ### + +# ****************************************************************** +# END BLACKLISTED IPS ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ****************************************************************** } -# Keep own IPs out of DDOS Filter -# Add your own IP addresses and ranges below to spare them from the rate -# limiting DDOS filter (one per line) -# This now automatically includes the whitelist-ips.conf file so you only -# need to edit that include file and have it include here too for you +# WHITELIST your own IPs from the DDOS Filter + # Add your own IP addresses and ranges into the custom include file whitelist-ips.conf + # to spare them from the rate limiting DDOS filter. + # This section includes the same / single whitelist-ips.conf file so you only + # need to edit that include file and have it include here for you too. geo $ratelimited { default 1; -# START WHITELISTED IP RANGES2 ### DO NOT EDIT THIS LINE AT ALL ### +# *************************************************************************** +# START WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# *************************************************************************** + include /etc/nginx/bots.d/whitelist-ips.conf; -# END WHITELISTED IP RANGES2 ### DO NOT EDIT THIS LINE AT ALL ### + +# ************************************************************************* +# END WHITELISTED IP RANGES2 ### DO NOT EDIT OR REMOVE THIS LINE AT ALL ### +# ************************************************************************* } @@ -6046,3 +6121,4 @@ geo $ratelimited { ### Also check out my Ultimate Apache Bad Bot Blocker on Github ### https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker + diff --git a/robots.txt/robots.txt b/robots.txt/robots.txt index fd8c3b89b..c06a06c47 100644 --- a/robots.txt/robots.txt +++ b/robots.txt/robots.txt @@ -1,7 +1,7 @@ ### Version Information # ################################################### -### Version: V3.2017.07.636 -### Updated: Thu Jul 6 16:34:46 SAST 2017 +### Version: V3.2017.07.637 +### Updated: Thu Jul 6 16:39:57 SAST 2017 ### Bad Bot Count: 479 ################################################### ### Version Information ##