diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a96827ef..1b0f5edba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ ###2016-12-11 - Added some repetitively bad IP's - Added extensive blocking for builtwith.com SEO analysis company that scrapes, accumulates and sells SEO web data. Have blocked all their domains and IP's including other domains linked to the owner. Also spent hours digging for IP's linked to this site that were very nicely hidden but through some research I uncovered them. www.builtwith.com DOES obey removal instructions at https://builtwith.com/removals and it is immediate BUT for those who have never heard of builtwith.com or even know their sites are indexed and new sites are being indexed and used by your competitors to outrank you, I have sufficient blocks in place to prevent them ever accessing your Nginx/Apache server. I suggest you FIRST see if your site is indexed, then request removal of each site which requires you to place a simple .html file in your webroot and once that is done, then update to the latest version of the script which will block them indexing new sites from thereon out. I will keep a watch on them and add new IP's as they change. -- In progress on growing the BAD IP block list which will be based off repetetive 444 errors from the bad referer domains in the blocker. These bad IP's will be auto added into the blocke and I will generate plain text IP lists and IP tables rules too which can be updated frequently to block them at firewall level and keep your web logs even cleaner. +- In progress on growing the BAD IP block list which will be based off repetetive 444 errors from the bad referer domains in the blocker. These bad IP's will be auto added into the blocker and I will generate plain text IP lists and IP tables rules too which can be updated frequently to block them at firewall level and keep your web logs even cleaner. ###2016-12-04 - Added creation of a Google Disavow File - google-disavow.txt diff --git a/README.md b/README.md index b6e0137f7..e93a42c9d 100644 --- a/README.md +++ b/README.md @@ -116,6 +116,7 @@ figure it out otherwise simply message me if you are having problems. ## FEATURES OF THE NGINX BAD BOT BLOCKER: - Extensive Lists of Bad and Known Bad Bots and Scrapers (updated almost daily) +- Blocking of SEO data collection companies like Semalt.com, Builtwith.com, WooRank.com and many others (updated regularly) - Alphabetically ordered for easier maintenance - Commented sections of certain important bots to be sure of before blocking - Includes the IP range of Cyveillance who are known to ignore robots.txt rules