UPDATE Bad Bots List / Dupe Removals / Extra Catches for Trendiction

This commit is contained in:
Mitchell Krog 2017-11-01 12:57:23 +02:00
parent f8373a4349
commit 47135f0e66
No known key found for this signature in database
GPG key ID: 2BB9A077C8DD6994

View file

@ -43,7 +43,6 @@ Boardreader
Bolt Bolt
BotALot BotALot
Brandprotect Brandprotect
BrandProtect
Brandwatch Brandwatch
Bubing Bubing
Buddy Buddy
@ -73,7 +72,6 @@ Copyscape
Cosmos Cosmos
Craftbot Craftbot
CrazyWebCrawler CrazyWebCrawler
CRAZYWEBCRAWLER
Crescent Crescent
CSHttp CSHttp
Curious Curious
@ -97,8 +95,6 @@ DomainCrawler
DomainSigmaCrawler DomainSigmaCrawler
DomainStatsBot DomainStatsBot
Dotbot Dotbot
Download\ Demon
Download\ Devil
Download\ Wonder Download\ Wonder
Dragonfly Dragonfly
Drip Drip
@ -109,8 +105,6 @@ eCatch
ECCP/1.0 ECCP/1.0
Ecxi Ecxi
EirGrabber EirGrabber
EMail\ Collector
EMail\ Extractor
EMail\ Siphon EMail\ Siphon
EMail\ Wolf EMail\ Wolf
EroCrawler EroCrawler
@ -170,7 +164,6 @@ IDBot
Id-search Id-search
IlseBot IlseBot
Image\ Fetch Image\ Fetch
Image\ Stripper
Image\ Sucker Image\ Sucker
Indy\ Library Indy\ Library
InfoNaviRobot InfoNaviRobot
@ -291,7 +284,6 @@ OutfoxBot
PageAnalyzer PageAnalyzer
Page\ Analyzer Page\ Analyzer
PageGrabber PageGrabber
Page\ Grabber
page\ scorer page\ scorer
PageScorer PageScorer
Panscient Panscient
@ -342,14 +334,12 @@ ScanAlert
Scanbot Scanbot
Scrapy Scrapy
Screaming Screaming
Screaming\ Frog\ SEO\ Spider
ScreenerBot ScreenerBot
Searchestate Searchestate
SearchmetricsBot SearchmetricsBot
Semrush Semrush
SemrushBot SemrushBot
SEOkicks SEOkicks
SEOkicks-Robot
SEOlyticsCrawler SEOlyticsCrawler
Seomoz Seomoz
SEOprofiler SEOprofiler
@ -357,7 +347,6 @@ seoscanners
SEOstats SEOstats
Siphon Siphon
SISTRIX SISTRIX
SISTRIX\ Crawler
Sitebeam Sitebeam
SiteExplorer SiteExplorer
Siteimprove Siteimprove
@ -412,12 +401,14 @@ TightTwatBot
Titan Titan
Toata Toata
Toweyabot Toweyabot
Trendiction
Trendictionbot Trendictionbot
trendiction.com
trendiction.de
True_Robot True_Robot
Turingos Turingos
Turnitin
TurnitinBot TurnitinBot
Turnitin\ Bot
Turnitin\ Robot
TwengaBot TwengaBot
Twice Twice
Typhoeus Typhoeus
@ -442,13 +433,10 @@ Webalta
WebAuto WebAuto
Web\ Auto Web\ Auto
WebBandit WebBandit
Web\ Bandit
WebCollage WebCollage
Web\ Collage Web\ Collage
WebCopier WebCopier
Web\ Copier
WEBDAV WEBDAV
WEBDAV\ Client
WebEnhancer WebEnhancer
Web\ Enhancer Web\ Enhancer
WebFetch WebFetch
@ -457,28 +445,23 @@ WebFuck
Web\ Fuck Web\ Fuck
WebGo\ IS WebGo\ IS
WebImageCollector WebImageCollector
Web\ Image\ Collector
WebLeacher WebLeacher
WebmasterWorldForumBot WebmasterWorldForumBot
webmeup-crawler webmeup-crawler
WebPix WebPix
Web\ Pix Web\ Pix
WebReaper WebReaper
Web\ Reaper
WebSauger WebSauger
Web\ Sauger Web\ Sauger
Webshag Webshag
WebsiteExtractor WebsiteExtractor
Website\ Extractor
WebsiteQuester WebsiteQuester
Website\ Quester Website\ Quester
Webster Webster
WebStripper WebStripper
Web\ Stripper
WebSucker WebSucker
Web\ Sucker Web\ Sucker
WebWhacker WebWhacker
Web\ Whacker
WebZIP WebZIP
WeSEE WeSEE
Whack Whack