V4.2021.07.2477 [ci skip]

dev-tools/test_units/random-bots-for-whitelist-test.list

@@ -1,250 +1,250 @@
 360Spider
+404checker
 404enemy
-ADmantX
 AIBOT
-Aboundexbot
+ASPSeek
+Abonti
+Aboundex
+AfD-Verbotsverfahren
 Aipbot
-Alligator
-AlphaBot
-Anarchie
+Alexibot
 Ankit
-Aspiegel
-Attach
-AwarioSmartBot
-BBBike
-BackStreet
-BacklinkCrawler
-Barkrowler
+BDCbot
+BackWeb
 Battleztar Bazinga
-BetaBot
 Bigfoot
+Bitacle
+Black Hole
 BlowFish
+Bolt
+BotALot
 Brandprotect
-Brandwatch
 Buck
-Bullseye
-BuzzSumo
-CATExplorador
-CazoodleBot
+Buddy
+BuiltBotTough
+BuiltWith
+CCBot
+Calculon
 CheTeam
-ChinaClaw
+Claritybot
 Cloud mapping
-Crescent
-Curious
-CyotekWebCopy
-DSearch
-DataCha0s
+Cogentbot
+Copier
+CopyRightCheck
+Copyscape
+DBLBot
 DatabaseDriverMysqli
-Dirbuster
+Demon
+Digincore
+DigitalPebble
 Disco
-Discobot
+Discoverybot
 Dispatch
 DnyzBot
-DomCopBot
-DomainAppender
+DomainCrawler
 DomainStatsBot
-Domains Project
-Dotbot
-Download Wonder
-EasyDL
-Ebingbong
+Dragonfly
+Drip
+ECCP/1.0
+EMail Siphon
+EMail Wolf
 Ecxi
-EirGrabber
-Exabot
-Express WebPictures
+EroCrawler
+ExtLinksBot
+Ezooms
 FDM
 FHscan
-FemtosearchBot
-Flunky
-Freeuploader
+Firefox/7.0
+FlashGet
 FyberSpider
-Fyrebot
-GetRight
-Go-Ahead-Got-It
+GT::WWW
+GalaxyBot
+Getintent
+Go!Zilla
 Gotit
 GrabNet
-Grabber
 Grafula
 GrapeFX
 GridBot
 HEADMasterSEO
+HMView
 HTTP::Lite
-HaosouSpider
+Harvest
+Havij
 Heritrix
 Humanlinks
-IRLbot
+IDBTE4M
 Id-search
-Image Fetch
-Intelliseek
+Image Sucker
+InfoNaviRobot
+InfoTekies
 InterGET
 Internet Ninja
 InternetSeer
-Iria
 IstellaBot
+JOC Web Spider
+JamesBOT
 Jbrofuzz
-Jetty
+JetCar
+JikeSpider
 Joomla
-Jorgee
 JustView
-LWP::Simple
-Lanshanbot
-Larbin
+Kozmosbot
+LNSpiderguy
 LeechGet
-Libwhisker
 Likse
-LinkScan
-Linkbot
-Linkdexbot
-LinkpadBot
+LinkextractorPro
 LinksManager
 LinqiaRSSBot
-Lipperhey Spider
-Lmspider
+Lipperhey
+Litemage_walker
 Ltx71
 MFC_Tear_Sample
-MIDown tool
-MSFrontPage
+MJ12bot
+MQQBrowser
 MSIECrawler
-Mag-Net
+Magnet
+Majestic SEO
 Majestic-SEO
 MarkWatch
+Mass Downloader
 Masscan
-MauiBot
-Meanpathbot
+MegaIndex.ru
+MicroMessenger
 Mister PiX
-Mojolicious
-Morfeus Fucking Scanner
-Msrabot
-Musobot
+Mr.4x3
+NPbot
 Name Intelligence
 Nameprotect
 Navroad
-Needle
+Nessus
 Net Vampire
+NetAnts
+NetLyzer
 NetMechanic
-Nettrack
-NextGenSearchBot
-Ninja
+NetSpider
+NimbleCrawler
 Nuclei
-Nutch
 Octopus
-Offline Explorer
-OpenLinkProfiler
+Offline Navigator
 Openfind
+Openvas
 OrangeSpider
-PECL::HTTP
+OutfoxBot
 PHPCrawl
 POE-Component-Client-HTTP
 PageAnalyzer
-PageScorer
 PageThing.com
-Panscient
-Pavuk
+Pandalytics
+Papa Foto
+PeoplePal
 Picscout
-Picsearch
-Piepmatz
-Pimonster
+Pixray
 ProWebWalker
-Probethenet
+Psbot
 Pump
 PyCurl
-Quick-Crawler
+QueryN Metasearch
+RSSingBot
 RankActive
-RankingBot2
-Re-re
-ReGet
-RealDownload
-RebelMouse
-Ripper
-Rogerbot
+RankingBot
+RankurBot
+RedesScrapy
+RepoMonkey
+RocketCrawler
 SEOlyticsCrawler
-ScoutJet
-Scrapy
-ScreenerBot
+SearchmetricsBot
 SemrushBot
 SentiBot
 SeoSiteCheckup
+Seomoz
+Site Sucker
 SiteCheckerBotCrawler
-SiteSucker
-Sitebeam
-Siteimprove
-Snake
+SiteSnagger
+Sitevigil
 Snoopy
-SocialRankIOBot
 Sottopop
-Spammen
-Spanner
+SpaceBison
 Spbot
+Spinn3r
 Sqlmap
+Sqlworm
 Sqworm
+Steeler
+Sucuri
+SuperHTTP
+Surfbot
+SurveyBot
 Suzuran
 T0PHackTeam
+T8Abot
+TeleportPro
 Telesoft
 Telesphorep
 The Intraformant
-TheNomad
+Thumbor
 Titan
+Toweyabot
 Trendiction
 Trendictionbot
 True_Robot
-Turnitin
-TwengaBot
 Typhoeus
-URLy.Warning
 Upflow
-V-BOT
-Vagabondo
-VelenPublicWebCrawler
-Voltron
-WASALive-Bot
-WEBDAV
-WWW-Collector-E
-WWW-Mechanize
+VCI
+Virusdie
+VoidEYE
+Voil
+WPScan
 WWW::Mechanize
-WWWOFFLE
-Wallpapers/3.0
-Web Auto
+WeSEE
 Web Collage
 Web Enhancer
-Web Fetch
-Web Pix
 Web Sucker
-WebEnhancer
-WebFuck
-WebImageCollector
+WebCopier
+WebGo IS
 WebLeacher
-WebZIP
-WebmasterWorldForumBot
+WebPix
+WebWhacker
 WebsiteQuester
+Webster
 Whack
 WinHTTrack
-WiseGuys Robot
-Xaldon WebSpider
-Xenu
-Zeus
-arquivo-web-crawler
+Wonderbot
+Wotbox
+Wprecon
+Xaldon_WebSpider
+YoudaoBot
+Zade
+Zauba
+Zermelo
+ZoominfoBot
 arquivo.pt
-crawl.sogou.com
+check1.exe
+coccocbot-web
 crawler.feedback
-crawler4j
-evc-batch
-facebookscraper
-heritrix
-instabid
-isitwp.com
+demandbase-bot
+eCatch
+gopher
+internetVista monitor
+ips-agent
 lwp-request
-magpie-crawler
+lwp-trivial
+mediawords
+muhstik-scan
 page scorer
+pcBrowser
 plumanalytics
 polaris version
 ripz
+s1z.ru
 scalaj-http
-seostar
-sitechecker.pro
+scan.lol
+seobility
+seoscanners
 siteripz
-sogouspider
-sp_auditbot
 spyfu
-sysscan
+trendiction.com
+trendiction.de
+webmeup-crawler
 xpymep1.exe
 zauba.io