V4.2021.10.2673 [ci skip]

dev-tools/test_units/random-bots-for-whitelist-test.list

@@ -1,249 +1,249 @@
-01h4x.com
 360Spider
-ADmantX
+404enemy
 ALittle Client
-Abonti
+Aboundexbot
 Acunetix
 AfD-Verbotsverfahren
+AhrefsBot
 AiHitBot
 Alexibot
-Anarchy
-Apexoo
-AwarioRssBot
-BDFetch
+Asterias
+Attach
+AwarioSmartBot
+BBBike
+BackStreet
 BackWeb
-BacklinkCrawler
-Badass
-Bitacle
-Boardreader
+Bandit
+BlowFish
 Bolt
-Buck
-BunnySlippers
+BuiltWith
 BuzzSumo
-CCBot
-Calculon
-CensysInspect
+CATExplorador
 CheeseBot
-ChinaClaw
-Chlooe
-Claritybot
-Cliqzbot
+CherryPicker
 Cogentbot
-Copier
-CrazyWebCrawler
+CopyRightCheck
+Copyscape
+Crawling at Home Project
 CrunchBot
-Curious
-CyotekWebCopy
 DBLBot
-DTS Agent
-DataCha0s
+DIIbot
+DatabaseDriverMysqli
 Deusu
-Devil
-Digincore
 DigitalPebble
+Disco
 Discobot
-DnBCrawler-Analytics
-DomainAppender
-DomainCrawler
+Discoverybot
+Dispatch
+DittoSpyder
+Domains Project
+Download Wonder
 Dragonfly
-EMail Wolf
+EMail Siphon
 Ecxi
 EirGrabber
 EroCrawler
-Express WebPictures
-ExtLinksBot
+Exabot
+ExtractorPro
 EyeNetIE
-Ezooms
 FDM
+FHscan
 FemtosearchBot
+Fimap
+Flunky
+FrontPage
+Fuzz
+FyberSpider
 G-i-g-a-b-o-t
-GT::WWW
-GalaxyBot
-Genieo
-GetRight
-Gigabot
 Go!Zilla
-GrabNet
+GrapeFX
 GrapeshotCrawler
 GridBot
-HMView
-HTTP::Lite
-Haansoft
-Humanlinks
-HybridBot
-IDBTE4M
-IDBot
-Iblog
+Havij
+IRLbot
 Id-search
 IlseBot
-Image Fetch
 IndeedBot
-Indy Library
 InfoTekies
-Internet Ninja
-Jbrofuzz
-JennyBot
-JetCar
+InterGET
+Iskanie
+JOC Web Spider
+JamesBOT
 Jetty
 JustView
-Keyword Density
-Kinza
-Larbin
-Leap
-LeechGet
+Kozmosbot
+LNSpiderguy
+LeechFTP
+LexiBot
+Lftp
+LibWeb
+Libwhisker
 LieBaoFast
 Lightspeedsystems
 Likse
-LinkWalker
-Linkbot
-LinkextractorPro
+LinkScan
 LinkpadBot
-LinqiaMetadataDownloaderBot
+LinksManager
+LinqiaScrapeBot
 Litemage_walker
-MIDown tool
-MIIxpc
+Lmspider
+MFC_Tear_Sample
 MJ12bot
 MQQBrowser
-MTRobot
-Majestic-SEO
 Majestic12
-MarkWatch
+MarkMonitor
+Mass Downloader
+Masscan
 Mata Hari
+Mb2345Browser
 MeanPath Bot
-Mediatoolkitbot
 MegaIndex.ru
-Metauri
 Microsoft URL Control
 Mister PiX
+Mojeek
 Mojolicious
 Morfeus Fucking Scanner
-Mozlila
 Mr.4x3
+Msrabot
+Musobot
 NPbot
+Name Intelligence
+Needle
+NetAnts
+NetLyzer
 NetMechanic
-NetSpider
-NetZIP
-Nikto
+Netvibes
+Niki-bot
 Nimbostratus
-Ninja
-Nmap
-Offline Explorer
+Nutch
+Offline Navigator
+OpenLinkProfiler
 OpenVAS
-Openvas
-OrangeBot
-PHPCrawl
-POE-Component-Client-HTTP
+Openfind
+OrangeSpider
+OutclicksBot
+PECL::HTTP
 Page Analyzer
-PageAnalyzer
+PageGrabber
+PageScorer
 PageThing.com
-Panscient
 Papa Foto
-Petalbot
+Pavuk
 Picscout
 PictureFinder
-Pockey
+Pimonster
+Pixray
 ProPowerBot
-ProWebWalker
-Psbot
+Probethenet
 Pump
-QueryN Metasearch
+PxBroker
+PyCurl
+Quick-Crawler
 RSSingBot
-RankFlex
-RankingBot2
-ReGet
-RedesScrapy
-RepoMonkey
+RankActive
+RankActiveLinkBot
+Rankivabot
+Re-re
+Recorder
+Ripper
+RocketCrawler
 Rogerbot
 SEOkicks
-SEOprofiler
 SISTRIX
 SMTBot
-SalesIntelligent
+ScoutJet
+Scrapy
+Screaming
 ScreenerBot
 ScrepyBot
-SemrushBot
-SentiBot
+Semrush
+Seomoz
+Shodan
 Siphon
-SiteExplorer
-SiteSnagger
-Sociscraper
-Sosospider
-Spanner
-Sucker
-SuperHTTP
-Suzuran
-Swiftbot
+Site Sucker
+SiteLockSpider
+Siteimprove
+Sitevigil
+Snoopy
+SocialRankIOBot
+Sogou web spider
+Sqlmap
+Sqlworm
+Sqworm
+Sucuri
+SuperBot
+Surfbot
+SurveyBot
 Szukacz
-T8Abot
-TeleportPro
+T0PHackTeam
+Teleport
 Telesoft
-Telesphorep
-The Intraformant
 TightTwatBot
-Titan
-Toata
+Toweyabot
 Tracemyfile
-Trendiction
-TurnitinBot
-Typhoeus
-Upflow
-V-BOT
-VB Project
-VCI
+True_Robot
+Turnitin
 Vacuum
 VelenPublicWebCrawler
 VidibleScraper
 Virusdie
-VoidEYE
-Voil
-Voltron
-WBSearchBot
-WISENutbot
-WWW::Mechanize
-Wallpapers/3.0
+WWW-Mechanize
+Wallpapers
+WallpapersHD
+WeSEE
+Web Auto
 Web Collage
-Web Fetch
-Web Fuck
-Web Pix
-Web Sucker
-WebAuto
-WebCollage
-WebImageCollector
-WebPix
-WebReaper
-WebSauger
+Web Sauger
+WebBandit
+WebEnhancer
+WebFetch
 WebStripper
-WebWhacker
-WebZIP
+WebSucker
+Webalta
+WebmasterWorldForumBot
 Webshag
-Website Quester
+WebsiteExtractor
+Webster
+Whack
 Whacker
 Whatweb
 Who.is Bot
-Widow
-YoudaoBot
+WiseGuys Robot
+Wotbox
+Wprecon
+Xaldon_WebSpider
+Zade
 Zauba
+Zermelo
 Zitebot
-ZmEu
-ZoomBot
 ZoominfoBot
-ZumBot
-autoemailspider
-clark-crawler
+ZyBorg
+cah.io.community
 cognitiveseo
+crawler.feedback
+crawler4j
 demandbase-bot
+eCatch
+evc-batch
 facebookscraper
-gopher
-ips-agent
-isitwp.com
-netEstate NE Crawler
-pcBrowser
+internetVista monitor
+mediawords
+oBot
+page scorer
 polaris version
-scan.lol
+ripz
+s1z.ru
+scalaj-http
 seobility
+sexsearcher
 sitechecker.pro
-tAkeOut
+sogouspider
+spyfu
+sysscan
+trendiction.de
 webmeup-crawler
+webpros.com
+x09Mozilla
 x22Mozilla
 xpymep1.exe
 zauba.io