V4.2022.01.2873 [ci skip]

dev-tools/test_units/random-bots-for-whitelist-test.list

@@ -1,250 +1,250 @@
 01h4x.com
-404enemy
-ASPSeek
-Aboundex
-Aboundexbot
+404checker
+ADmantX
+AIBOT
+Acunetix
 AhrefsBot
-AiHitBot
-Alexibot
 Anarchie
+Anarchy
 Ankit
-Anthill
+Apexoo
 Aspiegel
-AwarioRssBot
-BackWeb
-Bandit
-Barkrowler
-Bitacle
-Black Hole
-Blow
-Bolt
+Asterias
+BDFetch
+BackDoorBot
+Backlink-Ceck
+Badass
+Battleztar Bazinga
+Bigfoot
+Blackboard
+BlowFish
+Boardreader
 Brandprotect
-BunnySlippers
-CCBot
-Calculon
-Cegbfeieh
+Buck
+Buddy
+BuiltBotTough
+BuiltWith
+CATExplorador
 CheTeam
-ChinaClaw
-Cliqzbot
+Chlooe
+Citoid
+Claritybot
+Cocolyzebot
+Cogentbot
 Collector
-Copier
 CopyRightCheck
+Copyscape
 Cosmos
-CrazyWebCrawler
-CrunchBot
-Custo
+Crawling at Home Project
+Curious
 CyotekWebCopy
 DBLBot
-DataCha0s
-Demon
-Deusu
+DIIbot
+DTS Agent
+DatabaseDriverMysqli
+DigitalPebble
 Dirbuster
-Discobot
-Dispatch
-DomCopBot
+Discoverybot
+DnyzBot
 DomainCrawler
-Domains Project
-Dotbot
+DomainSigmaCrawler
+DomainStatsBot
+Download Wonder
 Dragonfly
-EMail Siphon
+Drip
+ECCP/1.0
+Ebingbong
 EirGrabber
-Evil
-ExtractorPro
+EroCrawler
+Extreme Picture Finder
 Ezooms
-FDM
-FHscan
 FemtosearchBot
-Fimap
-Foobot
-Freeuploader
-GT::WWW
-GalaxyBot
-GetRight
-Gotit
-GrabNet
-Grafula
-GrapeFX
+FlashGet
+FrontPage
+Genieo
+GermCrawler
+Go!Zilla
+Go-Ahead-Got-It
+Grabber
 GrapeshotCrawler
-GridBot
 HEADMasterSEO
-HTTP::Lite
 Haansoft
-HonoluluBot
-Image Fetch
-Image Sucker
-IndeedBot
-InfoNaviRobot
+Havij
+Hloader
+IDBot
+IRLbot
+Iblog
+Id-search
+IlseBot
+Indy Library
+InfoTekies
 Intelliseek
-InternetSeer
-Iskanie
-JOC Web Spider
-JamesBOT
-Jbrofuzz
+InterGET
+Iria
 JennyBot
-JetCar
-Jetty
+Jorgee
+JustView
 Jyxobot
-Kenjin Spider
-Kozmosbot
-LNSpiderguy
+Keyword Density
+Kinza
 LWP::Simple
+Lanshanbot
 LeechFTP
-LeechGet
-LexiBot
 LibWeb
 Libwhisker
-Lightspeedsystems
-Likse
-LinkScan
-LinkWalker
-Lmspider
+Linkdexbot
+LinkpadBot
+LinqiaScrapeBot
+Lipperhey
+Litemage_walker
+Ltx71
 MFC_Tear_Sample
-MIIxpc
-MQQBrowser
+MJ12bot
 MSIECrawler
+Mag-Net
 Magnet
 Mail.RU_Bot
 Majestic SEO
-Majestic12
-Masscan
+Majestic-SEO
+MarkMonitor
+MarkWatch
+Mass Downloader
 Mata Hari
-Mb2345Browser
-Metauri
 Microsoft Data Access
-Microsoft URL Control
+Minefield
 Mojolicious
-Morfeus Fucking Scanner
-Mozlila
-Msrabot
-Musobot
-Net Vampire
-Netcraft
-Netvibes
+MolokaiBot
+Mr.4x3
+NPbot
+Nameprotect
+Navroad
+NearSite
+Needle
+NetMechanic
+NetSpider
+NetZIP
 Nibbler
-Nimbostratus
+Niki-bot
+Nikto
 Ninja
-Nmap
+Not
 Nuclei
-Nutch
-Offline Navigator
+Offline Explorer
+OnCrawl
+OpenLinkProfiler
+Openfind
 OrangeSpider
 PECL::HTTP
-PageGrabber
-PageScorer
-Pandalytics
-Papa Foto
+PHPCrawl
+POE-Component-Client-HTTP
+Page Analyzer
+PageAnalyzer
+Pavuk
 Petalbot
-Picsearch
-Probethenet
+PictureFinder
+PleaseCrawl
+ProPowerBot
+Psbot
 Pump
-PxBroker
-QueryN Metasearch
+PyCurl
 RankActive
-RankActiveLinkBot
+RankingBot2
 Rankivabot
+Re-re
 ReGet
-Reaper
+RebelMouse
 RedesScrapy
-RepoMonkey
 Ripper
+RocketCrawler
+Rogerbot
 SBIder
-SISTRIX
-ScoutJet
+SEOkicks
+SEOlyticsCrawler
+SalesIntelligent
+Scrapy
+Screaming
+ScreenerBot
 ScrepyBot
-SearchmetricsBot
-SemanticJuice
-SemrushBot
+SentiBot
+SeoSiteCheckup
+SeobilityBot
 Seomoz
-Shodan
-Site Sucker
-SiteCheckerBotCrawler
-SiteSnagger
-Sitebeam
-Siteimprove
+SiteLockSpider
 SmartDownload
-Snake
-Sogou web spider
-Sosospider
+Snapbot
+Snoopy
+Sociscraper
+Sottopop
 SpaceBison
-Spammen
+SpankBot
 Spbot
-Spinn3r
 SputnikBot
-Sqlmap
-Sqlworm
-Sucker
-SuperHTTP
-Surfbot
+Sqworm
+Sucuri
 Suzuran
 Swiftbot
 Teleport
-TeleportPro
-Telesphoreo
-Telesphorep
-The Intraformant
-TheNomad
-Titan
+TightTwatBot
+Toata
+Tracemyfile
 Trendiction
-Trendictionbot
+True_Robot
+Turnitin
+TurnitinBot
+URLy Warning
 URLy.Warning
 UnisterBot
-Vacuum
+VCI
 VelenPublicWebCrawler
-VidibleScraper
-Virusdie
+VeriCiteCrawler
 VoidEYE
 Voil
-Voltron
 WASALive-Bot
-WBSearchBot
-WISENutbot
-WWW-Mechanize
+WEBDAV
+WPScan
 WWW::Mechanize
 WeSEE
 Web Collage
-Web Fetch
-Web Pix
-WebAuto
-WebBandit
-WebCollage
-WebCopier
-WebGo IS
+Web Fuck
+Web Sucker
+WebImageCollector
+WebLeacher
 WebSauger
 WebSucker
-WebmasterWorldForumBot
+Webalta
 WebsiteQuester
+Webster
 Whack
 Whatweb
 Who.is Bot
 WinHTTrack
 WiseGuys Robot
-Wonderbot
+Wotbox
 Wprecon
-Xaldon WebSpider
 Xaldon_WebSpider
 Xenu
-YoudaoBot
-Zeus
-ZmEu
+Zade
+ZoomBot
 ZumBot
-ZyBorg
+adscanner
 arquivo-web-crawler
 autoemailspider
-cognitiveseo
-crawl.sogou.com
-dataforseo.com
-eCatch
-evc-batch
+cah.io.community
+com.plumanalytics
+domainsproject.org
 gopher
+heritrix
+instabid
 internetVista monitor
-iubenda-radar
-meanpathbot
-muhstik-scan
-oBot
+lwp-trivial
+magpie-crawler
+netEstate NE Crawler
+pcBrowser
 polaris version
-s1z.ru
-satoristudio.net
-scan.lol
-trendiction.com
-voyagerx.com
+scalaj-http
+seostar
+sitechecker.pro
+spyfu
+trendiction.de
+x22Mozilla
 xpymep1.exe