V4.2021.10.2629 [ci skip]

dev-tools/test_units/random-bots-for-whitelist-test.list

@@ -1,250 +1,250 @@
 01h4x.com
-404enemy
-ALittle Client
+360Spider
+404checker
+AIBOT
 ASPSeek
 Acunetix
 AfD-Verbotsverfahren
-AiHitBot
-Alexibot
+Aipbot
+AllSubmitter
 Anarchie
+Anarchy
+Anarchy99
 Ankit
-Apexoo
 Aspiegel
-AwarioSmartBot
+Asterias
+AwarioRssBot
+BDCbot
 BDFetch
-BLEXBot
-Bigfoot
-BlackWidow
-Blackboard
-BuzzSumo
-CSHttp
-Calculon
+BackWeb
+Badass
+Bandit
+Barkrowler
+BatchFTP
+Battleztar Bazinga
+Black Hole
+Brandprotect
+BuiltBotTough
+Bullseye
+BunnySlippers
+CODE87
 CazoodleBot
-CensysInspect
-CheTeam
 CherryPicker
 Chlooe
 Claritybot
-Cliqzbot
 Cloud mapping
-Copier
+Cocolyzebot
+Collector
 CopyRightCheck
-Crescent
-Curious
+Cosmos
+Crawling at Home Project
+DBLBot
+DIIbot
+DSearch
+DataCha0s
 DatabaseDriverMysqli
-Devil
-DittoSpyder
-DnyzBot
+Dirbuster
+Disco
+Dispatch
 DomainAppender
-DomainSigmaCrawler
 Download Wonder
-Drip
-ECCP/1.0
+Dragonfly
 EMail Siphon
-EMail Wolf
-EirGrabber
+EasyDL
+Ebingbong
+Ecxi
+EroCrawler
 Evil
-Extreme Picture Finder
-Ezooms
-Fimap
-Flunky
-Fyrebot
-G-i-g-a-b-o-t
-Genieo
-GermCrawler
+EyeNetIE
+FDM
+FHscan
+FlashGet
+FyberSpider
 GetRight
 GetWeb
 Getintent
 Gigabot
 Go!Zilla
-GrapeshotCrawler
-HMView
-HTTP::Lite
+Go-Ahead-Got-It
+GrabNet
+Grabber
+Grafula
+GridBot
 HTTrack
-Haansoft
-Harvest
-HybridBot
-IDBTE4M
-IDBot
-IRLbot
-InfoNaviRobot
-InfoTekies
-Intelliseek
-InternetSeer
-Iria
+Heritrix
+Hloader
+Iblog
+Image Sucker
+IndeedBot
+Indy Library
+Internet Ninja
+Iskanie
 IstellaBot
 JOC Web Spider
+JamesBOT
 Jbrofuzz
-JikeSpider
+JetCar
+Jetty
+Joomla
 Jorgee
-JustView
-Kenjin Spider
 Keyword Density
+Kinza
+Kozmosbot
+LNSpiderguy
 LWP::Simple
-LexiBot
+Lanshanbot
+LeechFTP
 LibWeb
+Libwhisker
+LieBaoFast
 Lightspeedsystems
 LinkScan
-LinqiaMetadataDownloaderBot
-Lipperhey
-Lmspider
+Linkbot
+Linkdexbot
+LinkpadBot
+LinqiaScrapeBot
+Litemage_walker
 MIDown tool
 MJ12bot
-MQQBrowser
-MTRobot
-Mail.RU_Bot
+MSFrontPage
+MSIECrawler
+Mag-Net
+Majestic SEO
 Majestic-SEO
-Majestic12
-Mass Downloader
-Mata Hari
-Meanpathbot
-MicroMessenger
-Microsoft URL Control
+MauiBot
+MegaIndex.ru
+Metauri
 Mister PiX
-Moblie Safari
+Mojeek
 Mojolicious
 Morfeus Fucking Scanner
-Navroad
-NearSite
-Needle
+Mozlila
+Name Intelligence
+Nameprotect
 Nessus
-NetMechanic
-NetSpider
 NetZIP
-Nettrack
-NimbleCrawler
+Netcraft
+Netvibes
+Nibbler
+Niki-bot
+Nikto
 Nuclei
-Offline Navigator
+Nutch
+Offline Explorer
+OpenLinkProfiler
+OpenVAS
 Openfind
-OrangeBot
-OrangeSpider
 OutclicksBot
-POE-Component-Client-HTTP
 PageScorer
-Papa Foto
-Pi-Monster
-Picscout
-Picsearch
-Piepmatz
+PageThing.com
+PeoplePal
 Pimonster
+PleaseCrawl
 Probethenet
 Psbot
 Pu_iN
-PyCurl
-RSSingBot
+QueryN Metasearch
+Quick-Crawler
+RankActive
+RankActiveLinkBot
 RankFlex
+Rankivabot
 RankurBot
-Recorder
-RedesScrapy
-Ripper
-RocketCrawler
-Rogerbot
-SBIder
-ScanAlert
+ReGet
+Reaper
+RebelMouse
+RepoMonkey
+SEOkicks
+SEOlyticsCrawler
+SEOprofiler
+SEOstats
+SMTBot
+SalesIntelligent
 Scanbot
 ScoutJet
+Scrapy
 Screaming
-ScrepyBot
-Searchestate
+ScreenerBot
 SearchmetricsBot
-Semrush
-Siphon
-SiteCheckerBotCrawler
-Sitevigil
-Snoopy
-Sociscraper
-Sosospider
+SemrushBot
+SentiBot
+SeoSiteCheckup
+SeobilityBot
+Seomoz
+SiteLockSpider
+Siteimprove
+Snake
 Sottopop
 SpaceBison
-Spanner
 Spinn3r
-Stripper
-Sucuri
-Surfbot
+Sqlmap
+SuperHTTP
+SurveyBot
 Suzuran
-Szukacz
-T8Abot
-TeleportPro
+Swiftbot
+Telesphoreo
 Telesphorep
-TheNomad
-TightTwatBot
-Tracemyfile
-Trendiction
+The Intraformant
+Titan
+Toata
+Trendictionbot
 Turingos
 Turnitin
 TurnitinBot
-URLy Warning
+TwengaBot
+Twice
+Typhoeus
 URLy.Warning
-UnisterBot
-V-BOT
-VCI
-Vacuum
-VelenPublicWebCrawler
-VeriCiteCrawler
+VidibleScraper
 Virusdie
 VoidEYE
-WASALive-Bot
-WBSearchBot
+Voltron
 WISENutbot
 WWW-Collector-E
-WWW-Mechanize
-WWW::Mechanize
-Wallpapers
-Wallpapers/3.0
 WallpapersHD
 WeSEE
-Web Auto
 Web Collage
-Web Pix
-WebAuto
 WebBandit
-WebCollage
-WebCopier
-WebEnhancer
 WebFetch
 WebFuck
 WebImageCollector
-WebPix
-WebReaper
-WebSucker
-WebWhacker
+WebLeacher
+WebSauger
+WebStripper
 WebZIP
-Webalta
-WebmasterWorldForumBot
-WebsiteExtractor
-WebsiteQuester
+Webshag
 Webster
+Whack
+Whacker
 Who.is Bot
+Woobot
 Wotbox
-Wprecon
 Xaldon WebSpider
-Xaldon_WebSpider
-Xenu
 YoudaoBot
+Zade
+Zauba
+Zermelo
+Zeus
 Zitebot
-ZumBot
-ZyBorg
+ZmEu
 archive.org_bot
 arquivo.pt
-autoemailspider
-coccocbot-web
-com.plumanalytics
-crawler.feedback
-demandbase-bot
+cognitiveseo
+crawler4j
+domainsproject.org
 facebookscraper
-gopher
-instabid
 isitwp.com
 lwp-request
-lwp-trivial
-muhstik-scan
+magpie-crawler
 netEstate NE Crawler
-ripz
-seoscanners
-seostar
-siteripz
+oBot
+page scorer
+plumanalytics
+probe-image-size
+satoristudio.net
+scalaj-http
 sysscan
-trendiction.de
+tAkeOut
+webmeup-crawler
+webpros.com
 webprosbot
-xpymep1.exe
-zgrab