V4.2024.07.4601 [ci skip]

dev-tools/test_units/random-bots-for-whitelist-test.list

@@ -1,100 +1,100 @@
-ADmantX
+360Spider
 Acunetix
-AllSubmitter
+Aipbot
 AwarioBot
 BBBike
-BDCbot
-BackWeb
-Barkrowler
-BlackWidow
-ChinaClaw
-Cliqzbot
-Cocolyzebot
-Copyscape
-Curious
-Deusu
-DomainStatsBot
-Domains Project
-Drip
-Ecxi
-Express WebPictures
-GT::WWW
-GalaxyBot
-Genieo
-GermCrawler
-Gotit
-GrabNet
-GrapeshotCrawler
-Iblog
-Id-search
-Image Sucker
-Information Security Team InfraSec Scanner
+Bandit
+Bitacle
+Brandwatch
+Calculon
+Craftbot
+DataCha0s
+DnBCrawler-Analytics
+Dotbot
+EMail Wolf
+EasyDL
+EyeNetIE
+FDM
+FyberSpider
+Gigabot
+Google-Extended
 Intelliseek
-Kenjin Spider
-LeechGet
+InterGET
+Internet Ninja
+Iria
+JetCar
+Jyxobot
+Kinza
+Larbin
 Libwhisker
-LinkpadBot
-MFC_Tear_Sample
-MIDown tool
+LinkextractorPro
+LinqiaRSSBot
+MIIxpc
 MTRobot
-Majestic SEO
-Majestic12
-Mata Hari
-Mediatoolkitbot
-Metauri
-Mojeek
-MolokaiBot
-Mozlila
+Mag-Net
+Mass Downloader
+MicroMessenger
+Microsoft URL Control
 NICErsPRO
-Name Intelligence
-Nameprotect
-Needle
+Navroad
 NetAnts
-NetMechanic
-NetSpider
-NetZIP
-Nikto
-Nutch
-Openvas
 OutclicksBot
-Panscient
-Pavuk
-Pockey
-Rankivabot
-Recorder
-Ripper
+POE-Component-Client-HTTP
+PageGrabber
+Pandalytics
+Piepmatz
+PxBroker
+RSSingBot
+Rainbot
+RankurBot
+Re-re
+Rogerbot
+SEOprofiler
+Scanbot
+ScoutJet
 Scrapy
-Screaming
-SenutoBot
-SeoSiteCheckup
-SeobilityBot
-SiteSucker
-Sitebeam
-SocialRankIOBot
-SpaceBison
-Sqlworm
-Steeler
+SearchmetricsBot
+Semrush
+SentiBot
+Snapbot
+Sogou web spider
+Sottopop
 Stripper
-Suzuran
-URLy.Warning
-Wallpapers
-Website Quester
-WebsiteQuester
-Webster
+Sucuri
+Thumbor
+Toata
+Toweyabot
+Turingos
+TurnitinBot
+Typhoeus
+Voltron
+WWW::Mechanize
+Web Sucker
+WebBandit
+WebCollage
+WebGo IS
+WebPix
+WebReaper
+WebSucker
+Webshag
+WebsiteExtractor
+Whatweb
+WinHTTrack
 Wonderbot
-Wprecon
-adscanner
-arquivo-web-crawler
-arquivo.pt
-coccocbot
-dataforseo.com
-linkfluence
-muhstik-scan
-polaris version
+Woobot
+Xaldon_WebSpider
+YoudaoBot
+ZoomBot
+anthropic-ai
+check1.exe
+cohere-ai
+crawler4j
+dataforseobot
+domainsproject.org
+eCatch
+magpie-crawler
 s1z.ru
-scalaj-http
-scan.lol
-seocompany.store
-serpstatbot
-sexsearcher
-ubermetrics-technologies.com
+seoscanners
+sitechecker.pro
+tAkeOut
+trendiction.com