V4.2024.07.4647 [ci skip]

dev-tools/test_units/random-bots-for-whitelist-test.list

@@ -1,100 +1,100 @@
-Anarchie
-Anarchy
+360Spider
+ADmantX
+AIBOT
+Anarchy99
+Apexoo
+BDFetch
 Backlink-Ceck
-BatchFTP
-Battleztar Bazinga
-BlowFish
-Brandprotect
-Brandwatch
-BuiltWith
-CensysInspect
-Citoid
-Copyscape
-Crawling at Home Project
-Dispatch
-DomCopBot
-Drip
-Exabot
-FemtosearchBot
-Foobot
-GalaxyBot
-GetWeb
-Go-Ahead-Got-It
-Gotit
+Badass
+Black Hole
+BotALot
+BuzzSumo
+CODE87
+ChinaClaw
+Cocolyzebot
+CrunchBot
+Curious
+DIIbot
+DTS Agent
+DatabaseDriverMysqli
+Demon
+Deusu
+Discoverybot
+Dotbot
+EMail Siphon
+ExtLinksBot
+ExtractorPro
+FrontPage
+Google-Extended
 GrapeshotCrawler
-HTTrack
-IDBTE4M
-IlseBot
-Image Sucker
-ImagesiftBot
-Intelliseek
-Iskanie
-Jorgee
+Haansoft
+HaosouSpider
+IndeedBot
+Information Security Team InfraSec Scanner
+Internet Ninja
+JOC Web Spider
+JetCar
+JikeSpider
 Kenjin Spider
-Kinza
-Leap
-LeechFTP
-Lftp
-MJ12bot
-Mail.RU_Bot
-Majestic-SEO
+LNSpiderguy
+Likse
+Lipperhey Spider
+MQQBrowser
+Mag-Net
 Majestic12
-Mata Hari
-MicroMessenger
-NICErsPRO
-Nessus
-Net Vampire
-NetSpider
-Netcraft
+Meanpathbot
+Metauri
+Microsoft URL Control
+Mojolicious
+NearSite
+Needle
+Openvas
+POE-Component-Client-HTTP
 PageGrabber
-Petalbot
-PleaseCrawl
-Pu_iN
-PyCurl
-Rankivabot
-Re-re
-ReGet
-RealDownload
-RedesScrapy
+Piepmatz
+Pockey
+RankFlex
+SEOlyticsCrawler
+SEOprofiler
 SISTRIX
-Scrapy
-Screaming
-SemrushBot
-SentiBot
-Site Sucker
-SiteLockSpider
+ScreenerBot
+Seekport
+SeoSiteCheckup
 Siteimprove
-Snoopy
-Spinn3r
-SputnikBot
+Spbot
+SuperHTTP
 Surfbot
+Swiftbot
 Teleport
-Telesoft
+Telesphorep
+The Intraformant
+TightTwatBot
 Toweyabot
-Twice
+Tracemyfile
+TurnitinBot
 Typhoeus
-V-BOT
+URLy Warning
+URLy.Warning
+Vacuum
 VelenPublicWebCrawler
+Virusdie
 Voil
-WallpapersHD
+WWW-Mechanize
+Wallpapers
 Web Enhancer
-Web Pix
-Whacker
-Who.is Bot
-awario.com
-cah.io.community
-crawler.feedback
-crawler4j
+WebCopier
+WebFetch
+WebWhacker
+WebsiteExtractor
+Wprecon
+YaK
+ZmEu
+ZoominfoBot
+ZyBorg
+arquivo.pt
+autoemailspider
 facebookscraper
-heritrix
-instabid
-internetVista monitor
-lwp-trivial
-meanpathbot
-page scorer
-ripz
-seocompany.store
-serpstatbot
-voyagerx.com
-webmeup-crawler
+muhstik-scan
+spyfu
+x22Mozilla
 zgrab