V4.2021.10.2657 [ci skip]

dev-tools/test_units/random-bots-for-test.list

@@ -1,250 +1,250 @@
 01h4x.com
+360Spider
+404checker
 404enemy
 ASPSeek
 Aboundexbot
-Acunetix
 AhrefsBot
 AiHitBot
+Aipbot
 Alexibot
+AllSubmitter
+Alligator
 AlphaBot
+Anarchy
 Anarchy99
-Ankit
-Apexoo
-Aspiegel
+Anthill
+Attach
 AwarioSmartBot
+BDCbot
 BDFetch
 BackStreet
-BackWeb
 Backlink-Ceck
 Bandit
-Barkrowler
 BatchFTP
+BetaBot
+Bigfoot
+Black Hole
 BlackWidow
-BuiltBotTough
+Brandprotect
+Brandwatch
+Buck
 BuiltWith
-BunnySlippers
-CATExplorador
 CSHttp
-CensysInspect
-CheTeam
-Claritybot
-Cliqzbot
-Cloud mapping
+Calculon
+Cegbfeieh
+ChinaClaw
+Chlooe
 Cogentbot
-CopyRightCheck
-Cosmos
-Crawling at Home Project
-CrazyWebCrawler
+Copyscape
+Curious
 Custo
-DigitalPebble
+CyotekWebCopy
+DatabaseDriverMysqli
 Discobot
-Dispatch
+DittoSpyder
+DnyzBot
 DomCopBot
 Domains Project
-Dotbot
-Drip
-EasyDL
-EroCrawler
-Evil
+EMail Wolf
+Ecxi
+EirGrabber
+ExtLinksBot
 Extreme Picture Finder
-EyeNetIE
-FDM
 FemtosearchBot
-Firefox/7.0
+FlashGet
 Flunky
 Foobot
-FrontPage
-Fyrebot
+Fuzz
+FyberSpider
+GalaxyBot
 Genieo
-Go!Zilla
-Go-Ahead-Got-It
+Getintent
 Gotit
-GrabNet
 Grabber
 Grafula
-GrapeshotCrawler
-HEADMasterSEO
 HMView
 HTMLparser
-Hloader
+HTTP::Lite
+Heritrix
 Humanlinks
 HybridBot
+IDBTE4M
+IDBot
+IRLbot
 Iblog
+IlseBot
+Image Fetch
+Image Sucker
+IndeedBot
+InfoNaviRobot
 InfoTekies
-Intelliseek
-InternetSeer
-Iria
+InterGET
 Iskanie
+JOC Web Spider
 JamesBOT
-Jbrofuzz
-JennyBot
-Jyxobot
-Kenjin Spider
 Keyword Density
-Kozmosbot
+Kinza
 LNSpiderguy
 Larbin
-LexiBot
-LibWeb
+LeechGet
+Lftp
 Libwhisker
-Likse
+LieBaoFast
 LinkScan
-Linkbot
-LinqiaMetadataDownloaderBot
+LinkextractorPro
+LinkpadBot
+LinksManager
 LinqiaScrapeBot
-Lipperhey
 Lipperhey Spider
-Litemage_walker
-Lmspider
+Ltx71
 MFC_Tear_Sample
-MIIxpc
-MJ12bot
-Mag-Net
-Majestic SEO
-MarkMonitor
-MarkWatch
+MQQBrowser
+MSFrontPage
+Majestic12
 Mass Downloader
+Masscan
 Mata Hari
+MauiBot
+Mb2345Browser
 MeanPath Bot
+Meanpathbot
+Mediatoolkitbot
+MegaIndex.ru
+MicroMessenger
 Moblie Safari
-Mojolicious
-Morfeus Fucking Scanner
-Mozlila
-Mr.4x3
-Msrabot
-Musobot
-NICErsPRO
-Name Intelligence
 Nameprotect
-Navroad
 NearSite
-Needle
-NetAnts
+NetMechanic
 NetSpider
-Nettrack
-NextGenSearchBot
+Netvibes
 NimbleCrawler
-OnCrawl
-Openvas
+Offline Navigator
+Openfind
 OrangeSpider
+OutclicksBot
 OutfoxBot
-POE-Component-Client-HTTP
+PageScorer
+PageThing.com
+Panscient
+Papa Foto
 PeoplePal
-Petalbot
+Picscout
+Picsearch
 PictureFinder
-Pimonster
-Pixray
-Pockey
-Pu_iN
+Piepmatz
+PleaseCrawl
+ProPowerBot
+Probethenet
 PyCurl
-Quick-Crawler
+QueryN Metasearch
+RSSingBot
 RankActive
-RankActiveLinkBot
-RankFlex
-RankingBot
-RankingBot2
 ReGet
-Reaper
+RealDownload
+Recorder
 RedesScrapy
-Ripper
-RocketCrawler
-SEOkicks-Robot
+Rogerbot
+SBIder
+SEOkicks
 SEOlyticsCrawler
 SEOstats
+SISTRIX
 SalesIntelligent
 Scanbot
-ScoutJet
-Scrapy
-SearchmetricsBot
+Screaming
+Searchestate
 Semrush
+SemrushBot
 SeoSiteCheckup
-Shodan
+SeobilityBot
+Seomoz
 Site Sucker
 SiteExplorer
 SiteLockSpider
-Sitebeam
+SiteSucker
 Siteimprove
-Snapbot
-SocialRankIOBot
-Sogou web spider
+Sitevigil
+Sociscraper
+Sosospider
 Sottopop
+Spammen
+SpankBot
+Spbot
 Spinn3r
 SputnikBot
 Sqlmap
-Sqlworm
-Sqworm
+Steeler
+Sucuri
+SuperBot
 SuperHTTP
+Surfbot
 SurveyBot
+Suzuran
+Swiftbot
 Szukacz
-T8Abot
-Telesphorep
-The Intraformant
+T0PHackTeam
+Teleport
+TeleportPro
+Telesoft
+Thumbor
 TightTwatBot
-Titan
-Tracemyfile
-Trendictionbot
+Trendiction
+True_Robot
 Turingos
-Turnitin
-UnisterBot
-Upflow
 VB Project
-Vagabondo
-Voil
-WBSearchBot
+VCI
+VelenPublicWebCrawler
+Virusdie
+WASALive-Bot
+WEBDAV
+WPScan
 WWW-Collector-E
 WWW-Mechanize
-WWW::Mechanize
-WWWOFFLE
-WallpapersHD
-Web Fuck
+Wallpapers
+WeSEE
+Web Collage
+Web Pix
 Web Sauger
-Web Sucker
-WebAuto
+WebEnhancer
+WebFetch
 WebFuck
 WebGo IS
-WebImageCollector
-WebPix
+WebSauger
+WebStripper
+WebWhacker
 WebZIP
-Webshag
-WebsiteExtractor
-WebsiteQuester
+WebmasterWorldForumBot
 Whack
-Whacker
-Whatweb
 Who.is Bot
-Widow
+WiseGuys Robot
+Wprecon
 Xaldon_WebSpider
-YoudaoBot
+Xenu
 Zermelo
 Zeus
-ZoominfoBot
+ZumBot
+ZyBorg
 arquivo-web-crawler
-arquivo.pt
-autoemailspider
-cah.io.community
-coccocbot-web
+backlink-check
+clark-crawler
 com.plumanalytics
-crawl.sogou.com
 crawler.feedback
-dataforseo.com
-eCatch
-evc-batch
+crawler4j
+gopher
 instabid
-lwp-trivial
-meanpathbot
-mediawords
+lwp-request
+magpie-crawler
 oBot
 page scorer
 s1z.ru
-satoristudio.net
-scalaj-http
-seostar
-serpstatbot
+scan.lol
 sexsearcher
 sogouspider
 sp_auditbot
-tAkeOut
+spyfu
+sysscan
 ubermetrics-technologies.com
-webmeup-crawler
-x22Mozilla
-zgrab
+voyagerx.com
+webpros.com
+xpymep1.exe