DomainCrawler
InternetSeer
MTRobot
Needle
NetAnts
PageAnalyzer
SmartDownload
Sottopop
WebsiteExtractor
heritrix