DigitalPebble
Lftp
LinkWalker
Nutch
Sucker
TeleportPro
TheNomad
Turnitin
URLy Warning
clark-crawler