Introduction - If you have any usage issues, please Google them yourself
Heritrix: Internet Archive Web Crawler
The archive-crawler project is building a flexible, extensible, robust, and scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accesible content.