Internet Robot (Robot) also known as Spider, Worm, or Random, the core objective is to obtain information on the Internet. Home of the robot using hypertext links traversal Web, through the URL reference from a HTML document to another HTML document crawling. Online robot collected information can be a variety of purposes, such as indexing, HTML document verification, URL link verification, access to updated information, such as mirror sites.

Robot.txt
用C#语言构造蜘蛛程序 .txt

• We are an exchange download platform that only provides communication channels. The downloaded content comes from the internet. Except for download issues, please Google on your own.
• The downloaded content is provided for members to upload. If it unintentionally infringes on your copyright, please contact us.
• Please use Winrar for decompression tools
• If download fail, Try it againg or Feedback to us.
• If downloaded content did not match the introduction, Feedback to us，Confirm and will be refund.
• Before downloading, you can inquire through the uploaded person information

Nothing．