Spider Component Features

• Crawl a web site.
• Accumulate outbound links for crawling other web sites.
• Cache pages so future crawls can fetch from cache.
• Robots.txt compliant.
• Fetch the HTML content of each page crawled.
• Able to crawl HTTPS pages.
• Define "avoid" patterns to avoid URLs matching specific wildcard patterns.
• Define "avoid" patterns for avoiding matching outbound links.
• Read and connect timeouts.
• Maximum URL size to avoid ever-growing URLs.
• Maximum response size to avoid pages with very large or infinite content.
• Wind-down count to set a limit on pages spidered per site.