Can Crawl use a site map (or sitemap) as a crawling seed?

Yes. Both xml and gzipped sitemap xmls are supported.

Crawl supports both sitemaps (sitemap.xml) or the gzipped equivalent (e.g. sitemap.xml.gz) as seed URLs. Crawl will pursue all links contained within the sitemap — including other sitemaps — in adherence to any crawl restrictions (patterns, regular expressions, hops or other constraints) that may be in place for your crawl.


Did this page help you?