Can Crawl use a site map (or sitemap) as a crawling seed?
Yes. Both xml and gzipped sitemap xmls are supported.
Crawl supports both sitemaps (sitemap.xml
) or the gzipped equivalent (e.g. sitemap.xml.gz
) as seed URLs. Crawl will pursue all links contained within the sitemap — including other sitemaps — in adherence to any crawl restrictions (patterns, regular expressions, hops or other constraints) that may be in place for your crawl.
Updated over 2 years ago