Crawl supports both sitemaps (
sitemap.xml) or the gzipped equivalent (e.g.
sitemap.xml.gz) as seed URLs. Crawl will pursue all links contained within the sitemap — including other sitemaps — in adherence to any crawl restrictions (patterns, regular expressions, hops or other constraints) that may be in place for your crawl.
Updated 22 days ago