Can Diffbot crawl sites that use “infinite” or “endless” scrolling?

We offer an API parameter &scroll=slow that you can add to your Extract API calls to trigger page scrolling, to load lazy content/links. However, there is a maximum scrolling time of ~30s, so it is not currently possible to scroll endlessly.

In most cases sites will offer alternative means to find the same links:

• related links (to other posts or products) on individual post or product pages
• search filters or category links that narrow the number of results
• a sitemap file (e.g. sitemap.xml) or similar map to individual item pages

For assistance with any of the above, feel free to contact us at [email protected].