API Reference

Retrieve Crawl Job Data

get
https://api.diffbot.com/v3/crawl/data

Download the extracted results of a crawl job

To download the results of your crawl job's extracted URLs, make a GET request to this endpoint. By default the complete extracted JSON data will be downloaded.

Download a URL Report

The URL Report is a CSV file that provides a diagnostic on every URL crawled by a Crawl API job and does not include extraction output. To retrieve the URL report, pass type=urls to your GET request on this endpoint.

Query Params
string
required
Defaults to test-crawl

Name of the crawl job whose data you wish to download.

string

Request format=csv to download the extracted data in CSV format (default: json). Note that CSV files will only contain top-level fields.

string

Request type=urls to retrieve the URL Report (CSV).

string

Pass an integer value (e.g. num=100) to request a subset of URLs, most recently processed first.

Responses

Updated 8 months ago

Did this page help you?
Language
Credentials
Query
LoadingLoading…
Response
Click Try It! to start a request and see the response here! Or choose an example:
application/json

Updated 8 months ago

Did this page help you?