Diffbot maintains the largest independently crawled web index outside of Google and Bing (>100TB). The Web Search API allows for fast querying of this index.

Candidates are retrieved and reranked with a cross encoder model trained to rank factual relevance over popularity, primary sources over domain monopolies, and organic page rank over paid spend.

The net effect is search results that assume the user is not an idiot. A search for "hypothyroidism" returns relevant research study papers alongside content from publicly administered websites and reputable non-profit institutions.

Key Features

Latency: Currently sub-200ms hot latency, ~400ms cold. We are targeting sub 100ms P90 latency.
Highlights: Included by default and primed for agentic use. Chunk size can be dialed in with the maxTokens parameter for context-size sensitive applications.
Local Hosting: The entire web index, including the retrieval/reranking models, and the API, can be hosted on-prem, enabling internet "connectivity" in air-gapped or high risk workflows. Reach out to sales@diffbot.com for more information on this option.

Usage

Access the Web Search API via REST or through the diffbot-python library. For use with agents, install one of the harness plugins available in diffbot-skills.

Usage is currently unmetered while we open Web Search API up for public testing. Don't be the person to screw this up for everyone.

Limitations

The index is English only for the moment. Multi-language support is ongoing.
Content that has explicitly opted out to crawling (robots.txt) are not included in the index.
Platform-gated content is generally not included in the index, this includes content from sites like Reddit and most social media platforms.

200Successful response

400Bad Request

401Unauthorized