Diffbot maintains the largest independently crawled web index outside of Google and Bing (>100TB). The Web Search API allows for fast querying of this index.

Candidates are retrieved and reranked with a cross encoder model trained to rank factual relevance over popularity, primary sources over domain monopolies, and organic page rank over paid spend.

The net effect is search results that assume the user is not an idiot. A search for "hypothyroidism" returns relevant research study papers alongside content from publicly administered websites and reputable non-profit institutions.

Key Features

• Latency: Currently sub-200ms hot latency, ~400ms cold. We are targeting sub 100ms P90 latency.
• Highlights: Splices out the most relevant chunk of a page and returns it in the content of each search result. Not an AI summary; that would be slow and undeterminably inaccurate. Chunk size can be dialed in with the maxTokens parameter for context-size sensitive applications.
• Local Hosting: The entire web index, including the retrieval/reranking models, and the API, can be hosted on-prem, enabling internet "connectivity" in air-gapped or high risk workflows. Reach out to [email protected] for more information on this option.

Usage

Access the Web Search API via REST or through the diffbot-python library. For use with agents, install one of the harness plugins available in diffbot-skills.

Usage is currently unmetered while we open Web Search API up for public testing. Don't be the person to screw this up for everyone.

Operators

Use operators in your text query to refine your searches. Here are all currently supported operators:

Limitations

• The index is English only for the moment. Multi-language support is ongoing.
• Content that has explicitly opted out to crawling (robots.txt) are not included in the index.
• Platform-gated content is generally not included in the index, this includes content from sites like Reddit and most social media platforms.