Article Search Backends

type:Article queries are served by one of two backends:

• Fast - a low-latency index covering the last 6 months of articles by crawled date. Hit counts are stable and queries rarely time out.
• Archive - the full historical index of articles. It returns much older data, but queries are slower, hit counts may vary slightly between requests, and queries can occasionally fail with a transient timeouts that requires a retry.

By default, DQL picks the backend that is most likely to give you a fast, stable response. You can override that decision with the searchArchive=1 query parameter.

Default routing

For a type:Article query, DQL chooses the fast backend when any of the following is true:

• The query has a date filter (date>=…, date.timestamp>=…, etc.) whose lower bound falls inside the fast index 6 month window.
• The query uses sortBy:date (or revSortBy:date, or sortBy:crawlTimestamp) and the requested size is less than or equal to the default of 25.
• Otherwise, the query is routed to the archive backend.

Why sortBy:date defaults to the fast backend

Very large sorted scans are more likely to hit a transient timeout error on the backend. DQL therefore routes small sortBy:date requests to the fast backend by default. See Auto-rewrite of sortBy:date below for how this is surfaced in the response.

The searchArchive parameter

Set searchArchive=1 (or searchArchive=true) to force the query to run against the archive backend, regardless of the default routing:

query=type:Article author:"Marques Brownlee" sortBy:date&searchArchive=1

Use this when:

• You need article data older than the fast index 6 month window
• A sortBy:date query returns far fewer results than the same query without sortBy:date, and you want to see the older matches.

Trade-offs of searchArchive=1

Archive queries are slower, can return slightly different total hit counts between calls, and may occasionally fail with a transient timeouts. Retry the request if you get an empty result set that does not match an earlier successful response.

Auto-rewrite of sortBy:date

When a sortBy:date query is routed to the fast backend and you have not supplied an explicit lower-bound date filter, DQL adds one automatically so that the bound of the result set is explicit in the query itself. The rewritten query and the reason are returned in the response under rewrites:

  {
    "hits": 27,
    "results": 27,
    "diffbot_type": "Article",
    "rewrites": [
      {
        "query": "type:Article author:\"Marques Brownlee\" sortBy:date date.timestamp>=2025-12-11",
        "reason": "Results are limited to fast index. See https://docs.diffbot.com/reference/dql-article-backends"
      }
    ],
    "data": [ … ]
  }

The added clause uses the start of the fast index window at the time the query was processed. If you want results older than that lower bound, re-issue the query with searchArchive=1.

Articles without date field

Since sortBy:date requires date to be populated in the document, any Article without a date attribute will not be included in the hits or matches returned. See dates-and-timestamps .