commoncrawl.index

stable Advanced

Common Crawl index captures for a URL or domain in one crawl.

GET /v1/commoncrawl/index

What you get#

A list of Commoncrawl results in Onekey's unified shape, in data (an array). An empty array is a normal, successful answer and is charged like any other; a source that reports "nothing found" as an error returns a free 404 not_found instead.

What it costs#

Advanced · 5 credits for a fresh fetch, charged only when it succeeds.
Free when served from cache (0.2 credit for a cached response over 100 KB).
Free if the request is invalid (never charged) or the source fails or reports not found (refunded).

Parameters#

Name	Type	Required	Default	What it is	Constraints
`url`	string	yes	(required)	URL or domain (wildcards not allowed)	max 2000 chars; must match `(?:https?://)?[A-Za-z0-9](?:[A-Za-z0-9.-]{0,251}[A-Za-z0-9])?(?:[/?][\x21-\x5b\x5d-\x7e]*)?`; trimmed
`crawl`	string	no	`CC-MAIN-2026-39`	Crawl id, e.g. CC-MAIN-2026-39 (see commoncrawl.crawls)	max 15 chars; must match `CC-MAIN-[0-9]{4}-[0-9]{2}`; trimmed

Request#

curl "https://api.onekey.dev/v1/commoncrawl/index?url=example.com" \
  -H "x-api-key: YOUR_API_KEY"

Try it

Add your key to run this call right here. No key? Get one free. The key stays in this tab.

Recorded response#

A real Commoncrawl response, recorded and returned in Onekey's shape. Fields the platform does not provide are null.

json

{
  "success": true,
  "data": [
    {
      "entity": {
        "url": "https://www.example.com/",
        "createdAt": "2612-01-17T04:55:31Z",
        "language": "eng"
      },
      "ext": {
        "httpStatus": 200,
        "mimeType": "text/html",
        "filename": "crawl-data/CC-MAIN-2026-39/segments/1788492699475.64/warc/CC-MAIN-20260904131603-20260904161603-00865.warc.gz",
        "offset": 639725720,
        "length": 954
      }
    },
    {
      "entity": {
        "url": "https://example.com/",
        "createdAt": "2612-01-17T04:55:32Z",
        "language": "eng"
      },
      "ext": {
        "httpStatus": 200,
        "mimeType": "text/html",
        "filename": "crawl-data/CC-MAIN-2026-39/segments/1788492699475.64/warc/CC-MAIN-20260904131603-20260904161603-00162.warc.gz",
        "offset": 185952529,
        "length": 954
      }
    }
  ],
  "metadata": {
    "platform": "commoncrawl",
    "endpoint": "index",
    "cached": false,
    "stale": false,
    "age_s": 0,
    "fetched_at": "2026-01-01T12:00:00+00:00",
    "credits_charged": 5,
    "request_id": "7f3a9c2e5b1d4c8fa2e6d0b9c4f18e37"
  },
  "error": null
}

Field mapping#

How this endpoint's source fields become Onekey's unified schema. Fields the source lacks come back as null.

Source path	Onekey field	Transform
`url`	`entity.url`	`safeUrl`
`timestamp`	`entity.createdAt`	`toIsoDate`
`status`	`ext.httpStatus`	`toNumber`
`mime`	`ext.mimeType`	-
`languages`	`entity.language`	-
`filename`	`ext.filename`	-
`offset`	`ext.offset`	`toNumber`
`length`	`ext.length`	`toNumber`

Caching and freshness#

Fresh for about 1 hour. Repeat calls in that window are served from cache and are free.
Between 1 hour and 1 day the cached copy is still served instantly and marked stale: true while Onekey refreshes it in the background.
If the source is down when a refresh is needed, the last copy can be served for up to 1 day more (marked stale).
A "not found" is remembered for 5 minutes and is free.
The cache is shared across all customers, which is what keeps repeat calls cheap.

What can go wrong#

Status	Code	What to do
401	`unauthorized`	Check the `x-api-key` header.
402	`insufficient_credits`	Add credits; your balance must cover the price.
404	`not_found`	Unknown endpoint path.
422	`invalid_params`	Fix the parameters using the table above. Free.
429	`rate_limited`	Wait `Retry-After` seconds, then retry.
502	`upstream_error`	The source failed. Refunded; retry with backoff.

See Errors for the full list and retry guidance.

Status: stable (verified against the live source).