tavily.crawl

beta 2 credits

Crawl a website from a root URL and return the content of up to 10 pages.

GET /v1/tavily/crawl

What you get#

A list of Tavily results in Onekey's unified shape, in data (an array). An empty array is a normal, successful answer and is charged like any other; a source that reports "nothing found" as an error returns a free 404 not_found instead.

Also known as: /web/crawl on APIs that nest their paths. See Alternatives and equivalents.

What it costs#

2 credits for a fresh fetch, charged only when it succeeds.
Free when served from cache (0.2 credit for a cached response over 100 KB).
Free if the request is invalid (never charged) or the source fails or reports not found (refunded).

Parameters#

Name	Type	Required	Default	What it is	Constraints
`url`	string	yes	(required)	Root URL of the site to crawl	max 2048 chars; must match `https?://[A-Za-z0-9](?:[A-Za-z0-9.-]{0,251}[A-Za-z0-9])?(?::(?:80\|443))?(?:[/?][\x21-\x5b\x5d-\x7e]*)?`; trimmed

Request#

curl "https://api.onekey.dev/v1/tavily/crawl?url=https%3A%2F%2Fdocs.tavily.com" \
  -H "x-api-key: YOUR_API_KEY"

Try it

Add your key to run this call right here. No key? Get one free. The key stays in this tab.

Recorded response#

A real Tavily response, recorded and returned in Onekey's shape. Fields the platform does not provide are null.

json

{
  "success": true,
  "data": [
    {
      "content": {
        "url": "https://docs.tavily.com/welcome",
        "text": "# Welcome\n\nExample page content."
      }
    },
    {
      "content": {
        "url": "https://docs.tavily.com/documentation/about",
        "text": "# About\n\nExample."
      }
    }
  ],
  "metadata": {
    "platform": "tavily",
    "endpoint": "crawl",
    "cached": false,
    "stale": false,
    "age_s": 0,
    "fetched_at": "2026-01-01T12:00:00+00:00",
    "credits_charged": 2,
    "request_id": "7f3a9c2e5b1d4c8fa2e6d0b9c4f18e37"
  },
  "error": null
}

Field mapping#

How this endpoint's source fields become Onekey's unified schema. Fields the source lacks come back as null.

Source path	Onekey field	Transform
`url`	`content.url`	`safeUrl`
`raw_content`	`content.text`	-

Caching and freshness#

Fresh for about 10 minutes. Repeat calls in that window are served from cache and are free.
Between 10 minutes and 1 hour the cached copy is still served instantly and marked stale: true while Onekey refreshes it in the background.
If the source is down when a refresh is needed, the last copy can be served for up to 1 hour more (marked stale).
A "not found" is remembered for 2 minutes and is free.
The cache is shared across all customers, which is what keeps repeat calls cheap.

What can go wrong#

Status	Code	What to do
401	`unauthorized`	Check the `x-api-key` header.
402	`insufficient_credits`	Add credits; your balance must cover the price.
404	`not_found`	Unknown endpoint path.
422	`invalid_params`	Fix the parameters using the table above. Free.
429	`rate_limited`	Wait `Retry-After` seconds, then retry.
502	`upstream_error`	The source failed. Refunded; retry with backoff.

See Errors for the full list and retry guidance.

Status: beta (not yet verified against the live source).