# web.markdown

**stable** **2 credits**

Main content of a page as clean markdown (readability-style).

`GET /v1/web/markdown`

## What you get

One Web record in Onekey's unified shape, in `data`. If the record does not exist you get a `404 not_found` and are not charged.

**Also known as:** `/web/scrape`, `/web/extract` on APIs that nest their paths. See [Alternatives and equivalents](https://onekey.dev/docs/alternatives.md).

## What it costs

- **2 credits** for a fresh fetch, charged only when it succeeds.
- **Free when served from cache (0.2 credit for a cached response over 100 KB).**
- **Free** if the request is invalid (never charged) or the source fails or reports not found (refunded).

## Parameters

| Name | Type | Required | Default | What it is | Constraints |
| --- | --- | --- | --- | --- | --- |
| `url` | string | yes | (required) | Absolute http(s) URL of the page (ports 80/443 only; private/internal hosts are refused) | max 2048 chars; must match `https?://[A-Za-z0-9](?:[A-Za-z0-9.-]{0,251}[A-Za-z0-9])?(?::(?:80\|443))?(?:[/?][\x21-\x5b\x5d-\x7e]*)?`; trimmed |

## Request

```bash
curl "https://api.onekey.dev/v1/web/markdown?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FWeb_scraping" \
  -H "x-api-key: YOUR_API_KEY"
```

```python
import requests

res = requests.get(
    "https://api.onekey.dev/v1/web/markdown",
    params={"url": "https://en.wikipedia.org/wiki/Web_scraping"},
    headers={"x-api-key": "YOUR_API_KEY"},
)
body = res.json()

if body["success"]:
    print(body["data"])
else:
    print(body["error"]["code"], body["error"]["message"])
```

```javascript
const params = new URLSearchParams({ url: "https://en.wikipedia.org/wiki/Web_scraping" });
const res = await fetch(`https://api.onekey.dev/v1/web/markdown?${params}`, {
  headers: { "x-api-key": "YOUR_API_KEY" },
});
const body = await res.json();

if (body.success) console.log(body.data);
else console.error(body.error.code, body.error.message);
```

## Recorded response

A real Web response, recorded and returned in Onekey's shape. Fields the platform does not provide are `null`.

```json
{
  "success": true,
  "data": {
    "content": {
      "title": "Web scraping - Wikipedia",
      "text": "# Web scraping\n\nFrom Wikipedia, the free encyclopedia\n\nFor broader coverage of this topic, see [Data scraping](https://en.wikipedia.org/wiki/Data_scraping).\n\n\"Web scraper\" redirects here. For websites that scrape content, see [Scraper site](https://en.wikipedia.org/wiki/Scraper_site).\n\n|  | This article needs more citations. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed.Find sources: \"Web scraping\" – news · newspapers · books · scholar · JSTOR (April 2023) (Learn how and when to remove this message) |\n| --- | --- |\n\n**Web scraping**, **web harvesting**, or **web data extraction** is [data scraping](https://en.wikipedia.org/wiki/Data_scraping) used for [extracting data](https://en.wikipedia.org/wiki/Data_extraction) from [websites](https://en.wikipedia.org/wiki/Website).[1] Web scraping software may directly access the [World Wide Web](https://en.wikipedia.org/wiki/World_Wide_Web) using the [Hypertext Transfer Protocol](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) or a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a [bot](https://en.wikipedia.org/wiki/Internet_bot) or [web crawler](https://en.wikipedia.org/wiki/Web_crawler). It is a form of copying in which specific data is gathered and copied from the web, typically into a central local [database](https://en.wikipedia.org/wiki/Database) or [spreadsheet](https://en.wikipedia.org/wiki/Spreadsheet), for later [retrieval](https://en.wikipedia.org/wiki/Data_retrieval) or [analysis](https://en.wikipedia.org/wiki/Data_analysis).\n\nScraping a web page involves fetching it and then extracting data from it. Fetching is the downloading of a page (which a browser does when a user views a page). Therefore, web crawling is a main component of web scraping, to fetch pages for later processing. Having fetched, extraction can take place. The content of a page may be [parsed](https://en.wikipedia.org/wiki/Parsing), searched and reformatted, and its data copied into a spreadsheet or loaded into a database. Web scrapers typically take something out of a page, to make use of it for another purpose somewhere else. An example would be finding and copying names and telephone numbers, companies and their URLs, or e-mail addresses to a list (contact scraping). Another example is collecting competitors product prices for marketing purposes, which can involve gathering large-scale pricing datasets from e-commerce websites and analysing them using data science techniques such as trend analysis, predictive modelling, and competitive benchmarking.\n\n[Contact scraping](https://en.wikipedia.org/wiki/Contact_scraping) is a type of web scraping that is used as a component of applications used for [web indexing](https://en.wikipedia.org/wiki/Web_indexing), [web mining](https://en.wikipedia.org/wiki/Web_mining) and [data mining](https://en.wikipedia.org",
      "url": "https://en.wikipedia.org/wiki/Web_scraping",
      "language": "en",
      "createdAt": "2005-09-17T18:57:30Z"
    },
    "author": {
      "displayName": "Contributors to Wikimedia projects"
    },
    "entity": {
      "name": null
    },
    "ext": {
      "wordCount": 6206,
      "readingTimeMinutes": 29,
      "excerpt": "Web scraping From Wikipedia, the free encyclopedia For broader coverage of this topic, see Data scraping. \"Web scraper\" redirects here. For websites that scrape content, see Scraper site. This article needs more citations. Please help improve this article by adding citations to reliable sources. Uns"
    },
    "media": {
      "url": null
    }
  },
  "metadata": {
    "platform": "web",
    "endpoint": "markdown",
    "cached": false,
    "stale": false,
    "age_s": 0,
    "fetched_at": "2026-01-01T12:00:00+00:00",
    "credits_charged": 2,
    "request_id": "7f3a9c2e5b1d4c8fa2e6d0b9c4f18e37"
  },
  "error": null
}
```

## Field mapping

How this endpoint's source fields become Onekey's [unified schema](https://onekey.dev/docs/schema.md). Fields the source lacks come back as `null`.

| Source path | Onekey field | Transform |
| --- | --- | --- |
| `title` | `content.title` | - |
| `markdown` | `content.text` | - |
| `finalUrl` | `content.url` | `safeUrl` |
| `language` | `content.language` | - |
| `publishedAt` | `content.createdAt` | `toIsoDate` |
| `byline` | `author.displayName` | - |
| `siteName` | `entity.name` | - |
| `wordCount` | `ext.wordCount` | `toNumber` |
| `readingTimeMinutes` | `ext.readingTimeMinutes` | `toNumber` |
| `excerpt` | `ext.excerpt` | - |
| `imageUrl` | `media.url` | `safeUrl` |

## Caching and freshness

- Fresh for about 10 minutes. Repeat calls in that window are served from cache and are free.
- Between 10 minutes and 1 hour the cached copy is still served instantly and marked `stale: true` while Onekey refreshes it in the background.
- If the source is down when a refresh is needed, the last copy can be served for up to 1 hour more (marked stale).
- A "not found" is remembered for 2 minutes and is free.
- The cache is shared across all customers, which is what keeps repeat calls cheap.

## What can go wrong

| Status | Code | What to do |
| --- | --- | --- |
| 401 | `unauthorized` | Check the `x-api-key` header. |
| 402 | `insufficient_credits` | [Add credits](https://onekey.dev/pricing); your balance must cover the price. |
| 404 | `not_found` | That record does not exist. Free; do not retry. |
| 422 | `invalid_params` | Fix the parameters using the table above. Free. |
| 429 | `rate_limited` | Wait `Retry-After` seconds, then retry. |
| 502 | `upstream_error` | The source failed. Refunded; retry with backoff. |

See [Errors](https://onekey.dev/docs/errors.md) for the full list and retry guidance.

Status: stable (verified against the live source).