What you get#
Onekey exposes Common Crawl as 2 endpoints behind one API key: 2 endpoints return a list (data is an array). Every answer uses the same envelope (success, data, metadata, error), the cost is reported in metadata.credits_charged, and fields are mapped onto the unified schema so the code you write for Common Crawl also works for the rest of the web tools category.
Endpoints#
| Endpoint | What it does | Parameters | Credits | Status |
|---|---|---|---|---|
crawls | List Common Crawl crawl collections (id, name, date range, CDX API URL). | - | 1 | stable |
index | Common Crawl index captures for a URL or domain in one crawl. | url*, crawl | 5 | stable |
A * marks a required parameter. Each endpoint name links to its reference page with the full parameter table, caching window and a recorded response.
Make a request#
This calls commoncrawl.crawls (1 credit, stable). Replace YOUR_API_KEY with your key; sign up, no card. 1,000 free credits cover the open-platform endpoints. Hard-to-reach platforms (X, TikTok, Instagram, Reddit, and others marked on their page) need a verified email and a purchased pack.
curl "https://api.onekey.dev/v1/commoncrawl/crawls" \
-H "x-api-key: YOUR_API_KEY"Fields you get back#
These unified fields come back from Common Crawl, with the same names as on every other platform. The count is how many of its 2 mapped endpoints return the field; a field the source lacks is null, never missing. 8 platform-specific fields (ext.<name>) come on top, documented on each endpoint page.
| Field | Type | Endpoints |
|---|---|---|
entity.createdAt | ISO 8601 date | 1 |
entity.id | string | 1 |
entity.language | string | 1 |
entity.name | string | 1 |
entity.url | URL | 1 |
The full schema is in Unified schema.
Pricing#
Common Crawl calls cost 1 to 5 credits, flat per endpoint. Free when served from cache (0.2 credit for a cached response over 100 KB). Invalid, failed or not-found calls are not charged. See pricing for credit packs.