News and RSS tracking

The problem#

Feeds are supposed to be simple, but every publisher formats them a little differently, and turning a headline into readable text is a second project.

Onekey parses any feed URL into normalised items, searches news, and extracts an article to plain text or Markdown, so a news pipeline is three calls.

Endpoints to call#

Each endpoint name links to its reference page with every parameter, the caching window and a recorded response.

Your first call#

This is web.feed (3 credits). Replace YOUR_API_KEY with your key; sign up, no card. 1,000 free credits cover the open-platform endpoints. Hard-to-reach platforms (X, TikTok, Instagram, Reddit, and others marked on their page) need a verified email and a purchased pack.

curl "https://api.onekey.dev/v1/web/feed?url=https%3A%2F%2Fgithub.blog%2Ffeed%2F" \
  -H "x-api-key: YOUR_API_KEY"

A working script#

Read a set of feeds, remember what you have already seen, search news for a topic and fetch the full text of what is new. Persist seen between runs (a file or a database) and run the script on your own schedule.

import os
import requests

API = "https://api.onekey.dev"
HEADERS = {"x-api-key": os.environ["ONEKEY_KEY"]}


def call(path, **params):
    """GET /v1/<platform>/<endpoint>; returns the data or raises with the error code."""
    r = requests.get(f"{API}/v1/{path}", params=params, headers=HEADERS, timeout=30)
    body = r.json()
    if not body["success"]:
        raise RuntimeError(f"{path}: {body['error']['code']}")
    return body["data"]


def get(item, field):
    """get(item, 'content.title'): a field of the unified schema, or None."""
    for key in field.split("."):
        item = (item or {}).get(key)
    return item


FEEDS = ["https://github.blog/feed/"]
seen = set()

for url in FEEDS:
    for entry in call("web/feed", url=url):
        key = get(entry, "content.id") or get(entry, "content.url")
        if key in seen:
            continue
        seen.add(key)
        print(get(entry, "content.createdAt"), get(entry, "content.title"))

for item in call("google/news", q="battery recycling")[:5]:
    print("news", get(item, "author.displayName"), "|", get(item, "content.title"))

for item in call("gdelt/articles", q="battery recycling", n="10")[:5]:
    print("gdelt", get(item, "content.createdAt"), get(item, "content.title"))

# Full text of one article
article = call("web/article", url="https://en.wikipedia.org/wiki/RSS")
print(get(article, "content.title"), "-", get(article, "ext.wordCount"), "words")

What a month costs#

Illustrative volume, not a benchmark: 30 feeds read every 6 hours for 30 days, 10 news queries run 4 times a day, and 200 articles fetched as full text.

About 13,000 credits: the Builder pack (30,000 credits, $39) covers it, at $1.30 per 1,000 credits. Prices are in USD (regional pricing is planned); credits never expire. See pricing.

Every figure above assumes every call is a fresh fetch. These endpoints cache for 2 minutes to 10 minutes. Free when served from cache (0.2 credit for a cached response over 100 KB). Repeating a call inside that window costs nothing. Invalid, failed and not-found calls are not charged either.

Good to know#

• Feed responses are cached for a short window, so polling faster than that returns the same copy and costs nothing. Polling hourly instead of every 6 hours would multiply the feed line above by six.
• Only pages a normal visitor can read are extracted. Paywalled or login-only pages will not return full text.

Keep reading#

• Caching
• Pagination
• Credits and pricing
• Platform pages: Web pages, Google, GDELT