Scraping

POST /v1/scrape fetches a URL and returns its content in the format you ask for. The same request shape works for every site; the options you pass determine the output and how long the call takes.

Rendering & latency

Most scrapes return in well under a second. Some options require ByteKit to fully render the page before it can return content, which takes longer — typically a few seconds. Pass these only when you need them:

The country option routes the request through a geo-located proxy. On its own it does not add rendering latency; it only does so when combined with one of the options above.

Looking for wait_for_selector or wait_until? Those are not ScrapeRequest options — they belong to the Screenshot endpoint. Sending them to /v1/scrape returns 422.

cookies and headers are no longer request options on any capture endpoint. Sending either returns 400 validation_error.

# A simple, low-latency scrape
curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

# Slower: delay_ms forces a full render and waits before returning
curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "delay_ms": 2000
  }'

Options

Every POST /v1/scrape option is listed here by the job it does. Only url is required. The API reference carries the full request schema.

Output shaping

• url — the page to scrape.
• formats — what to return: any of raw (the default), markdown, links and images. The formats object in the response is keyed by what you asked for.
• markdown_mode — how the page becomes markdown: article (the default) extracts the article content, raw applies minimal cleanup, and llm produces compact, LLM-optimised output.
• markdown_query — a BM25 query string (up to 200 characters) that filters the markdown by relevance to it. Omit it or leave it empty to disable.
• include_tags and exclude_tags — HTML element names (not CSS selectors). include_tags keeps only the content inside those elements in the markdown output; exclude_tags strips them from it. include_tags is applied first, then exclude_tags. Neither affects raw, which always returns the full page.
• clean_markdown — when true and markdown is in formats, the converted markdown is sent to an LLM (Groq) to remove boilerplate and ads and improve formatting before it is returned. If that step fails for any reason (key unset, 4xx/5xx, timeout, empty response), you get the raw markdown unchanged plus a warning in the warnings array. Billing is 3× credits and bytes, applied only when cleaning succeeds.
• markdown_links — how links are written in the markdown: inline (the default) keeps [text](url), text keeps only the anchor text, none strips URLs and removes reference definitions, and references converts to numbered [text][N] references with a Links footer.
• markdown_images — how images are written in the markdown: inline (the default) keeps ![alt](url), text keeps only the alt text, none removes images entirely, and references converts to numbered ![alt][N] references with an Images footer. This is distinct from markdown_filter_images, which drops low-signal images before conversion.
• with_links_summary and with_images_summary — append a Links: or Images: footer, a numbered list of the extracted URLs, to the markdown. with_links_summary is enabled automatically when markdown_links is references.
• markdown_compact — collapse excessive whitespace for a more compact output.
• markdown_filter_images — filter low-signal images out of the markdown output.
• markdown_include_media — return formats.links and formats.images as rich scored objects instead of string arrays, and include a top-level tables array. Only effective when markdown is in formats.
• markdown_include_warnings — enable markdown-pipeline and tag-filter warnings for requests that include the markdown format. Artifact availability warnings such as artifact_unavailable may appear independently of this flag.
• remove_base64_images — base64-encoded data: image sources are stripped from the HTML before markdown conversion (the default, true). Set it to false to keep them. It does not affect raw, which always returns the original source unchanged.

Token control

• token_budget — the maximum token count for the markdown output. When set and markdown is in formats, the output is truncated at the last paragraph, then sentence, then token boundary that keeps the estimated count under the limit. Leave it out for no truncation. Setting it adds a markdown_tokens field to the response.
• token_encoding — the tokenizer hint for token_budget: cl100k_base (the default) matches GPT-4/4o and o200k_base matches GPT-4o-mini and the o-series. Both use the same 4-characters- per-token heuristic internally.
• markdown_include_stats — include a top-level stats object (characters, tokens, blocks) in the response. Only effective when markdown is in formats.

Rendering and access

• delay_ms — milliseconds to wait after page load before capturing, for content that renders late. See Rendering & latency above.
• country — a two-letter uppercase country code (default US) that routes the request through a geo-located proxy.
• mobile — emulate a mobile device: the request uses a mobile (Android) user agent and viewport so the upstream serves its mobile page. The default is false (desktop). The value is part of the cache key, so mobile and desktop responses are cached independently.
• solve_challenge — attempts to clear a Cloudflare challenge. Success is not guaranteed, and it is billed as a normal browser render.
• timeout_ms — the request timeout in milliseconds, from 1000 to 90000 (default 60000).

Caching and delivery

• cache_ttl — how long a freshly fetched URL may be served from cache. 0 (string or integer) skips the cache read, though cache-eligible results are still written. Nh or Nd sets a TTL, capped at 168 h / 7 d. The default is 48h. Safety-classified WAF husks are returned live but never stored in the shared cache. It is honoured on both the synchronous and asynchronous paths.
• async — queue the scrape and get a 202 receipt immediately; see Async scrapes below.
• webhook_url — an HTTPS endpoint to notify. An http:// URL is rejected with 400 invalid_url (reason scheme_unsupported).
• events — which webhook events to deliver when webhook_url is set: any of queued, completed and failed. All three are delivered by default; send a subset to receive fewer.
• custom — a JSON object of your own, echoed back on the success envelope (sync, async and webhook). It is capped at 4096 UTF-8 bytes and does not affect the cache key, so two requests that differ only in custom resolve to the same cache entry.

Examples

A markdown scrape trimmed to a token budget, with the response's token statistics:

curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "markdown_mode": "llm",
    "token_budget": 4000,
    "token_encoding": "o200k_base",
    "markdown_include_stats": true
  }'

A content-only scrape: keep the article and main elements, then strip navigation and footers from what is left.

curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "include_tags": ["article", "main"],
    "exclude_tags": ["nav", "footer", "aside"]
  }'

Response fields

The default format is raw — pass formats to request others (markdown, links, images). The formats object in the response is keyed by the format you requested.

The scrape_url tool in the MCP server differs: it returns markdown when formats is omitted. A request that includes markdown bills the 1.5× scrape_md endpoint factor; one that does not bills 1× (scrape). The API default stays raw so that omitting formats does not silently raise a request's cost.

Envelope fields are snake_case; the keys inside metadata are the page's own metadata and stay camelCase.

The X-Scrape-* response headers describe how each request was handled (cache status, timing, and the final URL) — useful for debugging.

Async scrapes

For long-running pages, pass "async": true to get a 202 Accepted response with a scrape id prefixed sc_. Poll the result with GET /v1/scrape/{id}.

# Start async scrape
curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "async": true}'

# Poll for result
curl https://api.bytekit.com/v1/scrape/sc_... \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE"

Webhook event header

Every outbound webhook POST includes an X-ByteKit-Event header with the event type:

Use this header to dispatch events without inspecting the body shape:

from flask import Flask, request

app = Flask(__name__)

@app.route("/webhooks/bytekit", methods=["POST"])
def handle():
    event = request.headers.get("X-ByteKit-Event")
    if event == "bulk.completed":
        handle_bulk(request.json)
    elif event == "scrape.completed":
        handle_scrape(request.json)
    return "", 200

X-ByteKit-Event is a reserved header — you cannot supply it via webhook_headers in monitor or bulk configuration. Attempts to do so will be rejected with a 422 at request time.

Next steps

• Rate Limits — usage counting and concurrency slots
• Errors — how to interpret and retry error responses
• API Reference: Scrape — full request/response schema