Scraping

Scrape any URL as HTML, markdown, or structured content — formats, options, latency, and async polling.

POST /v1/scrape fetches a URL and returns its content in the format you ask for. The same request shape works for every site; the options you pass determine the output and how long the call takes.

Rendering & latency

Most scrapes return in well under a second. Some options require ByteKit to fully render the page before it can return content, which takes longer — typically a few seconds. Pass these only when you need them:

Option	When it adds latency
`cookies`	Non-empty array
`headers`	Non-empty object
`delay_ms`	Greater than `0`
`async`	`true` (request is queued and processed off the fast path)

The country option routes the request through a geo-located proxy. On its own it does not add rendering latency; it only does so when combined with one of the options above.

Looking for wait_for_selector or wait_until? Those are not ScrapeRequest options — they belong to the Screenshot and Recording endpoints. Sending them to /v1/scrape returns 422.

# A simple, low-latency scrape
curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

# Slower: delay_ms forces a full render and waits before returning
curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "delay_ms": 2000
  }'

Response fields

The default format is raw_html — pass formats to request others (markdown, html, links, images). The formats object in the response is keyed by the format you requested.

Field	Description
`formats.raw_html`	Page content in the requested format (`raw_html` is the default when `formats` is omitted)
`contentLength`	Compressed wire bytes — used for bandwidth billing
`statusCode`	HTTP status code of the origin page
`metadata.title`	Page `<title>` element

The X-Scrape-* response headers describe how each request was handled (cache status, timing, and the final URL) — useful for debugging.

Async scrapes

For long-running pages, pass "async": true to get a 202 Accepted response with a scrape id prefixed sc_. Poll the result with GET /v1/scrape/{id}.

# Start async scrape
curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "async": true}'

# Poll for result
curl https://api.bytekit.com/v1/scrape/sc_... \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE"

Webhook event header

Every outbound webhook POST includes an X-ByteKit-Event header with the event type:

Event type	Trigger
`bulk.completed`	A bulk job finishes (all items processed)
`scrape.completed`	An async scrape succeeds
`scrape.failed`	An async scrape fails terminally
`monitor.change_detected`	A monitor fires and detects a change (screenshot or content)
`monitor.captured`	A monitor fires with no change detected (notify_on: every)
`sitemap.completed`	A sitemap job finishes
`sitemap.failed`	A sitemap job fails terminally

Use this header to dispatch events without inspecting the body shape:

from flask import Flask, request

app = Flask(__name__)

@app.route("/webhooks/bytekit", methods=["POST"])
def handle():
    event = request.headers.get("X-ByteKit-Event")
    if event == "bulk.completed":
        handle_bulk(request.json)
    elif event == "scrape.completed":
        handle_scrape(request.json)
    return "", 200

X-ByteKit-Event is a reserved header — you cannot supply it via webhook_headers in monitor or bulk configuration. Attempts to do so will be rejected with a 422 at request time.

Next steps

Rate Limits — quota model and concurrency slots
Errors — how to interpret and retry error responses
API Reference: Scrape — full request/response schema