# Scraping
Scrape any URL as HTML, markdown, or structured content — formats, options, latency, and async polling.
`POST /v1/scrape` fetches a URL and returns its content in the format you ask for. The same
request shape works for every site; the options you pass determine the output and how long the
call takes.
## Rendering & latency
Most scrapes return in well under a second. Some options require ByteKit to fully render the page
before it can return content, which takes longer — typically a few seconds. Pass these only when
you need them:
| Option | When it adds latency |
| ---------- | -------------------------------------------------------------- |
| `delay_ms` | Greater than `0` |
| `async` | `true` (the request is queued and processed in the background) |
The `country` option routes the request through a geo-located proxy. On its own it does not add
rendering latency; it only does so when combined with one of the options above.
Looking for `wait_for_selector` or `wait_until`? Those are not `ScrapeRequest` options — they
belong to the [Screenshot](https://bytekit.com/docs/api/screenshots/createscreenshot) endpoint. Sending them to
`/v1/scrape` returns `422`.
`cookies` and `headers` are no longer request options on any capture endpoint. Sending either
returns `400 validation_error`.
```bash
# A simple, low-latency scrape
curl -X POST https://api.bytekit.com/v1/scrape \
-H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
```
```bash
# Slower: delay_ms forces a full render and waits before returning
curl -X POST https://api.bytekit.com/v1/scrape \
-H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"delay_ms": 2000
}'
```
## Options
Every `POST /v1/scrape` option is listed here by the job it does. Only `url` is required. The
[API reference](https://bytekit.com/docs/api/scrape/createscrape) carries the full request schema.
### Output shaping
- `url` — the page to scrape.
- `formats` — what to return: any of `raw` (the default), `markdown`, `links` and `images`. The
`formats` object in the response is keyed by what you asked for.
- `markdown_mode` — how the page becomes markdown: `article` (the default) extracts the article
content, `raw` applies minimal cleanup, and `llm` produces compact, LLM-optimised output.
- `markdown_query` — a BM25 query string (up to 200 characters) that filters the markdown by
relevance to it. Omit it or leave it empty to disable.
- `include_tags` and `exclude_tags` — HTML element names (not CSS selectors). `include_tags` keeps
only the content inside those elements in the `markdown` output; `exclude_tags` strips them from
it. `include_tags` is applied first, then `exclude_tags`. Neither affects `raw`, which always
returns the full page.
- `clean_markdown` — when `true` and `markdown` is in `formats`, the converted markdown is sent to
an LLM (Groq) to remove boilerplate and ads and improve formatting before it is returned. If that
step fails for any reason (key unset, 4xx/5xx, timeout, empty response), you get the raw markdown
unchanged plus a warning in the `warnings` array. Billing is 3× credits and bytes, applied only
when cleaning succeeds.
- `markdown_links` — how links are written in the markdown: `inline` (the default) keeps
`[text](url)`, `text` keeps only the anchor text, `none` strips URLs and removes reference
definitions, and `references` converts to numbered `[text][N]` references with a Links footer.
- `markdown_images` — how images are written in the markdown: `inline` (the default) keeps
``, `text` keeps only the alt text, `none` removes images entirely, and `references`
converts to numbered `![alt][N]` references with an Images footer. This is distinct from
`markdown_filter_images`, which drops low-signal images before conversion.
- `with_links_summary` and `with_images_summary` — append a `Links:` or `Images:` footer, a
numbered list of the extracted URLs, to the markdown. `with_links_summary` is enabled
automatically when `markdown_links` is `references`.
- `markdown_compact` — collapse excessive whitespace for a more compact output.
- `markdown_filter_images` — filter low-signal images out of the markdown output.
- `markdown_include_media` — return `formats.links` and `formats.images` as rich scored objects
instead of string arrays, and include a top-level `tables` array. Only effective when `markdown`
is in `formats`.
- `markdown_include_warnings` — enable markdown-pipeline and tag-filter warnings for requests that
include the `markdown` format. Artifact availability warnings such as `artifact_unavailable` may
appear independently of this flag.
- `remove_base64_images` — base64-encoded `data:` image sources are stripped from the HTML before
markdown conversion (the default, `true`). Set it to `false` to keep them. It does not affect
`raw`, which always returns the original source unchanged.
### Token control
- `token_budget` — the maximum token count for the markdown output. When set and `markdown` is in
`formats`, the output is truncated at the last paragraph, then sentence, then token boundary that
keeps the estimated count under the limit. Leave it out for no truncation. Setting it adds a
`markdown_tokens` field to the response.
- `token_encoding` — the tokenizer hint for `token_budget`: `cl100k_base` (the default) matches
GPT-4/4o and `o200k_base` matches GPT-4o-mini and the o-series. Both use the same 4-characters-
per-token heuristic internally.
- `markdown_include_stats` — include a top-level `stats` object (characters, tokens, blocks) in the
response. Only effective when `markdown` is in `formats`.
### Rendering and access
- `delay_ms` — milliseconds to wait after page load before capturing, for content that renders
late. See [Rendering & latency](#rendering--latency) above.
- `country` — a two-letter uppercase country code (default `US`) that routes the request through a
geo-located proxy.
- `mobile` — emulate a mobile device: the request uses a mobile (Android) user agent and viewport
so the upstream serves its mobile page. The default is `false` (desktop). The value is part of
the cache key, so mobile and desktop responses are cached independently.
- `solve_challenge` — attempts to clear a Cloudflare challenge. Success is not guaranteed, and it
is billed as a normal browser render.
- `timeout_ms` — the request timeout in milliseconds, from `1000` to `90000` (default `60000`).
### Caching and delivery
- `cache_ttl` — how long a freshly fetched URL may be served from cache. `0` (string or integer)
skips the cache read, though cache-eligible results are still written. `Nh` or `Nd` sets a TTL,
capped at 168 h / 7 d. The default is `48h`. Safety-classified WAF husks are returned live but
never stored in the shared cache. It is honoured on both the synchronous and asynchronous paths.
- `async` — queue the scrape and get a `202` receipt immediately; see
[Async scrapes](#async-scrapes) below.
- `webhook_url` — an HTTPS endpoint to notify. An `http://` URL is rejected with `400 invalid_url`
(reason `scheme_unsupported`).
- `events` — which webhook events to deliver when `webhook_url` is set: any of `queued`,
`completed` and `failed`. All three are delivered by default; send a subset to receive fewer.
- `custom` — a JSON object of your own, echoed back on the success envelope (sync, async and
webhook). It is capped at 4096 UTF-8 bytes and does not affect the cache key, so two requests
that differ only in `custom` resolve to the same cache entry.
### Examples
A markdown scrape trimmed to a token budget, with the response's token statistics:
```bash
curl -X POST https://api.bytekit.com/v1/scrape \
-H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"formats": ["markdown"],
"markdown_mode": "llm",
"token_budget": 4000,
"token_encoding": "o200k_base",
"markdown_include_stats": true
}'
```
A content-only scrape: keep the `article` and `main` elements, then strip navigation and footers
from what is left.
```bash
curl -X POST https://api.bytekit.com/v1/scrape \
-H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"formats": ["markdown"],
"include_tags": ["article", "main"],
"exclude_tags": ["nav", "footer", "aside"]
}'
```
## Response fields
The default format is `raw` — pass `formats` to request others (`markdown`, `links`,
`images`). The `formats` object in the response is keyed by the format you requested.
The `scrape_url` tool in the [MCP server](https://bytekit.com/docs/mcp) differs: it returns markdown when `formats` is
omitted. A request that includes `markdown` bills the 1.5× `scrape_md` endpoint factor; one that
does not bills 1× (`scrape`). The API default stays `raw` so that omitting `formats` does not
silently raise a request's cost.
Envelope fields are `snake_case`; the keys inside `metadata` are the page's own metadata and
stay `camelCase`.
| Field | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `schema_version` | Response envelope version (currently `1`) |
| `formats.raw` | Page content in the requested format (`raw` is the default when `formats` is omitted) |
| `final_url` | URL the fetch landed on after redirects |
| `content_type` | `Content-Type` reported by the origin |
| `content_length` | Compressed wire bytes — the _basis_ for bandwidth billing, before the endpoint factor (see `billing`). A page served by a hosted browser reports the browser's full measured transfer |
| `status_code` | HTTP status code of the origin page |
| `retrieved_at` | ISO-8601 UTC timestamp of the fetch |
| `cache` | `"hit"` or `"miss"` |
| `cache_age_s` | Age of the cached entry in seconds — present only on a hit |
| `billing` | What the request cost: `raw_bytes`, the `factors` applied, the resulting `multiplier`, and `billed_bytes` |
| `metadata.title` | Page `` element |
The `X-Scrape-*` response headers describe how each request was handled (cache status, timing,
and the final URL) — useful for debugging.
## Async scrapes
For long-running pages, pass `"async": true` to get a `202 Accepted` response with a scrape
`id` prefixed `sc_`. Poll the result with `GET /v1/scrape/{id}`.
```bash
# Start async scrape
curl -X POST https://api.bytekit.com/v1/scrape \
-H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "async": true}'
# Poll for result
curl https://api.bytekit.com/v1/scrape/sc_... \
-H "Authorization: Bearer sk_live_YOUR_KEY_HERE"
```
## Webhook event header
Every outbound webhook POST includes an `X-ByteKit-Event` header with the event type:
| Event type | Trigger |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `bulk.completed` | A bulk job finishes (all items processed) |
| `scrape.completed` | An async scrape succeeds, or a scrape monitor delivers a capture (`has_change` tells a change from a no-change capture) |
| `scrape.failed` | An async scrape fails terminally |
| `monitor.change_detected` | A screenshot monitor fires and detects a change |
| `monitor.captured` | A screenshot monitor fires with no change detected (notify_on: every) |
| `monitor.suspended` | A monitor is suspended because the account ran out of balance (reason `insufficient_balance`) |
| `sitemap.completed` | A sitemap job finishes |
| `sitemap.failed` | A sitemap job fails terminally |
Use this header to dispatch events without inspecting the body shape:
```python
from flask import Flask, request
app = Flask(__name__)
@app.route("/webhooks/bytekit", methods=["POST"])
def handle():
event = request.headers.get("X-ByteKit-Event")
if event == "bulk.completed":
handle_bulk(request.json)
elif event == "scrape.completed":
handle_scrape(request.json)
return "", 200
```
`X-ByteKit-Event` is a reserved header — you cannot supply it via `webhook_headers` in
monitor or bulk configuration. Attempts to do so will be rejected with a `422` at request
time.
## Next steps
- [Rate Limits](https://bytekit.com/docs/guides/rate-limits) — usage counting and concurrency slots
- [Errors](https://bytekit.com/docs/guides/errors) — how to interpret and retry error responses
- [API Reference: Scrape](https://bytekit.com/docs/api/scrape/createscrape) — full request/response schema