# Scrape

Functions for interacting with the `scrape` API resource. Each operation ships in four forms — `sync` / `sync_detailed` and their `async` counterparts.

## Create Scrape

Scrape a URL

Fetches and processes a URL, returning content in one or more formats wrapped in a ScrapeEnvelope.

**The response shape is decided by your request, not by how we route it.** In precedence order:

1. `async: true` in the body, or `?async=true` in the query string (either one is enough; `false` in
one never cancels `true` in the other) → **202** with a `ScrapeQueuedEnvelope` receipt, returned
immediately. Poll `GET /v1/scrape/\{id\}` for the result.
2. Otherwise, a request that needs a full page render — today that is `delay_ms > 0` — is **held
open** until the render reaches a terminal state, then returns **200** with the content envelope
(`success` or `failed`). If the render is still running when the hold window expires (28 seconds by
default), the request returns the **202** receipt instead and the job continues in the background.
3. Otherwise the request is served by the fast HTTP path and returns **200** with the content
envelope, typically in under a second.

**A failed fetch is a result, not a transport error.** An unreachable host, a refused connection, or
a target whose bot protection defeats every proxy tier all come back on the same **200** as a
`failed` envelope — never a 5xx. Narrow on the envelope's `status` before reading the body. A
failure the fast path decides inline is recorded as a failed job, so its envelope normally carries
that job's `id` and `GET /v1/scrape/\{id\}` returns the same envelope; if recording the failed job
itself fails, the envelope still arrives on the same 200 with `id` set to `null` and there is
nothing to poll. A failure a held render reaches carries that render's real `id`. Either way the
request transfers no billable bytes and is charged nothing.

**A target that rate-limits us is not ByteKit rate-limiting you.** The `failed` envelope's
`error.code` is `target_rate_limited` when the target rate-limits the request. On the fast path that
means our proxy attempts end on an HTTP 429 and no bot protection is identified on any attempt, even
if the browser fallback then fails or reaches only an unidentified block page; the inline failure
carries `error.retry_after_seconds` when the target sent a usable `Retry-After`. On a held render
the browser's own answer decides: it observes a 429 on the page itself, or it fails outright after
the proxy attempts ended on an HTTP 429. A bot-protection verdict on an earlier fetch attempt does
not keep a held render `blocked` when the browser then observes the 429. A held render never carries
`error.retry_after_seconds`. Retrying a `target_rate_limited` failure later is likely to work. A
held render whose browser reaches an unidentified 403 or 503 page after a proxy 429 is `blocked`. A
429 that bot protection sends as a block stays `blocked` when it is the answer that decides the
code: the fast path's proxy attempts, or the browser render on a held render. Your own per-key limit
is a separate case: it answers HTTP 429 `rate_limited` with a `Retry-After` header (see the `429`
response).

So a 202 means one of exactly two things: you asked for one with `async: true` or `?async=true`, or
the hold window expired. `wait_for_selector` and `wait_until` are `ScreenshotRequest` options, not
`ScrapeRequest` options — sending either to this endpoint returns `422 validation_error`, so neither
can produce a 202 here.

**Parameters**

- `async_ (bool | Unset)`
- `body (ScrapeRequest)`
- `url` — `str`
- `formats` — `list[ScrapeRequestFormatsItem | ScrapeRequestJsonFormat] | Unset` — What to return. Each entry is either a string format (`raw`, `markdown`, `links` or `images`) or a `json` object entry that extracts structured data from the page. The string `"json"` is not a format value; the `json` entry is an object, described by `ScrapeRequestJsonFormat`. A list may hold at most one `json` entry, and a list holding only the `json` entry still captures `raw`. The `json` entry is accepted on `POST /v1/scrape`, `POST /v1/scrape/bulk` and `POST /v1/bulk` (scrape entries); monitor `formats` accept the string formats alone. Its result is `formats.json` on the response, and its charge is described on `ScrapeRequestJsonFormat`.
- `country` — `str | Unset`
- `delay_ms` — `int | Unset` — Milliseconds to wait after page load before capturing, for content that renders late. Any value above 0 puts the request on the full browser render path. Unless `async: true` is also set, the request is then HELD until the render finishes and returns 200 with the content envelope, or returns the 202 receipt if the hold window expires first. `0` (the default) keeps the request on the fast HTTP path, which returns 200 in one round trip.
- `timeout_ms` — `int | Unset`
- `async_` — `bool | Unset` — When true, the request returns 202 with a `ScrapeQueuedEnvelope` receipt immediately and the scrape runs in the background; poll `GET /v1/scrape/\{id\}` for the result. The `?async=true` query parameter is a second spelling of this field: the request is async when either one says true, and `false` in one never cancels `true` in the other. Asking for a receipt this way outranks every routing consequence: when neither is set (the default) a browser-path request is held to completion instead of being receipted. A 202 can still arrive without either, but only when the hold window expires.
- `webhook_url` — `str | Unset` — Webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported)
- `events` — `list[ScrapeRequestEventsItem] | Unset` — Webhook event types to deliver when webhook_url is set. Defaults to all three (queued, completed, failed); send an explicit subset to receive fewer.
- `markdown_mode` — `ScrapeRequestMarkdownMode | Unset` — Markdown processing mode. article=article extraction (default), raw=minimal cleanup, llm=compact LLM-optimised output.
- `markdown_query` — `str | Unset` — BM25 query string for relevance-ranked content filtering. Omit or leave empty to disable.
- `markdown_links` — `ScrapeRequestMarkdownLinks | Unset` — Link rendering style in the markdown output. 'inline' (default) keeps `[text](url)` as-is. 'text' strips URLs, keeping only anchor text. 'none' strips URLs and removes reference definitions. 'references' converts to numbered `[text][N]` refs with a Links footer.
- `markdown_images` — `ScrapeRequestMarkdownImages | Unset` — Image retention mode in the markdown output. 'inline' (default) keeps `![alt](url)` as-is. 'text' strips the URL, keeping only alt text. 'none' removes images entirely. 'references' converts to numbered `![alt][N]` refs with an Images footer. Distinct from markdown_filter_images (which drops low-signal images before conversion).
- `with_links_summary` — `bool | Unset` — When true, appends a 'Links:\n[1]: url\n...' footer to the markdown output, built from the extracted link list. Automatically enabled when markdown_links is 'references'.
- `with_images_summary` — `bool | Unset` — When true, appends an 'Images:\n[1]: url\n...' footer to the markdown output, built from the extracted image list.
- `markdown_compact` — `bool | Unset` — Collapse excessive whitespace for a more compact output.
- `markdown_filter_images` — `bool | Unset` — Filter low-signal images from the markdown output.
- `markdown_include_media` — `bool | Unset` — When true, `formats.links` and `formats.images` return `ScrapeScoredLink[]` / `ScrapeScoredImage[]` (rich objects) instead of `string[]`, and a top-level `tables` array is included. Only effective when `markdown` is in `formats`.
- `markdown_include_warnings` — `bool | Unset` — When true, enables markdown-pipeline and tag-filter warnings for requests that include the `markdown` format. Artifact availability warnings such as `artifact_unavailable` may appear independently of this flag.
- `markdown_include_stats` — `bool | Unset` — When true, the response includes a top-level `stats` object with `ScrapeStats` (chars, tokens, blocks). Only effective when `markdown` is in `formats`.
- `token_budget` — `int | Unset` — Maximum token count for the markdown output. When set and `markdown` is in `formats`, the output is truncated at the last paragraph → sentence → token boundary that keeps the estimated count under this limit. Absent → no truncation. Triggers a `markdown_tokens` field in the response.
- `token_encoding` — `ScrapeRequestTokenEncoding | Unset` — Tokenizer encoding hint for `token_budget`. `cl100k_base` (default) matches GPT-4/4o; `o200k_base` matches GPT-4o-mini / o-series. Both use the same 4-char/token heuristic internally.
- `mobile` — `bool | Unset` — When true, emulate a mobile device — the request uses a mobile (Android) user agent and a mobile viewport, so the upstream serves the mobile version of the page. Default false (desktop). The value is part of the cache key, so mobile and desktop responses are cached independently.
- `solve_challenge` — `bool | Unset` — Attempts to clear a Cloudflare challenge; success is not guaranteed. Billed as a normal browser render.
- `remove_base64_images` — `bool | Unset` — When true (default), base64-encoded `data:` image sources are stripped from the HTML before markdown conversion. Set to false to preserve base64 images in the pipeline output. Does not affect `raw`, which always returns the original source unchanged.
- `include_tags` — `list[str] | Unset` — HTML element names (not CSS selectors) to keep. When non-empty, only content within these elements is included in the `markdown` output. Applied before `exclude_tags`. Does not affect `raw`, which always returns the full page.
- `exclude_tags` — `list[str] | Unset` — HTML element names (not CSS selectors) to strip from the `markdown` output. Applied after `include_tags` when both are present. Does not affect `raw`, which always returns the full page.
- `cache_ttl` — `ScrapeRequestCacheTtlType1 | str | Unset` — How long a freshly fetched URL may be served from cache. `0` (string or integer) skips the cache read; cache-eligible results are still written. `Nh`/`Nd` set a TTL (capped at 168 h / 7 d). Safety-classified WAF husks are returned live but never stored in the shared cache. Honoured on both the synchronous and asynchronous paths; the async success envelope reports the lookup outcome via the `cache` field.
- `custom` — `ScrapeRequestCustom | Unset` — User-supplied JSON payload, echoed back on the success envelope (sync, async, webhook). Capped at 4096 UTF-8 bytes (`Buffer.byteLength`). Distinct from the system-owned `metadata` column. Does NOT affect cache-key inputs — two requests differing only in `custom` resolve to the same cache entry.
- `clean_markdown` — `bool | Unset` — When `true` and `markdown` is in `formats`, the converted markdown is sent to Groq LLM for cleaning (boilerplate removal, ad stripping, formatting improvement) before being returned. Graceful degradation: any Groq failure (key unset, 4xx/5xx, timeout, empty response) returns the raw markdown unchanged plus a warning in the `warnings` array. Billing: 3× credits and bytes applied only when cleaning succeeds.

**Call styles**

```python
sync_detailed(client=client, body=body, async_=async_) -> Response[Error | ScrapeErrorEnvelope | ScrapeSuccessEnvelope | ScrapeQueuedEnvelope]
sync(client=client, body=body, async_=async_) -> Error | ScrapeErrorEnvelope | ScrapeSuccessEnvelope | ScrapeQueuedEnvelope | None
await asyncio_detailed(client=client, body=body, async_=async_) -> Response[Error | ScrapeErrorEnvelope | ScrapeSuccessEnvelope | ScrapeQueuedEnvelope]
await asyncio(client=client, body=body, async_=async_) -> Error | ScrapeErrorEnvelope | ScrapeSuccessEnvelope | ScrapeQueuedEnvelope | None
```

## Get Scrape

Get scrape result

Always returns 200. Check the `status` discriminator for the scrape state.

**Parameters**

- `id (str)`

**Call styles**

```python
sync_detailed(id=id, client=client) -> Response[Error | ScrapeErrorEnvelope | ScrapeQueuedEnvelope | ScrapeSuccessEnvelope]
sync(id=id, client=client) -> Error | ScrapeErrorEnvelope | ScrapeQueuedEnvelope | ScrapeSuccessEnvelope | None
await asyncio_detailed(id=id, client=client) -> Response[Error | ScrapeErrorEnvelope | ScrapeQueuedEnvelope | ScrapeSuccessEnvelope]
await asyncio(id=id, client=client) -> Error | ScrapeErrorEnvelope | ScrapeQueuedEnvelope | ScrapeSuccessEnvelope | None
```