Scrape

Functions for interacting with the scrape API resource. Each operation ships in four forms — sync / sync_detailed and their async counterparts.

Create Scrape

Scrape a URL

Fetches and processes a URL, returning content in one or more formats wrapped in a ScrapeEnvelope.

The response shape is decided by your request, not by how we route it. In precedence order:

1. async: true in the body, or ?async=true in the query string (either one is enough; false in one never cancels true in the other) → 202 with a ScrapeQueuedEnvelope receipt, returned immediately. Poll GET /v1/scrape/\{id\} for the result.
2. Otherwise, a request that needs a full page render — today that is delay_ms > 0 — is held open until the render reaches a terminal state, then returns 200 with the content envelope (success or failed). If the render is still running when the hold window expires (28 seconds by default), the request returns the 202 receipt instead and the job continues in the background.
3. Otherwise the request is served by the fast HTTP path and returns 200 with the content envelope, typically in under a second.

A failed fetch is a result, not a transport error. An unreachable host, a refused connection, or a target whose bot protection defeats every proxy tier all come back on the same 200 as a failed envelope — never a 5xx. Narrow on the envelope's status before reading the body. A failure the fast path decides inline is recorded as a failed job, so its envelope normally carries that job's id and GET /v1/scrape/\{id\} returns the same envelope; if recording the failed job itself fails, the envelope still arrives on the same 200 with id set to null and there is nothing to poll. A failure a held render reaches carries that render's real id. Either way the request transfers no billable bytes and is charged nothing.

A target that rate-limits us is not ByteKit rate-limiting you. The failed envelope's error.code is target_rate_limited when the target rate-limits the request. On the fast path that means our proxy attempts end on an HTTP 429 and no bot protection is identified on any attempt, even if the browser fallback then fails or reaches only an unidentified block page; the inline failure carries error.retry_after_seconds when the target sent a usable Retry-After. On a held render the browser's own answer decides: it observes a 429 on the page itself, or it fails outright after the proxy attempts ended on an HTTP 429. A bot-protection verdict on an earlier fetch attempt does not keep a held render blocked when the browser then observes the 429. A held render never carries error.retry_after_seconds. Retrying a target_rate_limited failure later is likely to work. A held render whose browser reaches an unidentified 403 or 503 page after a proxy 429 is blocked. A 429 that bot protection sends as a block stays blocked when it is the answer that decides the code: the fast path's proxy attempts, or the browser render on a held render. Your own per-key limit is a separate case: it answers HTTP 429 rate_limited with a Retry-After header (see the 429 response).

So a 202 means one of exactly two things: you asked for one with async: true or ?async=true, or the hold window expired. wait_for_selector and wait_until are ScreenshotRequest options, not ScrapeRequest options — sending either to this endpoint returns 422 validation_error, so neither can produce a 202 here.

Parameters

• async_ (bool | Unset)
• body (ScrapeRequest)
• url — str
• formats — list[ScrapeRequestFormatsItem | ScrapeRequestJsonFormat] | Unset — What to return. Each entry is either a string format (raw, markdown, links or images) or a json object entry that extracts structured data from the page. The string "json" is not a format value; the json entry is an object, described by ScrapeRequestJsonFormat. A list may hold at most one json entry, and a list holding only the json entry still captures raw. The json entry is accepted on POST /v1/scrape, POST /v1/scrape/bulk and POST /v1/bulk (scrape entries); monitor formats accept the string formats alone. Its result is formats.json on the response, and its charge is described on ScrapeRequestJsonFormat.
• country — str | Unset
• delay_ms — int | Unset — Milliseconds to wait after page load before capturing, for content that renders late. Any value above 0 puts the request on the full browser render path. Unless async: true is also set, the request is then HELD until the render finishes and returns 200 with the content envelope, or returns the 202 receipt if the hold window expires first. 0 (the default) keeps the request on the fast HTTP path, which returns 200 in one round trip.
• timeout_ms — int | Unset
• async_ — bool | Unset — When true, the request returns 202 with a ScrapeQueuedEnvelope receipt immediately and the scrape runs in the background; poll GET /v1/scrape/\{id\} for the result. The ?async=true query parameter is a second spelling of this field: the request is async when either one says true, and false in one never cancels true in the other. Asking for a receipt this way outranks every routing consequence: when neither is set (the default) a browser-path request is held to completion instead of being receipted. A 202 can still arrive without either, but only when the hold window expires.
• webhook_url — str | Unset — Webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported)
• events — list[ScrapeRequestEventsItem] | Unset — Webhook event types to deliver when webhook_url is set. Defaults to all three (queued, completed, failed); send an explicit subset to receive fewer.
• markdown_mode — ScrapeRequestMarkdownMode | Unset — Markdown processing mode. article=article extraction (default), raw=minimal cleanup, llm=compact LLM-optimised output.
• markdown_query — str | Unset — BM25 query string for relevance-ranked content filtering. Omit or leave empty to disable.
• markdown_links — ScrapeRequestMarkdownLinks | Unset — Link rendering style in the markdown output. 'inline' (default) keeps [text](url) as-is. 'text' strips URLs, keeping only anchor text. 'none' strips URLs and removes reference definitions. 'references' converts to numbered [text][N] refs with a Links footer.
• markdown_images — ScrapeRequestMarkdownImages | Unset — Image retention mode in the markdown output. 'inline' (default) keeps ![alt](url) as-is. 'text' strips the URL, keeping only alt text. 'none' removes images entirely. 'references' converts to numbered ![alt][N] refs with an Images footer. Distinct from markdown_filter_images (which drops low-signal images before conversion).
• with_links_summary — bool | Unset — When true, appends a 'Links:\n[1]: url\n...' footer to the markdown output, built from the extracted link list. Automatically enabled when markdown_links is 'references'.
• with_images_summary — bool | Unset — When true, appends an 'Images:\n[1]: url\n...' footer to the markdown output, built from the extracted image list.
• markdown_compact — bool | Unset — Collapse excessive whitespace for a more compact output.
• markdown_filter_images — bool | Unset — Filter low-signal images from the markdown output.
• markdown_include_media — bool | Unset — When true, formats.links and formats.images return ScrapeScoredLink[] / ScrapeScoredImage[] (rich objects) instead of string[], and a top-level tables array is included. Only effective when markdown is in formats.
• markdown_include_warnings — bool | Unset — When true, enables markdown-pipeline and tag-filter warnings for requests that include the markdown format. Artifact availability warnings such as artifact_unavailable may appear independently of this flag.
• markdown_include_stats — bool | Unset — When true, the response includes a top-level stats object with ScrapeStats (chars, tokens, blocks). Only effective when markdown is in formats.
• token_budget — int | Unset — Maximum token count for the markdown output. When set and markdown is in formats, the output is truncated at the last paragraph → sentence → token boundary that keeps the estimated count under this limit. Absent → no truncation. Triggers a markdown_tokens field in the response.
• token_encoding — ScrapeRequestTokenEncoding | Unset — Tokenizer encoding hint for token_budget. cl100k_base (default) matches GPT-4/4o; o200k_base matches GPT-4o-mini / o-series. Both use the same 4-char/token heuristic internally.
• mobile — bool | Unset — When true, emulate a mobile device — the request uses a mobile (Android) user agent and a mobile viewport, so the upstream serves the mobile version of the page. Default false (desktop). The value is part of the cache key, so mobile and desktop responses are cached independently.
• solve_challenge — bool | Unset — Attempts to clear a Cloudflare challenge; success is not guaranteed. Billed as a normal browser render.
• remove_base64_images — bool | Unset — When true (default), base64-encoded data: image sources are stripped from the HTML before markdown conversion. Set to false to preserve base64 images in the pipeline output. Does not affect raw, which always returns the original source unchanged.
• include_tags — list[str] | Unset — HTML element names (not CSS selectors) to keep. When non-empty, only content within these elements is included in the markdown output. Applied before exclude_tags. Does not affect raw, which always returns the full page.
• exclude_tags — list[str] | Unset — HTML element names (not CSS selectors) to strip from the markdown output. Applied after include_tags when both are present. Does not affect raw, which always returns the full page.
• cache_ttl — ScrapeRequestCacheTtlType1 | str | Unset — How long a freshly fetched URL may be served from cache. 0 (string or integer) skips the cache read; cache-eligible results are still written. Nh/Nd set a TTL (capped at 168 h / 7 d). Safety-classified WAF husks are returned live but never stored in the shared cache. Honoured on both the synchronous and asynchronous paths; the async success envelope reports the lookup outcome via the cache field.
• custom — ScrapeRequestCustom | Unset — User-supplied JSON payload, echoed back on the success envelope (sync, async, webhook). Capped at 4096 UTF-8 bytes (Buffer.byteLength). Distinct from the system-owned metadata column. Does NOT affect cache-key inputs — two requests differing only in custom resolve to the same cache entry.
• clean_markdown — bool | Unset — When true and markdown is in formats, the converted markdown is sent to Groq LLM for cleaning (boilerplate removal, ad stripping, formatting improvement) before being returned. Graceful degradation: any Groq failure (key unset, 4xx/5xx, timeout, empty response) returns the raw markdown unchanged plus a warning in the warnings array. Billing: 3× credits and bytes applied only when cleaning succeeds.

Call styles

sync_detailed(client=client, body=body, async_=async_) -> Response[Error | ScrapeErrorEnvelope | ScrapeSuccessEnvelope | ScrapeQueuedEnvelope]
sync(client=client, body=body, async_=async_) -> Error | ScrapeErrorEnvelope | ScrapeSuccessEnvelope | ScrapeQueuedEnvelope | None
await asyncio_detailed(client=client, body=body, async_=async_) -> Response[Error | ScrapeErrorEnvelope | ScrapeSuccessEnvelope | ScrapeQueuedEnvelope]
await asyncio(client=client, body=body, async_=async_) -> Error | ScrapeErrorEnvelope | ScrapeSuccessEnvelope | ScrapeQueuedEnvelope | None

Get Scrape

Get scrape result

Always returns 200. Check the status discriminator for the scrape state.

Parameters

• id (str)

Call styles

sync_detailed(id=id, client=client) -> Response[Error | ScrapeErrorEnvelope | ScrapeQueuedEnvelope | ScrapeSuccessEnvelope]
sync(id=id, client=client) -> Error | ScrapeErrorEnvelope | ScrapeQueuedEnvelope | ScrapeSuccessEnvelope | None
await asyncio_detailed(id=id, client=client) -> Response[Error | ScrapeErrorEnvelope | ScrapeQueuedEnvelope | ScrapeSuccessEnvelope]
await asyncio(id=id, client=client) -> Error | ScrapeErrorEnvelope | ScrapeQueuedEnvelope | ScrapeSuccessEnvelope | None