# Scrape a URL

Fetches and processes a URL, returning content in one or more formats wrapped in a ScrapeEnvelope.

**The response shape is decided by your request, not by how we route it.** In precedence order:

1. `async: true` in the body, or `?async=true` in the query string (either one is enough; `false` in one never cancels `true` in the other) → **202** with a `ScrapeQueuedEnvelope` receipt, returned immediately. Poll `GET /v1/scrape/{id}` for the result.
2. Otherwise, a request that needs a full page render — today that is `delay_ms > 0` — is **held open** until the render reaches a terminal state, then returns **200** with the content envelope (`success` or `failed`). If the render is still running when the hold window expires (28 seconds by default), the request returns the **202** receipt instead and the job continues in the background.
3. Otherwise the request is served by the fast HTTP path and returns **200** with the content envelope, typically in under a second.

**A failed fetch is a result, not a transport error.** An unreachable host, a refused connection, or a target whose bot protection defeats every proxy tier all come back on the same **200** as a `failed` envelope — never a 5xx. Narrow on the envelope's `status` before reading the body. A failure the fast path decides inline is recorded as a failed job, so its envelope normally carries that job's `id` and `GET /v1/scrape/{id}` returns the same envelope; if recording the failed job itself fails, the envelope still arrives on the same 200 with `id` set to `null` and there is nothing to poll. A failure a held render reaches carries that render's real `id`. Either way the request transfers no billable bytes and is charged nothing.

**A target that rate-limits us is not ByteKit rate-limiting you.** The `failed` envelope's `error.code` is `target_rate_limited` when the target rate-limits the request. On the fast path that means our proxy attempts end on an HTTP 429 and no bot protection is identified on any attempt, even if the browser fallback then fails or reaches only an unidentified block page; the inline failure carries `error.retry_after_seconds` when the target sent a usable `Retry-After`. On a held render the browser's own answer decides: it observes a 429 on the page itself, or it fails outright after the proxy attempts ended on an HTTP 429. A bot-protection verdict on an earlier fetch attempt does not keep a held render `blocked` when the browser then observes the 429. A held render never carries `error.retry_after_seconds`. Retrying a `target_rate_limited` failure later is likely to work. A held render whose browser reaches an unidentified 403 or 503 page after a proxy 429 is `blocked`. A 429 that bot protection sends as a block stays `blocked` when it is the answer that decides the code: the fast path's proxy attempts, or the browser render on a held render. Your own per-key limit is a separate case: it answers HTTP 429 `rate_limited` with a `Retry-After` header (see the `429` response).

So a 202 means one of exactly two things: you asked for one with `async: true` or `?async=true`, or the hold window expired. `wait_for_selector` and `wait_until` are `ScreenshotRequest` options, not `ScrapeRequest` options — sending either to this endpoint returns `422 validation_error`, so neither can produce a 202 here.

**POST** `/v1/scrape`

## Parameters

- `async` (query, boolean) — Query-string spelling of the body `async` field. `true` returns the 202 receipt immediately; any other value is ignored. Either spelling turns async on, and `false` in one never cancels `true` in the other.

## Request Body

- `url` (required)
- `formats`
- `formats.type`
- `formats.schema`
- `formats.model`
- `formats.prompt`
- `country`
- `delay_ms`
- `timeout_ms`
- `async`
- `webhook_url`
- `events`
- `markdown_mode`
- `markdown_query`
- `markdown_links`
- `markdown_images`
- `with_links_summary`
- `with_images_summary`
- `markdown_compact`
- `markdown_filter_images`
- `markdown_include_media`
- `markdown_include_warnings`
- `markdown_include_stats`
- `token_budget`
- `token_encoding`
- `mobile`
- `solve_challenge`
- `remove_base64_images`
- `include_tags`
- `exclude_tags`
- `cache_ttl`
- `custom`
- `clean_markdown`

## Responses

### 200 — Scrape reached a terminal state and the body is that terminal envelope: `success` carrying content, or `failed` carrying the error envelope. Returned by the fast HTTP path in one round trip, and by a browser-path request without `async: true` or `?async=true` once its render reaches a terminal state inside the hold window. A held render that ends in failure answers 200 with the same `failed` envelope `GET /v1/scrape/{id}` returns — narrow on `status` before reading the body. A definitive upstream outcome is a result, not a transport error: an unreachable host, a refused connection, or a target whose bot protection defeats every proxy tier all answer 200 with the `failed` envelope, never a 5xx. An inline failure is recorded as a failed job, so its envelope normally carries that job's `id` and polling it returns the same envelope; `id` is `null` only when recording the failed job itself failed. `error.http_status` is present only when a real upstream HTTP status is known.
- `schema_version`
- `status`
- `id`
- `url`
- `final_url`
- `content_type`
- `content_length`
- `status_code`
- `retrieved_at`
- `formats`
- `formats.raw`
- `formats.markdown`
- `formats.links`
- `formats.links.url`
- `formats.links.text`
- `formats.links.isInternal`
- `formats.links.rel`
- `formats.images`
- `formats.images.url`
- `formats.images.alt`
- `formats.images.width`
- `formats.images.height`
- `formats.images.format`
- `formats.images.score`
- `formats.json`
- `formats.json.data`
- `formats.json.validated`
- `formats.json.model`
- `metadata`
- `metadata.title`
- `metadata.description`
- `metadata.language`
- `metadata.canonicalUrl`
- `metadata.ogTitle`
- `metadata.ogDescription`
- `metadata.ogImage`
- `metadata.ogUrl`
- `metadata.ogSiteName`
- `metadata.robots`
- `metadata.byline`
- `metadata.publishedAt`
- `tables`
- `tables.kind`
- `tables.gfm`
- `tables.rows`
- `tables.cols`
- `warnings`
- `warnings.code`
- `warnings.message`
- `warnings.element`
- `stats`
- `stats.chars`
- `stats.tokens`
- `stats.blocks`
- `markdown_tokens`
- `markdown_tokens.kept`
- `markdown_tokens.total`
- `markdown_tokens.truncated`
- `billing`
- `billing.raw_bytes`
- `billing.factors`
- `billing.factors.name`
- `billing.factors.value`
- `billing.factors.reason`
- `billing.multiplier`
- `billing.billed_bytes`
- `custom`
- `cache`
- `cache_age_s`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.retry_after_seconds`
- `error.failed_at`

### 202 — Scrape accepted for background processing; the body is a receipt, not content. Returned when `async: true` or `?async=true` was requested, or when a browser-path request's hold window expired before the render finished. Poll `GET /v1/scrape/{id}` for the result.
- `schema_version`
- `status`
- `id`
- `url`
- `status_url`
- `queued_at`
- `events`

### 400 — Bad request. When the `url` field fails strict validation the error
code is `invalid_url` and the `details` object contains:
- `field` — the request field that failed (e.g. `"url"`)
- `url` — the rejected value
- `reason` — one of: `scheme_missing`, `scheme_unsupported`,
  `scheme_separator_typo`, `host_empty`, `host_invalid`,
  `host_blocked_private`, `host_blocked_loopback`, `host_too_long`,
  `host_unicode_not_punycode`, `userinfo_present`, `url_too_long`,
  `control_chars`, `not_a_url`
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 401 — Unauthorized. The request carried no valid credential, and no work was
performed. The unified `Error` envelope carries one of the codes below
in `error.code`; match on the code, never on the prose.

- `invalid_api_key` — a bearer API key was presented and rejected:
  unknown, revoked, expired or malformed. The bearer-only operations
  (`/v1/webhook-deliveries*` among them) also return it when no
  `Authorization` header is sent. `GET /v1/account` is the exception
  to "malformed": it returns `invalid_api_key` only for a key that
  starts `sk_test_` or `sk_live_` and is unknown, revoked or expired.
- `unauthorized` — the `Authorization` header is absent or malformed on
  an operation that also accepts a dashboard session (the dual-auth and
  billing operations). `GET /v1/account` also returns it, for no
  `Authorization` header and for any bearer value that does not start
  `sk_test_` or `sk_live_` (that value is checked as a dashboard
  session token).

Not retryable as-is: supply a valid API key first.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 402 — Payment required. The account cannot fund this request, and no work was
performed. The unified `Error` envelope carries one of the billing codes
below in `error.code`; match on the code, never on the prose.

- `quota_exceeded` — the enforced balance for this request's meter
  cannot fund it. Capture endpoints (`/v1/scrape`, `/v1/fetch`,
  `/v1/screenshots`, `/v1/sitemap` and the bulk siblings) meter
  bandwidth: they reject when the balance cannot fund the request's
  byte reservation, and `error.details.required_bytes` and
  `error.details.available_bytes` report the bytes the request reserves
  and the bytes available. `/v1/search` meters credits.
- `spending_cap_reached` — the request would push overage spend past the
  account's configured cap. Only reachable on a custom overage-enabled
  plan; standard plans stop at `quota_exceeded`.
- `billing_entitlement_impaired` — the paid subscription is in a payment
  impairment (e.g. `past_due`). `error.message` names the reason.
- `billing_entitlement_expired` — the paid subscription entitlement
  period has ended, or its dunning grace period has run out.

Not retryable as-is: settle the balance, raise the cap, or restore the
subscription first. This response carries no `X-Quota-*` headers — they
travel with the 2xx responses of the operations that emit them.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 422 — Validation error. A `json` format entry whose `schema` the extractor cannot take is
refused with `schema_incompatible`, before anything is drawn.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 429 — Rate limit (`rate_limited`) or concurrency limit (`concurrency_limit`) exceeded. Carries a `Retry-After` header — seconds derived from the same clock as `X-RateLimit-Reset` for the rate-limit variant, a fixed small back-off for the concurrency variant.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 500 — Internal server error.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 503 — The request carried a `json` format entry and JSON extraction is unavailable on this
deployment (`provider_unavailable`). It is refused before anything is drawn. A
terminal upstream failure is never a `503`: it is a `200` result, described above.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`