# Scrape

Fetch a URL as raw HTML, clean markdown, or structured content. Accessed via `client.scrape`.

## create

```ts
client.scrape.create(opts: ScrapeOpts, requestOptions?: RequestOptions): Promise<ScrapeCreateResponse>
```

POST /v1/scrape — create a scrape job.

Resolves to the 200 `success` content envelope or the 202 queued receipt;
narrow on `status` to tell them apart. A TERMINAL FAILURE THROWS: the server
delivers it as a `failed` envelope on HTTP 200 (a definitive upstream outcome
is a result, not a transport error), and this method converts it into the
`ByteKitError` you already catch — `err.code`, `err.message`, and `err.status`
200. `scrape.get(id)` is unchanged and still returns `failed` as data.

See `ScrapeCreateResponse` for the full precedence: `async: true` receipts
immediately, a browser-path request without it is held open until the render
reaches a terminal state (200) or the server's hold window expires (202 — a
receipt you did not ask for), and everything else takes the fast path (200).

A held request therefore occupies the connection for as long as the render
takes, up to the server's hold window (28 s by default) — well inside this
SDK's 120 s default `timeoutMs`, but worth raising `requestTimeoutMs` if you
lowered it.

`requestOptions` carries the CLIENT-side abort budget (`requestTimeoutMs`) and an
optional caller `signal`; both override the constructor's `timeoutMs` default. They
are a different axis from `opts.timeout_ms`, which is the SERVER-side scrape budget.

## get

```ts
client.scrape.get(id: ScrapeId, requestOptions?: RequestOptions): Promise<ScrapeGetResponse>
```

GET /v1/scrape/\{id\} — poll a scrape by ID

## Options (`ScrapeOpts`)

- `url` (required) — `string`
- `formats` — `Array<'raw' | 'markdown' | 'links' | 'images' | ScrapeJsonFormat>` — String formats, plus at most one `\{ type: 'json', schema \}` entry for structured extraction (the entry's result is `formats.json`).
- `country` — `string`
- `delay_ms` — `number`
- `timeout_ms` — `number` — SERVER-side scrape budget in ms — how long the server may spend on the scrape itself. For the CLIENT-side abort budget (when this SDK gives up on the wire), see `RequestOptions.requestTimeoutMs`.
- `async` — `boolean`
- `webhook_url` — `string`
- `events` — `Array<'queued' | 'completed' | 'failed'>`
- `markdown_mode` — `MarkdownMode` — Markdown processing mode. article=article extraction (default), raw=minimal cleanup, llm=compact LLM-optimised output.
- `markdown_query` — `string` — BM25 query string for relevance-ranked content filtering. Omit or leave empty to disable.
- `markdown_links` — `MarkdownLinks` — Link rendering style in the markdown output.
- `markdown_images` — `MarkdownImages` — Image retention mode in the markdown output. Distinct from `markdown_filter_images` (which drops low-signal images before conversion).
- `with_links_summary` — `boolean` — When true, appends a `Links:` footer to the markdown output. Automatically enabled when `markdown_links` is `'references'`.
- `with_images_summary` — `boolean` — When true, appends an `Images:` footer to the markdown output.
- `markdown_compact` — `boolean` — Collapse excessive whitespace for a more compact output.
- `markdown_filter_images` — `boolean` — Filter low-signal images from the markdown output.
- `markdown_include_media` — `boolean` — When true, `formats.links` and `formats.images` return `ScrapeScoredLink[]` / `ScrapeScoredImage[]` (rich objects) instead of `string[]`, and a top-level `tables` array is included. Only effective when `markdown` is in `formats`.
- `markdown_include_warnings` — `boolean` — When true, the response includes markdown-pipeline and tag-filter warnings. `artifact_unavailable` may appear independently when a requested persisted format cannot be rehydrated.
- `markdown_include_stats` — `boolean` — When true, the response includes a top-level `stats` object with `ScrapeStats` (chars, tokens, blocks). Only effective when `markdown` is in `formats`.
- `token_budget` — `number` — Maximum token count for the markdown output. Truncates at the last paragraph → sentence → token boundary under the limit. Absent → no truncation.
- `token_encoding` — `TokenEncoding` — Tokenizer encoding hint for `token_budget`. `cl100k_base` (default) matches GPT-4/4o; `o200k_base` matches GPT-4o-mini / o-series.
- `solve_challenge` — `boolean` — Attempts to clear a Cloudflare challenge; success is not guaranteed. Billed as a normal browser render.
- `mobile` — `boolean` — When true, emulate a mobile device (mobile UA + viewport). Default false (desktop). Part of the cache key, so mobile and desktop responses cache independently.
- `remove_base64_images` — `boolean` — When true (default), base64-encoded `data:` image sources are stripped before markdown conversion. Set false to preserve them. Does not affect `raw`.
- `include_tags` — `string[]` — HTML element names (not CSS selectors) to keep. When non-empty, only content within these elements is included in `markdown` output. Applied before `exclude_tags`. Does not affect `raw`.
- `exclude_tags` — `string[]` — HTML element names (not CSS selectors) to strip from `markdown` output. Applied after `include_tags`. Does not affect `raw`.
- `clean_markdown` — `boolean` — When true and `markdown` is in `formats`, the converted markdown is sent to Groq for cleaning before return. Graceful degradation on any Groq failure. Billing: 3× credits and bytes applied only when cleaning succeeds.
- `cache_ttl` — `string | 0` — How long a freshly fetched URL may be served from cache. `'0'`/`0` skips the
cache read (the result is still written), `'Nh'`/`'Nd'` set a TTL (capped at
168h / 7d). Default `'48h'`. Honoured on both the synchronous and asynchronous
paths; the async success envelope reports the outcome via the `cache` field.
- `custom` — `Record<string, unknown>` — User-supplied JSON payload, echoed back on the success envelope so callers
can correlate the response to caller-side state (job IDs, batch metadata).
Capped at 4096 UTF-8 bytes after JSON serialization. Does NOT affect
cache-key inputs — two requests differing only in `custom` share the same
cache slot.