Scrape

Fetch a URL as raw HTML, clean markdown, or structured content. Accessed via client.scrape.

create

client.scrape.create(opts: ScrapeOpts, requestOptions?: RequestOptions): Promise<ScrapeCreateResponse>

POST /v1/scrape — create a scrape job.

Resolves to the 200 success content envelope or the 202 queued receipt; narrow on status to tell them apart. A TERMINAL FAILURE THROWS: the server delivers it as a failed envelope on HTTP 200 (a definitive upstream outcome is a result, not a transport error), and this method converts it into the ByteKitError you already catch — err.code, err.message, and err.status 200. scrape.get(id) is unchanged and still returns failed as data.

See ScrapeCreateResponse for the full precedence: async: true receipts immediately, a browser-path request without it is held open until the render reaches a terminal state (200) or the server's hold window expires (202 — a receipt you did not ask for), and everything else takes the fast path (200).

A held request therefore occupies the connection for as long as the render takes, up to the server's hold window (28 s by default) — well inside this SDK's 120 s default timeoutMs, but worth raising requestTimeoutMs if you lowered it.

requestOptions carries the CLIENT-side abort budget (requestTimeoutMs) and an optional caller signal; both override the constructor's timeoutMs default. They are a different axis from opts.timeout_ms, which is the SERVER-side scrape budget.

get

client.scrape.get(id: ScrapeId, requestOptions?: RequestOptions): Promise<ScrapeGetResponse>

GET /v1/scrape/{id} — poll a scrape by ID

Options (ScrapeOpts)

• url (required) — string
• formats — Array<'raw' | 'markdown' | 'links' | 'images' | ScrapeJsonFormat> — String formats, plus at most one \{ type: 'json', schema \} entry for structured extraction (the entry's result is formats.json).
• country — string
• delay_ms — number
• timeout_ms — number — SERVER-side scrape budget in ms — how long the server may spend on the scrape itself. For the CLIENT-side abort budget (when this SDK gives up on the wire), see RequestOptions.requestTimeoutMs.
• async — boolean
• webhook_url — string
• events — Array<'queued' | 'completed' | 'failed'>
• markdown_mode — MarkdownMode — Markdown processing mode. article=article extraction (default), raw=minimal cleanup, llm=compact LLM-optimised output.
• markdown_query — string — BM25 query string for relevance-ranked content filtering. Omit or leave empty to disable.
• markdown_links — MarkdownLinks — Link rendering style in the markdown output.
• markdown_images — MarkdownImages — Image retention mode in the markdown output. Distinct from markdown_filter_images (which drops low-signal images before conversion).
• with_links_summary — boolean — When true, appends a Links: footer to the markdown output. Automatically enabled when markdown_links is 'references'.
• with_images_summary — boolean — When true, appends an Images: footer to the markdown output.
• markdown_compact — boolean — Collapse excessive whitespace for a more compact output.
• markdown_filter_images — boolean — Filter low-signal images from the markdown output.
• markdown_include_media — boolean — When true, formats.links and formats.images return ScrapeScoredLink[] / ScrapeScoredImage[] (rich objects) instead of string[], and a top-level tables array is included. Only effective when markdown is in formats.
• markdown_include_warnings — boolean — When true, the response includes markdown-pipeline and tag-filter warnings. artifact_unavailable may appear independently when a requested persisted format cannot be rehydrated.
• markdown_include_stats — boolean — When true, the response includes a top-level stats object with ScrapeStats (chars, tokens, blocks). Only effective when markdown is in formats.
• token_budget — number — Maximum token count for the markdown output. Truncates at the last paragraph → sentence → token boundary under the limit. Absent → no truncation.
• token_encoding — TokenEncoding — Tokenizer encoding hint for token_budget. cl100k_base (default) matches GPT-4/4o; o200k_base matches GPT-4o-mini / o-series.
• solve_challenge — boolean — Attempts to clear a Cloudflare challenge; success is not guaranteed. Billed as a normal browser render.
• mobile — boolean — When true, emulate a mobile device (mobile UA + viewport). Default false (desktop). Part of the cache key, so mobile and desktop responses cache independently.
• remove_base64_images — boolean — When true (default), base64-encoded data: image sources are stripped before markdown conversion. Set false to preserve them. Does not affect raw.
• include_tags — string[] — HTML element names (not CSS selectors) to keep. When non-empty, only content within these elements is included in markdown output. Applied before exclude_tags. Does not affect raw.
• exclude_tags — string[] — HTML element names (not CSS selectors) to strip from markdown output. Applied after include_tags. Does not affect raw.
• clean_markdown — boolean — When true and markdown is in formats, the converted markdown is sent to Groq for cleaning before return. Graceful degradation on any Groq failure. Billing: 3× credits and bytes applied only when cleaning succeeds.
• cache_ttl — string | 0 — How long a freshly fetched URL may be served from cache. '0'/0 skips the cache read (the result is still written), 'Nh'/'Nd' set a TTL (capped at 168h / 7d). Default '48h'. Honoured on both the synchronous and asynchronous paths; the async success envelope reports the outcome via the cache field.
• custom — Record<string, unknown> — User-supplied JSON payload, echoed back on the success envelope so callers can correlate the response to caller-side state (job IDs, batch metadata). Capped at 4096 UTF-8 bytes after JSON serialization. Does NOT affect cache-key inputs — two requests differing only in custom share the same cache slot.