Scrape

Fetch a URL as raw HTML, clean markdown, or structured content. Accessed via client.scrape.

create

client.scrape.create(opts: ScrapeOpts, requestOptions?: RequestOptions): Promise<ScrapeCreateResponse>

POST /v1/scrape — create a scrape job.

Resolves to the 200 success envelope, or the 202 queued envelope when async: true. Narrow on status before reading the result.

requestOptions carries the CLIENT-side abort budget (requestTimeoutMs) and an optional caller signal; both override the constructor's timeoutMs default. They are a different axis from opts.timeout_ms, which is the SERVER-side scrape budget.

get

client.scrape.get(id: ScrapeId, requestOptions?: RequestOptions): Promise<ScrapeGetResponse>

GET /v1/scrape/{id} — poll a scrape by ID

KNOWN DRIFT (#2426): live staging spot-checks (two independent fetches) found the gateway serializes billing_multiplier as a JSON string (e.g. "0.50") on this GET path, while the OpenAPI spec — and this derived ScrapeGetResponse type — declare it number. POST /v1/scrape's success envelope for the same job serializes the identical field as a proper number, so the drift is specific to the GET path (likely a Postgres NUMERIC column round-tripping as a string on this query path). A consumer doing arithmetic on billing_multiplier after get() (e.g. content_length * billing_multiplier) should coerce with Number(...) defensively rather than trust the static type. This is a gateway/worker serialization bug, not a client bug — the fix belongs upstream in the gateway and is out of this ticket's scope.

Options (ScrapeOpts)

• url (required) — string
• formats — Array<'raw_html' | 'html' | 'markdown' | 'links' | 'images'>
• country — string
• cookies — Array<Record<string, unknown>>
• headers — Record<string, string>
• delay_ms — number
• timeout_ms — number — SERVER-side scrape budget in ms — how long the gateway may spend on the scrape itself. For the CLIENT-side abort budget (when this SDK gives up on the wire), see RequestOptions.requestTimeoutMs.
• async — boolean
• webhook_url — string
• events — Array<'queued' | 'completed' | 'failed'>
• markdown_mode — MarkdownMode — Markdown processing mode. article=article extraction (default), raw=minimal cleanup, llm=compact LLM-optimised output.
• markdown_query — string — BM25 query string for relevance-ranked content filtering. Omit or leave empty to disable.
• markdown_links — MarkdownLinks — Link rendering style in the markdown output.
• markdown_compact — boolean — Collapse excessive whitespace for a more compact output.
• markdown_filter_images — boolean — Filter low-signal images from the markdown output.
• markdown_include_media — boolean — When true, formats.links and formats.images return ScrapeScoredLink[] / ScrapeScoredImage[] (rich objects) instead of string[], and a top-level tables array is included. Only effective when markdown is in formats.
• markdown_include_warnings — boolean — When true, the response includes markdown-pipeline and tag-filter warnings. artifact_unavailable may appear independently when a requested persisted format cannot be rehydrated.
• markdown_include_stats — boolean — When true, the response includes a top-level stats object with ScrapeStats (chars, tokens, blocks). Only effective when markdown is in formats.
• cache_ttl — string | 0 — How long a freshly fetched URL may be served from cache. '0'/0 skips the cache read (the result is still written), 'Nh'/'Nd' set a TTL (capped at 168h / 7d). Default '48h'. Honoured on both the synchronous and asynchronous paths; the async success envelope reports the outcome via the cache field.
• custom — Record<string, unknown> — User-supplied JSON payload, echoed back on the success envelope so callers can correlate the response to caller-side state (job IDs, batch metadata). Capped at 4096 UTF-8 bytes after JSON serialization. Does NOT affect cache-key inputs — two requests differing only in custom share the same cache slot.