Scrape a URL

Fetches and processes a URL, returning content in one or more formats wrapped in a ScrapeEnvelope.

The response shape is decided by your request, not by how we route it. In precedence order:

async: true in the body, or ?async=true in the query string (either one is enough; false in one never cancels true in the other) → 202 with a ScrapeQueuedEnvelope receipt, returned immediately. Poll GET /v1/scrape/{id} for the result.
Otherwise, a request that needs a full page render — today that is delay_ms > 0 — is held open until the render reaches a terminal state, then returns 200 with the content envelope (success or failed). If the render is still running when the hold window expires (28 seconds by default), the request returns the 202 receipt instead and the job continues in the background.
Otherwise the request is served by the fast HTTP path and returns 200 with the content envelope, typically in under a second.

A failed fetch is a result, not a transport error. An unreachable host, a refused connection, or a target whose bot protection defeats every proxy tier all come back on the same 200 as a failed envelope — never a 5xx. Narrow on the envelope's status before reading the body. A failure the fast path decides inline is recorded as a failed job, so its envelope normally carries that job's id and GET /v1/scrape/{id} returns the same envelope; if recording the failed job itself fails, the envelope still arrives on the same 200 with id set to null and there is nothing to poll. A failure a held render reaches carries that render's real id. Either way the request transfers no billable bytes and is charged nothing.

A target that rate-limits us is not ByteKit rate-limiting you. The failed envelope's error.code is target_rate_limited when the target rate-limits the request. On the fast path that means our proxy attempts end on an HTTP 429 and no bot protection is identified on any attempt, even if the browser fallback then fails or reaches only an unidentified block page; the inline failure carries error.retry_after_seconds when the target sent a usable Retry-After. On a held render the browser's own answer decides: it observes a 429 on the page itself, or it fails outright after the proxy attempts ended on an HTTP 429. A bot-protection verdict on an earlier fetch attempt does not keep a held render blocked when the browser then observes the 429. A held render never carries error.retry_after_seconds. Retrying a target_rate_limited failure later is likely to work. A held render whose browser reaches an unidentified 403 or 503 page after a proxy 429 is blocked. A 429 that bot protection sends as a block stays blocked when it is the answer that decides the code: the fast path's proxy attempts, or the browser render on a held render. Your own per-key limit is a separate case: it answers HTTP 429 rate_limited with a Retry-After header (see the 429 response).

So a 202 means one of exactly two things: you asked for one with async: true or ?async=true, or the hold window expired. wait_for_selector and wait_until are ScreenshotRequest options, not ScrapeRequest options — sending either to this endpoint returns 422 validation_error, so neither can produce a 202 here.

For AI agents: https://bytekit.com/llms.txt — every docs page is available as markdown by appending .md

Authorization

AuthorizationRequiredBearer <token>

API key. Manage keys from the dashboard at app.bytekit.com.

In: header

Request Body

application/jsonRequired

urlRequiredstring

Format: "uri"

formatsarray<string | object>

What to return. Each entry is either a string format (raw, markdown, links or images) or a json object entry that extracts structured data from the page. The string "json" is not a format value; the json entry is an object, described by ScrapeRequestJsonFormat. A list may hold at most one json entry, and a list holding only the json entry still captures raw. The json entry is accepted on POST /v1/scrape, POST /v1/scrape/bulk and POST /v1/bulk (scrape entries); monitor formats accept the string formats alone. Its result is formats.json on the response, and its charge is described on ScrapeRequestJsonFormat.

Default: ["raw"]

countrystring

Default: "US"Pattern: "^[A-Z]{2}$"

delay_msinteger

Milliseconds to wait after page load before capturing, for content that renders late. Any value above 0 puts the request on the full browser render path. Unless async: true is also set, the request is then HELD until the render finishes and returns 200 with the content envelope, or returns the 202 receipt if the hold window expires first. 0 (the default) keeps the request on the fast HTTP path, which returns 200 in one round trip.

Default: 0Minimum: 0Maximum: 10000

timeout_msinteger

Default: 60000Minimum: 1000Maximum: 90000

asyncboolean

When true, the request returns 202 with a ScrapeQueuedEnvelope receipt immediately and the scrape runs in the background; poll GET /v1/scrape/{id} for the result. The ?async=true query parameter is a second spelling of this field: the request is async when either one says true, and false in one never cancels true in the other. Asking for a receipt this way outranks every routing consequence: when neither is set (the default) a browser-path request is held to completion instead of being receipted. A 202 can still arrive without either, but only when the hold window expires.

Default: false

webhook_urlstring

Webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported)

Pattern: "^https://"Format: "uri"

eventsarray<string>

Webhook event types to deliver when webhook_url is set. Defaults to all three (queued, completed, failed); send an explicit subset to receive fewer.

Default: ["queued","completed","failed"]

markdown_modestring

Markdown processing mode. article=article extraction (default), raw=minimal cleanup, llm=compact LLM-optimised output.

Value in: "article" | "raw" | "llm"

markdown_querystring

BM25 query string for relevance-ranked content filtering. Omit or leave empty to disable.

Maximum length: 200

markdown_linksstring

Link rendering style in the markdown output. 'inline' (default) keeps [text](url) as-is. 'text' strips URLs, keeping only anchor text. 'none' strips URLs and removes reference definitions. 'references' converts to numbered [text][N] refs with a Links footer.

Value in: "inline" | "references" | "none" | "text"

markdown_imagesstring

Image retention mode in the markdown output. 'inline' (default) keeps ![alt](url) as-is. 'text' strips the URL, keeping only alt text. 'none' removes images entirely. 'references' converts to numbered ![alt][N] refs with an Images footer. Distinct from markdown_filter_images (which drops low-signal images before conversion).

Value in: "inline" | "references" | "none" | "text"

with_links_summaryboolean

When true, appends a 'Links:\n[1]: url\n...' footer to the markdown output, built from the extracted link list. Automatically enabled when markdown_links is 'references'.

with_images_summaryboolean

When true, appends an 'Images:\n[1]: url\n...' footer to the markdown output, built from the extracted image list.

markdown_compactboolean

Collapse excessive whitespace for a more compact output.

markdown_filter_imagesboolean

Filter low-signal images from the markdown output.

markdown_include_mediaboolean

When true, formats.links and formats.images return ScrapeScoredLink[] / ScrapeScoredImage[] (rich objects) instead of string[], and a top-level tables array is included. Only effective when markdown is in formats.

markdown_include_warningsboolean

When true, enables markdown-pipeline and tag-filter warnings for requests that include the markdown format. Artifact availability warnings such as artifact_unavailable may appear independently of this flag.

markdown_include_statsboolean

When true, the response includes a top-level stats object with ScrapeStats (chars, tokens, blocks). Only effective when markdown is in formats.

token_budgetinteger

Maximum token count for the markdown output. When set and markdown is in formats, the output is truncated at the last paragraph → sentence → token boundary that keeps the estimated count under this limit. Absent → no truncation. Triggers a markdown_tokens field in the response.

Minimum: 0

token_encodingstring

Tokenizer encoding hint for token_budget. cl100k_base (default) matches GPT-4/4o; o200k_base matches GPT-4o-mini / o-series. Both use the same 4-char/token heuristic internally.

Default: "cl100k_base"Value in: "cl100k_base" | "o200k_base"

mobileboolean

When true, emulate a mobile device — the request uses a mobile (Android) user agent and a mobile viewport, so the upstream serves the mobile version of the page. Default false (desktop). The value is part of the cache key, so mobile and desktop responses are cached independently.

Default: false

solve_challengeboolean

Attempts to clear a Cloudflare challenge; success is not guaranteed. Billed as a normal browser render.

Default: false

remove_base64_imagesboolean

When true (default), base64-encoded data: image sources are stripped from the HTML before markdown conversion. Set to false to preserve base64 images in the pipeline output. Does not affect raw, which always returns the original source unchanged.

Default: true

include_tagsarray<string>

HTML element names (not CSS selectors) to keep. When non-empty, only content within these elements is included in the markdown output. Applied before exclude_tags. Does not affect raw, which always returns the full page.

Default: []

exclude_tagsarray<string>

HTML element names (not CSS selectors) to strip from the markdown output. Applied after include_tags when both are present. Does not affect raw, which always returns the full page.

Default: []

cache_ttlstring | integer

How long a freshly fetched URL may be served from cache. 0 (string or integer) skips the cache read; cache-eligible results are still written. Nh/Nd set a TTL (capped at 168 h / 7 d). Safety-classified WAF husks are returned live but never stored in the shared cache. Honoured on both the synchronous and asynchronous paths; the async success envelope reports the lookup outcome via the cache field.

Default: "48h"

customobject

User-supplied JSON payload, echoed back on the success envelope (sync, async, webhook). Capped at 4096 UTF-8 bytes (Buffer.byteLength). Distinct from the system-owned metadata column. Does NOT affect cache-key inputs — two requests differing only in custom resolve to the same cache entry.

Default: {}

clean_markdownboolean

When true and markdown is in formats, the converted markdown is sent to Groq LLM for cleaning (boilerplate removal, ad stripping, formatting improvement) before being returned. Graceful degradation: any Groq failure (key unset, 4xx/5xx, timeout, empty response) returns the raw markdown unchanged plus a warning in the warnings array. Billing: 3× credits and bytes applied only when cleaning succeeds.

Default: false

Query Parameters

asyncboolean

Query-string spelling of the body async field. true returns the 202 receipt immediately; any other value is ignored. Either spelling turns async on, and false in one never cancels true in the other.

Default: false

curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "formats": [
    "raw",
    "markdown"
  ],
  "markdown_mode": "article",
  "markdown_links": "inline",
  "remove_base64_images": true,
  "include_tags": [
    "article",
    "main"
  ],
  "exclude_tags": [
    "nav",
    "footer"
  ]
}'

Scrape reached a terminal state and the body is that terminal envelope: success carrying content, or failed carrying the error envelope. Returned by the fast HTTP path in one round trip, and by a browser-path request without async: true or ?async=true once its render reaches a terminal state inside the hold window. A held render that ends in failure answers 200 with the same failed envelope GET /v1/scrape/{id} returns — narrow on status before reading the body. A definitive upstream outcome is a result, not a transport error: an unreachable host, a refused connection, or a target whose bot protection defeats every proxy tier all answer 200 with the failed envelope, never a 5xx. An inline failure is recorded as a failed job, so its envelope normally carries that job's id and polling it returns the same envelope; id is null only when recording the failed job itself failed. error.http_status is present only when a real upstream HTTP status is known.

The fast path exhausted every proxy tier without a usable response (DNS failure, refused connection, or terminal bot-protection block, which reports blocked). The failure is recorded as a failed job, so id names it and GET /v1/scrape/{id} returns this same envelope. error.http_status is omitted because no upstream HTTP status was received. The request transfers no billable bytes and is charged nothing.

{
  "schema_version": 1,
  "status": "failed",
  "id": "sc_01j9abc123",
  "error": {
    "code": "upstream_error",
    "message": "Failed to fetch URL.",
    "failed_at": "2025-01-15T12:00:00Z"
  }
}

Scrape

Scrape API — endpoints, requests, and responses.

GET /v1/scrape/{id}Get scrape result

Always returns 200. Check the `status` discriminator for the scrape state.

Scrape a URL

Authorization

Request Body

Query Parameters

Terminal upstream fetch failure, delivered inline

The target site rate-limited the request, delivered inline