Scrape a URL

Fetches and processes a URL, returning content in one or more formats wrapped in a ScrapeEnvelope. Most requests return synchronously in under a second; requests needing a full page render return HTTP 202 and complete asynchronously.

Authorization

AuthorizationRequiredBearer <token>

API key. Manage keys from the dashboard at app.bytekit.com.

In: header

Request Body

application/jsonRequired

urlRequiredstring

Format: "uri"

formatsarray<string>

Default: ["raw_html"]

countrystring

Default: "US"Pattern: "^[A-Z]{2}$"

cookiesarray<object>

Default: []

headersobject

Default: {}

delay_msinteger

Default: 0Minimum: 0Maximum: 10000

timeout_msinteger

Default: 30000Minimum: 1000Maximum: 60000

asyncboolean

Default: false

webhook_urlstring

Webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported)

Pattern: "^https://"Format: "uri"

eventsarray<string>

Default: ["completed","failed"]

markdown_modestring

Markdown processing mode. article=article extraction (default), raw=minimal cleanup, llm=compact LLM-optimised output.

Value in: "article" | "raw" | "llm"

markdown_querystring

BM25 query string for relevance-ranked content filtering. Omit or leave empty to disable.

Maximum length: 200

markdown_linksstring

Link rendering style in the markdown output. 'inline' (default) keeps [text](url) as-is. 'text' strips URLs, keeping only anchor text. 'none' strips URLs and removes reference definitions. 'references' converts to numbered [text][N] refs with a Links footer.

Value in: "inline" | "references" | "none" | "text"

markdown_imagesstring

Image retention mode in the markdown output. 'inline' (default) keeps ![alt](url) as-is. 'text' strips the URL, keeping only alt text. 'none' removes images entirely. 'references' converts to numbered ![alt][N] refs with an Images footer. Distinct from markdown_filter_images (which drops low-signal images before conversion).

Value in: "inline" | "references" | "none" | "text"

with_links_summaryboolean

When true, appends a 'Links:\n[1]: url\n...' footer to the markdown output, built from the extracted link list. Automatically enabled when markdown_links is 'references'.

with_images_summaryboolean

When true, appends an 'Images:\n[1]: url\n...' footer to the markdown output, built from the extracted image list.

markdown_compactboolean

Collapse excessive whitespace for a more compact output.

markdown_filter_imagesboolean

Filter low-signal images from the markdown output.

markdown_include_mediaboolean

When true, formats.links and formats.images return ScrapeScoredLink[] / ScrapeScoredImage[] (rich objects) instead of string[], and a top-level tables array is included. Only effective when markdown is in formats.

markdown_include_warningsboolean

When true, enables markdown-pipeline and tag-filter warnings for affected html and markdown requests. Artifact availability warnings such as artifact_unavailable may appear independently of this flag.

markdown_include_statsboolean

When true, the response includes a top-level stats object with ScrapeStats (chars, tokens, blocks). Only effective when markdown is in formats.

token_budgetinteger

Maximum token count for the markdown output. When set and markdown is in formats, the output is truncated at the last paragraph → sentence → token boundary that keeps the estimated count under this limit. Absent → no truncation. Triggers a markdown_tokens field in the response.

Minimum: 0

token_encodingstring

Tokenizer encoding hint for token_budget. cl100k_base (default) matches GPT-4/4o; o200k_base matches GPT-4o-mini / o-series. Both use the same 4-char/token heuristic internally.

Default: "cl100k_base"Value in: "cl100k_base" | "o200k_base"

mobileboolean

When true, emulate a mobile device — the request uses a mobile (Android) user agent and a mobile viewport, so the upstream serves the mobile version of the page. Default false (desktop). The value is part of the cache key, so mobile and desktop responses are cached independently.

Default: false

remove_base64_imagesboolean

When true (default), base64-encoded data: image sources are stripped from the HTML before markdown conversion and from the html format output. Set to false to preserve base64 images in the pipeline output. Does not affect raw_html, which always returns the original HTML unchanged.

Default: true

include_tagsarray<string>

HTML element names (not CSS selectors) to keep. When non-empty, only content within these elements is included in the html and markdown output. Applied before exclude_tags. Does not affect raw_html, which always returns the full page.

Default: []

exclude_tagsarray<string>

HTML element names (not CSS selectors) to strip from the html and markdown output. Applied after include_tags when both are present. Does not affect raw_html, which always returns the full page.

Default: []

cache_ttlstring | integer

How long a freshly fetched URL may be served from cache. 0 (string or integer) skips the cache read; cache-eligible results are still written. Nh/Nd set a TTL (capped at 168 h / 7 d). Safety-classified WAF husks are returned live but never stored in the shared cache. Honoured on both the synchronous and asynchronous paths; the async success envelope reports the lookup outcome via the cache field.

Default: "48h"

customobject

User-supplied JSON payload, echoed back on the success envelope (sync, async, webhook). Capped at 4096 UTF-8 bytes (Buffer.byteLength). Distinct from the system-owned metadata column. Does NOT affect cache-key inputs — two requests differing only in custom resolve to the same cache entry.

Default: {}

clean_markdownboolean

When true and markdown is in formats, the converted markdown is sent to Groq LLM for cleaning (boilerplate removal, ad stripping, formatting improvement) before being returned. Graceful degradation: any Groq failure (key unset, 4xx/5xx, timeout, empty response) returns the raw markdown unchanged plus a warning in the warnings array. Billing: 3× credits and bytes applied only when cleaning succeeds.

Default: false

x_internal_scrape_pathstring

Internal-only: forces the request onto the worker enqueue branch and the lean-forced code path in scrape-worker. Requires X-Internal-Probe-Token header. Not exposed via SDK generators.

Value in: "lean-forced"

curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["raw_html","markdown"],"markdown_mode":"article","markdown_links":"inline","remove_base64_images":true,"include_tags":["article","main"],"exclude_tags":["nav","footer"]}'

Scrape completed synchronously.

{
  "schema_version": 1,
  "status": "success",
  "id": "sc_01j9abc123",
  "url": "https://example.com",
  "final_url": "https://example.com/",
  "content_type": "text/html; charset=UTF-8",
  "content_length": 1256,
  "status_code": 200,
  "retrieved_at": "2025-01-15T12:00:00Z",
  "formats": {
    "raw_html": "<html><body><h1>Hello World</h1><p>Example page with a <a href=\"/about\">link</a> and an image.</p></body></html>",
    "html": "<html><body><h1>Hello World</h1><p>Example page with a <a href=\"/about\">link</a> and an image.</p></body></html>",
    "markdown": "# Hello World\n\nExample page with a [link](/about) and an image.\n\n![Alt text](https://example.com/photo.png)",
    "links": [
      {
        "url": "https://example.com/about",
        "text": "link",
        "isInternal": true,
        "rel": null
      }
    ],
    "images": [
      {
        "url": "https://example.com/photo.png",
        "alt": "Alt text",
        "width": 800,
        "height": 600,
        "format": "png",
        "score": 0.95
      }
    ]
  },
  "metadata": {
    "title": "Example Domain",
    "description": "An example page for demonstration",
    "language": "en",
    "canonicalUrl": "https://example.com",
    "ogTitle": "Example Domain",
    "ogDescription": null,
    "ogImage": null,
    "byline": "Jane Doe",
    "publishedAt": "2025-01-15T09:00:00Z"
  },
  "tables": [
    {
      "kind": "data",
      "gfm": "| Col1 | Col2 |\n|------|------|\n| A | B |",
      "rows": 2,
      "cols": 2
    }
  ],
  "warnings": [
    {
      "code": "lossy_table",
      "message": "Table at index 3 was simplified",
      "element": "<table>...</table>"
    }
  ],
  "stats": {
    "chars": 1256,
    "tokens": 312,
    "blocks": 4
  }
}

Scrape a URL

Authorization

Request Body

Response