API ReferenceSitemap

Discover URLs on a domain

Crawls a domain to build a sitemap. Returns cached results if available within cache_ttl. Results delivered via webhook and downloadable from results_url. Webhook deliveries include X-ByteKit-Event: sitemap.completed or X-ByteKit-Event: sitemap.failed so receivers can dispatch on the header without parsing the body shape.

The webhook body is { event, data }. For a job the worker ran, data is not a SitemapResponse:

  • sitemap.completed data carries id, url, strategy, status, total_urls, sources, total_bytes, results_url, warnings, metadata, compact and billing.
  • sitemap.failed data carries id, url, strategy, status, error_code, error_message, warnings and metadata.

In both, url is the host as scheme://host, and warnings is the worker's own warnings or null. When the request is answered from a cached job (cache: hit), the sitemap.completed data is instead the full job object, the same fields as a SitemapResponse.

Limits: request bodies are capped at 512 KB. An oversized body returns a 400 with error code request_too_large.

For AI agents: https://bytekit.com/llms.txt — every docs page is available as markdown by appending .md

POST
/v1/sitemap

Authorization

AuthorizationRequiredBearer <token>

API key. Manage keys from the dashboard at app.bytekit.com.

In: header

Request Body

application/jsonRequired
urlRequiredstring

The site to discover. Sitemap discovery is host-scoped (robots.txt and /sitemap.xml live at the host root), so only the scheme and host are used: a path, query string or fragment is accepted but ignored, and the 202 response carries a warnings entry saying so. A bare host, or a host with just a trailing /, produces no warning.

Format: "uri"
strategystring

How much discovery work the job does.

  • quick — reads robots.txt and the XML sitemaps only. Every result has source: sitemap.
  • standard — the quick sources, then fills any remaining max_urls budget with URLs from your account's completed scrapes of the host in the last 90 days (source: known).
  • deep — the standard sources, then crawls page links outward from the URLs found so far, up to max_depth levels, until max_urls is reached (source: links).
Default: "standard"Value in: "quick" | "standard" | "deep"
max_depthinteger
Default: 2Minimum: 1Maximum: 5
max_urlsinteger
Default: 5000Minimum: 1Maximum: 100000
processobject | null

Accepted, stored and echoed back on the job, but it has no effect: the worker logs that the trigger is not implemented and ignores it.

Default: null
webhook_urlstring

Optional webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported). When omitted, no sitemap.completed/sitemap.failed webhook is delivered and the job is poll-only via GET /v1/sitemap/{id}. When present, delivery is unchanged. webhook_secret is ignored when webhook_url is absent.

Pattern: "^https://"Format: "uri"
webhook_secretstring | null

HMAC secret used to sign webhook deliveries. Each delivery carries X-Webhook-Timestamp and X-Webhook-Signature (sha256= + the hex HMAC-SHA256 of timestamp.body). See https://bytekit.com/docs/guides/webhooks for verification code.

Default: null
cache_ttlstring

Max 90 days.

Default: "30d"Pattern: "^(0|\\d+[dh])$"
compactboolean
Default: false
metadataobject

Arbitrary key-value metadata. Max 4096 bytes when JSON-serialized.

curl -X POST https://api.bytekit.com/v1/sitemap \
  -H "Authorization: Bearer sk_live_your_api_key_here"

Sitemap job accepted (or cached result returned).

{
  "id": "sm_01j9abc123",
  "url": "string",
  "strategy": "string",
  "max_depth": 0,
  "max_urls": 0,
  "status": "pending",
  "total_urls": 0,
  "sources": {},
  "total_bytes": 0,
  "results_url": "string",
  "cache": "miss",
  "cache_age_s": 0,
  "billing": {
    "raw_bytes": 1024,
    "factors": [
      {
        "name": "endpoint",
        "value": 1.5,
        "reason": "scrape_md"
      }
    ],
    "multiplier": 0.75,
    "billed_bytes": 768
  },
  "warnings": [
    "string"
  ],
  "error_code": "string",
  "error_message": "string",
  "compact": true,
  "process": {},
  "metadata": {},
  "created_at": "2019-08-24T14:15:22Z",
  "completed_at": "2019-08-24T14:15:22Z",
  "expires_at": "2019-08-24T14:15:22Z"
}

Sitemap

Sitemap API — endpoints, requests, and responses.

GET /v1/sitemap/{id}Get sitemap job status

Next Page