Discover URLs on a domain

Crawls a domain to build a sitemap. Returns cached results if available within cache_ttl. Results delivered via webhook and downloadable from results_url. Webhook deliveries include `X-ByteKit-Event: sitemap.completed` or `X-ByteKit-Event: sitemap.failed` so receivers can dispatch on the header without parsing the body shape.

Authorization

AuthorizationRequiredBearer <token>

API key. Manage keys from the dashboard at app.bytekit.com.

In: header

Request Body

application/jsonRequired

urlRequiredstring

Format: "uri"

strategystring

Default: "standard"Value in: "quick" | "standard" | "deep"

max_depthinteger

Default: 2Minimum: 1Maximum: 5

max_urlsinteger

Default: 5000Minimum: 1Maximum: 100000

processobject | null

Default: null

webhook_urlstring

Optional webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported). When omitted, no sitemap.completed/sitemap.failed webhook is delivered and the job is poll-only via GET /v1/sitemap/{id}. When present, delivery is unchanged. webhook_secret is ignored when webhook_url is absent.

Pattern: "^https://"Format: "uri"

webhook_secretstring | null

Default: null

cache_ttlstring

Max 90 days.

Default: "30d"Pattern: "^(0|\\d+[dh])$"

compactboolean

Default: false

metadataobject

Arbitrary key-value metadata. Max 4096 bytes when JSON-serialized.

curl -X POST https://api.bytekit.com/v1/sitemap \
  -H "Authorization: Bearer sk_live_your_api_key_here"

Sitemap job accepted (or cached result returned).

{
  "id": "sm_01j9abc123",
  "url": "string",
  "strategy": "string",
  "max_depth": 0,
  "max_urls": 0,
  "status": "pending",
  "total_urls": 0,
  "sources": {},
  "total_bytes": 0,
  "results_url": "string",
  "cache": "miss",
  "cache_age_s": 0,
  "warnings": [
    "string"
  ],
  "error_code": "string",
  "error_message": "string",
  "compact": true,
  "process": {},
  "metadata": {},
  "created_at": "2019-08-24T14:15:22Z",
  "completed_at": "2019-08-24T14:15:22Z",
  "expires_at": "2019-08-24T14:15:22Z"
}

Discover URLs on a domain

Authorization

Request Body

Response