# Discover URLs on a domain

Crawls a domain to build a sitemap. Returns cached results if available
within cache_ttl. Results delivered via webhook and downloadable from results_url.
Webhook deliveries include `X-ByteKit-Event: sitemap.completed` or
`X-ByteKit-Event: sitemap.failed` so receivers can dispatch on the
header without parsing the body shape.

The webhook body is `{ event, data }`. For a job the worker ran, `data` is not a
`SitemapResponse`:

- `sitemap.completed` `data` carries `id`, `url`, `strategy`, `status`,
  `total_urls`, `sources`, `total_bytes`, `results_url`, `warnings`,
  `metadata`, `compact` and `billing`.
- `sitemap.failed` `data` carries `id`, `url`, `strategy`, `status`,
  `error_code`, `error_message`, `warnings` and `metadata`.

In both, `url` is the host as `scheme://host`, and `warnings` is the
worker's own warnings or `null`. When the request is answered from a cached job
(`cache: hit`), the `sitemap.completed` `data` is instead the full job object, the
same fields as a `SitemapResponse`.

**Limits:** request bodies are capped at 512 KB. An oversized body
returns a `400` with error code `request_too_large`.

**POST** `/v1/sitemap`

## Request Body

- `url` (required)
- `strategy`
- `max_depth`
- `max_urls`
- `process`
- `process.action`
- `process.options`
- `webhook_url`
- `webhook_secret`
- `cache_ttl`
- `compact`
- `metadata`

## Responses

### 202 — Sitemap job accepted (or cached result returned).
- `id`
- `url`
- `strategy`
- `max_depth`
- `max_urls`
- `status`
- `total_urls`
- `sources`
- `total_bytes`
- `results_url`
- `cache`
- `cache_age_s`
- `billing`
- `billing.raw_bytes`
- `billing.factors`
- `billing.factors.name`
- `billing.factors.value`
- `billing.factors.reason`
- `billing.multiplier`
- `billing.billed_bytes`
- `warnings`
- `error_code`
- `error_message`
- `compact`
- `process`
- `metadata`
- `created_at`
- `completed_at`
- `expires_at`

### 400 — Bad request. When the `url` field fails strict validation the error
code is `invalid_url` and the `details` object contains:
- `field` — the request field that failed (e.g. `"url"`)
- `url` — the rejected value
- `reason` — one of: `scheme_missing`, `scheme_unsupported`,
  `scheme_separator_typo`, `host_empty`, `host_invalid`,
  `host_blocked_private`, `host_blocked_loopback`, `host_too_long`,
  `host_unicode_not_punycode`, `userinfo_present`, `url_too_long`,
  `control_chars`, `not_a_url`
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 401 — Unauthorized. The request carried no valid credential, and no work was
performed. The unified `Error` envelope carries one of the codes below
in `error.code`; match on the code, never on the prose.

- `invalid_api_key` — a bearer API key was presented and rejected:
  unknown, revoked, expired or malformed. The bearer-only operations
  (`/v1/webhook-deliveries*` among them) also return it when no
  `Authorization` header is sent. `GET /v1/account` is the exception
  to "malformed": it returns `invalid_api_key` only for a key that
  starts `sk_test_` or `sk_live_` and is unknown, revoked or expired.
- `unauthorized` — the `Authorization` header is absent or malformed on
  an operation that also accepts a dashboard session (the dual-auth and
  billing operations). `GET /v1/account` also returns it, for no
  `Authorization` header and for any bearer value that does not start
  `sk_test_` or `sk_live_` (that value is checked as a dashboard
  session token).

Not retryable as-is: supply a valid API key first.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 402 — Payment required. The account cannot fund this request, and no work was
performed. The unified `Error` envelope carries one of the billing codes
below in `error.code`; match on the code, never on the prose.

- `quota_exceeded` — the enforced balance for this request's meter
  cannot fund it. Capture endpoints (`/v1/scrape`, `/v1/fetch`,
  `/v1/screenshots`, `/v1/sitemap` and the bulk siblings) meter
  bandwidth: they reject when the balance cannot fund the request's
  byte reservation, and `error.details.required_bytes` and
  `error.details.available_bytes` report the bytes the request reserves
  and the bytes available. `/v1/search` meters credits.
- `spending_cap_reached` — the request would push overage spend past the
  account's configured cap. Only reachable on a custom overage-enabled
  plan; standard plans stop at `quota_exceeded`.
- `billing_entitlement_impaired` — the paid subscription is in a payment
  impairment (e.g. `past_due`). `error.message` names the reason.
- `billing_entitlement_expired` — the paid subscription entitlement
  period has ended, or its dunning grace period has run out.

Not retryable as-is: settle the balance, raise the cap, or restore the
subscription first. This response carries no `X-Quota-*` headers — they
travel with the 2xx responses of the operations that emit them.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 403 — The target is on ByteKit's blocklist. Nothing is queued and nothing is billed.
The unified `Error` envelope carries one of two codes:

- `host_forbidden` — a blocked domain, any `.gov` host, or a blocked port.
  `details` is `{ host }`.
- `path_forbidden` — an authentication-style path such as a login page.
  `details` is `{ path }`.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 422 — Validation error.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`

### 429 — Rate limit (`rate_limited`) or concurrency limit (`concurrency_limit`) exceeded. Carries a `Retry-After` header — seconds derived from the same clock as `X-RateLimit-Reset` for the rate-limit variant, a fixed small back-off for the concurrency variant.
- `status`
- `error`
- `error.code`
- `error.message`
- `error.http_status`
- `error.failed_at`
- `error.details`