# Sitemap Discover the URLs on a domain — start a job, read its status, download the results file, get a webhook, and what it costs. `POST /v1/sitemap` finds the URLs of a website, and `GET /v1/sitemap/{id}` reports on the job. A valid request answers `202` with a job id. ByteKit discovers the URLs in the background and saves them as a results file that you download from `results_url`. ## Start a job ```bash curl -i -X POST https://api.bytekit.com/v1/sitemap \ -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \ -H "Content-Type: application/json" \ -d '{"url": "https://bytekit.com", "strategy": "quick", "max_urls": 20}' ``` ```json { "id": "sm_c013b3fa71f94ff292f5c2fa74f8d77b", "url": "https://bytekit.com", "strategy": "quick", "max_depth": 2, "max_urls": 20, "status": "pending", "total_urls": null, "sources": null, "total_bytes": null, "results_url": null, "warnings": null, "error_code": null, "error_message": null, "compact": false, "process": null, "metadata": null, "created_at": "2026-10-10T14:59:44.602Z", "completed_at": null, "expires_at": null, "cache": "miss", "cache_age_s": null } ``` > This request was run for real and returned `202`. `cache` is `miss` because no earlier job could answer it; see [Cache](#cache). Only `url` is required. Keep the `id`: it is how you read the job. | Field | Default | What it does | | ---------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `url` | (required) | The site to discover. Only the scheme and host are used: a path, query string or fragment is accepted but ignored, and the `202` response carries a `warnings` entry saying so | | `strategy` | `standard` | `quick`, `standard` or `deep`: where ByteKit looks for URLs. See [Strategies](#strategies) | | `max_depth` | `2` | From 1 to 5. How many levels of page links a `deep` job follows | | `max_urls` | `5000` | From 1 to 100000. A new job stops collecting at this many URLs | | `process` | `null` | An object with an `action` and `options`. ByteKit accepts it and returns it on the job, and it has no other effect | | `compact` | `false` | `true` makes the results file a plain list of URLs, one per line, with no header | | `metadata` | none | An object of your own, up to 4096 bytes when serialized. It is returned on the job and in webhook deliveries. A key that starts with `_` is rejected with `422 validation_error` | | `webhook_url` | none | An `https://` URL to notify about the job. An `http://` URL is rejected with `400 invalid_url` and the reason `scheme_unsupported`. See [Webhooks](#webhooks) | | `webhook_secret` | `null` | Signs the webhook deliveries. It is ignored when there is no `webhook_url` | | `cache_ttl` | `30d` | How old a cached job may be and still answer the request. It is a string: `0`, hours such as `48h`, or days such as `30d`, up to 90 days. A number is rejected with `422 validation_error` | The `202` response reports `url` after ByteKit has normalised it: the scheme and host are lowercased, an international host is in punycode, non-ASCII characters are percent-encoded, and surrounding spaces are trimmed. If the result is only a scheme and host, with or without a trailing `/`, it is returned as `scheme://host`. If it carries a path, query string or fragment, it is returned whole, together with the `warnings` entry. `GET` returns the host as `scheme://host`. ## Strategies | Strategy | Where it looks | | ---------- | ---------------------------------------------------------------------------------------------------------------------------- | | `quick` | The sitemap files the site publishes. Every URL has the source `sitemap` | | `standard` | The same, plus URLs from your account's completed scrapes of the host in the last 90 days. These have the source `known` | | `deep` | The same as `standard`, plus URLs found by following links on pages, up to `max_depth` levels. These have the source `links` | The sources run in that order, and each one fills only the room that `max_urls` leaves. A URL found by more than one source is counted once, under the first. A `deep` job follows links only from URLs the earlier sources found, so on a site with no sitemap files and no `known` URLs it finds no `links`. The job's `sources` field counts the URLs per source. For a `deep` job run for real it read `{"known": 1, "links": 4, "sitemap": 119}`. ## Check the job `GET /v1/sitemap/{id}` answers `200` with the job whatever its `status`: `pending`, `processing`, `completed` or `failed`. Poll it every few seconds until the status is `completed` or `failed`. The jobs on this page finished within a few seconds. ```bash curl -i https://api.bytekit.com/v1/sitemap/sm_c013b3fa71f94ff292f5c2fa74f8d77b \ -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" ``` ```json { "id": "sm_c013b3fa71f94ff292f5c2fa74f8d77b", "url": "https://bytekit.com", "strategy": "quick", "max_depth": 2, "max_urls": 20, "status": "completed", "total_urls": 20, "sources": { "sitemap": 20 }, "total_bytes": 196, "results_url": "https://…", "warnings": [], "error_code": null, "error_message": null, "compact": false, "process": null, "metadata": null, "created_at": "2026-10-10T14:59:44.602Z", "completed_at": "2026-10-10T14:59:47.405Z", "expires_at": "2027-01-08T14:59:47.405Z", "billing": { "factors": [{ "name": "endpoint", "value": 1, "reason": "sitemap" }], "raw_bytes": 196, "multiplier": 1, "billed_bytes": 196 } } ``` > This request was run for real and returned `200`. The signed `results_url` is shortened. | Field | Meaning | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `status` | `pending`, `processing`, `completed` or `failed` | | `total_urls` | How many URLs the results hold. `null` until the job completes | | `sources` | The URLs per source: `sitemap`, `known` or `links` | | `total_bytes` | The size in bytes of the gzip-compressed results file. This is what the job is billed on; see [Cost](#cost) | | `results_url` | A signed link to the results file, present once the job is `completed`. Each `GET` of a completed job returns a new link, valid for 7 days | | `warnings` | Notes about the run, such as a source that could not be read | | `error_code`, `error_message` | Why a job failed. `null` otherwise | | `billing` | What the job was billed: `raw_bytes`, the `factors` applied, the `multiplier` and `billed_bytes` | | `expires_at` | When the job expires: 90 days after it completes, except that a cache hit keeps the expiry of the job that answered it. `null` until the job completes | A completed `200` also carries the headers `X-Raw-Bytes`, `X-Billing-Multiplier` and `X-Billable-Bytes`, which show the same figures as `billing`. ### When a job fails ```bash curl -i -X POST https://api.bytekit.com/v1/sitemap \ -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "strategy": "quick", "max_urls": 20}' ``` > This request was run for real and returned `202`, with the same fields as above and `cache` `miss`. The job then failed, and polling it returned `200`: ```json { "id": "sm_a380b0ea64dd4b19a854699c58933499", "url": "https://example.com", "strategy": "quick", "max_depth": 2, "max_urls": 20, "status": "failed", "total_urls": null, "sources": null, "total_bytes": null, "results_url": null, "warnings": ["robots.txt unavailable (non-success fetch)"], "error_code": "no_urls_found", "error_message": "No URLs discovered for this domain", "compact": false, "process": null, "metadata": null, "created_at": "2026-10-10T14:59:54.408Z", "completed_at": "2026-10-10T14:59:55.941Z", "expires_at": null } ``` A `robots.txt` or sitemap file that ByteKit cannot read does not fail a job by itself. It adds an entry to `warnings`, and discovery goes on with the other sources. A sitemap file that is not there (`404` or `410`) or is empty adds no warning, and neither does a page that cannot be fetched while a `deep` job follows links. These are the `error_code` values you may meet: | `error_code` | When | | ---------------- | ------------------------------------------------------------------------ | | `no_urls_found` | Discovery ended without finding a URL | | `blocked` | The site blocks ByteKit's requests, so discovery did not start | | `job_timeout` | The job did not finish in time | | `internal_error` | A failure on ByteKit's side | | `worker_crash` | The job stopped making progress, and ByteKit failed it after a long wait | A job that fails with `worker_crash` sends no `sitemap.failed` webhook, so poll the job as well as listening for the webhook. ## Download the results ```bash curl --compressed "$RESULTS_URL" ``` ```csv url,source https://bytekit.com/docs/changelog,sitemap https://bytekit.com/docs,sitemap https://bytekit.com/docs/introduction,sitemap ``` > This request was run for real and returned `200`. The file holds a header and 20 rows, and the first rows are shown. The results file is a gzip-compressed CSV, served with `Content-Encoding: gzip`, so `--compressed` makes curl decode it. A client that does not decode gzip receives the compressed bytes, and their count is `total_bytes`. It has the header `url,source` and one row per discovered URL, and `source` is `sitemap`, `known` or `links`. In that format a URL that contains a comma is wrapped in double quotes. With `compact: true` the file is a plain list of URLs, one per line, with no header. The `lastmod` and `depth` columns are not emitted. ## Cache Before it queues discovery, ByteKit looks for a completed job of your account that has the same scheme and host and the same `strategy`, that finished within `cache_ttl`, and whose `max_depth` and `max_urls` are at least the ones you sent. A job that was itself a cache hit is never matched, only the job that ran discovery is, so `cache_ttl` counts from when that job finished. The `cache` field of the `202` response says what it found. `GET` leaves out `cache` and `cache_age_s`. | `cache` | Meaning | | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `miss` | No such job. A new job is queued | | `hit` | A job matched. The response is already `completed` and has a `results_url`, and `cache_age_s` is the age of that job in seconds | | `bypass` | `cache_ttl` is `0`, so the cache was not read and a new job is queued | Sending the request above again a few seconds later returned a hit: ```json { "id": "sm_6aa7e5ea6a854f19976754b0fdc8f438", "url": "https://bytekit.com", "strategy": "quick", "max_depth": 2, "max_urls": 20, "status": "completed", "total_urls": 20, "sources": { "sitemap": 20 }, "total_bytes": 196, "results_url": "https://…", "warnings": null, "error_code": null, "error_message": null, "compact": false, "process": null, "metadata": null, "created_at": "2026-10-10T14:59:53.441Z", "completed_at": "2026-10-10T14:59:53.440Z", "expires_at": "2027-01-08T14:59:47.405Z", "billing": { "factors": [ { "name": "endpoint", "value": 1, "reason": "sitemap" }, { "name": "cache_hit", "value": 0.5, "reason": "hit" } ], "raw_bytes": 196, "multiplier": 0.5, "billed_bytes": 98 }, "cache": "hit", "cache_age_s": 6 } ``` > This request was run for real and returned `202`. The signed `results_url` is shortened. A hit has an `id` of its own and points at the same results file as the job that answered it. The file is returned as that job stored it. It can hold more URLs than the `max_urls` you sent, and it keeps the format that job was created with, whatever `compact` you send, because `compact` is not part of the match. The `compact` field of a hit's response repeats your request, not the format of the file, so tell the format from the file: only a non-compact file starts with the header line `url,source`. To get a new job that follows your request, send `cache_ttl` `0`. ## Webhooks Send a `webhook_url` and ByteKit POSTs the outcome to it. The header `X-ByteKit-Event` is `sitemap.completed` when the job completes and `sitemap.failed` when it fails, so you can dispatch on the header without reading the body. A cache hit sends `sitemap.completed` as well. A failure that sends no `sitemap.failed` is described under [When a job fails](#when-a-job-fails). Without a `webhook_url` the job is poll-only. Set `webhook_secret` to have the deliveries signed; see the [Webhooks guide](https://bytekit.com/docs/guides/webhooks) for verification. The [Scraping guide](https://bytekit.com/docs/guides/scraping#webhook-event-header) lists the other event types. The body has two keys, `event`, which repeats the header, and `data`. This page shows no captured delivery, only the fields: | Event | `data` holds | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `sitemap.completed` | `id`, `url`, `strategy`, `status`, `total_urls`, `sources`, `total_bytes`, `results_url`, `warnings`, `metadata`, `compact` and `billing` | | `sitemap.failed` | `id`, `url`, `strategy`, `status`, `error_code`, `error_message`, `warnings` and `metadata` | For a cache hit, `data` holds all the fields that `GET` returns. ## Cost A completed job is billed on the byte length of its gzip-compressed results file, which is `total_bytes`. The bytes ByteKit fetches while it discovers the URLs are not part of the charge. [Bandwidth billing](https://bytekit.com/docs/billing/bandwidth#sitemap) has the rules. In the job above, `raw_bytes` and `billed_bytes` are both `196`. A cache hit is billed at half: in the hit above `billing` lists a `cache_hit` factor of `0.5`, and `billed_bytes` is `98`. A job that ends `failed` is not billed. It has no `billing`, and its estimate is rolled back. When you submit a job, your usage counter rises by an estimate of 102,400 bytes for it. A completed job replaces the estimate with its billed bytes, and a failed job rolls it back, as the [Usage counting](https://bytekit.com/docs/guides/rate-limits#usage-counting) section describes. In a run for real the counter rose by 102,400 while a job was queued, and by 146 once it completed with `billed_bytes` `146`. ## Errors Errors use the envelope described in the [Errors guide](https://bytekit.com/docs/guides/errors). These are the ones you are most likely to meet on the sitemap routes: | Status | Code | When | | ------ | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `400` | `invalid_url` | `url` or `webhook_url` is not a valid HTTP(S) URL. `error.details.reason` says why | | `400` | `request_too_large` | The request body is over the limit. See [Request size limits](https://bytekit.com/docs/guides/rate-limits#request-size-limits) | | `401` | `unauthorized`, `invalid_api_key` | The API key is missing or not valid | | `402` | `quota_exceeded`, `spending_cap_reached`, `billing_entitlement_impaired`, `billing_entitlement_expired` | The balance or the subscription state blocks the request. See the [Errors guide](https://bytekit.com/docs/guides/errors) | | `403` | `host_forbidden`, `path_forbidden` | The URL's host or path is not allowed | | `404` | `not_found` | On `GET`, the job id does not exist, or belongs to another account | | `422` | `validation_error` | A field failed validation, such as `max_urls` out of range, an unknown `strategy`, a `cache_ttl` in the wrong form, or a malformed job id | | `429` | `rate_limited`, `concurrency_limit` | Too many requests per second, or too many in flight at once. See the [Errors guide](https://bytekit.com/docs/guides/errors) | ## Next steps - [Webhooks](https://bytekit.com/docs/guides/webhooks) — verify a delivery's signature - [Scraping](https://bytekit.com/docs/guides/scraping) — the event types - [Bandwidth billing](https://bytekit.com/docs/billing/bandwidth#sitemap) — how a sitemap job is billed - [Rate limits](https://bytekit.com/docs/guides/rate-limits) — concurrency slots and request size - [API Reference: Create a sitemap](https://bytekit.com/docs/api/sitemap/createsitemap), [Get a sitemap job](https://bytekit.com/docs/api/sitemap/getsitemap) — full request and response schemas