Sitemap

POST /v1/sitemap finds the URLs of a website, and GET /v1/sitemap/{id} reports on the job. A valid request answers 202 with a job id. ByteKit discovers the URLs in the background and saves them as a results file that you download from results_url.

Start a job

curl -i -X POST https://api.bytekit.com/v1/sitemap \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://bytekit.com", "strategy": "quick", "max_urls": 20}'

{
  "id": "sm_c013b3fa71f94ff292f5c2fa74f8d77b",
  "url": "https://bytekit.com",
  "strategy": "quick",
  "max_depth": 2,
  "max_urls": 20,
  "status": "pending",
  "total_urls": null,
  "sources": null,
  "total_bytes": null,
  "results_url": null,
  "warnings": null,
  "error_code": null,
  "error_message": null,
  "compact": false,
  "process": null,
  "metadata": null,
  "created_at": "2026-10-10T14:59:44.602Z",
  "completed_at": null,
  "expires_at": null,
  "cache": "miss",
  "cache_age_s": null
}

This request was run for real and returned 202. cache is miss because no earlier job could answer it; see Cache.

Only url is required. Keep the id: it is how you read the job.

The 202 response reports url after ByteKit has normalised it: the scheme and host are lowercased, an international host is in punycode, non-ASCII characters are percent-encoded, and surrounding spaces are trimmed. If the result is only a scheme and host, with or without a trailing /, it is returned as scheme://host. If it carries a path, query string or fragment, it is returned whole, together with the warnings entry. GET returns the host as scheme://host.

Strategies

The sources run in that order, and each one fills only the room that max_urls leaves. A URL found by more than one source is counted once, under the first. A deep job follows links only from URLs the earlier sources found, so on a site with no sitemap files and no known URLs it finds no links. The job's sources field counts the URLs per source. For a deep job run for real it read {"known": 1, "links": 4, "sitemap": 119}.

Check the job

GET /v1/sitemap/{id} answers 200 with the job whatever its status: pending, processing, completed or failed. Poll it every few seconds until the status is completed or failed. The jobs on this page finished within a few seconds.

curl -i https://api.bytekit.com/v1/sitemap/sm_c013b3fa71f94ff292f5c2fa74f8d77b \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE"

{
  "id": "sm_c013b3fa71f94ff292f5c2fa74f8d77b",
  "url": "https://bytekit.com",
  "strategy": "quick",
  "max_depth": 2,
  "max_urls": 20,
  "status": "completed",
  "total_urls": 20,
  "sources": { "sitemap": 20 },
  "total_bytes": 196,
  "results_url": "https://…",
  "warnings": [],
  "error_code": null,
  "error_message": null,
  "compact": false,
  "process": null,
  "metadata": null,
  "created_at": "2026-10-10T14:59:44.602Z",
  "completed_at": "2026-10-10T14:59:47.405Z",
  "expires_at": "2027-01-08T14:59:47.405Z",
  "billing": {
    "factors": [{ "name": "endpoint", "value": 1, "reason": "sitemap" }],
    "raw_bytes": 196,
    "multiplier": 1,
    "billed_bytes": 196
  }
}

This request was run for real and returned 200. The signed results_url is shortened.

A completed 200 also carries the headers X-Raw-Bytes, X-Billing-Multiplier and X-Billable-Bytes, which show the same figures as billing.

When a job fails

curl -i -X POST https://api.bytekit.com/v1/sitemap \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "strategy": "quick", "max_urls": 20}'

This request was run for real and returned 202, with the same fields as above and cache miss. The job then failed, and polling it returned 200:

{
  "id": "sm_a380b0ea64dd4b19a854699c58933499",
  "url": "https://example.com",
  "strategy": "quick",
  "max_depth": 2,
  "max_urls": 20,
  "status": "failed",
  "total_urls": null,
  "sources": null,
  "total_bytes": null,
  "results_url": null,
  "warnings": ["robots.txt unavailable (non-success fetch)"],
  "error_code": "no_urls_found",
  "error_message": "No URLs discovered for this domain",
  "compact": false,
  "process": null,
  "metadata": null,
  "created_at": "2026-10-10T14:59:54.408Z",
  "completed_at": "2026-10-10T14:59:55.941Z",
  "expires_at": null
}

A robots.txt or sitemap file that ByteKit cannot read does not fail a job by itself. It adds an entry to warnings, and discovery goes on with the other sources. A sitemap file that is not there (404 or 410) or is empty adds no warning, and neither does a page that cannot be fetched while a deep job follows links. These are the error_code values you may meet:

A job that fails with worker_crash sends no sitemap.failed webhook, so poll the job as well as listening for the webhook.

Download the results

curl --compressed "$RESULTS_URL"

url,source
https://bytekit.com/docs/changelog,sitemap
https://bytekit.com/docs,sitemap
https://bytekit.com/docs/introduction,sitemap

This request was run for real and returned 200. The file holds a header and 20 rows, and the first rows are shown.

The results file is a gzip-compressed CSV, served with Content-Encoding: gzip, so --compressed makes curl decode it. A client that does not decode gzip receives the compressed bytes, and their count is total_bytes. It has the header url,source and one row per discovered URL, and source is sitemap, known or links. In that format a URL that contains a comma is wrapped in double quotes. With compact: true the file is a plain list of URLs, one per line, with no header. The lastmod and depth columns are not emitted.

Cache

Before it queues discovery, ByteKit looks for a completed job of your account that has the same scheme and host and the same strategy, that finished within cache_ttl, and whose max_depth and max_urls are at least the ones you sent. A job that was itself a cache hit is never matched, only the job that ran discovery is, so cache_ttl counts from when that job finished. The cache field of the 202 response says what it found. GET leaves out cache and cache_age_s.

Sending the request above again a few seconds later returned a hit:

{
  "id": "sm_6aa7e5ea6a854f19976754b0fdc8f438",
  "url": "https://bytekit.com",
  "strategy": "quick",
  "max_depth": 2,
  "max_urls": 20,
  "status": "completed",
  "total_urls": 20,
  "sources": { "sitemap": 20 },
  "total_bytes": 196,
  "results_url": "https://…",
  "warnings": null,
  "error_code": null,
  "error_message": null,
  "compact": false,
  "process": null,
  "metadata": null,
  "created_at": "2026-10-10T14:59:53.441Z",
  "completed_at": "2026-10-10T14:59:53.440Z",
  "expires_at": "2027-01-08T14:59:47.405Z",
  "billing": {
    "factors": [
      { "name": "endpoint", "value": 1, "reason": "sitemap" },
      { "name": "cache_hit", "value": 0.5, "reason": "hit" }
    ],
    "raw_bytes": 196,
    "multiplier": 0.5,
    "billed_bytes": 98
  },
  "cache": "hit",
  "cache_age_s": 6
}

This request was run for real and returned 202. The signed results_url is shortened.

A hit has an id of its own and points at the same results file as the job that answered it. The file is returned as that job stored it. It can hold more URLs than the max_urls you sent, and it keeps the format that job was created with, whatever compact you send, because compact is not part of the match. The compact field of a hit's response repeats your request, not the format of the file, so tell the format from the file: only a non-compact file starts with the header line url,source. To get a new job that follows your request, send cache_ttl 0.

Webhooks

Send a webhook_url and ByteKit POSTs the outcome to it. The header X-ByteKit-Event is sitemap.completed when the job completes and sitemap.failed when it fails, so you can dispatch on the header without reading the body. A cache hit sends sitemap.completed as well. A failure that sends no sitemap.failed is described under When a job fails. Without a webhook_url the job is poll-only. Set webhook_secret to have the deliveries signed; see the Webhooks guide for verification. The Scraping guide lists the other event types.

The body has two keys, event, which repeats the header, and data. This page shows no captured delivery, only the fields:

For a cache hit, data holds all the fields that GET returns.

Cost

A completed job is billed on the byte length of its gzip-compressed results file, which is total_bytes. The bytes ByteKit fetches while it discovers the URLs are not part of the charge. Bandwidth billing has the rules.

In the job above, raw_bytes and billed_bytes are both 196. A cache hit is billed at half: in the hit above billing lists a cache_hit factor of 0.5, and billed_bytes is 98. A job that ends failed is not billed. It has no billing, and its estimate is rolled back.

When you submit a job, your usage counter rises by an estimate of 102,400 bytes for it. A completed job replaces the estimate with its billed bytes, and a failed job rolls it back, as the Usage counting section describes. In a run for real the counter rose by 102,400 while a job was queued, and by 146 once it completed with billed_bytes 146.

Errors

Errors use the envelope described in the Errors guide. These are the ones you are most likely to meet on the sitemap routes:

Next steps

• Webhooks — verify a delivery's signature
• Scraping — the event types
• Bandwidth billing — how a sitemap job is billed
• Rate limits — concurrency slots and request size
• API Reference: Create a sitemap, Get a sitemap job — full request and response schemas