API ReferenceScrape Bulk

Bulk scrape URLs

Enqueues multiple URLs for scraping. Results delivered via webhook. Completion deliveries use X-ByteKit-Event: bulk.completed and the BulkCompletedWebhook contract: at most 500 complete items and 1 MiB, followed by cursor recovery through GET /v1/scrape/bulk/{id} whenever recovery_required is true.

Unknown keys. An unknown key on an items[] entry, or at the top level of the body, is rejected with 422 validation_error. An unknown key inside defaults is not: it is dropped without a warning and the request is accepted with 202. The exception is cookies and headers, on an item or in defaults: they answer 400 validation_error.

For AI agents: https://bytekit.com/llms.txt — every docs page is available as markdown by appending .md

POST
/v1/scrape/bulk

Authorization

AuthorizationRequiredBearer <token>

API key. Manage keys from the dashboard at app.bytekit.com.

In: header

Request Body

application/jsonRequired
urlsarray<string>

Simple URL list. Provide urls OR items, not both.

itemsarray<object>

Per-item configs. Provide urls OR items, not both.

defaultsobject
webhook_urlRequiredstring

Webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported)

Pattern: "^https://"Format: "uri"
webhook_secretstring

HMAC secret used to sign webhook deliveries. Each delivery carries X-Webhook-Timestamp and X-Webhook-Signature (sha256= + the hex HMAC-SHA256 of timestamp.body). See https://bytekit.com/docs/guides/webhooks for verification code.

metadataobject

Arbitrary key-value metadata. Max 4096 bytes when JSON-serialized.

customobject

Top-level user-supplied JSON payload, capped at 4096 UTF-8 bytes. Applied to every scrape-type entry that does not provide its own per-entry custom (the per-entry value wins). Echoed on each entry's success envelope in the bulk results response (the key is omitted when no custom was supplied at either level). Distinct from the system-owned metadata column and never used as a cache-key input.

curl -X POST https://api.bytekit.com/v1/scrape/bulk \
  -H "Authorization: Bearer sk_live_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "url": "https://example.com",
      "formats": [
        "raw",
        "markdown"
      ],
      "custom": {
        "batch_id": 17
      }
    },
    {
      "url": "https://example.org",
      "formats": [
        "markdown"
      ]
    }
  ],
  "defaults": {
    "formats": [
      "raw"
    ],
    "country": "US"
  },
  "custom": {
    "campaign": "fall-2026"
  },
  "webhook_url": "https://hooks.example.com/bytekit"
}'

Bulk scrape job accepted.

{
  "id": "bulk_01j9abc123",
  "status": "processing",
  "total": 0,
  "completed": 0,
  "failed": 0,
  "created_at": "2019-08-24T14:15:22Z",
  "completed_at": "2019-08-24T14:15:22Z",
  "total_credits_charged": 0,
  "total_billed_bytes": 0,
  "estimated_credits": 0,
  "webhook_url": "string"
}

Scrape Bulk

Scrape Bulk API — endpoints, requests, and responses.

GET /v1/scrape/bulk/{id}Get scrape bulk job status

Returns the job envelope plus its rehydrated per-URL scrape envelopes. **Downloading item content.** A `raw` or `markdown` artifact within the 10-MiB inline cap is returned inline. An oversized one is replaced by a **pre-signed** URL, minted fresh on every read of this endpoint and valid for **86400 seconds (24 hours)**. The underlying objects are private, so an unsigned URL returns `401`: download the content inside that window, or re-poll this endpoint for a fresh signature. An artifact that was never stored, or whose object cannot be signed, is omitted from `formats` and reported as `artifact_unavailable` in `warnings`. The private unsigned storage URL is never returned. **Cancelled by the platform.** When the account can no longer spend (its bandwidth balance is exhausted, or the account is deleted) while items are still pending, ByteKit cancels the job itself. It then reads `status: cancelled` even though you never called `DELETE /v1/bulk/{id}`, and the cancellation sends no webhook. **Job ids from other create routes.** The job is looked up by account only, so the id of a job created by `POST /v1/fetch/bulk` or `POST /v1/bulk` is found here too, not a `404`. The items listed are the job's scrape rows: a fetch-bulk job's items appear, and a `POST /v1/bulk` job shows its scrape items but not its screenshots. An id from another account answers `404`.