# Scrape

Scrape URL content

Scrape URL content

### scrape › bulk

Bulk scrape operations

| Option | Description |
| --- | --- |
| `-h, --help` | display help for command |

#### scrape › bulk › create

Create a bulk scrape job

| Option | Description |
| --- | --- |
| `--file <path>` | Path to a newline-delimited file. Each line is a plain URL, or a JSON object carrying "url" plus per-item config. A line whose only key is "url" behaves exactly as before (sent as a bare URL string); any line with additional keys switches the payload to the per-item shape. |
| `--webhook-url <url>` | HTTPS webhook URL notified when the job completes |
| `--webhook-secret <secret>` | HMAC secret used to sign the completion webhook (omitted → unsigned) |
| `--defaults <json>` | Defaults applied to every entry, as a JSON object (omitted → none). Unknown keys are rejected by /v1/bulk (422) and ignored by scrape bulk / fetch bulk; fetch bulk additionally rejects browser-only keys (wait_until, wait_for_selector, delay_ms) with a 400. `cookies` and `headers` are no longer accepted by any capture endpoint and 400 (#3242). |
| `--metadata <json>` | Arbitrary metadata echoed on the job envelope, as a JSON object, e.g. '\{"run":"7"\}' (omitted → none) |
| `--custom <json>` | User payload echoed on every scrape entry, as a JSON object, e.g. '\{"job":"1"\}' (omitted → none; a per-entry value in --file wins) |
| `-h, --help` | display help for command |

```bash
$ bytekit scrape bulk create --file <path> --webhook-url <url> --webhook-secret <secret> --defaults <json> --metadata <json> --custom <json>
```

#### scrape › bulk › get

Get a bulk scrape job

| Option | Description |
| --- | --- |
| `--cursor <cursor>` | Continue from a cursor returned by the preceding page |
| `--limit <n>` | Maximum items to return (1-500) |
| `--status <status>` | Filter items by status: pending, processing, completed, failed, cancelled |
| `-h, --help` | display help for command |

```bash
$ bytekit scrape bulk get <id> --cursor <cursor> --limit <n> --status <status>
```

### scrape › create

Create a scrape job

| Option | Description |
| --- | --- |
| `--url <url>` | URL to scrape |
| `--format <format>` | Output format; repeat or comma-separate for multiple (e.g. --format raw --format markdown, or --format raw,markdown). Sent as the formats array; omitted → server default. |
| `--country <cc>` | Two-letter country code to route the request through (e.g. us, de) |
| `--timeout-ms <n>` | Per-request timeout in milliseconds |
| `--markdown-mode <mode>` | Markdown extraction mode: article, raw, llm |
| `--markdown-query <q>` | BM25 query string for relevance-ranked markdown filtering (omitted → disabled) |
| `--cache-ttl <ttl>` | Cache freshness: 0, Nh (hours), or Nd (days), e.g. 48h, 2d, 0 (omitted → server default) |
| `--delay-ms <n>` | Post-load delay in milliseconds before capture |
| `--events <csv>` | Webhook events (comma-separated): queued, completed, failed (omitted → server default) |
| `--token-encoding <enc>` | Tokenizer for --token-budget: cl100k_base, o200k_base (omitted → server default) |
| `--custom <json>` | User payload echoed on the envelope as a JSON object, e.g. '\{"job":"1"\}' (omitted → none) |
| `--remove-base64-images` | Strip base64 data: images before conversion (server default) |
| `--no-remove-base64-images` | Preserve base64 data: images in the pipeline output |
| `--markdown-links <mode>` | Link rendering: inline, references, none, text |
| `--markdown-images <mode>` | Image retention: inline, references, none, text |
| `--with-links-summary` | Append a Links footer to the markdown output |
| `--with-images-summary` | Append an Images footer to the markdown output |
| `--markdown-compact` | Collapse excessive whitespace in the markdown output |
| `--markdown-filter-images` | Filter low-signal images from the markdown output |
| `--markdown-include-media` | Return rich link/image objects and a tables array |
| `--markdown-include-warnings` | Include markdown-pipeline and tag-filter warnings |
| `--markdown-include-stats` | Include a top-level stats object (chars, tokens, blocks) |
| `--async` | Queue the scrape and return a job envelope immediately (no wait) |
| `--webhook-url <url>` | HTTPS webhook URL notified when an async scrape completes |
| `--token-budget <n>` | Maximum tokens of markdown to return |
| `--clean-markdown` | Post-process markdown with the LLM cleaner (billed 3x) |
| `--include-tags <csv>` | Comma-separated HTML tags to keep (e.g. article,main) |
| `--exclude-tags <csv>` | Comma-separated HTML tags to drop (e.g. nav,footer) |
| `--mobile` | Render with a mobile viewport/user-agent |
| `--solve-challenge` | Attempt to clear a Cloudflare challenge; success is not guaranteed. Billed as a normal browser render. |
| `--raw` | Print the requested format body raw, with no JSON wrapper |
| `-o, --output <file>` | Write the output to the given file path (implies --raw unless --json is set) |
| `-h, --help` | display help for command |

```bash
$ bytekit scrape create --url <url> --format <format> --country <cc> --timeout-ms <n> --markdown-mode <mode> --markdown-query <q> --cache-ttl <ttl> --delay-ms <n> --events <csv> --token-encoding <enc> --custom <json> --remove-base64-images --no-remove-base64-images --markdown-links <mode> --markdown-images <mode> --with-links-summary --with-images-summary --markdown-compact --markdown-filter-images --markdown-include-media --markdown-include-warnings --markdown-include-stats --async --webhook-url <url> --token-budget <n> --clean-markdown --include-tags <csv> --exclude-tags <csv> --mobile --solve-challenge --raw
```

### scrape › get

Get a scrape job by ID (use --raw/-o/--format to retrieve a completed body)

| Option | Description |
| --- | --- |
| `--raw` | Print the requested format body raw, with no JSON wrapper |
| `-o, --output <file>` | Write the output to the given file path (implies --raw unless --json is set) |
| `--format <format>` | Which persisted format to emit — only takes effect with --raw/-o; ignored otherwise |
| `-h, --help` | display help for command |

```bash
$ bytekit scrape get <id> --raw --format <format>
```