> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tazpal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape

> POST /v1/scrape — scrape a URL and return content in Markdown, screenshot, PDF, CSV, or HTML.

# Scrape a URL

`POST /v1/scrape` fetches a URL with a stealth browser and returns its content in one
of five output formats. Markdown is the default and is optimized for LLM consumption.

<Info>
  **Authentication:** API key **or** x402 payment. On x402 requests, payment is verified
  before the scrape and settled only on success.
</Info>

## Endpoint

```
POST https://api.tazpal.com/v1/scrape
```

## Request

### Headers

| Header                                | Required | Description                                                           |
| ------------------------------------- | -------- | --------------------------------------------------------------------- |
| `Authorization: Bearer baas_live_...` | *one of* | API key authentication                                                |
| `X-PAYMENT: <base64 permit>`          | *one of* | x402 payment                                                          |
| `Content-Type: application/json`      | yes      | Must be `application/json`                                            |
| `X-Request-ID`                        | no       | Client-supplied request ID (echoed in `X-Request-ID` response header) |

### Body

| Field               | Type    | Required | Default    | Description                                           |
| ------------------- | ------- | -------- | ---------- | ----------------------------------------------------- |
| `url`               | string  | ✅        | —          | Target URL to scrape                                  |
| `output`            | string  | no       | `markdown` | One of `markdown`, `screenshot`, `pdf`, `csv`, `html` |
| `options`           | object  | no       | `{}`       | Output-specific options (see below)                   |
| `wait_for_selector` | string  | no       | —          | CSS selector to wait for before extraction            |
| `timeout_ms`        | integer | no       | `20000`    | Navigation timeout in ms (1000–120000)                |
| `block_media`       | boolean | no       | `true`     | Block image/font/video requests for speed             |
| `proxy_url`         | string  | no       | —          | Explicit proxy (`socks5://` or `http://`)             |
| `wait_strategy`     | string  | no       | auto       | `default`, `spa`, `heavy`, or `cloudflare`            |
| `retry`             | boolean | no       | `true`     | Retry on failure with exponential backoff             |
| `bypass_cache`      | boolean | no       | `false`    | Skip the 5-minute response cache                      |
| `javascript`        | string  | no       | —          | Custom JS to execute after page load                  |

### Options by output format

<AccordionGroup>
  <Accordion title="screenshot options">
    | Key                | Type    | Default | Description                      |
    | ------------------ | ------- | ------- | -------------------------------- |
    | `format`           | string  | `png`   | `png` or `jpeg`                  |
    | `full_page`        | boolean | `true`  | Capture the full page (PNG only) |
    | `width` / `height` | integer | —       | Explicit viewport size           |
    | `quality`          | integer | `90`    | JPEG quality                     |
  </Accordion>

  <Accordion title="pdf options">
    | Key                | Type    | Description                              |
    | ------------------ | ------- | ---------------------------------------- |
    | `format`           | string  | Page size (e.g. `A4`, `Letter`)          |
    | `landscape`        | boolean | Landscape orientation                    |
    | `print_background` | boolean | Print background graphics                |
    | `scale`            | number  | Render scale (e.g. `0.8`)                |
    | `margin`           | object  | `{top, bottom, left, right}` margins     |
    | `quality`          | integer | JPEG quality for the screenshot fallback |
  </Accordion>

  <Accordion title="csv options">
    | Key              | Type   | Default | Description                        |
    | ---------------- | ------ | ------- | ---------------------------------- |
    | `table_selector` | string | `table` | CSS selector for tables to extract |
  </Accordion>
</AccordionGroup>

## Response

### Success — `200 OK`

| Field               | Type           | Description                                               |
| ------------------- | -------------- | --------------------------------------------------------- |
| `status`            | integer        | Always `200`                                              |
| `url`               | string         | The scraped URL                                           |
| `output`            | string         | The output format used                                    |
| `data`              | object         | `ScrapeData` for markdown; format-specific dict otherwise |
| `execution_time_ms` | integer        | End-to-end time including browser launch                  |
| `payment`           | object \| null | x402 settlement result (null for API-key auth)            |

**Markdown `data` shape:**

| Field             | Type    | Description            |
| ----------------- | ------- | ---------------------- |
| `title`           | string  | Page title             |
| `markdown`        | string  | Clean Markdown content |
| `character_count` | integer | Length of the Markdown |

## Examples

### Markdown (default)

<CodeGroup>
  ```bash theme={null}
  curl -X POST https://api.tazpal.com/v1/scrape \
    -H "Authorization: Bearer baas_live_YOUR_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://news.ycombinator.com"}'
  ```

  ```python theme={null}
  import httpx

  resp = httpx.post(
      "https://api.tazpal.com/v1/scrape",
      headers={"Authorization": "Bearer baas_live_YOUR_KEY"},
      json={"url": "https://news.ycombinator.com"},
  )
  resp.raise_for_status()
  data = resp.json()["data"]
  print(data["title"])
  print(data["markdown"][:500])
  ```

  ```javascript Node.js theme={null}
  const resp = await fetch("https://api.tazpal.com/v1/scrape", {
    method: "POST",
    headers: {
      Authorization: "Bearer baas_live_YOUR_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url: "https://news.ycombinator.com" }),
  });
  const { data } = await resp.json();
  console.log(data.title, data.markdown.slice(0, 500));
  ```
</CodeGroup>

### Screenshot (base64 PNG)

<Tabs>
  <Tab title="API key">
    <CodeGroup>
      ```bash theme={null}
      curl -X POST https://api.tazpal.com/v1/scrape \
        -H "Authorization: Bearer baas_live_YOUR_KEY" \
        -H "Content-Type: application/json" \
        -d '{"url": "https://example.com", "output": "screenshot", "options": {"full_page": true}}'
      ```
    </CodeGroup>
  </Tab>

  <Tab title="x402">
    <CodeGroup>
      ```bash theme={null}
      curl -X POST https://api.tazpal.com/v1/scrape \
        -H "X-PAYMENT: eyJwYXlsb2FkIjp7Im5vbmNlIjoi..." \
        -H "Content-Type: application/json" \
        -d '{"url": "https://example.com", "output": "screenshot"}'
      ```
    </CodeGroup>
  </Tab>
</Tabs>

```json Response theme={null}
{
  "status": 200,
  "url": "https://example.com",
  "output": "screenshot",
  "data": {
    "data": "iVBORw0KGgoAAAANSUhEUgAA...",
    "format": "png",
    "size_bytes": 48102
  },
  "execution_time_ms": 2210,
  "payment": null
}
```

### PDF

```bash theme={null}
curl -X POST https://api.tazpal.com/v1/scrape \
  -H "Authorization: Bearer baas_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "output": "pdf", "options": {"format": "A4", "print_background": true}}'
```

### CSV (table extraction)

```bash theme={null}
curl -X POST https://api.tazpal.com/v1/scrape \
  -H "Authorization: Bearer baas_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/data", "output": "csv", "options": {"table_selector": "table.prices"}}'
```

### Custom JavaScript + wait strategy

```python theme={null}
import httpx

resp = httpx.post(
    "https://api.tazpal.com/v1/scrape",
    headers={"Authorization": "Bearer baas_live_YOUR_KEY"},
    json={
        "url": "https://example.com",
        "wait_strategy": "spa",
        "javascript": "document.querySelector('#load-more').click()",
    },
)
```

## Errors

| Status | `detail.error`      | Meaning                                       |
| ------ | ------------------- | --------------------------------------------- |
| `400`  | `invalid_output`    | `output` is not a valid format                |
| `400`  | `connection_failed` | The URL could not be reached (DNS/connection) |
| `401`  | `unauthorized`      | Missing or invalid API key / `X-PAYMENT`      |
| `429`  | `rate_limited`      | Per-IP rate limit exceeded                    |
| `500`  | `scrape_failed`     | Unexpected engine failure                     |
| `504`  | `timeout`           | Navigation timed out                          |

```json Error example theme={null}
{
  "detail": {
    "error": "timeout",
    "message": "Navigation timed out after 20000ms: https://slow.example.com",
    "url": "https://slow.example.com",
    "payment_charged": false
  }
}
```

<Note>
  Error responses carry a `payment_charged` boolean so x402 callers can confirm they
  were not charged for a failed scrape.
</Note>