> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tazpal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl

> Multi-page crawl jobs — start, list, poll status, fetch results, and cancel.

# Crawl multiple pages

The crawl API runs **asynchronous, multi-page** jobs in the background. You start a
job, receive a `job_id`, then poll for status and results.

<Info>
  **Authentication:** API key required for all crawl endpoints.
</Info>

## Overview

| Method   | Endpoint                     | Description                       |
| -------- | ---------------------------- | --------------------------------- |
| `POST`   | `/v1/crawl`                  | Start a crawl job (returns `202`) |
| `GET`    | `/v1/crawl`                  | List your jobs, newest first      |
| `GET`    | `/v1/crawl/{job_id}`         | Job status + progress             |
| `GET`    | `/v1/crawl/{job_id}/results` | Paginated results                 |
| `DELETE` | `/v1/crawl/{job_id}`         | Cancel a pending/running job      |

## Crawl modes

| Mode      | Discovery                                   | Key fields                 |
| --------- | ------------------------------------------- | -------------------------- |
| `sitemap` | Parse `sitemap.xml` (incl. sitemap indexes) | `start_url`                |
| `link`    | Follow on-page links up to `depth`          | `start_url`, `depth`       |
| `pattern` | Match URLs against a glob/regex pattern     | `start_url`, `url_pattern` |
| `batch`   | Crawl an explicit list of URLs in parallel  | `urls`                     |

When `mode` is omitted it's auto-detected: `batch` if `urls` is present, otherwise
`link`.

***

## Start a crawl

```
POST https://api.tazpal.com/v1/crawl
```

### Body

| Field            | Type      | Required         | Default | Description                              |
| ---------------- | --------- | ---------------- | ------- | ---------------------------------------- |
| `start_url`      | string    | *mode-dependent* | —       | Starting URL (sitemap/link/pattern)      |
| `mode`           | string    | no               | auto    | `sitemap`, `link`, `pattern`, or `batch` |
| `max_pages`      | integer   | no               | `100`   | Max pages to crawl (1–1000)              |
| `depth`          | integer   | no               | `3`     | Link crawl depth (0–10)                  |
| `url_pattern`    | string    | *pattern mode*   | —       | Glob/regex URL pattern                   |
| `urls`           | string\[] | *batch mode*     | —       | Explicit URL list                        |
| `options`        | object    | no               | `{}`    | Crawl options (see below)                |
| `callback_url`   | string    | no               | —       | Webhook URL for completion               |
| `webhook_secret` | string    | no               | —       | HMAC-SHA256 signing secret               |

### `options` object

| Field                | Type      | Default | Description                                |
| -------------------- | --------- | ------- | ------------------------------------------ |
| `respect_robots_txt` | boolean   | `true`  | Respect `robots.txt` (incl. `Crawl-delay`) |
| `delay_ms`           | integer   | `1000`  | Delay between page fetches (0–60000)       |
| `timeout_per_page`   | integer   | `30000` | Per-page fetch timeout (1000–120000)       |
| `parallel`           | integer   | `5`     | Concurrency for batch mode (1–50)          |
| `follow_external`    | boolean   | `false` | Follow off-domain links (link mode)        |
| `include_patterns`   | string\[] | —       | Include glob/regex patterns (link mode)    |
| `exclude_patterns`   | string\[] | —       | Exclude glob/regex patterns (link mode)    |

### Example

<Tabs>
  <Tab title="Sitemap">
    <CodeGroup>
      ```bash theme={null}
      curl -X POST https://api.tazpal.com/v1/crawl \
        -H "Authorization: Bearer baas_live_YOUR_KEY" \
        -H "Content-Type: application/json" \
        -d '{"start_url": "https://example.com/sitemap.xml", "mode": "sitemap", "max_pages": 500}'
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Batch">
    <CodeGroup>
      ```bash theme={null}
      curl -X POST https://api.tazpal.com/v1/crawl \
        -H "Authorization: Bearer baas_live_YOUR_KEY" \
        -H "Content-Type: application/json" \
        -d '{"mode": "batch", "urls": ["https://a.com/1", "https://a.com/2"], "options": {"parallel": 10}}'
      ```
    </CodeGroup>
  </Tab>

  <Tab title="With webhook">
    <CodeGroup>
      ```bash theme={null}
      curl -X POST https://api.tazpal.com/v1/crawl \
        -H "Authorization: Bearer baas_live_YOUR_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "start_url": "https://example.com",
          "mode": "link",
          "callback_url": "https://yourapp.example.com/hooks/baas",
          "webhook_secret": "your_signing_secret"
        }'
      ```
    </CodeGroup>
  </Tab>
</Tabs>

```json Response — 202 Accepted theme={null}
{
  "job_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "status": "pending",
  "mode": "sitemap",
  "start_url": "https://example.com/sitemap.xml",
  "max_pages": 500,
  "depth": 3,
  "created_at": "2026-08-28T14:30:00Z"
}
```

***

## Get job status

```
GET https://api.tazpal.com/v1/crawl/{job_id}
```

```bash theme={null}
curl https://api.tazpal.com/v1/crawl/9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d \
  -H "Authorization: Bearer baas_live_YOUR_KEY"
```

```json Response theme={null}
{
  "job_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "status": "running",
  "mode": "sitemap",
  "start_url": "https://example.com/sitemap.xml",
  "max_pages": 500,
  "depth": 3,
  "progress": {
    "pages_crawled": 120,
    "pages_total": 300,
    "pages_failed": 2,
    "skipped": 0,
    "current_url": "https://example.com/page/121"
  },
  "result_count": 120,
  "error_count": 2,
  "error": null,
  "callback_url": null,
  "started_at": "2026-08-28T14:30:00Z",
  "completed_at": null,
  "created_at": "2026-08-28T14:30:00Z"
}
```

### Job statuses

| Status      | Meaning                              |
| ----------- | ------------------------------------ |
| `pending`   | Queued, not yet started              |
| `running`   | Discovery or fetching in progress    |
| `completed` | Finished successfully                |
| `failed`    | Failed during discovery or execution |
| `canceled`  | Canceled via `DELETE`                |

***

## Get job results

```
GET https://api.tazpal.com/v1/crawl/{job_id}/results?limit=100&offset=0
```

| Query    | Default | Range  |
| -------- | ------- | ------ |
| `limit`  | `100`   | 1–1000 |
| `offset` | `0`     | ≥ 0    |

```bash theme={null}
curl "https://api.tazpal.com/v1/crawl/9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d/results?limit=50" \
  -H "Authorization: Bearer baas_live_YOUR_KEY"
```

```json Response theme={null}
{
  "job_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "count": 50,
  "results": [
    {
      "id": "1f4e8...",
      "url": "https://example.com/page/1",
      "status": 200,
      "markdown": "# Page 1\n\n...",
      "metadata": {},
      "error": null,
      "crawled_at": "2026-08-28T14:30:05Z"
    }
  ]
}
```

***

## List all jobs

```
GET https://api.tazpal.com/v1/crawl
```

```bash theme={null}
curl https://api.tazpal.com/v1/crawl -H "Authorization: Bearer baas_live_YOUR_KEY"
```

***

## Cancel a job

```
DELETE https://api.tazpal.com/v1/crawl/{job_id}
```

Cancels a `pending` or `running` job cooperatively and returns `204 No Content`.

```bash theme={null}
curl -X DELETE https://api.tazpal.com/v1/crawl/9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d \
  -H "Authorization: Bearer baas_live_YOUR_KEY"
```

***

## Errors

| Status | `detail.error`    | Meaning                                    |
| ------ | ----------------- | ------------------------------------------ |
| `400`  | `invalid_mode`    | Unknown crawl mode                         |
| `400`  | `invalid_request` | Missing required field for the chosen mode |
| `401`  | —                 | Missing/invalid API key                    |
| `404`  | —                 | Job not found (or not owned by caller)     |
| `429`  | `rate_limited`    | Rate limit exceeded                        |

<Note>
  Crawl jobs are scoped to the API key's owner. You can only list, read, or cancel
  jobs you created — other users' `job_id`s return `404`.
</Note>