Crawl multiple pages

The crawl API runs asynchronous, multi-page jobs in the background. You start a job, receive a job_id, then poll for status and results.

Authentication: API key required for all crawl endpoints.

Overview

Method	Endpoint	Description
`POST`	`/v1/crawl`	Start a crawl job (returns `202`)
`GET`	`/v1/crawl`	List your jobs, newest first
`GET`	`/v1/crawl/{job_id}`	Job status + progress
`GET`	`/v1/crawl/{job_id}/results`	Paginated results
`DELETE`	`/v1/crawl/{job_id}`	Cancel a pending/running job

Crawl modes

Mode	Discovery	Key fields
`sitemap`	Parse `sitemap.xml` (incl. sitemap indexes)	`start_url`
`link`	Follow on-page links up to `depth`	`start_url`, `depth`
`pattern`	Match URLs against a glob/regex pattern	`start_url`, `url_pattern`
`batch`	Crawl an explicit list of URLs in parallel	`urls`

When mode is omitted it’s auto-detected: batch if urls is present, otherwise link.

Start a crawl

POST https://api.tazpal.com/v1/crawl

Body

Field	Type	Required	Default	Description
`start_url`	string	mode-dependent	—	Starting URL (sitemap/link/pattern)
`mode`	string	no	auto	`sitemap`, `link`, `pattern`, or `batch`
`max_pages`	integer	no	`100`	Max pages to crawl (1–1000)
`depth`	integer	no	`3`	Link crawl depth (0–10)
`url_pattern`	string	pattern mode	—	Glob/regex URL pattern
`urls`	string[]	batch mode	—	Explicit URL list
`options`	object	no	`{}`	Crawl options (see below)
`callback_url`	string	no	—	Webhook URL for completion
`webhook_secret`	string	no	—	HMAC-SHA256 signing secret

`options` object

Field	Type	Default	Description
`respect_robots_txt`	boolean	`true`	Respect `robots.txt` (incl. `Crawl-delay`)
`delay_ms`	integer	`1000`	Delay between page fetches (0–60000)
`timeout_per_page`	integer	`30000`	Per-page fetch timeout (1000–120000)
`parallel`	integer	`5`	Concurrency for batch mode (1–50)
`follow_external`	boolean	`false`	Follow off-domain links (link mode)
`include_patterns`	string[]	—	Include glob/regex patterns (link mode)
`exclude_patterns`	string[]	—	Exclude glob/regex patterns (link mode)

Example

Sitemap
Batch
With webhook

curl -X POST https://api.tazpal.com/v1/crawl \
  -H "Authorization: Bearer baas_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"start_url": "https://example.com/sitemap.xml", "mode": "sitemap", "max_pages": 500}'

curl -X POST https://api.tazpal.com/v1/crawl \
  -H "Authorization: Bearer baas_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"mode": "batch", "urls": ["https://a.com/1", "https://a.com/2"], "options": {"parallel": 10}}'

curl -X POST https://api.tazpal.com/v1/crawl \
  -H "Authorization: Bearer baas_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "start_url": "https://example.com",
    "mode": "link",
    "callback_url": "https://yourapp.example.com/hooks/baas",
    "webhook_secret": "your_signing_secret"
  }'

Response — 202 Accepted

{
  "job_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "status": "pending",
  "mode": "sitemap",
  "start_url": "https://example.com/sitemap.xml",
  "max_pages": 500,
  "depth": 3,
  "created_at": "2026-08-28T14:30:00Z"
}

Get job status

GET https://api.tazpal.com/v1/crawl/{job_id}

curl https://api.tazpal.com/v1/crawl/9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d \
  -H "Authorization: Bearer baas_live_YOUR_KEY"

Response

{
  "job_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "status": "running",
  "mode": "sitemap",
  "start_url": "https://example.com/sitemap.xml",
  "max_pages": 500,
  "depth": 3,
  "progress": {
    "pages_crawled": 120,
    "pages_total": 300,
    "pages_failed": 2,
    "skipped": 0,
    "current_url": "https://example.com/page/121"
  },
  "result_count": 120,
  "error_count": 2,
  "error": null,
  "callback_url": null,
  "started_at": "2026-08-28T14:30:00Z",
  "completed_at": null,
  "created_at": "2026-08-28T14:30:00Z"
}

Job statuses

Status	Meaning
`pending`	Queued, not yet started
`running`	Discovery or fetching in progress
`completed`	Finished successfully
`failed`	Failed during discovery or execution
`canceled`	Canceled via `DELETE`

Get job results

GET https://api.tazpal.com/v1/crawl/{job_id}/results?limit=100&offset=0

Query	Default	Range
`limit`	`100`	1–1000
`offset`	`0`	≥ 0

curl "https://api.tazpal.com/v1/crawl/9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d/results?limit=50" \
  -H "Authorization: Bearer baas_live_YOUR_KEY"

Response

{
  "job_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "count": 50,
  "results": [
    {
      "id": "1f4e8...",
      "url": "https://example.com/page/1",
      "status": 200,
      "markdown": "# Page 1\n\n...",
      "metadata": {},
      "error": null,
      "crawled_at": "2026-08-28T14:30:05Z"
    }
  ]
}

List all jobs

GET https://api.tazpal.com/v1/crawl

curl https://api.tazpal.com/v1/crawl -H "Authorization: Bearer baas_live_YOUR_KEY"

Cancel a job

DELETE https://api.tazpal.com/v1/crawl/{job_id}

Cancels a pending or running job cooperatively and returns 204 No Content.

curl -X DELETE https://api.tazpal.com/v1/crawl/9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d \
  -H "Authorization: Bearer baas_live_YOUR_KEY"

Errors

Status	`detail.error`	Meaning
`400`	`invalid_mode`	Unknown crawl mode
`400`	`invalid_request`	Missing required field for the chosen mode
`401`	—	Missing/invalid API key
`404`	—	Job not found (or not owned by caller)
`429`	`rate_limited`	Rate limit exceeded

Crawl jobs are scoped to the API key’s owner. You can only list, read, or cancel jobs you created — other users’ job_ids return 404.

Getting Started

Endpoints

Reference

SDKs & Guides

Crawl

Crawl multiple pages

Overview

Crawl modes

Start a crawl

Body

`options` object

Example

Get job status

Job statuses

Get job results

List all jobs

Cancel a job

Errors

​Crawl multiple pages

​Overview

​Crawl modes

​Start a crawl

​Body

​options object

​Example

​Get job status

​Job statuses

​Get job results

​List all jobs

​Cancel a job

​Errors

Crawl multiple pages

Overview

Crawl modes

Start a crawl

Body

`options` object

Example

Get job status

Job statuses

Get job results

List all jobs

Cancel a job

Errors