# Web crawl

> Read many pages of one website as markdown, as a long-running job.

`web_crawl` reads many pages of one website: it follows links from the URL and returns each page’s content as markdown. Use it to read a whole docs section or blog; use [`web_map`](https://clonvo.com/docs/web-map.md) first if you only need the URLs, and [`web_scrape`](https://clonvo.com/docs/web-scrape.md) for a single page.

Call it with `POST /v1/tools/web_crawl`, or as `web_crawl` over [MCP](https://clonvo.com/docs/mcp.md). A crawl can take minutes, so it runs as a [job](https://clonvo.com/docs/jobs.md).

## Arguments

-   `url` string Required
    
    Where to start crawling: a site or a section of it.
    
-   `max_pages` integer
    
    The most pages to read (1–100, default 10). Clonvo credits serve up to 20.
    
-   `include_paths` string\[\]
    
    Only crawl URL paths matching these regular expressions, e.g. \["^/docs/"\].
    
-   `exclude_paths` string\[\]
    
    Skip URL paths matching these regular expressions.
    
-   `instructions` string
    
    What to look for, in words, e.g. "pages about authentication". Providers use it to pick pages.
    
-   `max_chars_per_page` integer
    
    The most characters of content per page (default 5,000).
    

## Example

```bash
curl https://api.clonvo.com/v1/tools/web_crawl \
  -H "Authorization: Bearer $CLONVO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "arguments": { "url": "https://www.postgresql.org/docs/18/", "include_paths": ["^/docs/18/"], "max_pages": 2 }
  }'
```

Crawls on Clonvo credits, and on some providers, finish within the request, so this one answers `200` with the finished job. Otherwise the call answers `202` with a running job; check it with `GET /v1/tools/jobs/{id}` until its `status` is final.

```json
{
  "id": "gen-01M6B2K4M6P8R0T2V4X6Z8B0D2",
  "object": "tool.job",
  "tool": "web_crawl",
  "status": "completed",
  "provider": "clonvo",
  "created_at": "2026-11-09T12:00:00.000Z",
  "completed_at": "2026-11-09T12:00:38.000Z",
  "result": {
    "url": "https://www.postgresql.org/docs/18/",
    "pages": [
      {
        "url": "https://www.postgresql.org/docs/18/index.html",
        "title": "PostgreSQL 18 Documentation",
        "content": "# PostgreSQL 18 Documentation\n\n## Table of Contents\n\n- Preface…",
        "truncated": true
      },
      {
        "url": "https://www.postgresql.org/docs/18/tutorial.html",
        "title": "Part I. Tutorial",
        "content": "# Part I. Tutorial\n\nWelcome to the PostgreSQL Tutorial…",
        "truncated": false
      }
    ]
  },
  "usage": { "cost": 0.0096, "charged": 0.009792, "is_byok": false }
}
```

## Result

Once the job is `completed`, its `result` has:

| Field | Description |
| --- | --- |
| `url` | The URL the crawl started from. |
| `pages[]` | The pages read, each with `url`, `title` (or `null`), `content` as markdown, and `truncated`, which is `true` when the page was longer than `max_chars_per_page` and was cut. |
| `notes` | Present when the provider handled an argument differently. |

The job’s other fields are described in [Jobs](https://clonvo.com/docs/jobs.md#the-job-object).

## Notes

-   Clonvo credits serve crawls of up to 20 pages. A larger `max_pages` needs your own key for a provider that crawls; see [Bring your own tool keys](https://clonvo.com/docs/tool-keys.md).
-   A crawl can return fewer pages than `max_pages` when the site has fewer pages within reach of the URL.
-   `instructions` is free text you wrote, so it’s never stored as a [web tool query](https://clonvo.com/docs/privacy.md#web-tool-queries); providers use it to pick pages.
-   On Clonvo credits, the cap for the call grows with `max_pages`; see [Tool billing](https://clonvo.com/docs/tool-billing.md#on-clonvo-credits).