API reference
Map
Discover every URL on a site without scraping page content.
POST /v1/map
Map combines three sources of URLs, in order:
- Sitemaps —
robots.txtSitemap:directives,/sitemap.xmland/sitemap_index.xml, following nested sitemap indexes. - Link discovery — a bounded breadth-first walk of the seed page and its same-site links.
- Filtering — the same include/exclude, depth and file-type rules the crawler uses.
| Field | Type | Default | Description |
|---|---|---|---|
url | string | — | Site to map. |
limit | number | 5000 | Maximum links returned (1–5000). |
includeSubdomains | boolean | true | Include subdomains of the base URL. |
search | string | — | Case-insensitive substring filter on results. |
ignoreSitemap | boolean | true | Skip sitemap discovery. |
includePaths / excludePaths | string[] | [] | Glob filters, as in /v1/crawl. |
maxDepth | number | 10 | Maximum path depth. |
allowExternalLinks | boolean | false | Allow other domains. |
{
"success": true,
"links": [
"https://example.com/pricing",
"https://example.com/docs",
"https://example.com/blog/hello"
]
}Map returns URLs only — it does not convert pages to markdown. Use /v1/crawl when you need content, or chain map into batch scrapes when you want to choose the pages yourself.