Webhooks

Webhooks push results to you instead of making you poll. Give llmcrawl a URL and it POSTs a JSON payload when the work is done — useful for long crawls, background pipelines and notifications.

Subscribe to events

Two ways, and you can use both:

• Saved endpoints — create endpoints in the dashboard under Webhooks. Each endpoint has a name, a payload URL and an event subscription; an endpoint with no events selected receives every event.
• Per request — pass webhookUrls (and optional metadata) on POST /v1/scrape or POST /v1/crawl. This delivers that request's result to the given URLs without creating a saved endpoint.

Events currently delivered:

The dashboard can also send a test delivery to an endpoint so you can verify your receiver before relying on it.

Payloads

Every delivery is a POST with content-type: application/json. A scrape delivery carries the finished document:

{
  "success": true,
  "event": "scrape.completed",
  "data": {
    "markdown": "...",
    "metadata": { "sourceURL": "https://example.com", "statusCode": 200 }
  },
  "metadata": { "orderId": "A-1024" }
}

A crawl delivery carries the summary and the documents collected so far (up to the first 200):

{
  "success": true,
  "event": "crawl.completed",
  "data": {
    "type": "crawl.completed",
    "crawlId": "crawl_9f2c...",
    "status": "completed",
    "completed": 34,
    "total": 34,
    "data": [{ "markdown": "...", "metadata": { "sourceURL": "https://example.com/about" } }]
  },
  "metadata": { "run": "nightly" }
}

metadata is whatever you passed as metadata on a scrape or webhookMetadata on a crawl — use it to correlate deliveries with your own jobs.

Delivery and retries

• Deliveries are attempted up to 3 times, with a short backoff between attempts, and each attempt times out after 10 seconds.
• Return any 2xx status to acknowledge a delivery. Anything else is retried and then recorded as failed.
• A failing webhook never fails the scrape or crawl that triggered it. Scrape responses carry a warning when a delivery could not be made, and every attempt — status code, attempts and failures — shows up in the dashboard's delivery log.

Local testing

To try a receiver locally, expose it with a tunnel and add the public URL as an endpoint, then hit Send test delivery in the dashboard. The delivery log shows what was sent and how your endpoint responded.