LLMCrawl Documentation

Appearance

Quickstart ​

LLMCrawl empowers you to convert entire websites into markdown format ready for Large Language Models (LLM).

Welcome to LLMCrawl ​

LLMCrawl is an API service designed to crawl a given URL and transform the content into clean markdown. It thoroughly navigates all accessible subpages, providing you with well-structured markdown for each, eliminating the need for a pre-existing sitemap.

Features ​

How to Use LLMCrawl ​

We offer a robust and user-friendly API. Our hosted version comes with comprehensive documentation and a playground for you to experiment with. Additionally, you can opt to self-host the backend.

  • API Documentation

Obtaining an API Key ​

To access the API, register on LLMCrawl and obtain an API key. This key will authenticate your requests.

Scraping a Single URL ​

To scrape content from a single URL and obtain the data as a dictionary:

cURL ​

To initiate a crawl using cURL:

bash
curl -G https://llmcrawl.dev/api/v1/scrape \
-H "Authorization: Bearer {token}" \
-d '{"url": "https://example.com"}'

This will return the crawl status and results:

json
{
  "success": true,
  "data": {
    "markdown": "# Markdown Content",
    "metadata": {
      "title": "Page Title",
      <...>
    }
  }
}