---
url: /api-reference.md
---


---

---
url: /examples.md
---

# Examples & Use Cases

Explore practical examples of using LLMCrawl for various scraping and data extraction tasks.

## E-commerce & Product Data

### Product Information Extraction

Extract structured product data from e-commerce websites:

```typescript
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";

const client = new LLMCrawl({ apiKey: "your-api-key" });

const productSchema = {
  type: "object",
  properties: {
    name: { type: "string" },
    price: { type: "number" },
    originalPrice: { type: "number" },
    discount: { type: "number" },
    rating: { type: "number" },
    reviewCount: { type: "number" },
    inStock: { type: "boolean" },
    description: { type: "string" },
    specifications: {
      type: "object",
      properties: {
        brand: { type: "string" },
        model: { type: "string" },
        color: { type: "string" },
        size: { type: "string" },
      },
    },
    images: {
      type: "array",
      items: { type: "string" },
    },
  },
  required: ["name", "price", "inStock"],
};

const result = await client.scrape("https://store.example.com/product/123", {
  formats: ["markdown"],
  extract: { schema: productSchema },
});

if (result.success) {
  const product = JSON.parse(result.data.extract);
  console.log(`Product: ${product.name}`);
  console.log(`Price: $${product.price}`);
  console.log(`In Stock: ${product.inStock}`);
}
```

### Price Monitoring

Monitor product prices across multiple websites:

```typescript
interface Product {
  name: string;
  price: number;
  url: string;
  timestamp: Date;
}

async function monitorPrices(productUrls: string[]): Promise<Product[]> {
  const priceSchema = {
    type: "object",
    properties: {
      name: { type: "string" },
      price: { type: "number" },
      currency: { type: "string" },
    },
    required: ["name", "price"],
  };

  const products: Product[] = [];

  for (const url of productUrls) {
    try {
      const result = await client.scrape(url, {
        extract: { schema: priceSchema },
      });

      if (result.success && result.data.extract) {
        const data = JSON.parse(result.data.extract);
        products.push({
          name: data.name,
          price: data.price,
          url,
          timestamp: new Date(),
        });
      }
    } catch (error) {
      console.error(`Failed to scrape ${url}:`, error);
    }

    // Rate limiting
    await new Promise((resolve) => setTimeout(resolve, 1000));
  }

  return products;
}

// Usage
const productUrls = [
  "https://store1.example.com/product/123",
  "https://store2.example.com/item/456",
  "https://store3.example.com/product/789",
];

const prices = await monitorPrices(productUrls);
console.log("Current prices:", prices);
```

## News & Content Aggregation

### News Article Extraction

Extract structured data from news articles:

```typescript
const articleSchema = {
  type: "object",
  properties: {
    headline: { type: "string" },
    subheadline: { type: "string" },
    author: { type: "string" },
    publishDate: { type: "string" },
    content: { type: "string" },
    tags: { type: "array", items: { type: "string" } },
    category: { type: "string" },
    readTime: { type: "number" },
    relatedArticles: {
      type: "array",
      items: {
        type: "object",
        properties: {
          title: { type: "string" },
          url: { type: "string" },
        },
      },
    },
  },
  required: ["headline", "content"],
};

const newsResult = await client.scrape("https://news.example.com/article/123", {
  formats: ["markdown"],
  extract: { schema: articleSchema },
});

if (newsResult.success) {
  const article = JSON.parse(newsResult.data.extract);
  console.log(`Article: ${article.headline}`);
  console.log(`Author: ${article.author}`);
  console.log(`Published: ${article.publishDate}`);
}
```

### Blog Content Crawling

Crawl and extract content from blog sites:

```typescript
const blogCrawl = await client.crawl("https://blog.example.com", {
  limit: 200,
  includePaths: ["/posts/*", "/articles/*"],
  excludePaths: ["/admin/*", "/author/*"],
  scrapeOptions: {
    formats: ["markdown"],
    extract: {
      schema: {
        type: "object",
        properties: {
          title: { type: "string" },
          author: { type: "string" },
          publishDate: { type: "string" },
          content: { type: "string" },
          tags: { type: "array", items: { type: "string" } },
          summary: { type: "string" },
        },
      },
    },
  },
});

// Monitor progress
if (blogCrawl.success) {
  let status = await client.getCrawlStatus(blogCrawl.id);

  while (status.success && status.status === "scraping") {
    console.log(`Progress: ${status.completed}/${status.total} articles`);
    await new Promise((resolve) => setTimeout(resolve, 10000));
    status = await client.getCrawlStatus(blogCrawl.id);
  }

  if (status.success && status.status === "completed") {
    console.log(`Successfully crawled ${status.data.length} articles`);

    // Process articles
    const articles = status.data
      .filter((page) => page.extract)
      .map((page) => JSON.parse(page.extract));

    // Create content database
    const contentDB = articles.map((article, index) => ({
      id: index + 1,
      title: article.title,
      author: article.author,
      publishDate: article.publishDate,
      wordCount: article.content.split(" ").length,
      tags: article.tags || [],
    }));

    console.log("Content database created:", contentDB.length, "entries");
  }
}
```

## Documentation & Knowledge Base

### API Documentation Scraping

Extract API documentation with code examples:

```typescript
const apiDocSchema = {
  type: "object",
  properties: {
    title: { type: "string" },
    description: { type: "string" },
    endpoint: { type: "string" },
    method: { type: "string" },
    parameters: {
      type: "array",
      items: {
        type: "object",
        properties: {
          name: { type: "string" },
          type: { type: "string" },
          required: { type: "boolean" },
          description: { type: "string" },
        },
      },
    },
    responseExample: { type: "string" },
    codeExamples: {
      type: "array",
      items: {
        type: "object",
        properties: {
          language: { type: "string" },
          code: { type: "string" },
        },
      },
    },
  },
};

const docsCrawl = await client.crawl("https://docs.api.example.com", {
  limit: 500,
  includePaths: ["/reference/*", "/endpoints/*"],
  scrapeOptions: {
    formats: ["markdown"],
    extract: { schema: apiDocSchema },
  },
});
```

### Knowledge Base Creation

Build a searchable knowledge base from documentation:

```typescript
interface KnowledgeEntry {
  id: string;
  title: string;
  content: string;
  url: string;
  section: string;
  keywords: string[];
}

async function buildKnowledgeBase(baseUrl: string): Promise<KnowledgeEntry[]> {
  const crawl = await client.crawl(baseUrl, {
    limit: 1000,
    includePaths: ["/docs/*", "/guides/*", "/tutorials/*"],
    scrapeOptions: {
      formats: ["markdown"],
      extract: {
        schema: {
          type: "object",
          properties: {
            title: { type: "string" },
            section: { type: "string" },
            content: { type: "string" },
            keywords: { type: "array", items: { type: "string" } },
          },
        },
      },
    },
  });

  if (!crawl.success) return [];

  // Wait for completion
  let status = await client.getCrawlStatus(crawl.id);
  while (status.success && status.status === "scraping") {
    await new Promise((resolve) => setTimeout(resolve, 5000));
    status = await client.getCrawlStatus(crawl.id);
  }

  if (!status.success || status.status !== "completed") return [];

  // Process results
  const knowledgeBase: KnowledgeEntry[] = status.data
    .filter((page) => page.extract && page.markdown)
    .map((page, index) => {
      const extracted = JSON.parse(page.extract);
      return {
        id: `kb_${index + 1}`,
        title: extracted.title || page.metadata?.title || "Untitled",
        content: page.markdown,
        url: page.metadata?.url || "",
        section: extracted.section || "General",
        keywords: extracted.keywords || [],
      };
    });

  return knowledgeBase;
}

// Usage
const kb = await buildKnowledgeBase("https://docs.myapp.com");
console.log(`Knowledge base created with ${kb.length} entries`);
```

## Real Estate & Property Data

### Property Listing Extraction

Extract property details from real estate websites:

```typescript
const propertySchema = {
  type: "object",
  properties: {
    title: { type: "string" },
    price: { type: "number" },
    address: { type: "string" },
    bedrooms: { type: "number" },
    bathrooms: { type: "number" },
    sqft: { type: "number" },
    propertyType: { type: "string" },
    description: { type: "string" },
    features: { type: "array", items: { type: "string" } },
    images: { type: "array", items: { type: "string" } },
    agent: {
      type: "object",
      properties: {
        name: { type: "string" },
        phone: { type: "string" },
        email: { type: "string" },
      },
    },
  },
  required: ["title", "price", "address"],
};

const property = await client.scrape("https://realty.example.com/listing/123", {
  formats: ["markdown"],
  extract: { schema: propertySchema },
});
```

### Market Analysis

Analyze property market trends:

```typescript
async function analyzeMarket(searchUrls: string[]) {
  const properties = [];

  for (const url of searchUrls) {
    const result = await client.scrape(url, {
      extract: { schema: propertySchema },
    });

    if (result.success && result.data.extract) {
      const property = JSON.parse(result.data.extract);
      properties.push(property);
    }

    await new Promise((resolve) => setTimeout(resolve, 2000));
  }

  // Calculate market metrics
  const prices = properties.map((p) => p.price).filter((p) => p > 0);
  const avgPrice = prices.reduce((a, b) => a + b, 0) / prices.length;
  const medianPrice = prices.sort((a, b) => a - b)[
    Math.floor(prices.length / 2)
  ];

  return {
    totalProperties: properties.length,
    averagePrice: avgPrice,
    medianPrice: medianPrice,
    priceRange: {
      min: Math.min(...prices),
      max: Math.max(...prices),
    },
    propertyTypes: [...new Set(properties.map((p) => p.propertyType))],
  };
}
```

## Job & Career Data

### Job Listing Aggregation

Extract job postings from career sites:

```typescript
const jobSchema = {
  type: "object",
  properties: {
    title: { type: "string" },
    company: { type: "string" },
    location: { type: "string" },
    salary: { type: "string" },
    type: { type: "string" }, // full-time, part-time, contract
    remote: { type: "boolean" },
    description: { type: "string" },
    requirements: { type: "array", items: { type: "string" } },
    benefits: { type: "array", items: { type: "string" } },
    postedDate: { type: "string" },
    applicationUrl: { type: "string" },
  },
  required: ["title", "company"],
};

// Crawl job boards
const jobsCrawl = await client.crawl("https://jobs.example.com", {
  limit: 1000,
  includePaths: ["/jobs/*", "/careers/*"],
  excludePaths: ["/apply/*", "/profile/*"],
  scrapeOptions: {
    formats: ["markdown"],
    extract: { schema: jobSchema },
  },
});
```

## Social Media & Reviews

### Review Extraction

Extract customer reviews and ratings:

```typescript
const reviewSchema = {
  type: "object",
  properties: {
    rating: { type: "number" },
    title: { type: "string" },
    content: { type: "string" },
    author: { type: "string" },
    date: { type: "string" },
    verified: { type: "boolean" },
    helpful: { type: "number" },
    product: { type: "string" },
  },
};

const reviews = await client.scrape("https://reviews.example.com/product/123", {
  extract: { schema: reviewSchema },
});
```

## Financial Data

### Stock & Financial Information

Extract financial data from company pages:

```typescript
const financialSchema = {
  type: "object",
  properties: {
    symbol: { type: "string" },
    companyName: { type: "string" },
    currentPrice: { type: "number" },
    change: { type: "number" },
    changePercent: { type: "number" },
    volume: { type: "number" },
    marketCap: { type: "string" },
    peRatio: { type: "number" },
    dividendYield: { type: "number" },
    earningsDate: { type: "string" },
  },
};

const stockData = await client.scrape(
  "https://finance.example.com/stock/AAPL",
  {
    extract: { schema: financialSchema },
  }
);
```

## Advanced Patterns

### Retry Logic with Exponential Backoff

```typescript
async function scrapeWithRetry(
  url: string,
  options: any,
  maxRetries = 3
): Promise<any> {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const result = await client.scrape(url, options);
      if (result.success) return result;

      if (attempt < maxRetries) {
        const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
        await new Promise((resolve) => setTimeout(resolve, delay));
      }
    } catch (error) {
      if (attempt === maxRetries) throw error;
      const delay = Math.pow(2, attempt) * 1000;
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
}
```

### Batch Processing with Concurrency Control

```typescript
async function batchScrape(urls: string[], concurrency = 5) {
  const results = [];

  for (let i = 0; i < urls.length; i += concurrency) {
    const batch = urls.slice(i, i + concurrency);
    const batchResults = await Promise.allSettled(
      batch.map((url) => client.scrape(url))
    );

    results.push(...batchResults);

    // Rate limiting between batches
    if (i + concurrency < urls.length) {
      await new Promise((resolve) => setTimeout(resolve, 1000));
    }
  }

  return results;
}
```

### Data Validation & Cleaning

```typescript
import Ajv from "ajv";

const ajv = new Ajv();

function validateAndCleanData(data: any, schema: any) {
  const validate = ajv.compile(schema);
  const valid = validate(data);

  if (!valid) {
    console.warn("Validation errors:", validate.errors);
    // Attempt to clean/fix data
    return cleanData(data, validate.errors);
  }

  return data;
}

function cleanData(data: any, errors: any[]) {
  // Implement data cleaning logic based on validation errors
  const cleaned = { ...data };

  errors.forEach((error) => {
    if (error.keyword === "type" && error.params.type === "number") {
      const path = error.instancePath.replace("/", "");
      if (typeof cleaned[path] === "string") {
        const num = parseFloat(cleaned[path].replace(/[^0-9.-]/g, ""));
        if (!isNaN(num)) cleaned[path] = num;
      }
    }
  });

  return cleaned;
}
```

These examples demonstrate the versatility of LLMCrawl for various data extraction and web scraping use cases. The AI-powered extraction feature makes it easy to transform unstructured web content into structured, usable data for your applications.

---

---
url: /introduction.md
---


---

---
url: /sdk-javascript.md
---

# JavaScript/TypeScript SDK

[![npm version](https://badge.fury.io/js/@llmcrawl%2Fllmcrawl-js.svg)](https://badge.fury.io/js/@llmcrawl%2Fllmcrawl-js)

The official JavaScript SDK for LLMCrawl provides a simple and powerful way to scrape websites, crawl multiple pages, and extract structured data using AI from your JavaScript or TypeScript applications.

## Installation

Install the SDK using npm or your preferred package manager:

```bash
npm install @llmcrawl/llmcrawl-js
```

```bash
yarn add @llmcrawl/llmcrawl-js
```

```bash
pnpm add @llmcrawl/llmcrawl-js
```

## Quick Start

```typescript
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";

const client = new LLMCrawl({
  apiKey: "your-api-key-here",
});

// Scrape a single page
const result = await client.scrape("https://example.com");
console.log(result.data?.markdown);
```

## Authentication

Initialize the client with your API key:

```typescript
const client = new LLMCrawl({
  apiKey: "your-api-key",
  baseUrl: "https://api.llmcrawl.dev", // Optional custom base URL
});
```

You can obtain an API key from your [LLMCrawl Dashboard](https://llmcrawl.dev/dashboard/api-keys).

## Core Features

### 🌐 Single Page Scraping

Extract content from individual web pages with multiple format options.

### 🕷️ Website Crawling

Crawl entire websites with customizable depth and filtering options.

### 🗺️ Site Mapping

Get all URLs from a website without scraping content.

### 🤖 AI-Powered Extraction

Extract structured data using custom JSON schemas and AI.

### 📷 Screenshot Capture

Take screenshots of web pages during scraping.

### ⚙️ Flexible Configuration

Extensive customization options for headers, timeouts, and more.

## API Reference

### Scraping Single Pages

The `scrape()` method extracts content from a single webpage:

```typescript
const result = await client.scrape("https://example.com", {
  formats: ["markdown", "html", "links"],
  headers: {
    "User-Agent": "Mozilla/5.0 (compatible; LLMCrawl)",
  },
  waitFor: 3000, // Wait 3 seconds for page to load
  timeout: 30000, // 30 second timeout
  extract: {
    schema: {
      type: "object",
      properties: {
        title: { type: "string" },
        price: { type: "number" },
        description: { type: "string" },
      },
      required: ["title", "price"],
    },
  },
});

if (result.success) {
  console.log("Markdown:", result.data.markdown);
  console.log("Extracted data:", result.data.extract);
  console.log("Links:", result.data.links);
}
```

#### Scrape Options

| Option        | Type             | Description                                                                                             |
| ------------- | ---------------- | ------------------------------------------------------------------------------------------------------- |
| `formats`     | `string[]`       | Output formats: `'markdown'`, `'html'`, `'rawHtml'`, `'links'`, `'screenshot'`, `'screenshot@fullPage'` |
| `headers`     | `object`         | Custom HTTP headers                                                                                     |
| `includeTags` | `string[]`       | HTML tags to include in output                                                                          |
| `excludeTags` | `string[]`       | HTML tags to exclude from output                                                                        |
| `timeout`     | `number`         | Request timeout in milliseconds (1000-90000)                                                            |
| `waitFor`     | `number`         | Delay before capturing content (0-60000ms)                                                              |
| `extract`     | `ExtractOptions` | AI extraction configuration                                                                             |
| `webhookUrls` | `string[]`       | URLs to send results to                                                                                 |
| `metadata`    | `object`         | Additional metadata to include                                                                          |

### Website Crawling

Start a crawl job to scrape multiple pages:

```typescript
const crawlResult = await client.crawl("https://example.com", {
  limit: 100,
  maxDepth: 3,
  includePaths: ["/blog/*", "/docs/*"],
  excludePaths: ["/admin/*", "/login"],
  allowBackwardLinks: false,
  allowExternalLinks: false,
  scrapeOptions: {
    formats: ["markdown"],
    extract: {
      schema: {
        type: "object",
        properties: {
          title: { type: "string" },
          content: { type: "string" },
          author: { type: "string" },
        },
      },
    },
  },
});

if (crawlResult.success) {
  console.log("Crawl started with ID:", crawlResult.id);

  // Monitor crawl progress
  const status = await client.getCrawlStatus(crawlResult.id);
  console.log(`Progress: ${status.completed}/${status.total}`);
}
```

#### Crawl Options

| Option               | Type            | Description                                   |
| -------------------- | --------------- | --------------------------------------------- |
| `limit`              | `number`        | Maximum number of pages to crawl              |
| `maxDepth`           | `number`        | Maximum crawl depth                           |
| `includePaths`       | `string[]`      | Path patterns to include (supports wildcards) |
| `excludePaths`       | `string[]`      | Path patterns to exclude                      |
| `allowBackwardLinks` | `boolean`       | Allow crawling backward links                 |
| `allowExternalLinks` | `boolean`       | Allow crawling external domains               |
| `ignoreSitemap`      | `boolean`       | Ignore robots.txt and sitemap.xml             |
| `scrapeOptions`      | `ScrapeOptions` | Scraping options for each page                |
| `webhookUrls`        | `string[]`      | URLs for webhook notifications                |
| `webhookMetadata`    | `object`        | Additional webhook metadata                   |

### Monitoring Crawl Jobs

Check the status of a running crawl:

```typescript
const status = await client.getCrawlStatus("crawl-job-id");

if (status.success) {
  console.log("Status:", status.status); // 'scraping', 'completed', 'failed'
  console.log("Progress:", `${status.completed}/${status.total}`);

  if (status.status === "completed") {
    console.log("Scraped pages:", status.data.length);
    status.data.forEach((page, index) => {
      console.log(`Page ${index + 1}:`, page.metadata?.title);
    });
  }
}
```

Cancel a running crawl:

```typescript
const result = await client.cancelCrawl("crawl-job-id");

if (result.success) {
  console.log("Crawl cancelled:", result.message);
}
```

### Site Mapping

Get all URLs from a website without scraping content:

```typescript
const mapResult = await client.map("https://example.com", {
  limit: 1000,
  includeSubdomains: true,
  search: "documentation",
  includePaths: ["/docs/*", "/api/*"],
  excludePaths: ["/internal/*"],
});

if (mapResult.success) {
  console.log(`Found ${mapResult.links.length} URLs`);
  mapResult.links.forEach((link) => {
    console.log(link);
  });
}
```

#### Map Options

| Option              | Type       | Description                                |
| ------------------- | ---------- | ------------------------------------------ |
| `limit`             | `number`   | Maximum number of links to return (1-5000) |
| `includeSubdomains` | `boolean`  | Include subdomain URLs                     |
| `search`            | `string`   | Filter links by search query               |
| `ignoreSitemap`     | `boolean`  | Ignore robots.txt and sitemap.xml          |
| `includePaths`      | `string[]` | Path patterns to include                   |
| `excludePaths`      | `string[]` | Path patterns to exclude                   |

## AI-Powered Data Extraction

LLMCrawl's most powerful feature is AI-powered structured data extraction using custom JSON schemas:

### E-commerce Example

```typescript
const result = await client.scrape("https://store.example.com/product/123", {
  formats: ["markdown"],
  extract: {
    mode: "llm",
    schema: {
      type: "object",
      properties: {
        productName: { type: "string" },
        price: { type: "number" },
        originalPrice: { type: "number" },
        discount: { type: "number" },
        inStock: { type: "boolean" },
        rating: { type: "number" },
        reviewCount: { type: "number" },
        description: { type: "string" },
        specifications: {
          type: "object",
          properties: {
            color: { type: "string" },
            size: { type: "string" },
            brand: { type: "string" },
          },
        },
        images: {
          type: "array",
          items: { type: "string" },
        },
      },
      required: ["productName", "price", "inStock"],
    },
    systemPrompt: "Extract product information from this e-commerce page.",
    prompt: "Focus on getting accurate pricing and availability information.",
  },
});

if (result.success && result.data.extract) {
  const product = JSON.parse(result.data.extract);
  console.log("Product:", product.productName);
  console.log("Price:", product.price);
  console.log("In Stock:", product.inStock);
}
```

### News Article Example

```typescript
const article = await client.scrape("https://news.example.com/article/123", {
  formats: ["markdown"],
  extract: {
    schema: {
      type: "object",
      properties: {
        headline: { type: "string" },
        subheadline: { type: "string" },
        author: { type: "string" },
        publishDate: { type: "string", format: "date-time" },
        content: { type: "string" },
        tags: {
          type: "array",
          items: { type: "string" },
        },
        category: { type: "string" },
        readTime: { type: "number" },
        relatedArticles: {
          type: "array",
          items: {
            type: "object",
            properties: {
              title: { type: "string" },
              url: { type: "string" },
            },
          },
        },
      },
      required: ["headline", "content", "publishDate"],
    },
  },
});
```

### Complex Schema Example

```typescript
const schema = {
  type: "object",
  properties: {
    company: {
      type: "object",
      properties: {
        name: { type: "string" },
        description: { type: "string" },
        founded: { type: "string" },
        employees: { type: "number" },
        location: {
          type: "object",
          properties: {
            city: { type: "string" },
            country: { type: "string" },
            address: { type: "string" },
          },
        },
      },
    },
    leadership: {
      type: "array",
      items: {
        type: "object",
        properties: {
          name: { type: "string" },
          position: { type: "string" },
          bio: { type: "string" },
        },
      },
    },
    products: {
      type: "array",
      items: {
        type: "object",
        properties: {
          name: { type: "string" },
          description: { type: "string" },
          price: { type: "number" },
        },
      },
    },
  },
};
```

## Advanced Examples

### Crawling Documentation Sites

```typescript
// Start crawling documentation
const crawl = await client.crawl("https://docs.example.com", {
  limit: 500,
  maxDepth: 4,
  includePaths: ["/docs/*", "/api/*", "/guides/*"],
  excludePaths: ["/docs/internal/*", "/admin/*"],
  scrapeOptions: {
    formats: ["markdown"],
    excludeTags: ["nav", "footer", "aside"],
    extract: {
      schema: {
        type: "object",
        properties: {
          title: { type: "string" },
          section: { type: "string" },
          content: { type: "string" },
          codeExamples: {
            type: "array",
            items: {
              type: "object",
              properties: {
                language: { type: "string" },
                code: { type: "string" },
              },
            },
          },
        },
      },
    },
  },
});

// Poll for completion
if (crawl.success) {
  let status = await client.getCrawlStatus(crawl.id);

  while (status.success && status.status === "scraping") {
    console.log(`Progress: ${status.completed}/${status.total} pages`);
    await new Promise((resolve) => setTimeout(resolve, 5000));
    status = await client.getCrawlStatus(crawl.id);
  }

  if (status.success && status.status === "completed") {
    console.log("Documentation crawl completed!");
    console.log(`Total pages: ${status.data.length}`);

    // Process the results
    status.data.forEach((page) => {
      if (page.extract) {
        const pageData = JSON.parse(page.extract);
        console.log(`Section: ${pageData.section}`);
        console.log(`Title: ${pageData.title}`);
      }
    });
  }
}
```

### Batch Processing with Custom Headers

```typescript
const urls = [
  "https://example.com/page1",
  "https://example.com/page2",
  "https://example.com/page3",
];

const customHeaders = {
  "User-Agent": "Mozilla/5.0 (compatible; LLMCrawl)",
  "Accept-Language": "en-US,en;q=0.9",
};

const results = await Promise.all(
  urls.map(async (url) => {
    try {
      const result = await client.scrape(url, {
        formats: ["markdown", "links"],
        headers: customHeaders,
        timeout: 30000,
        waitFor: 2000,
      });
      return { url, success: true, data: result.data };
    } catch (error) {
      return { url, success: false, error: error.message };
    }
  })
);

results.forEach((result) => {
  if (result.success) {
    console.log(`✅ ${result.url}: ${result.data?.markdown?.length} chars`);
  } else {
    console.log(`❌ ${result.url}: ${result.error}`);
  }
});
```

### Screenshot Capture

```typescript
const result = await client.scrape("https://example.com", {
  formats: ["screenshot@fullPage", "markdown"],
  waitFor: 3000, // Wait for page to fully load
});

if (result.success && result.data.screenshot) {
  // Screenshot is returned as base64 encoded string
  const screenshotBuffer = Buffer.from(result.data.screenshot, "base64");

  // Save to file (Node.js)
  await fs.writeFile("screenshot.png", screenshotBuffer);

  // Or create download link (Browser)
  const blob = new Blob([screenshotBuffer], { type: "image/png" });
  const url = URL.createObjectURL(blob);
}
```

## Error Handling

All SDK methods return a response object with a `success` field for consistent error handling:

```typescript
const result = await client.scrape("https://example.com");

if (result.success) {
  // Handle successful response
  console.log("Content:", result.data.markdown);
  console.log("Metadata:", result.data.metadata);
} else {
  // Handle error
  console.error("Error:", result.error);
  console.error("Details:", result.details);

  // Common error types:
  // - Authentication errors (invalid API key)
  // - Rate limiting
  // - Network timeouts
  // - Invalid URLs
  // - Server errors
}
```

### Error Types

| Error Type        | Description                        |
| ----------------- | ---------------------------------- |
| `Authentication`  | Invalid or missing API key         |
| `RateLimit`       | Too many requests, try again later |
| `Timeout`         | Request exceeded timeout limit     |
| `InvalidURL`      | Malformed or unreachable URL       |
| `ServerError`     | Internal server error              |
| `ValidationError` | Invalid parameters or schema       |

## Type Definitions

The SDK includes comprehensive TypeScript types for better development experience:

```typescript
import type {
  // Response types
  ScrapeResponse,
  CrawlResponse,
  CrawlStatusResponse,
  MapResponse,

  // Data types
  Document,
  ExtractOptions,
  ScrapeOptions,
  CrawlerOptions,
  MapOptions,

  // Configuration types
  LLMCrawlConfig,
} from "@llmcrawl/llmcrawl-js";

// Example usage with types
const scrapeOptions: ScrapeOptions = {
  formats: ["markdown", "html"],
  timeout: 30000,
  extract: {
    schema: {
      type: "object",
      properties: {
        title: { type: "string" },
      },
    },
  },
};

const result: ScrapeResponse = await client.scrape(url, scrapeOptions);
```

## Environment-Specific Usage

### Node.js

```typescript
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";
import fs from "fs/promises";

const client = new LLMCrawl({
  apiKey: process.env.LLMCRAWL_API_KEY,
});

const result = await client.scrape("https://example.com");
if (result.success) {
  await fs.writeFile("output.md", result.data.markdown);
}
```

### Browser/React

```typescript
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";

function MyComponent() {
  const [content, setContent] = useState("");

  const handleScrape = async () => {
    const client = new LLMCrawl({
      apiKey: process.env.REACT_APP_LLMCRAWL_API_KEY,
    });

    const result = await client.scrape("https://example.com");
    if (result.success) {
      setContent(result.data.markdown);
    }
  };

  return (
    <div>
      <button onClick={handleScrape}>Scrape Website</button>
      <pre>{content}</pre>
    </div>
  );
}
```

### Next.js API Route

```typescript
// pages/api/scrape.ts
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";
import type { NextApiRequest, NextApiResponse } from "next";

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  if (req.method !== "POST") {
    return res.status(405).json({ error: "Method not allowed" });
  }

  const { url } = req.body;

  const client = new LLMCrawl({
    apiKey: process.env.LLMCRAWL_API_KEY!,
  });

  try {
    const result = await client.scrape(url, {
      formats: ["markdown"],
    });

    if (result.success) {
      res.status(200).json({ content: result.data.markdown });
    } else {
      res.status(400).json({ error: result.error });
    }
  } catch (error) {
    res.status(500).json({ error: "Internal server error" });
  }
}
```

## Best Practices

### 1. Rate Limiting

```typescript
// Implement client-side rate limiting
const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

async function scrapeWithRateLimit(urls: string[]) {
  const results = [];

  for (const url of urls) {
    const result = await client.scrape(url);
    results.push(result);

    // Wait 1 second between requests
    await delay(1000);
  }

  return results;
}
```

### 2. Error Recovery

```typescript
async function scrapeWithRetry(url: string, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const result = await client.scrape(url);
      if (result.success) return result;

      if (attempt < maxRetries) {
        await delay(1000 * attempt); // Exponential backoff
      }
    } catch (error) {
      if (attempt === maxRetries) throw error;
      await delay(1000 * attempt);
    }
  }
}
```

### 3. Schema Validation

```typescript
import Ajv from "ajv";

const ajv = new Ajv();

const productSchema = {
  type: "object",
  properties: {
    name: { type: "string" },
    price: { type: "number" },
  },
  required: ["name", "price"],
};

const validate = ajv.compile(productSchema);

const result = await client.scrape(url, {
  extract: { schema: productSchema },
});

if (result.success && result.data.extract) {
  const data = JSON.parse(result.data.extract);
  if (validate(data)) {
    console.log("Valid product data:", data);
  } else {
    console.error("Invalid data:", validate.errors);
  }
}
```

## Migration Guide

### From v0.x to v1.0.0

**Before (v0.x):**

```typescript
import LLMCrawl from "@llmcrawl/llmcrawl-js";

const client = new LLMCrawl("your-api-key");
```

**After (v1.0.0):**

```typescript
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";

const client = new LLMCrawl({
  apiKey: "your-api-key",
});
```

## Support and Resources

* 📧 **Email Support**: <contact@llmcrawl.dev>
* 📚 **Documentation**: <https://docs.llmcrawl.dev>
* 🎮 **Playground**: <https://llmcrawl.dev/tools>
* 🐛 **Issues**: [GitHub Issues](https://github.com/LLMCrawl/llmcrawl-js/issues)
* 💬 **Community**: [Discord Server](https://discord.gg/llmcrawl)

## License

MIT License - see [LICENSE](https://github.com/LLMCrawl/llmcrawl-js/blob/main/LICENSE) file for details.

---

---
url: /sdk-python.md
---

# Python SDK (Coming Soon)

We're actively working on an official Python SDK for LLMCrawl. It will provide the same powerful features as our JavaScript SDK with Pythonic APIs.

## Expected Features

* **Simple Integration**: Easy-to-use Python classes and methods
* **Type Hints**: Full typing support for better IDE experience
* **Async Support**: Both synchronous and asynchronous APIs
* **AI Extraction**: Structured data extraction with Pydantic models
* **Comprehensive**: Full API coverage including scraping, crawling, and mapping

## Preview API

```python
# Expected API design (subject to change)
from llmcrawl import LLMCrawl

client = LLMCrawl(api_key="your-api-key")

# Scrape a single page
result = await client.scrape("https://example.com")
print(result.data.markdown)

# AI-powered extraction with Pydantic
from pydantic import BaseModel

class Product(BaseModel):
    name: str
    price: float
    in_stock: bool

result = await client.scrape(
    "https://store.example.com/product/123",
    extract_model=Product
)
product = result.data.extract  # Type: Product
```

## Current Alternative: REST API

While we work on the official Python SDK, you can use our REST API directly:

```python
import requests
import json

class LLMCrawlClient:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.base_url = "https://api.llmcrawl.dev/v1"
        self.headers = {
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        }

    def scrape(self, url: str, **options):
        data = {"url": url, **options}
        response = requests.post(
            f"{self.base_url}/scrape",
            headers=self.headers,
            json=data
        )
        return response.json()

    def crawl(self, url: str, **options):
        data = {"url": url, **options}
        response = requests.post(
            f"{self.base_url}/crawl",
            headers=self.headers,
            json=data
        )
        return response.json()

    def get_crawl_status(self, job_id: str):
        response = requests.get(
            f"{self.base_url}/crawl/{job_id}",
            headers=self.headers
        )
        return response.json()

# Usage
client = LLMCrawlClient("your-api-key")

# Scrape with AI extraction
result = client.scrape(
    "https://example.com",
    extract={
        "schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "content": {"type": "string"}
            }
        }
    }
)

if result["success"]:
    extracted_data = json.loads(result["data"]["extract"])
    print(f"Title: {extracted_data['title']}")
```

## Using with Popular Python Libraries

### With Pydantic for Type Safety

```python
from pydantic import BaseModel
from typing import List, Optional
import json

class Article(BaseModel):
    title: str
    author: str
    content: str
    tags: List[str]
    publish_date: Optional[str] = None

# Define extraction schema
article_schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "author": {"type": "string"},
        "content": {"type": "string"},
        "tags": {"type": "array", "items": {"type": "string"}},
        "publish_date": {"type": "string"}
    },
    "required": ["title", "author", "content"]
}

# Scrape and validate
result = client.scrape(
    "https://news.example.com/article",
    extract={"schema": article_schema}
)

if result["success"]:
    # Parse and validate with Pydantic
    extracted_data = json.loads(result["data"]["extract"])
    article = Article(**extracted_data)
    print(f"Article: {article.title} by {article.author}")
```

### With AsyncIO for Concurrent Scraping

```python
import asyncio
import aiohttp
from typing import List, Dict

class AsyncLLMCrawlClient:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.base_url = "https://api.llmcrawl.dev/v1"
        self.headers = {
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        }

    async def scrape(self, session: aiohttp.ClientSession, url: str, **options):
        data = {"url": url, **options}
        async with session.post(
            f"{self.base_url}/scrape",
            headers=self.headers,
            json=data
        ) as response:
            return await response.json()

    async def scrape_multiple(self, urls: List[str], **options):
        async with aiohttp.ClientSession() as session:
            tasks = [self.scrape(session, url, **options) for url in urls]
            return await asyncio.gather(*tasks)

# Usage
async def main():
    client = AsyncLLMCrawlClient("your-api-key")

    urls = [
        "https://example1.com",
        "https://example2.com",
        "https://example3.com"
    ]

    results = await client.scrape_multiple(urls, formats=["markdown"])

    for i, result in enumerate(results):
        if result["success"]:
            print(f"URL {i+1}: {len(result['data']['markdown'])} characters")

# Run
asyncio.run(main())
```

### Integration with Data Processing Libraries

```python
import pandas as pd
from concurrent.futures import ThreadPoolExecutor, as_completed

def scrape_and_extract(url: str, schema: dict) -> dict:
    """Scrape a URL and extract structured data"""
    result = client.scrape(url, extract={"schema": schema})
    if result["success"]:
        return {
            "url": url,
            "success": True,
            "data": json.loads(result["data"]["extract"])
        }
    return {"url": url, "success": False, "error": result.get("error")}

# Product schema for e-commerce scraping
product_schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "price": {"type": "number"},
        "rating": {"type": "number"},
        "reviews_count": {"type": "number"}
    }
}

# URLs to scrape
product_urls = [
    "https://store.example.com/product/1",
    "https://store.example.com/product/2",
    # ... more URLs
]

# Parallel scraping
results = []
with ThreadPoolExecutor(max_workers=5) as executor:
    future_to_url = {
        executor.submit(scrape_and_extract, url, product_schema): url
        for url in product_urls
    }

    for future in as_completed(future_to_url):
        result = future.result()
        results.append(result)

# Convert to DataFrame for analysis
successful_results = [r for r in results if r["success"]]
df = pd.DataFrame([r["data"] for r in successful_results])
print(df.describe())
```

## Want to Contribute?

We welcome contributions to the Python SDK development! If you're interested in helping build the official Python SDK, please:

1. **Join the Discussion**: Reach out to us at <contact@llmcrawl.dev>
2. **Share Requirements**: Tell us what features are most important for your Python use cases

## Stay Updated

* 📧 **Email**: <contact@llmcrawl.dev>
* 💬 **Discord**: [Join our community](https://discord.gg/llmcrawl)
* 🐙 **GitHub**: Watch our repositories for updates

In the meantime, the REST API examples above provide a solid foundation for using LLMCrawl in your Python applications!

---

---
url: /sdks.md
---

# SDKs & Libraries

LLMCrawl provides official SDKs and libraries to make integration with your applications seamless and developer-friendly.

## Official SDKs

### JavaScript/TypeScript SDK

Our most comprehensive SDK for JavaScript and TypeScript applications.

* **Package**: `@llmcrawl/llmcrawl-js`
* **Platform**: Node.js, Browser, React, Next.js, Vue, Angular
* **Features**: Full API coverage, TypeScript support, AI extraction, webhooks
* **Documentation**: [JavaScript/TypeScript SDK Guide](/sdk-javascript)

```bash
npm install @llmcrawl/llmcrawl-js
```

```typescript
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";

const client = new LLMCrawl({ apiKey: "your-api-key" });
const result = await client.scrape("https://example.com");
```

[View Full Documentation →](/sdk-javascript)

## Framework-Specific Examples

### React/Next.js

Perfect for building web applications with scraping capabilities:

```typescript
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";

function ScrapingComponent() {
  const [result, setResult] = useState(null);

  const handleScrape = async () => {
    const client = new LLMCrawl({
      apiKey: process.env.NEXT_PUBLIC_LLMCRAWL_API_KEY,
    });
    const data = await client.scrape("https://example.com");
    setResult(data);
  };

  return (
    <div>
      <button onClick={handleScrape}>Scrape Website</button>
      {result && <pre>{result.data?.markdown}</pre>}
    </div>
  );
}
```

### Node.js

Ideal for backend services, automation, and data processing:

```typescript
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";
import fs from "fs/promises";

const client = new LLMCrawl({ apiKey: process.env.LLMCRAWL_API_KEY });

// Scrape and save to file
const result = await client.scrape("https://docs.example.com");
if (result.success) {
  await fs.writeFile("content.md", result.data.markdown);
}
```

### Express.js API

Build RESTful APIs with scraping capabilities:

```typescript
import express from "express";
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";

const app = express();
const client = new LLMCrawl({ apiKey: process.env.LLMCRAWL_API_KEY });

app.post("/api/scrape", async (req, res) => {
  const { url } = req.body;
  const result = await client.scrape(url);

  if (result.success) {
    res.json({ content: result.data.markdown });
  } else {
    res.status(400).json({ error: result.error });
  }
});
```

## Community SDKs

We welcome community contributions! If you've built an SDK for another language or framework, [let us know](mailto:contact@llmcrawl.dev).

### Python SDK (Coming Soon)

Official Python SDK in development with full typing support and Pydantic integration.

* **Status**: In Development (Q2 2025 Beta)
* **Features**: Type hints, async support, Pydantic models
* **Documentation**: [Python SDK Guide](/sdk-python)

```python
# Preview API (subject to change)
from llmcrawl import LLMCrawl

client = LLMCrawl(api_key="your-api-key")
result = await client.scrape("https://example.com")
```

[View Python Documentation →](/sdk-python)

### Go (Community)

Looking for Go developers to contribute an official SDK.

### PHP (Community)

Looking for PHP developers to contribute an official SDK.

## REST API

All SDKs are built on top of our REST API. You can also use the API directly:

```bash
curl -X POST https://api.llmcrawl.dev/v1/scrape \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

[View API Reference →](/api-reference/)

## WebSocket Support

For real-time crawl updates and streaming results:

```typescript
import { LLMCrawl } from "@llmcrawl/llmcrawl-js";

const client = new LLMCrawl({ apiKey: "your-api-key" });

// Start crawl with webhook
const crawl = await client.crawl("https://example.com", {
  webhookUrls: ["https://your-app.com/webhook"],
  limit: 100,
});

// Or poll for updates
let status = await client.getCrawlStatus(crawl.id);
while (status.status === "scraping") {
  await new Promise((resolve) => setTimeout(resolve, 5000));
  status = await client.getCrawlStatus(crawl.id);
  console.log(`Progress: ${status.completed}/${status.total}`);
}
```

## Getting Started

1. **Get an API Key**: Sign up at [llmcrawl.dev](https://llmcrawl.dev) and get your API key from the [dashboard](https://llmcrawl.dev/dashboard/api-keys).

2. **Install SDK**: Choose your preferred language/framework and install the SDK.

3. **Initialize Client**: Create a client instance with your API key.

4. **Start Scraping**: Use the SDK methods to scrape, crawl, or map websites.

## Support

* 📚 **Documentation**: Comprehensive guides and API reference
* 🎮 **Playground**: Test the API interactively at [llmcrawl.dev/tools](https://llmcrawl.dev/tools)
* 💬 **Community**: Join our [Discord](https://discord.gg/llmcrawl) for help and discussions
* 📧 **Email**: Direct support at <contact@llmcrawl.dev>

## Contributing

Interested in contributing to our SDKs or building one for a new language? Check out our [contribution guidelines](https://github.com/LLMCrawl/llmcrawl-js/blob/main/CONTRIBUTING.md) or reach out to us!