[Home](/) / [How To](/how-to/) / AI and SEO

How to

# How to turn a website into data for a chatbot

A chatbot is only as good as the content it reads. Here is how to turn a docs site or help center into clean chunks.

**Short answer**

Give the **Web Crawler for LLM & RAG** a start URL and a page limit. It crawls the site, strips menus, footers and cookie banners, and returns each page as Markdown, split into chunks with overlap, ready for your vector database.

## A real use case

A SaaS company builds a support bot. They crawl their 180-page help center with 1,000-character chunks and load the result into their vector store. The bot answers from their own docs instead of guessing.

## Step by step

1. **Enter the start URL** and a max page count.
2. **Set crawl scope** to stay inside the site.
3. **Set a chunk size** that fits your embedding model.
4. **Run** and export JSON to your vector store.

### Input (paste into the JSON tab)

```
{
  "startUrls": [
    {
      "url": "https://help.example.com/"
    }
  ],
  "maxPages": 200,
  "crawlScope": "hostname",
  "outputFormats": [
    "markdown"
  ],
  "chunkSize": 1000,
  "chunkOverlap": 200,
  "useSitemaps": true
}
```

### What you get back

```
{
  "url": "https://help.example.com/getting-started",
  "title": "Getting started",
  "wordCount": 640,
  "markdown": "# Getting started\n\nCreate your first project...",
  "chunks": [
    {
      "index": 0,
      "text": "# Getting started..."
    }
  ]
}
```

Trimmed example. Names and numbers are illustrative; your run returns real data with more fields.

## Tips

- Use exclude globs to skip /blog or /changelog if they add noise.
- Turn on sitemaps for sites with weak internal linking.
- Add video content with the [YouTube transcripts guide](/how-to/download-youtube-transcripts-in-bulk/).

## What it costs

$2 per 1,000 pages. A 200-page site costs about **$0.40**.

Plus a small per-run start fee. Apify's free plan includes monthly credit, enough to test. Check the listing for current pricing.

## FAQ

**Does it handle JavaScript sites?**

Yes. Crawler type auto switches to a browser when a page needs it.

**Does it respect robots.txt?**

Yes, by default.

## Related

- [**Web Crawler for LLM & RAG**](https://apify.com/scrape-lads/website-content-crawler-rag?utm_source=scrapelads&utm_medium=site&utm_campaign=how-to-crawl-website-for-ai-chatbot&utm_content=website-content-crawler-rag) Crawl any site into clean Markdown and embedding-ready chunks. $2 / 1,000 pages.
- [How to check if ChatGPT mentions your brand](/how-to/check-if-chatgpt-mentions-your-brand/)
- [How to use a scraper MCP server in Claude](/how-to/use-scraper-mcp-server-in-claude/)
- [All How To guides](/how-to/)

## Try it on your own data

Paste the input above, run it, and see your results in a couple of minutes.
[Open Web Crawler for LLM & RAG](https://apify.com/scrape-lads/website-content-crawler-rag?utm_source=scrapelads&utm_medium=site&utm_campaign=how-to-crawl-website-for-ai-chatbot&utm_content=website-content-crawler-rag)