ScrapeLads
Run on Apify

Home / How To / AI and SEO

How to

How to turn a website into data for a chatbot

A chatbot is only as good as the content it reads. Here is how to turn a docs site or help center into clean chunks.

Short answer

Give the Web Crawler for LLM & RAG a start URL and a page limit. It crawls the site, strips menus, footers and cookie banners, and returns each page as Markdown, split into chunks with overlap, ready for your vector database.

A real use case

A SaaS company builds a support bot. They crawl their 180-page help center with 1,000-character chunks and load the result into their vector store. The bot answers from their own docs instead of guessing.

Step by step

  1. Enter the start URL and a max page count.
  2. Set crawl scope to stay inside the site.
  3. Set a chunk size that fits your embedding model.
  4. Run and export JSON to your vector store.

Input (paste into the JSON tab)

{
  "startUrls": [
    {
      "url": "https://help.example.com/"
    }
  ],
  "maxPages": 200,
  "crawlScope": "hostname",
  "outputFormats": [
    "markdown"
  ],
  "chunkSize": 1000,
  "chunkOverlap": 200,
  "useSitemaps": true
}

What you get back

{
  "url": "https://help.example.com/getting-started",
  "title": "Getting started",
  "wordCount": 640,
  "markdown": "# Getting started\n\nCreate your first project...",
  "chunks": [
    {
      "index": 0,
      "text": "# Getting started..."
    }
  ]
}

Trimmed example. Names and numbers are illustrative; your run returns real data with more fields.

Tips

  • Use exclude globs to skip /blog or /changelog if they add noise.
  • Turn on sitemaps for sites with weak internal linking.
  • Add video content with the YouTube transcripts guide.

What it costs

$2 per 1,000 pages. A 200-page site costs about $0.40.

Plus a small per-run start fee. Apify's free plan includes monthly credit, enough to test. Check the listing for current pricing.

FAQ

Does it handle JavaScript sites?

Yes. Crawler type auto switches to a browser when a page needs it.

Does it respect robots.txt?

Yes, by default.

Related

Try it on your own data

Paste the input above, run it, and see your results in a couple of minutes.

Open Web Crawler for LLM & RAG