How to Turn a Website into an LLM Wiki

Short answer

Run the LLM Wiki & OKF Bundle Generator on the docs URL. It crawls the site, writes one markdown file per page with an AI summary, key facts, tags and links to related pages, adds an index.md in every folder and returns the whole wiki as a zip, in Google's Open Knowledge Format (OKF).

A real use case

A developer wants Claude Code to answer questions about an internal SDK without pasting docs into every chat. They run the actor on the SDK docs, unzip the bundle into the repo under docs/wiki/, and tell the agent to start from index.md. The agent opens two or three concept files per question instead of loading everything.

Step by step

Open the actor on Apify and sign in (the free plan is enough to test).
Add the docs URL as a start URL. The crawl stays under that path by default.
Set Max pages. Each page becomes one wiki entry, so this is also your price cap.
Pick a folder layout: mirror the site, or group by type (guides, tutorials, API references).
Start, then download bundle.zip from the Output tab and unzip it where your agent can read it.

Input (paste into the JSON tab)

{
  "startUrls": [
    {
      "url": "https://docs.example.com"
    }
  ],
  "maxPages": 100,
  "aiEnrichment": true,
  "groupBy": "type"
}

What you get back

{
  "conceptId": "guides/quick-start",
  "type": "Guide",
  "title": "Quick start",
  "description": "Install the SDK, set an API key and make a first request.",
  "tags": [
    "install",
    "authentication",
    "quick-start"
  ],
  "url": "https://docs.example.com/quick-start",
  "aiEnriched": true
}

Trimmed example. Names and numbers are illustrative; your run returns real data with more fields.

Tips

Turn off includePageContent for a compact wiki of summaries and key facts that fits in one context window.
Open viz.html from the run's storage to see the wiki as a graph before you hand it to an agent.
Schedule the task weekly. File paths come from URLs, so a git diff shows exactly which pages changed.

What it costs

$2 per 1,000 pages, plus $4 per 1,000 for AI summaries. A 100-page docs site with AI costs about $0.61.

Plus a small per-run start fee. Apify's free plan includes monthly credit, enough to test. Check the listing for current pricing.

FAQ

What is an LLM wiki?

A folder of short markdown pages, one per topic, linked to each other, that an AI agent browses like a person browses a wiki. Karpathy popularized the idea; OKF is Google's open format for it.

LLM wiki vs RAG: which is better?

RAG cuts pages into chunks and finds them by similarity, which loses structure. A wiki keeps one page per topic with links and sources, so the agent can navigate and cite. You can still index the same pages in a vector database.

Does it work with Claude, Cursor or MCP?

Yes. Unzip the bundle into your project or point a file-system MCP server at the folder. Agents start at index.md.

LLM Wiki & OKF Bundle Generator Turn any docs site or dataset into an LLM wiki your AI agent can browse. $2 / 1,000 pages, $6 with AI.
How to check if ChatGPT mentions your brand
How to turn a website into data for a chatbot
How to use a scraper MCP server in Claude
How to create an OKF bundle from a website
All How To guides

How to turn a website into an LLM wiki