RAG Grounding

When you ask Praxis924 to generate a lesson, the AI can write from general knowledge — or it can write from your material. RAG grounding is what makes the second option possible. Attach a dataset to a framework, and every lesson generated under that framework retrieves the most relevant passages from your source material, writes from them, and cites them. This guide is for admins building curriculum in the /admin console. By the end you'll know how to upload a dataset, confirm it indexed, generate a grounded lesson, and verify the citations.

What you'll need

• Admin access to the /admin console (grounding is configured per framework, and dataset routes are guarded by require_admin).
• A Technology → Framework already created — grounding attaches at the framework level, so every lesson under it inherits the same source pool.
• A dataset in a supported format: plain text or alpaca-style JSON. The chunker is alpaca-aware, so instruction/response records are split sensibly rather than blindly by character count.
• At least one LLM provider key configured for generation. Embeddings are separate — Praxis924 always embeds with Hugging Face (EMBED_DIM=384) into pgvector, independent of whichever provider writes the prose.

Step 1 — Upload a dataset to the framework

In /admin, open the Technology, then the Framework you want to ground. Add a DataSource by uploading your text or alpaca file. On upload, Praxis924 does three things in sequence:

1. Chunks the document into passages (alpaca-aware).
2. Embeds each chunk via Hugging Face into a 384-dimension vector.
3. Stores those vectors in pgvector, attached to this framework.

Indexing is a background step, so the DataSource moves through indexing states before it's ready. Wait for it to finish before generating — a dataset that's still indexing (or that failed) won't be retrievable.

Step 2 — Confirm the dataset indexed

Before generating anything, check the DataSource's state. A healthy source shows as indexed with chunks stored. If it shows a failed/empty state, re-upload — a half-indexed dataset silently degrades retrieval, because the workflow will fall back to weaker matches or to the lesson's raw content instead of your material.

Step 3 — Generate a grounded lesson

Open a lesson (a ContentItem) under the grounded framework and run Generate. Lesson generation is a durable DBOS workflow, and grounding changes only the first step:

1. Build the grounded source. Because the framework has a dataset, the workflow retrieves the top-K chunks most relevant to this lesson by cosine similarity in pgvector. If vector search comes back thin, it falls back to a keyword (ILIKE) match, then to a first-N slice — so generation never stalls for lack of a perfect hit.
2. Cap the source at ~3000 characters. That grounded source is re-sent in every section prompt, so it's capped to protect your tokens-per-minute budget. Retrieval quality matters more than volume here.
3. Generate the 4 sections — main_explanation, practical_explanation, key_principles, summary — each through the review loop (score below 7 → improve, max 2 rounds).
4. Generate exercises, then persist the lesson plus its citations and flip generation_status to generated.

Because the workflow is durable, a crash or provider hiccup resumes from the last completed step rather than restarting — and an LLMUnavailableError leaves the lesson re-runnable.

Step 4 — Verify the citations

Open the generated lesson in the viewer. Grounded lessons render their Sources inline. Skim them: the cited passages should trace back to the dataset you uploaded. If citations look generic or absent, the lesson probably generated before the dataset finished indexing — re-check Step 2 and regenerate.

When it goes wrong

• Citations are missing or generic. The dataset wasn't indexed at generation time, or the DataSource failed. Confirm the indexed state, then regenerate.
• Retrieved passages feel off-topic. Vector search fell back to keyword or first-N because nothing scored well. Tighten the dataset to the framework's actual subject, or split unrelated material into separate frameworks.
• Lesson generated with no key. With no provider configured, generation emits deterministic placeholder content. Add a key and regenerate.
• Nothing indexed after upload. Re-upload. Embeddings run through Hugging Face regardless of your generation provider, so an HF-side failure blocks indexing even when generation would otherwise work.

What you get

Grounded lessons read like they came from your source of truth because they did — retrieved, written, and cited from material you control. Once a framework is grounded, everything downstream inherits it: bulk Generate all lessons stay on-source, Lucy answers from the lesson's own content when learners open her from it, and published lessons carry their citations into the content pool export. Next, ground a second framework, or run Generate all to build an entire grounded syllabus in one pass.