The AI Layer

The single most fragile dependency in any AI product is the model provider. Free tiers rate-limit, quotas run dry, and services go down — usually at the worst moment. Praxis924 routes every LLM call through one layer that expects this and survives it. This is how that layer is built, and how you'd build the same thing.

The problem

Every feature needs a model: lesson generation, Lucy the AI teacher, mock interviews, code grading. The naive version has each of them import a provider SDK and call it directly. That gives you three problems, all of which bite in production:

• N places to configure — every feature needs its own key wiring.
• N places to break — when a provider returns 429, each call site fails on its own, differently.
• No shared caching — the same prompt gets paid for again and again.

One provider hiccup and half the product goes dark. So the very first rule is: features never talk to a provider directly.

The design in one sentence

A single LLMService singleton chooses a provider by priority at startup, and on every call tries providers in order until one succeeds — with a Redis cache in front, and embeddings pinned to their own separate provider.

Provider selection happens once, at startup

The priority order is NVIDIA → Groq → Gemini → Hugging Face. Whichever has a key configured becomes the primary; the rest form the failover chain. There's no per-request negotiation — the chain is fixed when the process boots.

1# The chain is just the configured providers, in priority order.
2PROVIDERS = [p for p in (Nvidia, Groq, Gemini, HuggingFace) if p.has_key()]

The failover loop is the whole trick

Every call goes through one internal method. It checks the cache, then walks the chain: try a provider, and on a quota-or-availability error, move to the next. Only when the entire chain is exhausted does it raise a typed error.

1def _chat(messages, **opts):
2    k = cache_key(messages, opts)
3    if (hit := cache.get(k)):          # 1. Redis first — free and instant
4        return hit
5    for provider in PROVIDERS:         # 2. try each in priority order
6        try:
7            out = provider.chat(messages, **opts)
8            cache.set(k, out)
9            return out
10        except (QuotaError, ProviderDown):
11            continue                    # 3. failover to the next provider
12    raise LLMUnavailableError           # 4. all dead — fail closed, typed

That last line matters more than it looks. When everything is down, generation raises LLMUnavailableError — a specific type the durable workflow understands. Instead of persisting half a lesson or a placeholder, the workflow leaves the lesson re-runnable and stops cleanly. Fail closed, never fail messy.

Embeddings are deliberately on a separate track

Text generation fails over across four providers. Embeddings do not — they always run on Hugging Face, at a fixed dimension (EMBED_DIM = 384). Two reasons this is intentional:

1. Independence. Retrieval has to keep working even when your text provider is having a bad day. Coupling them means one outage takes out both.
2. You can't hot-swap an embedding model. The vector dimension is baked into the pgvector column. Switching models means a full re-index, so this is a decision you make once, not per request.

The gotchas that cost real hours

Build the minimal version yourself

You don't need a framework for this. The whole pattern is a list of providers, a cache lookup, and a try/except loop — exactly the two snippets above. Add providers by appending to the chain; the failover logic never changes.

What you'd add at scale

The free-tier design optimises for "keep working without paying." Past that, you'd reach for paid tiers with real headroom, a proper job queue instead of in-process workflows, per-provider circuit breakers so a flapping provider gets skipped automatically, and token budgeting per tenant. But the core — one door, priority chain, fail over, fail closed — stays exactly the same.