Notes from building InfiniteGrammar.de

Building AI Products That Deliver on Their Promise

I’m a product manager who builds. InfiniteGrammar.de is my German grammar platform: language models write every exercise, AI agents write most of the code, and I own the decisions in between. These articles are about those decisions: defining quality for AI output, testing it, paying for it and running it in production.

Start with the case study → About me

Drafts that pass every check

17% →78%

held-out tests, old vs current pipeline

Accepted exercises graded flawed

0 of 31

blind audit of the current pipeline

Cost per accepted exercise

$0.23 →$0.058

all checks and repairs included

Organic search clicks

+77%

after the first wave of agent-made SEO fixes

Start here

The whole quality story in one article, with the numbers and what I’d do differently.

Case study · 6 min read

From one in six to three in four: making an LLM content pipeline trustworthy

My generated exercises looked fine and quietly let learners down. I built the checks first, tested the judge, and let cost per clean exercise choose the pipeline. Then I re-checked the whole live library against the new standard.

Evals Model choice Unit economics Production

Quality and evals

Defining “good” for LLM output, testing the judge, and deciding with evidence.

02Before I trusted my LLM judge, I made it grade the answer keyMy first verifier failed 3 of 5 official telc texts. How I tested the judge and rebuilt it.5 min read5 October 2026 03In a gap-fill exercise, the wrong answers are the productThe distractors decide what an exercise teaches. How I turned “good wrong answers” into checks, rules and two product decisions.4 min read5 March 2026 04Writing the rule before the test: how I chose prompts and modelsPass rules written before each test, held-out tasks, blind audits. What I adopted, what I killed, and why.4 min read6 October 2026

Running it in production

Unit economics, a library-wide re-check, and the dashboards that watch it all.

05Cost per clean exercise: the number that changed my model choicePrice per token is the wrong unit. Where the money goes in an LLM content pipeline, and why checking costs more than writing.4 min read9 October 2026 06Why I took 84% of my live exercises offlineA re-check passed 133 of 829 live exercises. How I replaced the rest without empty sections or broken learner progress.3 min read9 October 2026 07Three questions, three dashboards: a cockpit for an AI content productWill anyone run out? Is it good? What does it cost? How I specified three admin dashboards, and the definition that mattered most.3 min read10 October 2026

Product and learners

Variety that learners feel, and reminders that respect them.

08Variety is a sequencing problem, not just a content problemWhy embeddings measured the wrong similarity, the metric that lied to me, and a reorder that never breaks learner progress.4 min read18 March 2026 09Reminders that know what you practisedA section-based reminder schedule with guardrails. Most of its value is in the emails it decides not to send.3 min read20 March 2026

Building with AI agents

How a coding agent does the engineering while I keep the decisions and the risk.

10My engineer is an AI agent. This is how I manage itClaude Code writes most of the code. The operating model that keeps that safe: decisions, budgets, guardrails, and three recent mistakes.5 min read10 August 2026 11My SEO agent isn’t allowed to grade its own homeworkAn autonomous agent fixes technical SEO, but only against a frozen verifier that must fail first. Organic clicks +77% after the first wave.5 min read25 September 2026 12What fast AI-built frontends hide: an SEO post-mortemAI tools shipped the UI fast and hid an SEO architecture problem. What went wrong, and the rules I now give coding agents.4 min read5 June 2026

How I work

Three habits that run through every article.

Evidence before opinions

Before a change reaches users, I decide what result would prove it works, then test it on cases the system hasn’t seen. If it misses the bar, it doesn’t ship, however much I liked the idea.

Quality is a product decision

I turn “good” into checks and scores before building, test the evaluator on cases with known answers, and let cost per good result decide which model wins.

Agents build, I own the risk

AI agents write most of the code. Product decisions, budgets, production data and the final sign-off stay with me.