MCQ Generation Pipeline

A curriculum needs a large pool of questions, and it needs them at a consistent quality bar. Every subtopic needs several. Every difficulty band needs coverage. The pool has to keep growing as the curriculum does, and it has to stay honest about what it covers, because a question pool that quietly misses a third of the syllabus is worse than no pool at all. It looks like coverage.

Hand authoring does not scale here, and the reason is not the writing. A good author writes a question in a few minutes. The bottleneck is review, and review is where the whole thing comes apart. Send the same twenty questions to three reviewers and you get three different verdicts. One rejects for ambiguous stems. One rejects for distractors nobody would ever pick. One rejects for being pitched two levels below where the curriculum says the student is. None of them are wrong, and no two of them agree, so the quality of any given question ends up being a function of who happened to open it.

So the obvious move is to have a model write the questions. That part works on the first afternoon. A model will write questions all day, in volume, in house style, on any subtopic you name. Fluency was never the constraint. The constraint is that nobody can tell you which of those questions is safe to put in front of a student, and the moment you try to answer that at scale you find out you never had a definition of good in the first place. You had four reviewers with four private ones.

That reframing is the whole project. The deliverable is not a generator. It is a set of criteria precise enough that a machine applies them the same way twice, plus the architecture that applies them in an order where each pass catches what the pass before it let through.

Six generation agents write. Twelve evaluation agents read. What follows is the evaluation layer running: a field of generated questions thinning out as each layer rejects, with the reason each item died attached to it. Four layers of three evaluators, in the order they actually run.

The counts are a representative sample rather than a published statistic. Read the shape and not the numbers. Most of what gets generated does not survive, the drop is steep at every layer, and no two layers kill for the same reason. That last part is the design.

Defining good twelve different ways sounds like a taxonomy exercise. It was mostly argument. Each definition had to be narrow enough that two different readers would rule the same way on the same item, and broad enough to catch more than one specific failure. Ambiguous stem is easy to say and very hard to grade, so it had to become something like can the stem be read only one way, which is answerable. Multiply that by twelve, then check the twelve do not quietly overlap, because two evaluators enforcing the same criterion is not redundancy. It is one wasted evaluator and a blind spot somewhere else.

Then there is what happens when an agent produces junk. Occasionally a generation agent returns something that is not really a question: four options where two say the same thing, a stem referencing a code block it never included, an answer key pointing at an option that is not there. Deciding what the pipeline does with those mattered more than reducing how often they happen. Dropping them silently hides a generation problem. Passing them through spends twelve verdicts on something that was never a question. So malformed output fails structurally, before evaluation, and it gets counted, because that count is the only early signal that a generation prompt has drifted.

The harder failure mode is the opposite one: an evaluation layer so strict that nothing survives. Every time a criterion got tightened survival dropped, and from the inside a well calibrated bar and an overfit one look identical. A layer that rejects almost everything even reads as rigour on a dashboard. The only check that worked was reading the rejections rather than the survivors. If a strict evaluator is throwing out questions a competent human author would have shipped without hesitation, the criterion is wrong, not the questions.

Underneath all of it is an asymmetry that decides every close call. A question thrown away costs one more generation pass, which is cheap. A wrong question reaching a student teaches them something false, and they will not argue with it, because they trust that the platform is teaching them correctly. They will remember the wrong answer as the right one and be marked correct for it later.

That asymmetry is why the evaluation layer has twice as many agents as the generation layer, and why every ambiguous verdict resolves toward rejection.

I would write the twelve criteria before writing a single generation prompt. We built generation first because it was the half that visibly worked, then spent far longer than we needed to discovering that most of our disagreements about output quality were undefined criteria wearing a disguise. The definitions were the actual spec. Written first, the generation prompts would have fallen out of them almost for free, and we would have had a way to tell a strict layer from a broken one immediately instead of only once there was a pile of rejections to argue over.