Ask a general-purpose AI for “five sixth-grade questions on ratios” and you’ll get five plausible questions. Read them closely and a third are testing seventh-grade proportional reasoning, one is really a fractions item wearing a ratio costume, and the “hard” one is hard only because the arithmetic is annoying — not because the thinking is deeper.
This is the alignment gap, and it’s the single reason most AI assessment tools don’t survive contact with a curriculum coordinator. The model is optimizing for “looks like a math question about ratios.” A teacher needs “assesses exactly this standard, at this depth, and nothing else.” Those are different targets.
Three ways alignment quietly breaks
The failures are consistent enough to name. Once you can see them, you can design against them.
1. Grade drift
The model reaches for the most representative version of a concept, which skews toward whatever is most common in its training data — often a grade or two above target. A standard about describing ratio relationships becomes a problem about solving proportions, which is a later skill.
Grade drift is hard to catch by skimming because the drifted item is usually a good item. It reads as rigorous. It just belongs in a different unit, and a student who hasn’t been taught cross-multiplication yet will fail it for reasons that tell you nothing about whether they understand ratios.
A quick test: for each item, ask what a student would need to have already been taught in order to answer it. If the answer includes anything downstream of the standard you selected, the item has drifted.
2. DOK inflation
“Make it rigorous” gets interpreted as “make it harder,” and harder usually means heavier computation. But Depth of Knowledge is about the kind of thinking, not the size of the numbers. A DOK 3 item asks a student to reason, compare, and justify; a DOK 1 item with ugly decimals is still DOK 1 — now with an arithmetic tax attached.
Depth of Knowledge is about the kind of thinking a question demands — not how tedious the arithmetic is.
The practical cost of inflation is measurement noise. When you write a “hard” recall item, students who understand the concept still miss it, and you can no longer tell a reasoning gap from a computation slip. Your reteach plan gets built on bad data.
3. Standard substitution
The most subtle one: the item is well-written, grade-appropriate, and rigorous — but it assesses an adjacent standard. It’s a good question for the wrong learning target, which is worse than a bad question, because it passes a quick read.
Substitution clusters around standards that share surface vocabulary. Ratio items slide into unit-rate items. “Compare two texts” items slide into “summarize a text” items. Nothing about the wording looks wrong; the construct being measured has simply moved.
Designing for alignment instead of hoping for it
The fix isn’t a better prompt. It’s constraint. An item generator that stays aligned treats the standard as a specification, not a topic hint:
- Load the standard’s actual intent and boundaries before generating — what it does and does not ask students to do.
- Commit to a DOK level up front, in a blueprint, so rigor is a target the item is checked against — not a vibe.
- Generate distractors from the standard’s known misconceptions, so wrong answers stay on-standard too.
- Verify each finished item back against the standard and DOK it claims — and regenerate the ones that drift.
That last step matters more than it looks. Generation and verification should not be the same pass. When a system is asked to produce an item and judge it in one breath, it grades its own homework generously. Separating the two — write, then check against the spec — is what turns “plausible” into “defensible.”
standard: 6.RP.A.3 — ratio & rate reasoning
dok_target: 2 · items: 5
types: mcq ×3, constructed ×1, figure ×1
distractors: from_misconception_model
verify: trace_to_standard = required
What “defensible” means in practice
A defensible assessment is one you can hand to an instructional coach and explain item by item: this measures this standard, at this depth, and here’s what each wrong answer tells you. That’s the bar for anything you’ll use to make an instructional decision.
It’s also, not coincidentally, the bar for anything a district will adopt. Coverage claims that can’t be traced don’t survive a curriculum review — and shouldn’t.
None of this removes the teacher. It removes the busywork between a teacher and a defensible draft — the part that eats a Sunday. The judgment stays exactly where it should: with the person who knows the class.