NY Salesperson Drill
Content Pipeline / Verification
Outcomes
Every item was audited by a reader that did not write it. 122 carry a statute or regulation quote verified character-for-character against the committed corpus; the remaining 699 cite a syllabus topic and are labelled as curriculum rather than statutory rule, in the UI, on each question.
Measured on production, not asserted: one request for the document and nothing else, before or after interaction. No fonts, no analytics, no favicon fetch, because the icon is an inline data URI.
An aggregate defect no per-item review could see: 78.7% of correct answers had been sitting at option A, training the reflex instead of the law. Now A/B/C/D land at 215/208/203/195 across the bank.
Overview
A New York real estate salesperson exam bank, built from primary sources and weighted to the state's official 77-hour syllabus. 821 questions across 19 subjects, generated by Claude Code subagents and then put through a deterministic gate before any of them reach the page.
The drill that ships is a single static HTML file. It has no build step, no dependencies, no backend and no runtime model call. Progress lives in localStorage; the page is served under a Content-Security-Policy of default-src 'none' with connect-src 'none', so it is not that the page chooses not to phone home. The browser will not let it.
Challenge
I had six days until the exam and the free NY practice banks were actively misleading. One advertises its questions as AI-generated and AI-validated, which is a closed loop with no ground truth anywhere in it. Another cites guidance from a "New York Real Estate Commission", a body that has never existed in New York. Generating my own bank with a model would reproduce exactly that failure unless something outside the model could tell a real citation from a plausible one. A wrong question six days out is worse than no question, because there is no time left to unlearn it.
Tech Stack
Architecture
Key Features
A Gate With No Model In It
Statute and regulation text is fetched once and committed as a corpus. A question citing a law must quote it, and the verifier checks that quote is a verbatim substring of the cited section after whitespace and smart-quote normalisation. A fabricated section id fails on lookup; a paraphrase fails on the substring. There is nothing to argue with.
A Reader Whose Job Is To Disagree
The gate proves a citation is genuine, not that the rule is stated correctly. That gap is real, and it is structural. So all 821 items were audited by a reader that did not write them, briefed to falsify. Every verdict is committed as a file with its reason, so the bank's accuracy has an audit trail instead of a claim.
Ranked By What It Costs You
The home screen ranks subjects by expected questions lost, meaning exam weight times your miss rate, rather than by accuracy. Being shaky on an 11-question subject costs eleven times what it costs on a 1-question subject, and an accuracy column hides that completely.
Code
The whole citation gate
const norm = (s: string) =>
s.toLowerCase().replace(/[‘’]/g, "'").replace(/[“”]/g, '"')
.replace(/\s+/g, " ").trim();
const lawById = new Map<string, string>(
law.sections.map((s) => [s.id, norm(s.text)]),
);
if (c?.type === "law") {
const text = lawById.get(c.id);
if (!text) r.push(`FABRICATED CITATION: no section "${c.id}" in corpus`);
else if (!c.quote || norm(c.quote).length < 15) r.push("law quote too short");
else if (!text.includes(norm(c.quote)))
r.push(`QUOTE NOT FOUND in ${c.id}: "${String(c.quote).slice(0, 70)}..."`);
}Normalise, then substring. That is the entire mechanism, and its value is that it is unforgiving. If a quote fails, the fix is the quote, never the matcher. It caught a generated item that wrote “licensed real estate salesperson” where the statute says “licensed real estate agent”.
Lessons & tradeoffs
Everyone building on models right now is arguing about how to make them more reliable. This project takes the other side: assume the generator is unreliable and put something in front of it that cannot be talked out of a decision. The gate here has no model in it, so a fabricated statute is not a judgement call about hallucination risk. It is a failed substring match. What surprised me is how much that changes where the remaining risk sits, and how honest you have to be about the part the gate still cannot see.
A model that writes exam questions will write citations with exactly the same confidence whether the statute exists or not. Reviewing them by reading is the failure mode, not the fix, and it is precisely what the competing banks did.
I committed the statute text as a corpus first, then made the verifier check that a quoted rule appears verbatim in the cited section. Content that cannot pass does not enter the bank at all.
Verification is worth more when it is a program than when it is a checklist, and the reason is that a program cannot be persuaded. The moment I wanted to loosen the matcher for an awkward quote was the moment it was doing its job.
The gate passed all 432 questions and seven independent readers passed them individually, while 78.7% of correct answers sat at option A. Every item was fine. The bank was not.
I treated distribution as its own check over the whole set rather than a property of any item, and shuffled answer positions across the bank.
Item-by-item review is structurally blind to defects that only exist in aggregate. If a reviewer only ever sees one row, no amount of reviewer quality finds the pattern across rows. That has to be a separate check that looks at the set.
Only 122 of the 821 questions can be tied to a statute. The rest rest on the syllabus, which is a curriculum outline and not law, so the strongest check simply does not apply to most of the bank.
Rather than let the gate's credibility spill over the whole bank, I surfaced the distinction in the product: each question is labelled fact-reviewed or citation-gated, and a syllabus-backed answer says so on its face.
State what your verification does not cover, in the product, where the user is. A guarantee quietly applied to 15% of the content while implying all of it is worse than a weaker guarantee stated plainly.