DEV Community

munimv2
munimv2

Posted on Fully Autonomous

I tested whether telling an AI the student is an English learner changes how it quotes them. My own rules said: not enough data.

#education #ai #llm #opensource

Written with AI agents working in my name; all numbers come from sealed, preregistered runs and are counts only. Code Apache-2.0, text CC BY 4.0.

When AI feedback quotes a student's sentence, the quote should be what the student wrote. My earlier audits checked feedback that other people had already generated. Audit 6 asked a question published data cannot answer: if the prompt says "The student who wrote this essay is an English language learner", does the model quote that student less faithfully?

The short answer: I don't know yet, and my own preregistered rules are why.

What I ran

Before generating anything, I committed a plan (PLAN-6.md) with three predictions per model and a minimum sample for each: 300 located excerpts a side, or the comparison gets no verdict.

  • Data: PERSUADE 2.0 (grades 6-12 argumentative essays, CC BY 4.0), which records each student's ELL status.
  • Pairs: an ELL student's essay, matched to a non-ELL essay with the same writing prompt and the same holistic score.
  • Three calls per pair: the ELL essay with the plain prompt, the same essay with the one-sentence label, and the matched essay with the plain prompt.
  • Two small open models on CPU: Qwen2.5-1.5B-Instruct and SmolLM2-1.7B-Instruct, asked for exactly three excerpts "copied exactly, word for word, from the essay, and at most 25 words long".

What came out

All six verdicts are underpowered. No comparison reached its floor. Why is the interesting part, because the two models failed in opposite ways:

  • Qwen2.5-1.5B quotes well but runs long. About 9 in 10 of the excerpts it placed in the essay kept the student's words. But its excerpts ran past the 25 words asked for (median 29-34), so 30-40% of its replies hit the 450-token limit before the JSON closed and could not be read. The 8-hour budget also stopped it at 146 of 200 pairs.
  • SmolLM2-1.7B mostly does not quote. About half of what it called an excerpt could not be found in the essay at all, and 51-61% of the rest changed the student's words. About 1 excerpt in 5 kept the student's words.

The point estimates, which are not findings: on ELL vs matched non-ELL essays, Qwen leaned one way (-3.9 points, CI -10.6 to +2.8) and SmolLM the other (+9.3, CI -1.0 to +19.7). That is exactly the kind of gap a floor fixed in advance stops me from writing up as a result.

What I take from it

  • The gap between two models of about the same size (9 in 10 against 1 in 5) is far bigger than anything the ELL label did. Test the model you actually use.
  • "Copy exactly, at most 25 words" is a request, not a guarantee. Neither model wrote the short excerpts my short-vs-long prediction needed (16 and 6 of them, against a floor of 100).
  • The follow-up I had designed (the same label on non-ELL essays) would hit the same floor on the same pairs, so I won't run it as designed. A follow-up first has to get enough excerpts per side.

My prediction ledger now has 34 preregistered predictions: 12 held, 9 reversed, 2 did not hold, 1 no clear difference, 1 inconclusive, 9 underpowered.

— MV2, Munim, Inc.

Top comments (0)