Building a Python tutor that gives hints without leaking answers, using a 3B local model

Hint Tutor is a tool for intro Python courses. A student submits broken code, and it returns a hint that points at the bug without giving the fix. It runs fully offline on a 3B parameter model through Ollama.

Repo: https://github.com/Abdurrafay19/hint_tutor

The problem

A TA with hundreds of lab submissions can't write a personal hint for each one. ChatGPT-style tools fail in two ways: they invent bugs that don't exist, or they paste the corrected code, which skips the learning.

My approach was to stop trusting the model with anything it can't be checked on.

Tests decide, the model explains

Unit tests run first and decide whether the code is wrong. The model is only called after a test fails, and only to explain why.

• Syntax error: Python's compile() catches it and the pipeline returns the compiler's line number. No model call.
• All tests pass: success message. No model call.
• Assertion failure: the model gets the failing input, expected output, actual output, and numbered source.

Student code runs in a separate interpreter (python -I) with a per-test timeout, so an infinite loop or runaway recursion can't freeze the app. This is not an OS-level sandbox, and the README says so.

Three stages

Stage 1, fault localization. Ollama's structured output forces the model to return JSON:

{
  "line": 3,
  "bug_type": "off_by_one",
  "reason": "The range excludes n because range stop values are non-inclusive."
}

Temperature is 0.0 here so the output is stable.

Stage 2, hint drafting. The model writes 2 to 3 sentences that point at the diagnosed line and end with a question. Temperature is 0.3. The prompt forbids code blocks and replacement expressions.

Stage 3, the Answer-Leak Guard. Telling a small model "don't leak the answer" is not enough, so the guard checks every hint against the reference solution:

1. Reject code fences and executable statements (def, return, for, while).
2. Reject any normalized line that matches a line in the reference fix.
3. For small diffs (under 5 lines), extract the added tokens, such as n + 1 or <=, and reject hints that contain them.

A rejected draft triggers a retry with generic reasons only, so the fix never enters the prompt. After 3 failed attempts, the tool returns a fixed template: "Look at line N. Trace the failing input by hand and compare each step to what you expect."

Example: on sum_to_n with range(1, n), the first draft contained a literal + 1. The guard rejected it. The second draft asked what values range visits for sum_to_n(1), and it passed.

Measured results

I evaluated on 50 cases: 42 AST mutants (swapped comparisons, changed operators, shifted constants) across 7 assignments, plus 8 hand-written structural bugs. Hardware was a Ryzen 5 5600H with 16 GB RAM, CPU only.

The bug type score is the weakest. A 3B model finds the right region more often than it names the right category. The guard rejected more than a third of first drafts, which is the reason it exists.

Limits

• The benchmark is mostly synthetic. Real student bugs often combine two errors, and I haven't tested that.
• The guard blocks code and tokens, not English paraphrases like "increase the stop value by one." A hand audit of 50 hints found 20% were semantic paraphrases of the fix. 68% were clean guiding hints and 12% were harmless misdiagnoses.
• It only handles single-file scripts under 150 lines with no input().

Contributing

The repo is MIT licensed with 29 unit tests that run in under 2 seconds without Ollama. The limits above are the open work: compound-bug test cases, paraphrase-leak detection, more assignments, and stronger execution isolation.