This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
🦥 TL;DR. Paste a week of volunteer chat messages. Gem...
For further actions, you may consider blocking this person and/or reporting abuse
The part I would steal for my own work is not the Gemma-plus-rules split, it is keeping that first frozen score exactly as it came out. I have watched teams quietly re-run a test after a fix and then report the improved number as if it were the first result, and the whole comparison quietly stops meaning anything. Labelling the set C rerun post-hoc, after the Python fixes, is what makes the 1.0 honest rather than impressive. Do you keep the frozen run and the post-fix run side by side in the repo, so a reader can see both without trusting your summary?
Thanks, that was exactly the trap I was worried about. It’s easy to fix things, re-run, and quietly let the new number become “the” number.
Yes, both are in the repo as separate files, so you don’t have to trust my summary (github.com/bipul724/availability-ledger):
The first score is in evaluation/results/frozen-setC/. Set C was scored once early on 21 Oct, and those files were committed later that morning, before any of the fixes. Git shows no change to them since. Their hashes are in evaluation/protected.sha256, and CI fails if any of them changes or if a file is added to or removed from that folder.
The post-fix numbers are separate files in evaluation/results/, from a later commit. They were re-scored from the same saved Gemma replies, with zero new model calls.
RESULTS.md puts them side by side (0.527 → 1.0) and lists which fix moved which message.
You can check it yourself without a model: python3 -m evaluation.run --set C --extractor hybrid --model gemma4:e2b-it-qat --cache-only --out /tmp/c replays the saved replies and should show the post-fix result (slot agreement 1.0, 0 fresh calls). CI runs the same re-score on every push and fails if any score differs from what’s committed. If you check out the commit where the first score was frozen and re-score from the same replies, you get the original numbers back exactly (slot agreement 0.433 rules only, 0.403 Gemma only, 0.527 Gemma + rules).
Your question actually made me find a gap. The raw Gemma replies for set C were committed once and never touched, but CI wasn’t hashing them, and it wasn’t checking the set C fixtures against their manifest either (the manifest was committed before the scoring run). So you’d have been trusting git history there. I’ve now added both to the CI integrity check.
And I agree about what the 1.0 means. It’s me fixing failures I’d already seen, so it says very little about new messages. The honest next step is a fresh set D, written and frozen the same way, and scored once. I haven’t done that yet.
The CI integrity check is the right fix, and the part I would still pin down is the manifest itself: it was committed before the scoring run, so if the manifest can be regenerated after a fixture change, the check passes while the evidence moved. I would hash the manifest in CI and fail on any regeneration, not just on a mismatch. The same logic applies to set D: freeze the raw Gemma replies before the extractor is touched again, so model drift and extractor fixes stay separable. That way, if set D also reaches 1.0, it is measured on outputs the extractor never saw during fixing. Will set D freeze its raw replies before any extractor change, or will the extractor and the set evolve together again?
Good point, and the manifest part is already pinned. fixtures/setC.manifest has had its own line in evaluation/protected.sha256 since the integrity check was added on 21 Oct. So regenerating it after a fixture change fails CI, even if the new manifest and the fixtures agree with each other. The manifest itself has a single commit, from before the scoring run.
The one thing CI can’t stop is someone editing the hash list and the manifest in the same commit. That’s where it falls back on public git history. The hash list has had two commits so far, and both only added lines; no existing hash has ever changed. A regeneration would show up as a changed line in that file’s diff.
On set D: yes, raw replies first. Set C already went in that order in practice: its raw Gemma replies were committed with the first score, before the first extractor fix. The pipeline is split that way too. Replies are cached per message and everything after them is Python, so a cache-only re-score never calls the model. For set D I’ve made the order explicit:
1.commit the manifest and pin its hash;
2.run Gemma once;
3.commit the raw replies and the first score, with their hashes added to CI;
4.only then touch the extractor.
That way, if set D also reaches 1.0 later, it’s measured on replies the fixes never influenced. A model change would show up as different replies, not as a different score from the same replies.
Honest status: set D doesn’t exist yet. It’s an open task, and your “freeze the raw replies before any extractor change” order is now written into it.
The append-only hash list is the part that settles it for me: two commits, both only adding lines, means a same-commit rewrite cannot hide, it shows up as a changed line in the diff. I have leaned on that same property in a deploy log where append-only history was the only witness I trusted. For set D, will you also freeze the extractor version alongside the raw replies, so a later extractor change cannot quietly re-score the same replies?
Yes. Today the result files record the model and the prompt version, but not the code commit. For set C, the extractor version is written in RESULTS.md’s provenance (the exact commit), and the reproduction I mentioned works because git still has that commit. But that’s prose plus git history, not something pinned in the result file itself.
For set D I’ve added it to the task: the first-score files will name the exact commit that produced them, and that commit has to reproduce the frozen score from the frozen replies, the same way the set C one does.
On “quietly re-score”: a later extractor can re-score the same replies, and it should, because that’s how the post-hoc numbers get made. What it can’t do is replace the first score. CI re-scores into a temporary folder and compares against the current results files, while the frozen ones are hash-locked separately. So a new extractor can only add a new, separately labelled number next to the old one, never overwrite it.
Appreciate you pushing on this. Append-only history being “the only witness I trusted” is a good way to put it.
The bit where Gemma completely missed Maria’s cancellation and the keyword rules caught it is the most convincing part of the post. Two dumb-ish readers beating one smart one is a real lesson.
Curious how you decide when the two readers disagree on a normal availability (not a cancellation). Do both versions show up, or does one win by default?
The “change” word bug is brutal, and it’s exactly the kind of thing you only find with a test set you didn’t write yourself. Hours right, sheet wrong.
Curious if you thought about letting Gemma flag replacements too and having the rules confirm, the same way cancellations work?
For normal availability, Gemma’s version is the default proposal. If the readers disagree on the hours/status for the same person and day, it gets a “Model and rules read this differently” warning.
The UI shows that proposal beside the original message, but doesn’t show the rules’ alternative alongside it yet. The reviewer can edit, accept or reject it; disagreements stay out of bulk acceptance. Only accepted records reach the final export.
On replacements, Gemma actually already has a replaces_earlier field, and the prompt explicitly mentions “change”! On that message it returned the correct hours but set the flag to false. My keyword fallback recognised “change of plans,” but missed bare “change,” so both paths failed together.
The fix widened the Python replacement cues. When that fallback removes earlier hours, it now shows “Earlier answer replaced — check,” including what was removed. Either the model hint or a code cue can trigger a provisional replacement; I don’t require agreement from both.
Those cues can overreach too, so human review still matters. The original failing score stays frozen, and the fix is labelled post-hoc.
nice, keeping disagreements out of bulk accept is a good call. one small thing: when it says “model and rules read this differently”, showing the rules version right under gemma’s would make it a one click decision instead of reading the message again. feels like it’d cut a lot of the 28 flagged ones down to quick picks
That’s the comparison the current UI is missing. Showing Gemma’s version and the rules’ version together, with a button to choose either, would make straightforward disagreements easier to resolve. I’d keep the original message visible and an edit/reject option for cases where both are wrong.
One caveat: the 28 flagged suggestions also include ambiguous names, dates and other warnings, so this wouldn’t simplify all of them. I’d want to measure review time before claiming how much it saves.
The four calls hitting the 300s timeout back to back is interesting. Was Ollama still loading the model, or was some other process using up the RAM in the 8 GB Mac? Curious whether a warm-up call before the batch would have avoided it.
I couldn’t establish the cause. The failures happened late in the batch, at positions 37–40, after earlier calls had completed. The code already sends keep_alive: "30m", although there’s no separate warm-up step.
I did see a macOS memory warning naming Claude and ChatGPT on that 8 GB Mac. After closing other apps, restarting Ollama and the app server, and running a two-message check, a later full run completed 49 calls with zero errors in 1.42 minutes. But that small check had already warmed the model, so several things changed together.
I didn’t capture RAM/swap during the stalls, and the timed-out calls didn’t return Ollama’s load-time breakdown. Memory pressure is plausible, but I can’t attribute the failures to it.
A warm-up is worth testing; I haven’t demonstrated that it alone would have prevented them.
makes sense, failing at 37-40 after everything before worked doesn’t look like a loading issue. one cheap thing for next time: ollama returns load_duration and eval_duration on the calls that do succeed, so logging those per call would show if things were slowly getting slower before the stall. plus memory_pressure or vm_stat running in another terminal. then you’d actually have the data instead of guessing
Yes—the missing piece is lining those signals up. The tracing adapter already records load, prompt and generation times for successful calls as load_s, prompt_s and output_s, but I haven’t established whether they were trending upward before the stall.
I also didn’t capture memory pressure or swap activity alongside that batch. Correlating those timings with memory_pressure/vm_stat would make the next investigation much more useful. It would show what preceded the failures, without treating the successful retry as proof of the cause.