Hoi hoi! 👋
I'm @nyaomaru, a frontend engineer currently building an AI-assisted CHANGELOG generator and experimenting with Jev. 😸
Recently, I hav...
For further actions, you may consider blocking this person and/or reporting abuse
the framing of WHY as a selection problem rather than a generation problem is the right call. once you let the model paraphrase it starts hallucinating intent and that ends up worse than leaving the field blank. the candidate snippet approach keeps the context window tight and forces the model to pick instead of invent.
what I am curious about: how does the bot handle PRs that skip all the rationale markers entirely? does Jev still get a chance to infer from diff context or does it just return an empty WHY and let the human fill it in?
Yep, no diff inference in the
Jevpath 😸If there’s no recognized WHY section, it can still use the PR body as a fallback candidate source, but only if it passes the local trust checks. If nothing explicit and trustworthy survives, it returns no WHY rather than inventing one.
I’d rather leave that gap for a human than turn implementation context into a plausible story.
the 'rather leave that gap' instinct is where a lot of tooling goes wrong in the other direction — they fill it plausibly and ship noise nobody can audit later.
curious about the local trust checks though. are those heuristics (minimum content length, no boilerplate patterns) or something trained on labeled PR data?
The local trust checks are deterministic heuristics, not something trained on labeled PR data.
They look for signals like explicit
Why/Motivation/Context/Problemsections, rationale markers such as “because” or “to prevent”, issue references, and a bit of content-length signal. We also down-rank low-signal maintenance text like typo/format/lint-only descriptions and skip candidates that don’t clear the trust threshold.So
Jevstill only selects from text the author actually wrote.The heuristics decide whether that text is trustworthy enough to even become a candidate, they don’t try to infer a missing rationale.
I’d rather have a false negative there than manufacture a plausible WHY 😸
the 'because' / 'to prevent' detection is where i'd expect the most noise across codebases — we ran into this trying something similar. some engineers write 'this fixes X to prevent Y', others just write 'fix X', same intent, zero rationale signal in text.
we ended up treating explicit issue references as the stronger gate and using marker presence as a tiebreaker.
do you weight those signals differently or is it an OR condition to clear the trust threshold?
It’s weighted rather than an OR condition.
Right now explicit rationale markers carry more weight than issue references: markers add more to the local trust score, while an issue reference is only a smaller supporting signal.
Your experience is interesting though. Making issue references the stronger gate and using rationale markers more as a tiebreaker might reduce noise across different writing styles.
I may experiment with that against the evaluation corpus 😸
Thanks for the generous shoutout and writeup, @nyaomaru! 😸
Collaborating on this has been an absolute blast. When tackling the WHY problem, your core decision to treat it as an evidence-selection problem - keeping the author's original words instead of letting an LLM hallucinate or paraphrase reasons is what makes the output genuinely trustworthy.
Designing the 50-case evaluation harness together in PR #212 was a great experience. Your insistence that the ground truth corpus must stay independent of any specific threshold was spot on: it gave us a true yardstick to measure precision and recall instead of guessing.
Seeing this integrate cleanly into the deterministic-first v1 pipeline (structuring the enrichment before rendering instead of patching Markdown) really brings the whole architecture together.
Excited to keep building and refining changelog-bot with you! 🚀
Thank you so much, John! 😸
Working on this together has been a lot of fun for me too!!
Your work on the 50-case evaluation harness and threshold sweeps made this experiment much stronger. It gave us a real way to measure the approach instead of just guessing what “feels safe.”
I’m really happy with how the deterministic-first v1 direction is coming together, and I’m excited to keep building changelog-bot with you! 🚀
Thanks for this article, really useful! 😸
Funny thing: it took me slightly off track, in a good way. Reading about WHY extraction made me realize my own project was missing something basic: a CHANGELOG. I maintain a small vanilla JS 2D game engine (BeeEngine) and went from v2.0 to v2.10.0 with only a README 😅 So thanks to your post, it now has a proper CHANGELOG.md, and future developers will actually know what changed and why.
The idea of treating WHY as a selection problem instead of a generation problem, and keeping the original PR text as evidence, is really smart. changelog-bot looks like a big and well-thought-out project. Good luck with v1, and kudos to John for the benchmark! 🙌
Thank you so much! 😸
This genuinely makes me really happy to hear.
The fact that the article actually pushed you to add a proper
CHANGELOG.mdtoBeeEngineis such a nice outcome 😹I’m really glad the “WHY as selection, not generation” idea resonated with you too.
Thanks as well for the kind words about
changelog-botand the shoutout to John! 🙌Good luck with
BeeEngineand future releases withCHANGELOG.md! 🚀Chiming in here - thanks so much for the kind words, @antonioprosperi2svg !
Skipping changelogs until a project matures is painfully relatable! Really awesome that the post nudged you to set up a proper CHANGELOG.md for BeeEngine.
Teaming up with @nyaomaru on that benchmark was a blast, especially making sure we tested against real-world edge cases rather than guessing. Wishing you all the best with BeeEngine and future releases!
Thank you so much for stopping by! Your post was exactly the push I needed. BeeEngine finally has a proper CHANGELOG.md, and I'll keep it updated with every release from now on.
Testing real edge cases instead of guessing is a great lesson too. I'll try to apply the same approach to my own engine. Thanks again to both of you!
"Deterministic code owns the structure, AI only enriches it" is the line we ended up drawing too. Our first version let the agent take more of the pipeline steps; it was slower and worse at the boring parts, so now a script opens the PR when a developer moves the ticket, and the reason for the change sits in the ticket a person wrote. Which raises a question for your selector: when an agent writes the PR body, its "This reduces duplication" line is still original text. Does the 50-case corpus include agent-written PR descriptions?
Good point 😸 Not explicitly yet.
The current corpus mixes real PR-derived cases with adversarial fixtures, but it doesn’t track whether the source PR description was human or agent written.
I think that’s worth adding, because “quoted from the PR” doesn’t necessarily mean “grounded in human intent” if the PR body itself was generated.
I’ve opened an issue for this, and I’d like to support it in the future. 👍
Glad it turned into an issue. On our side the PR and commit links get posted back into the ticket thread, so the human ask is always one hop from the PR, never inside it. How do you plan to tell the two apart: a flag from the tool that wrote the body, or a guess from the text?
I wouldn’t try to guess from the text.
My current direction is to treat the PR itself as the source of truth. If some intent exists only in an external ticket or tool metadata and isn’t reflected in the PR, I don’t plan to fetch it for WHY extraction. 😸
So if an agent wrote the PR body, the important question is whether the rationale is actually present and explicit there, not whether I can infer who wrote it.
Keeping the quoted sentence is useful for traceability. Would you also store the PR description’s revision or a hash alongside it? A PR body can be edited after release, so a link alone may no longer show the evidence that the selector actually saw.
That’s a really good point 😸
Right now I preserve the exact selected source text, but I don’t store an immutable revision or hash of the PR body alongside it.
A hash or snapshot of the evidence seen at extraction time would make the provenance much stronger, especially if the PR description is edited later. I think that’s worth exploring. 👍
Thanks for clarifying! I'd keep the extraction-time snapshot as well as its hash. The hash can detect a change, but the snapshot lets someone inspect the original evidence after the PR body has been edited. A small 'source changed since extraction' note could then make that visible without rewriting the old changelog entry.
Agreed 😸 Keeping both the snapshot and the hash would give much stronger provenance than the hash alone.
I also really like the idea of a small “source changed since extraction” indicator without rewriting the original changelog entry.
I opened an issue to explore this further, including how much source data should be retained by default 👇
github.com/nyaomaru/changelog-bot/...
Thanks again for the thoughtful suggestion!
The selection-not-generation framing is the right call. I tried pure LLM summarization on changelogs last year and roughly 30% of entries ended up being plausible-sounding reasons the author never actually wrote — worse than no explanation at all. One thing that bit me: regex markers like
so thatandin order toalso match a lot of commit messages that describe how, not why. You'll probably need a second filter pass on the matched sentences or the noise will creep back in.Exactly! That’s the failure mode I was worried about too 😸
The regex markers are only used to find possible WHY candidates, not to accept them.
After that,
Jevasks two separate questions:So something containing so that or in order to can still be rejected if it’s really describing implementation rather than rationale.
I also added implementation-only negative cases to the evaluation corpus for exactly this reason. Really good point though that candidate-extraction layer can get noisy very quickly if it starts doing too much. 🙌
The "no WHY is better than an invented one" rule is the part I'd steal. I build a website audit tool and landed on the same thing... when a site blocks my checker it comes back as null with "couldn't audit" instead of a score, so a blocked site never passes for a healthy one
curious whether the benchmark has any cases where the why only lives in a linked issue ("fixes #123") and not in the PR body
Yeah 😸 The benchmark already has a similar case:
Resolves #123. Closes #456...is expected to produce no WHY.Right now,
changelog-botdoesn’t follow linked issues for rationale. The actual WHY candidates come from the PR body, so if the reasoning only lives in the issue, it returns nothing rather than guessing.I could see linked issues becoming an explicit evidence source later, though. 👍
A useful counterfactual test would keep the cache-fix title and implementation text fixed, then swap only the rationale: one version explains stale cache data, another explains an unrelated README change, and a third has no rationale. The latter two should produce no WHY even when the supplied sentence is an exact quote.
Would you keep those related variants together in a held-out group, tune thresholds on separate cases, and report wrong-change acceptance separately from missing-reason rejection? That would test relevance beyond whether the output came from the PR.
AI-generated test suggestion. I have not run Jev or this benchmark.
That’s a really useful test shape 😸
Jevalready scores two things separately for each candidate "whether it explicitly states a WHY", and "whether that WHY actually applies to the changelog change". So the “wrong change, but perfectly valid quote” case is exactly the kind of false positive I want to reject.For this use case, I intentionally care more about precision than recall, missing a valid WHY is acceptable, but attaching a wrong WHY is not.
The benchmark currently sweeps thresholds and reports aggregate precision/recall/F1, but I’m not yet grouping counterfactual variants as held-out families or reporting wrong-change acceptance separately from missing-reason rejection.
I like that direction though, especially keeping related variants together so threshold tuning can’t accidentally overfit the same example family. I'll try add this test! 👍
Extracting the architectural 'why' rather than just summarizing the git diff diffs is the holy grail of automated changelogs. Most LLM-based release tools just list changed functions, which creates noise for end users. Focusing on semantic intent and user impact makes the resulting notes genuinely readable. Love seeing this level of curation!
Thank you! 😸
That’s exactly the goal.
Diffs are good at telling us what changed, but not always WHY it matters.
The hard part is extracting that intent without letting the model invent it, which is why I’m leaning toward evidence selection over free-form generation.
Really glad that came through! 🙌
Love that the core tension you picked is 'don't let the AI invent a why'. That's the same line I ended up drawing when we started summarizing AI-assisted work.
One guardrail that has held up well for us: treat a missing 'why' as a signal, not a gap to fill. If the PR trail doesn't contain the reasoning, the generator should leave the field empty (or flag it) rather than synthesize something plausible - a changelog with a confidently wrong 'why' is worse than one with no 'why', because readers will trust it.
The second guardrail is making the why cheap to write: a two-line 'intent' section in the PR template, authored by the person who did the work, gives the extractor real material instead of forcing it to guess from the diff. Extraction gets dramatically more reliable once the input is honest.
Curious whether you've experimented with a confidence signal - e.g., marking entries where the why was inferred vs. quoted - and how readers respond to that.
Yes, that’s very close to the conclusion I reached too 😸
AI inference can produce a WHY that sounds completely plausible and is even logically consistent with the diff. The problem is that “plausible” is not the same as “what actually motivated the change.”
Once that inferred explanation gets written into the
CHANGELOG.md, the documented intent can quietly drift away from the real specification or decision that was made.So for the
Jevpath, my current rule is pretty strict. "If the WHY isn’t grounded in actual PR evidence, don’t use it". Missing WHY is better than a convincing but invented one.I do keep confidence internally for candidate selection, but the output itself is still the original PR text. I’m not currently mixing quoted and inferred WHYs.
And I really like your idea of making the WHY cheap to write in the PR template. Better input is probably much more valuable than trying to make the extractor increasingly clever. 😸
Not wanting the model to invent a nice-looking reason is the right instinct, because a plausible WHY is more dangerous than a blank one; once it reads well, nobody re-checks it against the diff. I would ground the WHY against the PR discussion and the linked issues as much as the code change, since the intent usually lives in the review comments more than the patch. Are you pulling from the thread, or just the diff?
Exactly, and the current
Jevpath is actually narrower than that 😸Right now it doesn’t read the diff or the PR discussion thread. It extracts candidate rationale from the PR description, then scores each candidate separately for:
Linked issues and review comments aren’t evidence sources yet. I agree they’re valuable though, especially because intent often gets clarified during review.
The important constraint for me would be keeping the same rule "use them as additional source evidence, not as material for the model to invent a rationale from."
I’ll definitely consider exploring that in the future!
Really cool article! Loved reading this 👏 and its on Devtv :)
Wow, that’s so cool! It’s my first time being featured on a broadcast. 😹
Thx! 😸
The "first two sentences are WHAT/HOW" insight is so practical. Most PR descriptions already have the WHY buried deeper—you just need to find it.
Exactly 😸 The WHY is often there, just not at the top.
The tricky part is finding it without mistaking WHAT/HOW for rationale. 😸
hahahahaha
😹
You are famous!☺️
😸
Very interesting read and approach, thanks for sharing.
Thanks so much! 😸
Really glad you found the approach interesting. Appreciate you reading!
Really interesting Article 😎
Cover image I have no words! 😯
Balancing strict LLM constraints with raw PR text
How Do We Extract the “Why” from a PR into a CHANGELOG with Jev? 🤔
Thx! 😸 John uses magic hand 🦾
Multi-change PRs: How does the model map specific rationale snippets to individual bullet points when a single pull request addresses multiple distinct issues or features at once?
Good question!! 😸
Right now,
changelog-botdoesn’t try to solve this at sub-PR granularity.A PR is effectively the atomic release unit. Changes are deduplicated by PR number, so one PR becomes one changelog item. WHY candidates are then evaluated against that item and the PR title for relevance.
So if one PR contains several unrelated changes, it won’t reliably map separate rationale snippets to separate bullets, those separate change identities don’t exist in the current model.
And to be fair, I think that’s also a useful signal that the PR may be doing too much 😹 For independent user-facing changes, separate PRs give both reviewers and the changelog a much cleaner semantic boundary.
If I support true multi-change PRs later, I’d want to model the sub-changes explicitly with their own stable IDs and run WHY selection per change, rather than asking the model to infer the split from prose. 👍
Chính xác nắm bắt được "why" mới là phần khó nhất, chứ không phải "what" hay "how". Team mình từng thử enforce conventional commits + PR template bắt buộc điền
Reason:/Context:nhưng dev vẫn lười điền hoặc viết một dòng "fix bug" xong.Thực tế hiệu quả nhất là: reviewer bắt hỏi ngay trong PR comment "Tại sao chọn approach này thay vì cái kia?" — câu trả lời đó mới là nguyên liệu vàng cho CHANGELOG. AI có thể summarize được diff, nhưng context quyết định (trade-off, constraint, business reason) thường chỉ nằm trong discussion thread, không có trong code.
Một tip nhỏ: ghi chú
Changelog:ngay trong commit message khi commit có breaking change hoặc decision quan trọng. Sau đó script extract ra thôi, đừng để AI đoán — nó hallucinate context rất nguy hiểm — found it via LabAgent, site: labagent .tech