I had an idea, more of a stress-test really.
I have been building my IDE for a few months now, it's actually getting to a pretty solid state, but there's a feature I really wanna try.
Essentially, point it at a repo, have it check the issues/PRs and manage it. It generates a team to manage the codebase and any issue gets delegated to the team member qualified to do it. That team member implements it and the 'team head' reviews it, before a PR is made.
If I run it, pointed at Forem, it'd be going through thousands and probably drain my bank faster than I can hit stop... So instead, what I propose, is I set it up and have it monitor a post's comments. tagged with [feature], [bug], [random], you write your request as a comment and it'll kick off a job to create a PR, then respond to your comment with the PR's link.
The idea being that it can maintain a repo... But private repos dont really have an issues board, you'd likely have a log-sink that errors go to and it'd in that case, look at recurring errors and patch them. Or an app might have a 'user feedback' system, where it'd take that feedback, judge whether it's worth implementing and do it.
The end-goal being when I get Dwarven Stronghold up and running, that'll be the single source of monetization. You can donate, sure, but if you want a feature, it's either free request if it's something genuinely broad-scope, but then it's in a queue, if it's something more urgent, or you want it just for you (who knows, maybe you want pink bearded dwarves), you send your request, it does a quote (literally how much it'd likely cost in development + 5%), then you can pay it and it'll make it. If you're not happy with it, you get 3 rounds of feedback, third being direct to human (me), if you're still not satisfied, then a full refund.
Well that's atleast how I plan on using it. But the core idea is that it becomes your triage team for any project, internal, or external and because you can wire up your own AI provider (including your own), you can essentially use it as your frontline support service... I mostly want to run the experiment to see what's the average cost and success rate, that way I can see whether my VC system, concurrency system and sitemap actually make development cheaper and more accurate. I'll then create a follow-up post after a week or 2 with all the stats, usages, etc. So it's not just a black-box experiment.
Who'd be interested in partaking in the experiment? For the sake of not overwhelming Forem's PR system, these PR's will be pointed at my fork for review, once I've reviewed them, then they'll get pushed through to Forem.
Top comments (15)
[feature] I have a structural edge-case challenge for your agent team.
Add a deterministic “Verification Receipt” widget to article pages for returning readers. Its job is to determine whether the current article state still matches the state the reader previously observed.
1. State Contract
The receipt must produce exactly one of three states:
VERIFIED— explicit hash/revision evidence shows that the content and defined core metadata are identical.CHANGED— explicit evidence shows that the content or defined core metadata has diverged.UNKNOWN— the underlying sensor/source is unavailable, stale, blocked, inconsistent, or otherwise unable to establish either result.Invariant:
UNKNOWNis a first-class state and must never fail-open or be coerced intoVERIFIED. A timeout, 404, missing storage value, malformed response, or unavailable sensor must produceUNKNOWN, not a default-green result.2. Baseline and Evaluation
Baseline creation/update must be a separate operation from evaluation.
The evaluation itself must be read-only: it must not modify, replace, or refresh the saved baseline while performing the comparison.
A first visit with no baseline therefore returns
UNKNOWNrather than silently creating a baseline and reportingVERIFIED.3. Edge Cases
A metadata-only change must count as
CHANGED. For example, a title, tag set, or author change with identical body text must not produceVERIFIED.The receipt should also define its source of truth explicitly: what exact representation is compared (for example, the canonical article payload or another stable revision representation), rather than relying on a UI timestamp alone.
4. Inspectability
Clicking “Why?” should open an audit drawer showing:
body_hash, revision,updated_at,tag_set, etc.);Live,Timeout,Unavailable,Conflicting, etc.);5. Mandatory Tests
CHANGEDCHANGEDVERIFIEDUNKNOWNUNKNOWNUNKNOWNUNKNOWNUNKNOWN, while the previous baseline remains unchangedContext: this comes directly from a failure mode I’ve been investigating in autonomous coding systems: a correct rule executing over a dead or degraded sensor can produce a perfectly green result.
I’m interested in whether your triage team can implement “I don't know” as a valid, inspectable outcome, rather than treating it as an exception to swallow.
Interesting one, that's definitely 1 of those cases where the intricacy is filling the narrower cracks, rather than fill a canvas. I think that's a pretty solid 1 to run, though I doubt the PR'll go live... Counta think of it, I'll look into potentially hosting a forem live (the active codebase), it'll need 1 regardless to actually test it live, but that way once you get your feature request added, you can review the outcome on the demo site? Though I'll need to look where I still have credits, think i still have some on Azure.
Exactly. The widget is just the substrate.
What I’m actually probing is whether your triage team can return
UNKNOWNwhen its sensor is dead or the evidence is insufficient. That’s the failure mode I’ve been studying and experimenting with over the last few weeks.So the merge is secondary for me; the real deliverable is the behaviour and the test suite. It either passes or fails on the dead-endpoint, no-baseline, and conflicting-revision cases.
That’s the result I’d be most interested in seeing in your follow-up post.
Will definitely be interesting!
Me! Still confused, but I am interested (in Dwarven Stronghold).
Basically, put my money where my mouth is. When I do it, write a comment about something you want or want fixed in dev.to and my IDE will implement it. I'll review it (for research purposes and QC) and then create the PR to forem for it. Simple. Eg. @francistrdev I imagine you'd like a gif search, so you dont have to google them?
Aaaah! I know exactly what I want!
dev.to/effessdev/comment/3f65m
A progress bar! Is this how I am supposed to post the comment?
[feature] add a minimal progress bar at the top of the post page to know how much of the post we have read (progress excludes the comment section)
If I got what you are doing correctly, this is a wonderful idea!
Sounds solid, that's exactly how it should be formatted too! That way it'll know how to tag the PR properly when it eventually makes it to forem.
A gif search is useful, but I rarely use the integrated search. I tend to search in Google so that I get a good variety of images to use from. More creativity on my end because you will never know what GIF/Images I use for my article. One day is Anime and the other is just something else.
So set your gif source? Either google (just a blanket gif search), or giphy. Either way, costs nothing server-side to implement, it's all in your browser. I think it'd help people be more animated in their comments and posts, having it easier than embedding or image uploading?
Oh that makes sense. I tend to have experience where the images/gif that is integrated is more "selected" than anything.
Nah, stuff like whatsapp's gif search is terrible, cuz it's too curated. A good ol google search tends to work better, so why try re-invent the wheel?
The architecture/QA split is probably the part that gets more important as agents improve. Writing the code is becoming cheaper, but deciding whether the change actually belongs in the system is still a separate job.
This is a really interesting experiment. I like that you're measuring the actual cost, success rate, and review process instead of treating AI coding as a black box. The follow-up stats should be especially interesting.