If you've pointed an AI agent at a live incident, you've seen the failure mode: it invents a plausible-sounding metric, queries it, gets an empty result, and confidently invents another. The Prometheus HTTP API is a bad interface for an LLM — it's powerful, but nothing helps the agent discover what's actually there before it starts guessing.
I work in production engineering, so I built the interface I wanted the agent to have: prom-mcp, an MCP (Model Context Protocol) server for Prometheus written in Go.
What it gives the agent
Three capabilities, shaped for triage rather than raw API access:
- PromQL queries with readable output. Instant and range queries, formatted like a triage note instead of a JSON wall.
- Alert explanations. When an alert is firing, the agent gets the alert joined with the rule definition that produced it — the expression, the labels, the annotations — so it can reason about the real condition.
- Discovery before querying. Series and label discovery tools let the agent check which metrics actually exist, which kills the hallucination loop at the source.
Plus a triage prompt that walks the agent through an incident in order: what's firing, since when, what's the blast radius.
Install
Single Go binary, zero dependencies, stdio transport:
claude mcp add prom -- prom-mcp
Set PROM_URL if your Prometheus isn't at http://localhost:9090. No server handy? prom-mcp demo runs against fixtures so you can see the tool shapes in action. Claude Desktop users can grab the MCPB bundle from the GitHub releases, and it's listed on the official MCP Registry as io.github.aniketatgithub/prom-mcp.
Repo: https://github.com/aniketatgithub/prom-mcp
What I'd like feedback on
The tool shapes. If you run agents against real infrastructure, what questions do you wish yours could ask your metrics stack mid-incident? That list is my roadmap.
Top comments (6)
Discovery-before-query is the right fix, and I'd test one case it can still miss: a metric that exists in the index but has stopped reporting. Series discovery over a time range will list it, an instant query returns nothing once the default lookback (5 minutes, if I remember the default right) passes, and the agent sees the same empty result it got from a hallucinated name, then guesses again.
Two cheap things for the tool output: return "exists, last sample N minutes ago" separately from "no such metric", and when a name isn't found, return the closest real names (edit distance over the metric list) so the agent corrects itself instead of inventing a second one. A test with a fixture that has one dead series and one typo'd name would show whether the loop really ends.
On your feedback question: the one I'd want mid-incident is "what changed in the last 30 minutes", i.e. series whose rate shifted most against the previous hour. Does
prom-mcp demoinclude a fixture with a stale series?Thanks, this is a proper test case and you are right about the failure mode. A series that stopped reporting looks identical to a hallucinated name today, and that restarts the guessing loop the tool exists to end.
Filed both fixes: stale series get reported with their last-seen time, separate from unknown metrics (github.com/aniketatgithub/prom-mcp...), and a not-found name returns the closest real names so the agent corrects itself (github.com/aniketatgithub/prom-mcp...).
On the demo question: no stale fixture yet. The current fixtures are all live series, so I am adding one with a dead series and a typo'd name as the regression test for exactly this.
And yes on "what changed in the last 30 minutes". Rate shift against the previous hour is the mid-incident question I keep coming back to as well. It is next on the list after these two.
@aniketatgithub Agreed, and it is the worst kind of failure to debug: an empty result looks like an answer. Checking names against discovery before any query means a wrong name is caught at the query, not three steps later.
Exactly. An empty result that looks like an answer is worse than an error, at least an error tells the agent to stop and rethink. Catching the wrong name at the query is the whole design.
Discovery before query is the right fix. Hallucinated metric names fail silently, so the list is the guardrail.
Thanks Paul. Silent failures are the worst kind in an agent loop; the list is what makes it stop guessing.