DEV Community

Cover image for Building AgentOps for Gemini Agents with Grafana, Tempo, and Prometheus
Hastimal Jangid
Hastimal Jangid Subscriber

Posted on AI-assisted

Building AgentOps for Gemini Agents with Grafana, Tempo, and Prometheus

#agents #prometheus #grafana #antigravity

In Part 5, I added OpenTelemetry tracing to the AI Search Journey Lab.

That changed the way I could debug a single execution.

Instead of seeing only:

User Query
    ↓
...
    ↓
Final Answer
Enter fullscreen mode Exit fullscreen mode

I could inspect:

journey.execution
├── intent parsing
├── query fan-out
├── retrieval
├── search grounding
├── evidence aggregation
├── ranking
└── synthesis
Enter fullscreen mode Exit fullscreen mode

Each stage had timing, status, attributes, and parent-child relationships.

That solved one problem:

What happened during this request?

But once I had traces, another question appeared:

How is the system behaving across many requests?

I wanted to see more than one execution.

I wanted:

  • request volume
  • average latency
  • failure rates
  • latency by workflow
  • V4 agent behavior
  • MCP tool activity
  • trace drill-down
  • historical operational trends

That pushed the project from observability toward AgentOps.

For this milestone, I built a local Docker-based observability stack using:

OpenTelemetry
Grafana
Tempo
Prometheus
Docker
Enter fullscreen mode Exit fullscreen mode

The entire stack can run alongside the application locally.

From tracing to AgentOps

OpenTelemetry gave me the instrumentation.

But instrumentation alone is not an operational system.

I needed somewhere for traces to go.

I needed somewhere for metrics to go.

And I needed a UI where I could investigate both.

The architecture became:

Gemini / AI Search Application
              ↓
        OpenTelemetry
              ↓
      OTel Collector
         ↙         ↘
      Tempo     Prometheus
      Traces      Metrics
         ↘         ↙
            Grafana
Enter fullscreen mode Exit fullscreen mode

That is the basic AgentOps stack I wanted.

The application remains responsible for generating telemetry.

The OpenTelemetry Collector handles telemetry transport.

Tempo stores traces.

Prometheus stores metrics.

Grafana becomes the operational view.

The local AgentOps architecture routes OpenTelemetry traces to Tempo and operational metrics to Prometheus, with Grafana providing a unified investigation layer.

Step 1: Keep the observability stack local

One goal was important from the beginning:

I did not need another cloud service just to inspect my AI application during development.

So I created a Docker Compose stack for the observability infrastructure.

The project includes:

docker-compose.observability.yml
Enter fullscreen mode Exit fullscreen mode

GitHub Code: Source Code

along with configuration for:

observability/
├── otel-collector/
├── prometheus/
├── tempo/
└── grafana/
Enter fullscreen mode Exit fullscreen mode

The Docker Compose stack runs the OpenTelemetry Collector, Prometheus, Grafana Tempo, and Grafana locally.

That gives me a repeatable development environment without manually installing each observability component.

Step 2: Use the OpenTelemetry Collector as the routing layer

I did not want the application to know how every backend worked.
The application should emit telemetry.

The collector should decide where that telemetry goes.

Conceptually:

Application
    ↓
OTLP
    ↓
OpenTelemetry Collector
    ├── traces → Tempo
    └── metrics → Prometheus
Enter fullscreen mode Exit fullscreen mode

The project has a dedicated collector configuration:

observability/otel-collector/otel-collector-config.yaml
Enter fullscreen mode Exit fullscreen mode

which routes traces and metrics into the local backends.

This is a useful separation because the application instrumentation does not have to be rewritten every time I change the backend.

Step 3: Export traces from the application

In V5, traces could already be inspected locally.

For AgentOps, I extended the telemetry initialization so traces can also be exported over OTLP when a local collector endpoint is configured.

The local endpoint is:

http://localhost:4318
Enter fullscreen mode Exit fullscreen mode

Export traces from the application

The telemetry client can conditionally attach the OTLP exporter while preserving the original in-memory tracing behavior.

Conceptually:

settings.otel_exporter_otlp_endpoint = "http://localhost:4318"
init_telemetry()
Enter fullscreen mode Exit fullscreen mode

Then normal instrumentation continues to use the same span helper:

with trace_span(
    "journey.search_to_decision",
    attributes={
        "workflow.stage": "v1",
    },
):
    ...
Enter fullscreen mode Exit fullscreen mode

The difference is that the completed trace can now leave the Python process and become visible in Tempo and Grafana.

Step 4: Send traces to Tempo

Tempo becomes the trace backend.

That gives me persistent trace inspection outside the Streamlit application.

For example, I can query Tempo directly:

curl -s "http://localhost:3200/api/search"
Enter fullscreen mode Exit fullscreen mode

Trace Curl

And if I know a trace ID:

curl -s \
  "http://localhost:3200/api/traces/<TRACE_ID>"
Enter fullscreen mode Exit fullscreen mode

During implementation, I used those API calls to verify that traces were actually reaching Tempo before relying on Grafana to visualize them.

I like this verification pattern:

Application generated span
        ↓
Collector received span
        ↓
Tempo stored trace
        ↓
Grafana displayed trace
Enter fullscreen mode Exit fullscreen mode

Note: If the Grafana panel is empty, I can work backwards through the pipeline.

Grafana Tempo showing a complete AI or agent execution as a distributed trace with stage-level spans and latency.

Step 5: Add operational metrics

Traces answer:

What happened in this request?

Metrics answer:

What is happening across the system?

I added a small set of intentionally low-cardinality metrics.
The current implementation emits:

ai_search_journey_requests_total
ai_search_journey_failures_total
ai_search_journey_duration_seconds_count
ai_search_journey_duration_seconds_sum
Enter fullscreen mode Exit fullscreen mode

The key labels are intentionally constrained to values such as:

stage = v1 | v3 | v4

status = ok | error
Enter fullscreen mode Exit fullscreen mode

I deliberately did not put things like these into metric labels:

prompt
raw query
session ID
trace ID
user text
Enter fullscreen mode Exit fullscreen mode

The project explicitly excludes those from metric labels for privacy and cardinality reasons.

That matters because a metric such as:

requests_total{prompt="..."}
Enter fullscreen mode Exit fullscreen mode

would quickly become both a privacy problem and a cardinality problem.

Step 6: Verify metrics before building dashboards

One thing I learned during this milestone was not to assume that because traces are working, metrics are working too.

They are separate telemetry paths.

I verified the collector's Prometheus endpoint directly:

curl -s http://localhost:8889/metrics \
  | grep ai_search_journey
Enter fullscreen mode Exit fullscreen mode

Prometheus Tracing

Also verify that the collector endpoint itself accepts OTLP traffic:

curl -i -X POST \
  http://localhost:4318/v1/metrics \
  -H "Content-Type: application/json" \
  -d '{"resourceMetrics":[]}'
Enter fullscreen mode Exit fullscreen mode

If that returns an HTTP response, 4318 is fine.

If after restarting the app and running one workflow you still get no metrics, run:

docker logs otel-collector --tail 100
Enter fullscreen mode Exit fullscreen mode

and:

grep -RniE 'OTEL_EXPORTER_OTLP_ENDPOINT|otel_exporter_otlp_endpoint|metrics' \
  .env .env.example src/ai_search_journey/config.py
Enter fullscreen mode Exit fullscreen mode

View collector metrics export

Then send me those outputs. The Docker stack itself is not the problem; we now need to confirm the application → collector metrics export path.

That debugging process was actually a good example of why observability needs independent verification.

Step 7: Query Prometheus directly

Before trusting Grafana, I tested Prometheus directly.

For example, total V4 requests:

curl -sG \
  http://localhost:9090/api/v1/query \
  --data-urlencode \
  'query=sum(ai_search_journey_requests_total{stage="v4"})'
Enter fullscreen mode Exit fullscreen mode

Again, Prometheus returned the expected value.

This gave me three independent checkpoints:

Collector
   ↓
Prometheus
   ↓
Grafana
Enter fullscreen mode Exit fullscreen mode

Note: If the Grafana panel looks wrong, you can check the underlying PromQL directly.

Let's open http://localhost:9090 and run:

sum(ai_search_journey_requests_total{stage="v4"})
Enter fullscreen mode Exit fullscreen mode

Prometheus receiving the custom AI Search Journey metrics exported through OpenTelemetry.

Step 8: Build the Grafana dashboard

Once traces and metrics were verified independently, I built the Grafana layer.

The project provisions the dashboard automatically through:

observability/grafana/provisioning/dashboards/
Enter fullscreen mode Exit fullscreen mode

and provisions the Prometheus and Tempo datasources through:

observability/grafana/provisioning/datasources/
Enter fullscreen mode Exit fullscreen mode

The main dashboard is:

AI Search Journey — Observability Overview

with UID:

ai-search-journey-overview

That means the dashboard is infrastructure-as-code rather than something I have to recreate manually every time I start Grafana.

Step 9: Track request rate by workflow

One useful panel is request rate grouped by workflow stage.

The dashboard uses PromQL such as:

sum by (stage) (
  rate(ai_search_journey_requests_total[1m])
)
Enter fullscreen mode Exit fullscreen mode

That lets me distinguish activity from:

v1  → Search to Decision
v3  → AI Visibility Scans
v4  → AI Visibility Agent
Enter fullscreen mode Exit fullscreen mode

This is more useful than a single global request counter.

I can immediately see which part of the application is actually generating activity.

Step 10: Track average execution duration

The next useful view is latency.

The dashboard computes average duration using the accumulated duration sum and count:

sum by (stage) (
  rate(ai_search_journey_duration_seconds_sum[1m])
)
/
sum by (stage) (
  rate(ai_search_journey_duration_seconds_count[1m])
)
Enter fullscreen mode Exit fullscreen mode

This gives me a stage-level operational view:

V1 average duration
V3 average duration
V4 average duration
Enter fullscreen mode Exit fullscreen mode

That is different from the span-level view in Tempo.

Prometheus answers:

Is this workflow getting slower over time?

Tempo answers:

Why was this particular request slow?

I want both.

Step 11: Combine metrics and traces

This is where Grafana becomes much more useful than either backend alone.

Prometheus gives me the macro view:

Request rate
Average latency
Failures
Workflow health
Enter fullscreen mode Exit fullscreen mode

Tempo gives me the micro view:

One request
One trace
Individual spans
Tool calls
Timing
Errors
Enter fullscreen mode Exit fullscreen mode

So the operational workflow becomes:

Grafana metric shows latency increase
             ↓
Choose suspicious time window
             ↓
Open related traces
             ↓
Inspect span hierarchy in Tempo
             ↓
Find slow tool / model / database operation
Enter fullscreen mode Exit fullscreen mode

That is much closer to how I want to debug agentic systems.

The Grafana AgentOps dashboard combines Prometheus operational metrics with Tempo trace investigation for V1 search, V3 visibility scans, and V4 agent interactions.

Step 12: Observe real V4 agent behavior

The AgentOps story becomes more interesting with V4.

A V4 request is not just: prompt → response

It can involve:

User Question
      ↓
Gemini
      ↓
Google ADK
      ↓
MCP tool
      ↓
BigQuery
      ↓
Tool Result
      ↓
Gemini Synthesis
Enter fullscreen mode Exit fullscreen mode

So I added child-span instrumentation around the V4 workflow.

The implementation preserves the root agent span and traces child operations for the agent workflow. A live V4 interaction was used to verify that those traces reached the local Docker observability stack.

That lets me inspect questions such as:
Which tool did the agent choose?
How long did the MCP call take?
Was BigQuery the bottleneck?
Did the agent call multiple tools?
Which stage failed?

That is much more meaningful AgentOps than simply charting LLM request counts.

A real V4 Visibility Agent interaction traced through its agent and tool execution stages.

Step 13: Use TraceQL for agent investigation

Once the V4 spans included consistent workflow metadata, I could filter traces by attributes rather than manually searching through everything.

For example, V4 spans can carry:

workflow.stage = "v4"
Enter fullscreen mode Exit fullscreen mode

The project verified the Grafana-to-Tempo query path against actual V4 agent traces using TraceQL filtering.

That gives me a useful investigation pattern:

Show me V4 traces
      ↓
Find slow request
      ↓
Open trace
      ↓
Inspect MCP child span
      ↓
Inspect BigQuery span
Enter fullscreen mode Exit fullscreen mode

This becomes increasingly valuable as the number of agents and tools grows.

Step 14: Start everything locally

One thing I wanted from this milestone was a simple local workflow.

From the repository root:

source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

Then:

python scripts/run_app_locally.py
Enter fullscreen mode Exit fullscreen mode

That is the normal local startup path for this project.

The stack gives me:

Streamlit
http://localhost:8502

Grafana
http://localhost:3000

Prometheus
http://localhost:9090

Tempo
http://localhost:3200
Enter fullscreen mode Exit fullscreen mode

So one command can give me the application and the local AgentOps environment.

Step 15: Verify that Docker is healthy

If Grafana does not open, I first inspect Docker instead of debugging the Python application.

docker ps
Enter fullscreen mode Exit fullscreen mode

I expect the local observability services to be running.

The project Compose definition can also be validated independently:

docker compose \
  -f docker-compose.observability.yml \
  config
Enter fullscreen mode Exit fullscreen mode

Step 17: Validate the complete stack

The AgentOps milestone is not complete just because Grafana loads.

I validate both code and infrastructure.

The verification suite includes:

./.venv/bin/ruff check .
./.venv/bin/mypy src
./.venv/bin/pytest -q
git diff --check
docker compose -f docker-compose.observability.yml config
Enter fullscreen mode Exit fullscreen mode

During the completed Dockerized observability implementation, the recorded results were:

Ruff
All checks passed!

Mypy
Success: no issues found in 38 source files

Pytest
381 passed, 3 warnings

Git diff check
Clean

Docker Compose config
Valid
Enter fullscreen mode Exit fullscreen mode

Validating both the Python application and the Dockerized AgentOps infrastructure before release.

What I learned building the stack

The first lesson was:

Traces and metrics answer different questions.

Trying to force one to do everything makes the system harder to use.

The second lesson was:

Verify every telemetry layer independently.

I now think in terms of:

Application instrumentation

→ Collector ingestion

→ Backend storage

→ Query result

→ Dashboard
Enter fullscreen mode Exit fullscreen mode

The third lesson was:

Agent observability needs tool-level visibility.

Only tracing the outer Gemini request would hide most of the interesting behavior.

For an agent, I want to see:

reasoning boundary

tool selection

tool execution

database call

response synthesis
Enter fullscreen mode Exit fullscreen mode

The fourth lesson was:

Local observability is extremely useful.

I can experiment with AgentOps architecture without keeping another cloud observability environment running continuously.

What I intentionally did not do

This project is not trying to rebuild a commercial observability platform.

I kept the architecture relatively small:

OpenTelemetry
Collector
Tempo
Prometheus
Grafana
Enter fullscreen mode Exit fullscreen mode

I also do not record full prompts as Prometheus labels.

I do not make trace IDs into high-cardinality metrics.

And I do not treat raw chain-of-thought as something that should be logged.

The goal is operational visibility into the system:

Which stage ran?
Which tool ran?
How long did it take?
Did it succeed?
How often does it fail?
Which trace should I inspect?

That is enough to make debugging substantially better without turning telemetry into a privacy problem.

From AgentOps to healthcare discovery

At this point the AI Search Journey Lab has evolved through several layers:

Grounded Search
      ↓
Journey Analysis
      ↓
AI Visibility
      ↓
AI Visibility Agent
      ↓
OpenTelemetry Tracing
      ↓
AgentOps
Enter fullscreen mode Exit fullscreen mode

For the final article in this series, I want to take the architecture into a domain where grounding and evidence matter even more:

healthcare discovery.

The question changes from:

Which coffee shop should I choose?

to something much more sensitive:

How can an AI-assisted search journey help someone discover appropriate care while preserving evidence, source attribution, and explainability?

That becomes Part 7.

Next: From Search to Care — Applying Gemini Search Journeys to Healthcare Discovery

In Part 7, I will apply the search journey pattern to healthcare discovery.

The focus will be on:

  • Gemini
  • healthcare provider discovery
  • grounded evidence
  • search journey fan-out
  • source attribution
  • deterministic evaluation
  • explainable recommendations
  • care-oriented constraints
  • the boundary between discovery and medical advice

The goal is not to build an AI doctor.

The goal is to explore how the same grounded search architecture can support a more evidence-oriented path from a healthcare question to provider discovery.

Source code

The project is open source: https://github.com/hastimal/ai-search-journey-lab

Top comments (0)