DEV Community

Cover image for From Query Fan-Out to Trace: Observing AI Search Journeys with OpenTelemetry
Hastimal Jangid
Hastimal Jangid Subscriber

Posted on AI-assisted

From Query Fan-Out to Trace: Observing AI Search Journeys with OpenTelemetry

#ai #gemini #antigravity #opensource

In the earlier parts of this series, I focused on what the system produced.

I started with grounded search journeys, then added historical visibility analysis, and eventually put an agent in front of that history with Gemini, Google ADK, MCP, and BigQuery.

But once the workflow became more complex, a different question started bothering me:

When something is slow, fails, or behaves unexpectedly, how do I know which stage caused it?

A single request could involve:

Intent parsing
      ↓
Query fan-out
      ↓
Places retrieval
      ↓
Search grounding
      ↓
Evidence aggregation
      ↓
Constraint evaluation
      ↓
Ranking
      ↓
Answer synthesis
Enter fullscreen mode Exit fullscreen mode

And in later versions:

Agent chat
      ↓
MCP tool call
      ↓
BigQuery read
      ↓
Gemini synthesis
Enter fullscreen mode Exit fullscreen mode

Looking only at the final answer was no longer enough.

I needed to see the execution journey itself.

That became V5: OpenTelemetry observability for the AI Search Journey Lab.

Why tracing became necessary

Consider a query like:

Find a coffee shop near Geekdom San Antonio for 6 people to work together, preferably quiet, and open after 8 PM.

From the UI, I can see whether the final recommendations look reasonable.

But suppose the request takes six seconds.

Where did those six seconds go?
Was Gemini intent extraction slow?
Did query fan-out generate too many retrieval tasks?
Did Google Places take most of the time?
Was Search grounding the bottleneck?
Did ranking become expensive?
Did final synthesis take longer than expected?

Without tracing, all I really know is:

Request started

...

Request completed
Enter fullscreen mode Exit fullscreen mode

That is not enough once an AI workflow has multiple stages, external APIs, tools, agents, and data stores.

I wanted this instead:

Request
├── intent parsing              410 ms
├── query fan-out               530 ms
├── places retrieval          1,240 ms
├── search grounding            870 ms
├── evidence aggregation        120 ms
├── constraint evaluation        70 ms
├── ranking                      95 ms
└── answer synthesis            690 ms
Enter fullscreen mode Exit fullscreen mode

Now I can reason about the system.

OpenTelemetry turns the AI search journey into a trace composed of spans for intent parsing, fan-out, retrieval, grounding, ranking, and synthesis.

OpenTelemetry turns the AI search journey into a trace composed of spans for intent parsing, fan-out, retrieval, grounding, ranking, and synthesis.

Step 1: Think in traces and spans

The first concept I needed was simple:

A trace represents one complete operation.

A span represents one step inside that operation.

For V1, the root operation is the search journey itself.

The implementation creates a root span around the full journey and then nests child spans for the major execution stages. The project also propagates a journey.run_id so the spans belonging to the same execution can be correlated.

Conceptually:

Trace: journey.execution
│
├── journey.intent_parsing
├── journey.query_fanout
├── journey.places_retrieval
├── journey.normalize
├── journey.search_grounding
├── journey.evidence_aggregation
├── journey.constraint_eval
├── journey.ranking
├── journey.static_map
└── journey.answer_synthesis
Enter fullscreen mode Exit fullscreen mode

Those names are not arbitrary UI labels.

They correspond to actual stages I wanted to inspect independently.

That immediately gave me something the original application did not have:

structure around execution time.

Step 2: Instrument the stages I actually care about

I did not want to create spans for every Python function.

That would create a lot of noise without adding much value.

Instead, I instrumented meaningful workflow boundaries.

For example:

1. Intent parsing

journey.intent_parsing
Enter fullscreen mode Exit fullscreen mode

This covers Gemini-based intent extraction.

Useful attributes include:

intent.category
intent.has_location
intent.constraint_count
Enter fullscreen mode Exit fullscreen mode

2. Query fan-out

journey.query_fanout
Enter fullscreen mode Exit fullscreen mode

This records the fan-out planning stage.

Useful attributes include:

fanout.total_queries
fanout.places_queries
fanout.search_queries
Enter fullscreen mode Exit fullscreen mode

3. Places retrieval

journey.places_retrieval
Enter fullscreen mode Exit fullscreen mode

Useful attributes include:

places.task_id
places.raw_count
Enter fullscreen mode Exit fullscreen mode

4. Search grounding

journey.search_grounding
Enter fullscreen mode Exit fullscreen mode

Useful attributes include:

search.task_id
search.sources_count
search.queries_count
Enter fullscreen mode Exit fullscreen mode

5. Ranking

journey.ranking
Enter fullscreen mode Exit fullscreen mode

Useful attributes include:

ranking.candidate_count
ranking.top_score
Enter fullscreen mode Exit fullscreen mode

6. Final synthesis 

journey.answer_synthesis
Enter fullscreen mode Exit fullscreen mode

Useful attributes include:

answer.recommendations_count
answer.caveats_count
Enter fullscreen mode Exit fullscreen mode

The project also traces normalization, evidence aggregation, constraint evaluation, and static map generation.

This gave me a trace model that follows the way I already think about the search journey.

Step 3: Add correlation IDs

Timing alone is useful.

Correlation is what makes the traces much more powerful.

The project uses identifiers such as:

journey.run_id
visibility.scan_id
session.id
Enter fullscreen mode Exit fullscreen mode

to connect related operations across the different application stages.

That matters because a V1 journey can later lead to a V3 visibility scan.

And a V4 agent interaction has its own session and turn IDs.

So instead of looking at disconnected events, I can follow relationships such as:

journey.run_id
      ↓
V1 Search Journey
      ↓
visibility.scan_id
      ↓
V3 Visibility Scan
Enter fullscreen mode Exit fullscreen mode

For V4:

session.id
      ↓
v4.session_id
      ↓
v4.turn_id
      ↓
MCP tool call
      ↓
BigQuery read
Enter fullscreen mode Exit fullscreen mode

The V4 tracing model includes spans such as:

v4.chat_turn

v4.mcp_tool.<tool_name>

v4.bigquery.read
Enter fullscreen mode Exit fullscreen mode

with metadata for tool names, brands, date ranges, read-only BigQuery operations, and rows returned.

This is where observability starts becoming useful across versions instead of only inside one function.

Step 4: Propagate trace context

Creating spans is easy.

Creating the correct parent-child relationships is more important.
OpenTelemetry context propagation keeps child operations attached to the correct parent span.

The implementation uses standard OpenTelemetry context handling, including span context attachment and propagation, so nested work such as ranking, BigQuery reads, and tool calls stays under the correct workflow.

That gives me a trace hierarchy like:

journey.execution
│
├── journey.intent_parsing
│
├── journey.query_fanout
│
│   ├── places query 1
│   ├── places query 2
│   └── search query 1
│
├── journey.search_grounding
│
├── journey.ranking
│
└── journey.answer_synthesis
Enter fullscreen mode Exit fullscreen mode

Without parent-child relationships, tracing becomes little more than a list of timestamps.

With them, I get an execution tree.

A complete AI search journey represented as a hierarchy of OpenTelemetry spans.

Step 5: Create a reusable tracing helper

I did not want tracing logic scattered through the application.
So the telemetry layer includes a reusable helper around span creation.

Conceptually, the code looks like this:

with trace_span(
    "journey.ranking",
    attributes={
        "ranking.candidate_count": len(candidates),
    },
):
    ranked = rank_candidates(candidates)
Enter fullscreen mode Exit fullscreen mode

The tracing helper is responsible for things such as:

span creation
attributes
duration
status
error capture
sanitization
Enter fullscreen mode Exit fullscreen mode

The telemetry design includes a dedicated tracer module plus a trace_span(...) context manager so instrumentation can stay small at each call site.

That matters because I did not want observability code to dominate the actual application logic.

The business code should still read like business code.

Step 6: Record status, not just duration

A slow span and a failed span are different problems.

So each span records status.

Conceptually:

StatusCode.OK

or

StatusCode.ERROR
Enter fullscreen mode Exit fullscreen mode

When an exception occurs, the span captures the failure before the exception continues through the normal error-handling path.

That means I can distinguish:

journey.places_retrieval
Duration: 1.8 s
Status: OK
Enter fullscreen mode Exit fullscreen mode

from:

journey.places_retrieval
Duration: 1.8 s
Status: ERROR
Enter fullscreen mode Exit fullscreen mode

That is much more useful during debugging.

The observability tests were designed to verify normal span lifecycle as well as error capture and parent-child relationships.

Step 7: Treat telemetry as sensitive data

One observability mistake I wanted to avoid was dumping everything into traces.

An AI application can easily expose:

API keys
authorization headers
tokens
raw prompts
private user input
URLs containing credentials
exception payloads
Enter fullscreen mode Exit fullscreen mode

So I added an explicit redaction layer.

Sensitive dictionary keys such as:

api_key
key
secret
authorization
password
token
Enter fullscreen mode Exit fullscreen mode

are masked.

The implementation also scans for credential-like patterns such as Google API keys, OAuth tokens, and bearer tokens.

I also avoid storing raw prompts as span attributes.

Instead, telemetry can record safer metadata such as:

query length
sanitized summary
hash
category
constraint count
Enter fullscreen mode Exit fullscreen mode

The design explicitly avoids verbatim system prompts and user prompts in the trace attributes.

That became an important principle for me:

Observability should make the system easier to inspect without turning telemetry into a second sensitive-data store.

Step 8: Keep V5 local first

For the initial observability milestone, I deliberately kept the tracing stack lightweight.

The design uses the OpenTelemetry API and SDK with an in-memory span exporter rather than requiring an external tracing platform.

The core dependencies are:

opentelemetry-api
opentelemetry-sdk
Enter fullscreen mode Exit fullscreen mode

That was enough to support:

TracerProvider
SimpleSpanProcessor
InMemorySpanExporter
Enter fullscreen mode Exit fullscreen mode

without requiring an external collector just to understand the tracing model.

This made V5 useful locally and kept the milestone focused:

Instrument
      ↓
Capture
      ↓
Inspect
      ↓
Debug
Enter fullscreen mode Exit fullscreen mode

The Grafana/Tempo/Prometheus stack comes later.

That becomes Part 6.

Step 9: Retain traces without leaking memory

Streamlit has its own execution model.

The script can rerun from top to bottom as users interact with the app.

So trace retention needs to survive UI interactions without growing forever.

The V5 design uses an in-memory telemetry store plus Streamlit session state.

Completed traces are kept in a bounded buffer instead of being retained indefinitely. The design considered limits such as the last 50 execution runs or roughly 2,000 spans, with older entries discarded automatically.

That gives me enough history for local debugging without creating an unbounded memory problem.

Step 10: Build an observability UI

Raw spans are useful to developers.

A visual timeline is faster.

So V5 adds an Observability [V5] tab to the Streamlit application.
The UI is designed around three things:

Run selection
Execution timeline
Span inspection
Enter fullscreen mode Exit fullscreen mode

The run selector can distinguish workflows such as:

V1 Search
V3 Visibility Scan
V4 Agent Chat
Enter fullscreen mode Exit fullscreen mode

The timeline shows:

span name
start offset
duration
status
Enter fullscreen mode Exit fullscreen mode

And the span inspector exposes safe attributes, correlation IDs, events, and sanitized exception details.

That means I can click a run and inspect the journey without leaving the application.

The V5 Streamlit observability view makes each search, visibility scan, or agent interaction inspectable as a timed execution trace.

Step 11: Trace V3 visibility scans too

Once the tracing foundation existed, I did not want it to stop at V1.

The visibility workflow also gets its own trace hierarchy.

The V3 root span is:

visibility.scan_execution
Enter fullscreen mode Exit fullscreen mode

with attributes such as:

visibility.scan_id
visibility.brand_id
journey.run_id
Enter fullscreen mode Exit fullscreen mode

Child spans include:

visibility.extract_bundle
visibility.save_bundle
Enter fullscreen mode Exit fullscreen mode

The persistence span can record metadata such as:

bigquery.dataset
bigquery.table_count
bigquery.saved
Enter fullscreen mode Exit fullscreen mode

That makes it possible to distinguish:

The search journey was slow
Enter fullscreen mode Exit fullscreen mode

from:

The visibility extraction was slow
Enter fullscreen mode Exit fullscreen mode

or:

The BigQuery persistence stage was slow
Enter fullscreen mode Exit fullscreen mode

Those are operationally very different problems.

Step 12: Trace V4 agent/tool behavior

V4 introduced another layer: agent orchestration.

Now a single user question can result in:

Gemini reasoning
      ↓
ADK tool selection
      ↓
MCP request
      ↓
BigQuery read
      ↓
tool result
      ↓
Gemini synthesis
Enter fullscreen mode Exit fullscreen mode

So V5 also traces agent activity.

The root agent interaction uses:

v4.chat_turn
Enter fullscreen mode Exit fullscreen mode

Tool calls use:

v4.mcp_tool.<tool_name>
Enter fullscreen mode Exit fullscreen mode

And BigQuery reads use:

v4.bigquery.read
Enter fullscreen mode Exit fullscreen mode

The BigQuery span includes a bigquery.read_only attribute so the observability layer can preserve and expose the read-only nature of the V4 agent.

This is useful because now I can answer questions such as:

Which MCP tool did the agent call?
How long did it take?
Which brand did it analyze?
What date range was requested?
How many rows came back?
Did the BigQuery read succeed?

without needing to infer that from the final answer.

OpenTelemetry makes the V4 agent workflow inspectable down to individual MCP tool calls and read-only BigQuery operations.

Step 13: Use traces to find bottlenecks

This is where tracing becomes practical.

Suppose a journey takes 5.4 seconds.

The trace might reveal:

intent parsing             310 ms
query fan-out              460 ms
places retrieval         2,310 ms
search grounding           890 ms
ranking                     80 ms
answer synthesis           740 ms
Enter fullscreen mode Exit fullscreen mode

Now the bottleneck is obvious.

I do not need to optimize ranking.

I do not need to optimize Gemini intent parsing.

I need to investigate Places retrieval.

On another run:

intent parsing             280 ms
query fan-out              440 ms
places retrieval           780 ms
search grounding         2,420 ms
ranking                     75 ms
answer synthesis           810 ms
Enter fullscreen mode Exit fullscreen mode

Now the problem is different.

Tracing turns:

The app feels slow.

into:

Search grounding contributed almost half the request duration.

That is a much more actionable statement.

Step 14: Use traces to debug failures

Latency is only one use case.

Failures become much easier to understand too.

Instead of:

Search failed.
Enter fullscreen mode Exit fullscreen mode

I can inspect:

journey.execution             ERROR
├── journey.intent_parsing     OK
├── journey.query_fanout       OK
├── journey.places_retrieval   OK
├── journey.search_grounding   ERROR
└── remaining stages           not executed
Enter fullscreen mode Exit fullscreen mode

Now I know exactly where to start.

The telemetry test plan also explicitly covers failure injection, sanitized error capture, and span status verification.

This becomes especially valuable once external services are involved.

Step 15: Run the application locally

For local development, I still use the normal project workflow.

Activate the virtual environment:

source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

Then start the project:

python scripts/run_app_locally.py
Enter fullscreen mode Exit fullscreen mode

Running the same search journey locally now produces a complete trace that can be inspected inside the application.

What I learned building the tracing layer

The first lesson was that observability works best when the spans follow the architecture.

If I create traces around random functions, the timeline becomes noise.

If I create spans around real workflow stages, the trace becomes documentation.

The second lesson was:

Correlation matters as much as timing.

A fast span is useful.

A fast span tied to the correct search journey, visibility scan, or chat turn is much more useful.

The third lesson was:

Telemetry needs its own privacy model.

Logging every prompt and exception is easy.

Designing traces that are useful without leaking secrets is harder—and more important.

The fourth lesson was that observability should be added before the system becomes impossible to reason about.

V5 was the point where the application had enough moving parts that tracing stopped being optional for me.

One thing I intentionally kept separate

OpenTelemetry gives me the instrumentation model.

It gives me:

traces
spans
attributes
statuses
context propagation
events
Enter fullscreen mode Exit fullscreen mode

But V5 does not need to solve the entire production monitoring problem.

I deliberately separated:

Instrumentation
Enter fullscreen mode Exit fullscreen mode

from:

Operational dashboards
metrics backend
trace backend
AgentOps
Enter fullscreen mode Exit fullscreen mode

That second layer is the next article.

The project later extends the same telemetry into a Dockerized stack with an OpenTelemetry Collector, Tempo, Prometheus, and Grafana.

But I wanted to understand the traces first.

From traces to AgentOps

Once the traces were available, the next set of questions became operational:

How many requests are running?
What is the average latency?
Which stage fails most often?
How many tool calls does the agent make?
Which traces should I investigate?
Can I see request metrics and traces together?
Can the entire observability stack run locally?

That led to the next milestone:

AgentOps.

Next: Building AgentOps for Gemini Agents with Grafana, Tempo, and Prometheus

In Part 6, I will take the OpenTelemetry instrumentation from V5 and connect it to a complete local observability stack.

The architecture becomes:

Gemini / Agent Application
          ↓
OpenTelemetry
          ↓
OTel Collector
       ↙       ↘
   Tempo      Prometheus
   Traces       Metrics
       ↘       ↙
        Grafana
Enter fullscreen mode Exit fullscreen mode

The focus will move from instrumenting individual runs to operating the system over time.

Source code

The project is open source: https://github.com/hastimal/ai-search-journey-lab

Top comments (1)

Collapse
 
hjangid profile image
Hastimal Jangid •

The project is open source: github.com/hastimal/ai-search-jour...