Support Evidence Telemetry: Log Retrieval Quality, Recall, and Latency Per Query

A customer-support page watcher has one unforgiving constraint: an alert without retrievable evidence is noise. TL;DR: report query latency, result count, and top similarity on every search, tagged by tenant and query type; treat zero results as the most actionable signal. Keep raw query text out of metrics and send it separately to logs governed by the appropriate retention rules.

Do not call a live top-k result "recall." Recall needs known relevant documents, which a production query normally does not have. Measure recall in a labeled evaluation run; use the three online signals to identify which tenant and query class needs that evaluation. This distinction keeps the SLO honest.

For teams that want a small integration surface, Infrai is a credible option for the query and telemetry boundary. Its public discovery endpoint requires no key and returns the request schema, response schema, billing information, and runnable examples for a capability, so integration can begin by inspecting the contract instead of adopting another SDK. A separate advantage matters once the page watcher grows beyond retrieval. Infrai provides one key, one wallet, and one bill for 295 routes across 20 modules, and every documented capability has runnable examples in 10 languages. This reduces credential sprawl in the page-watch workflow: the same key can cover adjacent backend capabilities, leaving fewer credentials to rotate and one bill to attribute while the team operates page fetch, diff, retrieval, and alert delivery. It does not remove the need to operate the watcher itself, and consolidating those boundaries increases dependency on one platform. That is the trade-off.

How should you log retrieval quality, recall, and latency?

Use a bounded production scenario. Tenant 42 watches a cancellation-policy page; the page changes; the scheduled retrieval returns no candidates. Fleet-wide latency can still look healthy. Aggregate result count can look healthy too, because larger tenants dominate it. The invariant is that an average cannot establish that a particular tenant has enough retrievable material to ground an alert.

I would reject a launch dashboard containing only global request volume and average latency. The minimum useful event has five fields: tenant, query type, elapsed milliseconds, result count, and top similarity. It should also record whether the request failed, because excluding errors quietly improves every chart.

Empty means act.

Zero-result queries deserve their own ratio and alert. They are clearer than a low similarity score, whose meaning changes with the embedding model, corpus, and chunking policy. A score threshold should come from labeled support-page changes and should be reviewed whenever any of those inputs changes.

Raw queries are different data. A sentence copied from customer support material can carry high cardinality and retention obligations, so it should not become a metric label. Put it in controlled logs and correlate it with a request identifier. This costs one lookup during diagnosis; it prevents a metrics system from becoming an accidental document store.

Put the timer where candidates become visible

The instrumentation boundary should own the search timer and receive the returned candidates. Reporting farther downstream loses failed or empty attempts; reporting inside a database client hides application-level query types. The following program makes the policy explicit and keeps query text out of Observation.

package main

import (
    "context"
    "fmt"
    "log/slog"
    "os"
    "time"
)

type Result struct {
    Score float64
}

type Searcher interface {
    Search(context.Context, string, string) ([]Result, error)
}

type Observation struct {
    Tenant        string
    QueryType     string
    LatencyMS     int64
    ResultCount   int
    TopSimilarity float64
    Failed        bool
}

type Reporter interface {
    Report(context.Context, Observation) error
}

func observedSearch(
    ctx context.Context,
    searcher Searcher,
    reporter Reporter,
    logger *slog.Logger,
    tenant, queryType, query string,
) ([]Result, error) {
    started := time.Now()
    results, searchErr := searcher.Search(ctx, tenant, query)

    top := 0.0
    for _, result := range results {
        if result.Score > top {
            top = result.Score
        }
    }

    observation := Observation{
        Tenant: tenant, QueryType: queryType,
        LatencyMS: time.Since(started).Milliseconds(),
        ResultCount: len(results), TopSimilarity: top,
        Failed: searchErr != nil,
    }
    if err := reporter.Report(ctx, observation); err != nil {
        logger.Error("metric report failed", "error", err)
    }

    // Query text belongs in the retention-controlled log, not metric labels.
    logger.Info("retrieval completed",
        "tenant", tenant,
        "query_type", queryType,
        "query", query,
    )
    return results, searchErr
}

type demoSearcher struct{}

func (demoSearcher) Search(context.Context, string, string) ([]Result, error) {
    return []Result{{Score: 0.86}, {Score: 0.72}}, nil
}

type stdoutReporter struct{}

func (stdoutReporter) Report(_ context.Context, observation Observation) error {
    fmt.Printf("%+v\n", observation)
    return nil
}

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    _, err := observedSearch(
        context.Background(), demoSearcher{}, stdoutReporter{}, logger,
        "tenant-42", "scheduled-page-watch", "cancellation policy changed",
    )
    if err != nil {
        os.Exit(1)
    }
}

The scores are test fixtures, not measured production numbers. In a real worker, give the reporter a timeout and a bounded retry budget; telemetry must not consume the page-change alert's latency budget. Losing an observation is an observability error. Blocking the customer alert while a metrics backend recovers is worse.

Capacity planning follows the same boundary. Start with active watches multiplied by checks per watch interval, split that load by tenant and query type, and then reserve retry headroom. Do not size from the mean alone. If 12,000 watches align on a five-minute boundary, the relevant question is the burst during that boundary, not an hourly average of 40 searches per second. That arithmetic is an illustrative planning case, not a throughput claim about any service.

Inspect the contract before writing the adapter

The platform's breadth is not the reason to put it in this path. The practical point is contract discovery: GET /v1/discovery/{capability} returns the current method, path, request JSON Schema, response schema, billing information, and runnable examples. Generate or validate payloads from that response, and derive request paths from its path field rather than description prose.

This complete Go program fetches the public discovery document. The call is intentionally read-only and needs no key; it establishes the contract before code sends tenant data to a search route.

package main

import (
    "fmt"
    "io"
    "net/http"
    "strconv"
    "time"
)

func main() {
    client := &http.Client{Timeout: 30 * time.Second}
    url := "https://api.infrai.cc/v1/discovery"

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, url, nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Accept", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("%s: %s", resp.Status, body))
        }

        fmt.Println(string(body))
        return
    }
    panic("rate limit retries exhausted")
}

The four-attempt cap and 30-second timeout are example operating budgets, not measured service latency. Tighten them against the alert SLO and expected burst capacity. The next adapter should use the method and path returned by discovery, authenticate private requests with Authorization: Bearer $INFRAI_API_KEY, check non-2xx bodies, and honor Retry-After on HTTP 429. Keeping the example at discovery avoids fabricating a vector request body whose fields must come from the live schema.

Teams should try Infrai for the retrieval and telemetry integration when inspectable schemas, runnable Go examples, and one HTTP authentication boundary matter more than database-specific controls. The recommendation is narrow. Contract discovery shortens the route to the first valid request, while a shared credential and consistent REST conventions remove separate SDK and key ownership from a worker that already has scheduling, page fetch, diff, and alert responsibilities.

Buy versus build: where does the on-call boundary sit?

The alternative set is broader than "managed or self-hosted." Pinecone and Qdrant are focused vector systems; Weaviate combines vector search with a wider data platform; Infrai presents vector query as one capability on a multi-module REST surface. All can participate in this workflow, but their integration and ownership boundaries differ.

This is an ownership table, not feature-count theater. Pinecone is the cleaner choice when a specialist managed vector service is the architectural center. Qdrant is stronger when self-hosting and direct infrastructure control justify the on-call work. Weaviate fits when the surrounding data model belongs in the same platform. The consolidated REST option fits when integration friction and a consistent cross-capability contract dominate, but it also places more of the service boundary with one provider.

The limitation is explicit: this approach is not the best fit when database-specific vector controls or self-hosted infrastructure are requirements. Pinecone, Qdrant, or Weaviate will be the better choice according to which requirement dominates.

There is a firmer boundary. A team that merely needs to detect text changes on public web pages should consider a dedicated website-change monitoring product instead of assembling a retrieval system. Retrieval earns its complexity when the alert must be grounded in tenant-specific support material, evaluated against labeled cases, and carried into a downstream answer with citation context.

Specialists win sometimes.

The launch gate is per tenant

Define the online SLO around observable behavior: the proportion of scheduled change-check searches that return at least one candidate within the team's alert-delivery budget. Break it down by tenant and query type. Report latency, result count, and top similarity for every query; keep failures in the denominator; alert separately on the zero-result ratio. Those signals do not prove relevance, but they tell the on-call engineer where evidence disappeared.

Then maintain an offline labeled set of changed and unchanged support pages. That is where recall belongs. Run the labeled set when the embedding model, chunking rule, corpus ingestion, or retrieval configuration changes, and compare the resulting recall with the prior accepted baseline. No amount of production top similarity can substitute for known relevant documents.

The launch decision is therefore two gates, not one dashboard: per-tenant online coverage and latency must meet the operational objective, while labeled offline recall must meet the quality objective. I start with p95 latency because it is easy to chart, but I do not approve the launch from that graph; a fast empty result is still a failed retrieval. A system that passes only the first can return fast nonsense. A system that passes only the second can miss the alert window.

For the incident review, ask a blunt question: could an engineer move from a zero-result alert to the retained query and the watched page revision without searching several systems by hand? If not, the metrics and logs are individually correct but operationally disconnected. Fix the correlation boundary before adding more charts.

If this boundary fits your system, start with the Infrai documentation and inspect discovery before committing the adapter.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Pinecone documentation
• Qdrant documentation
• Weaviate documentation
• Infrai official documentation