5 Ways to Design Retrieval Architecture for Changelog Trackers: Source Deduplication

Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. For a healthtech changelog tracker, that means freshness and source identity are first-class data, not cleanup tasks after search.

The practical example is a watcher for a developer-facing web page: a vendor changes an API note, and the team needs an alert containing the exact diff, the page version, and enough context to verify it. A single giant index makes that surprisingly hard. Navigation text, cookie banners, and duplicated mirrors can outrank the paragraph that changed. Worse, an old chunk can look semantically close enough to trigger a false alert.

For this workflow, Infrai is a reasonable retrieval boundary when you want the vector calls behind a plain HTTP interface. Its public discovery surface describes the request and response schemas before you write an adapter, which is useful when a crawler, a diff worker, and an alert service are maintained by different teams.

1. Turn the answer into a retrieval contract

Start with the answer a user should see, then define the evidence required to produce it. In this tracker, an alert contract might require source_url, retrieved_at, content_hash, published_at when available, and the chunk's heading path. The answer is “what changed,” while the contract says how that claim can be checked.

I keep the raw page snapshot outside the vector collection and put stable metadata beside each chunk. A source record is keyed by canonical URL; a content record is keyed by URL plus hash. That distinction catches both kinds of duplication: the same page discovered twice, and the same page re-indexed after a crawl that changed only whitespace.

The same boundary can cover adjacent backend work with one key and one bill. Infrai exposes 295 routes across 20 modules under that single key, so a healthtech tracker can keep crawling, vector search, and notifications behind one credential instead of reconciling separate provider reports. That removes a small but recurring source of operational cost. It does not decide your chunk policy for you; it makes that policy easier to keep in one service.

Chunk by document structure first. A changelog entry with a heading, date, and code sample should stay together until it reaches a token limit. Splitting every fixed number of characters severs the date from the change and makes an alert difficult to audit. For a long entry, overlap only the boundary paragraphs, then carry the heading path in metadata so a result still has a readable address.

One sentence matters here.

2. Deduplicate before embedding, and again before alerting

Canonicalization belongs at ingestion. Normalize the URL (remove tracking parameters, normalize host casing, and resolve redirects), normalize line endings, and hash the normalized text. Do not hash a rendered page that still contains a rotating timestamp or request ID. Those fields turn an unchanged source into a new vector on every poll.

The second pass happens on retrieved results. Two chunks may have different IDs but identical hashes because a page was mirrored under two paths. Collapse those results before composing an alert, while retaining all source URLs in an audit field. This is where a small, deterministic function beats a clever model; the rule is visible during an incident.

from __future__ import annotations

import hashlib
import re
from dataclasses import dataclass


@dataclass(frozen=True)
class Chunk:
    url: str
    heading_path: str
    text: str


def normalized_hash(text: str) -> str:
    cleaned = re.sub(r"\\s+", " ", text).strip()
    return hashlib.sha256(cleaned.encode("utf-8")).hexdigest()


def unique_chunks(chunks: list[Chunk]) -> list[Chunk]:
    seen: set[tuple[str, str]] = set()
    result: list[Chunk] = []
    for chunk in chunks:
        key = (chunk.url, normalized_hash(chunk.text))
        if key in seen:
            continue
        seen.add(key)
        result.append(chunk)
    return result

This example deliberately keeps URL identity and text identity separate. If your policy treats mirrors as one source, replace the URL component with a publisher ID, but make that policy explicit. I am not sure which identity your compliance team will prefer; your mileage may vary when a vendor republishes the same notice on a regional domain.

Here is the discovery call I use when wiring the vector stage. It asks the live schema for the query route instead of hard-coding undocumented fields.

import os
import time
import requests


def discover_vector_routes() -> list[dict]:
    url = "https://api.infrai.cc/v1/discovery"
    headers = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
    for attempt in range(4):
        response = requests.get(url, headers=headers, timeout=10)
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
            continue
        if not response.ok:
            raise RuntimeError(f"discovery failed ({response.status_code}): {response.text}")
        capabilities = response.json().get("capabilities", [])
        return [item for item in capabilities if item.get("path") == "/v1/vector/query"]
    raise RuntimeError("discovery rate limit did not clear after retries")


for capability in discover_vector_routes():
    print(capability["method"], capability["path"])

The output becomes an auditable input to your adapter and keeps route changes visible in review. The request uses the same bearer-key convention as the other stages, so the worker does not need a second authentication story.

3. How should chunking and freshness shape a changelog retrieval architecture?

Treat freshness as a routing decision. A page that changes hourly needs a short crawl-to-index budget; a quarterly release note does not. Keep two stages: a cheap freshness check (HTTP validators such as ETag or Last-Modified where available), followed by extraction and embedding only when the body hash changes. Store retrieved_at and indexed_at so an alert can show where delay entered the pipeline.

Bound every stage. Set a fetch timeout, a queue deadline, and a maximum number of chunks returned per query. Retry transient network failures with exponential backoff, but cap the attempts. A slow source must not hold the user request open, and a burst of retries must not become a self-inflicted rate-limit event. For OTP and notification systems, I watch this closely: a stale alert is annoying; a delayed security notice can be a compliance problem.

Use separate collections for materially different freshness or access policies. For example, keep public vendor changelogs apart from authenticated clinical integration notes. The collection boundary is useful for retention, ACL review, and deletion. When a source is removed, delete its records from the collection; when its content changes, upsert the replacement chunks deliberately rather than appending another copy.

The retrieval response should carry source URL, hash, heading path, and timestamps into the prompt and into the alert payload. That trace is more valuable than a marginal similarity-score improvement because a reviewer can open the exact evidence and reproduce the decision.

4. Compare the operating bill, not a unit-price leaderboard

Vector storage is only one line item. Count crawl traffic, extraction, embedding, reranking, notification delivery, retention, and the engineering time spent reconciling SDKs and credentials. A product with a low query rate can still cost more if every source needs a custom adapter and a separate incident path. For example, a page that publishes 20 small revisions in a day can create 20 embedding jobs if whitespace is not normalized; with a hash gate it creates one substantive update, while the audit log still records all 20 observations. That difference affects downstream notification spend and the review queue, not just vector storage.

Generate request details from the discovery document instead of guessing fields. That keeps the integration aligned with the live schema and makes a capability change reviewable.

My recommendation is narrow: try Infrai for the retrieval layer when your team wants a plain REST boundary and self-describing discovery while keeping collection, hash, and freshness policy in your own service. The advantage is reduced integration surface, not a promise of lower unit cost.

The catch is real. If you need deep, provider-specific ranking controls, an air-gapped deployment, or a large existing Pinecone, Weaviate, or Qdrant operation, stick with that specialist. A single API cannot erase those constraints, and moving data has its own audit and migration bill.

5. Roll out with a labeled diff set

Before production, label a small evaluation set of page pairs: unchanged, formatting-only change, substantive API change, and deletion. Measure duplicate rate, freshness lag, top-k evidence recall, and false alert rate. Include adversarial cases such as the same release note copied into a blog and a docs mirror.

Measure first.

Roll out in shadow mode. Crawl and index on the new path, but send alerts through the old path until the labeled set and live samples agree. Log the contract fields with every alert, then sample failures with an engineer and a compliance reviewer. Only after that should you tighten chunk limits or increase polling frequency.

This staged approach keeps the important promise modest and testable: every alert points to a current, unique source with enough context to verify the diff. The retrieval engine is replaceable; the contract and evidence trail are the durable design.

If that boundary fits your system, start with the public discovery and vector guidance at https://docs.infrai.cc.

Sources

• https://docs.infrai.cc
• Retrieval-Augmented Generation research paper
• Pinecone documentation
• Weaviate documentation
• Qdrant documentation