Rebuild the Conversation Store Before You Leave a Threaded Agent API

You should treat conversation persistence as a cutover blocker, not as a cleanup task after the paid runtime is gone. Paid thread APIs hide message identity, tool-call correlation, and replay rules inside vendor session objects you cannot export cleanly. If you switch models or hosts first, you will spend the next week reconstructing chats from logs that never stored those identifiers. Own a narrow conversation store, dual-write for a measured window, and only then retire the threaded API.

What a vendor thread is actually storing

A threaded agent API is not merely a chat transcript with a convenient identifier for support. It is a mutable document that binds user turns, assistant drafts, tool calls, and file handles to one opaque session. When the vendor retries a tool, it often reuses an internal call id that never appears in your application logs. When a user refreshes the page, the host rebuilds the UI from that document instead of from your database.

You usually discover this only during cutover, when the new path cannot find the last successful tool result. The replacement model then repeats a side effect that already happened, such as opening a second support ticket. That failure is a storage problem rather than a prompt problem, and a longer system message will not repair it. You need identifiers you minted, stored results you can replay, and a drain plan that outlives the vendor session.

Inventory the leftovers before you write a migrator

Walk the current product path with a notebook and capture every identifier the paid runtime minted without asking you. Number the findings so engineering, support, and the cutover owner share one leftover list instead of three conflicting ones. Those leftovers later become acceptance tests, so incomplete notes here will hide as production incidents after the SDK is removed.

1. List every thread_id, run_id, message_id, and tool_call_id that appears in logs, webhooks, or SDK objects.
2. Record which of those identifiers your own tables store, and which exist only inside the vendor session.
3. Note whether tool retries reuse the same call id or mint a new one after a timeout.
4. Capture attachment and image references that point at vendor-hosted blobs rather than your object storage.
5. Write down the exact UI behavior when a stream aborts halfway through an assistant message.

You now hold leftovers: orphan identifiers, unowned blobs, and assistant messages that ended without a terminal event. Those leftovers become the acceptance tests for the conversation store you are about to own before any model switch. If a leftover cannot be represented in your schema, the cutover plan is not ready and you should not drain the vendor.

The conversation contract you should own

You should keep this store intentionally boring so any future model host can replay the same turns without a second agent framework. You need a conversation row you mint, a strictly append-only message log, and a mapping table from vendor ids to your ids. The store is a cutover artifact first, and only later a product database, so resist adding memory, ranking, or prompt templates into the same tables. The column that saves you during a host restart is the idempotency key on each tool call.

The following SQL is a worked example contract you can run locally, not production DDL for a multi-region service. Read it as the minimum shape that can survive leaving a threaded API. If your current ORM already has a Chat model, map these fields onto it instead of creating a parallel mythology of sessions.

-- worked example: conversation store you own
CREATE TABLE conversations (
  conversation_id TEXT PRIMARY KEY,
  tenant_id TEXT NOT NULL,
  created_at TEXT NOT NULL,
  closed_at TEXT
);

CREATE TABLE messages (
  message_id TEXT PRIMARY KEY,
  conversation_id TEXT NOT NULL,
  role TEXT NOT NULL CHECK (role IN ('user', 'assistant', 'tool')),
  sequence INTEGER NOT NULL,
  body TEXT NOT NULL,
  created_at TEXT NOT NULL,
  UNIQUE (conversation_id, sequence)
);

CREATE TABLE tool_calls (
  tool_call_id TEXT PRIMARY KEY,
  message_id TEXT NOT NULL,
  conversation_id TEXT NOT NULL,
  name TEXT NOT NULL,
  arguments_json TEXT NOT NULL,
  result_json TEXT,
  idempotency_key TEXT NOT NULL UNIQUE,
  vendor_call_id TEXT,
  status TEXT NOT NULL CHECK (status IN ('pending', 'succeeded', 'failed', 'replayed'))
);

CREATE TABLE id_map (
  vendor_thread_id TEXT PRIMARY KEY,
  conversation_id TEXT NOT NULL
);

When the new host retries a tool because a process restarted, you answer from result_json instead of calling the world again. That is the entire point of owning the store before you own the model path. Vendor call ids remain in id_map and vendor_call_id only so late webhooks can be attached or ignored without creating duplicate rows.

Cutover plan: dual-write, then dual-read, then retire

Do not flip a feature flag that sends live users toward an empty store and a newly chosen model host. Run a three-phase cutover and keep the paid thread API as the source of truth until the drain tests pass. Fail closed on your writes during the first phase, because a missing user turn is worse than a slower request. Support should keep a rollback path to the vendor thread until phase three is boring.

1. Dual-write every accepted user turn into your store while the vendor still owns the thread, and fail the request if your write fails.
2. Start dual-read in a shadow path: rebuild the prompt from your messages, call the new host, and compare tool names plus argument keys against the vendor run.
3. Serve a small internal cohort from your store and the new host, while support still has the vendor thread as a rollback.
4. Freeze new vendor threads, drain in-flight runs, export remaining id maps, and only then disable the SDK.

Phase two is where a cheap shadow host earns its keep, because you need many envelope comparisons before you trust sequence numbers. You should not spend a premium runtime discovering that your first tool result never landed in messages. Shadow traffic is a storage rehearsal, not a model bake-off, so keep the scoring rule narrow.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you are already leaving a paid thread API, MonkeyCode's free model access and free server option can host that shadow path while the conversation store is still under test. The product does not replace the store, the idempotency keys, or the drain checklist above.

Worked example: append, map, and refuse a duplicate tool

The following Python is a local worked example you can save as conversation_store.py and run without a network. It is not a framework, and it does not speak any vendor SDK. Use it to prove that a vendor thread maps once and that a repeated tool key cannot reenter the world. If you already have a database, port the methods rather than shipping SQLite to production.

# worked example: conversation_store.py
from __future__ import annotations

import json
import sqlite3
import uuid
from dataclasses import dataclass
from typing import Any

SCHEMA = """
CREATE TABLE IF NOT EXISTS conversations (
  conversation_id TEXT PRIMARY KEY,
  tenant_id TEXT NOT NULL,
  created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS messages (
  message_id TEXT PRIMARY KEY,
  conversation_id TEXT NOT NULL,
  role TEXT NOT NULL,
  sequence INTEGER NOT NULL,
  body TEXT NOT NULL,
  UNIQUE (conversation_id, sequence)
);
CREATE TABLE IF NOT EXISTS tool_calls (
  tool_call_id TEXT PRIMARY KEY,
  conversation_id TEXT NOT NULL,
  name TEXT NOT NULL,
  arguments_json TEXT NOT NULL,
  result_json TEXT,
  idempotency_key TEXT NOT NULL UNIQUE,
  vendor_call_id TEXT,
  status TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS id_map (
  vendor_thread_id TEXT PRIMARY KEY,
  conversation_id TEXT NOT NULL
);
"""

@dataclass(frozen=True)
class ToolReplay:
    tool_call_id: str
    result_json: str
    replayed: bool

class ConversationStore:
    def __init__(self, path: str = ":memory:") -> None:
        self.conn = sqlite3.connect(path)
        self.conn.execute("PRAGMA foreign_keys = ON")
        self.conn.executescript(SCHEMA)

    def open_from_vendor(self, tenant_id: str, vendor_thread_id: str) -> str:
        row = self.conn.execute(
            "SELECT conversation_id FROM id_map WHERE vendor_thread_id = ?",
            (vendor_thread_id,),
        ).fetchone()
        if row:
            return row[0]
        conversation_id = str(uuid.uuid4())
        self.conn.execute(
            "INSERT INTO conversations VALUES (?, ?, datetime('now'))",
            (conversation_id, tenant_id),
        )
        self.conn.execute(
            "INSERT INTO id_map VALUES (?, ?)",
            (vendor_thread_id, conversation_id),
        )
        self.conn.commit()
        return conversation_id

    def append(self, conversation_id: str, role: str, body: str) -> str:
        seq_row = self.conn.execute(
            "SELECT COALESCE(MAX(sequence), 0) FROM messages WHERE conversation_id = ?",
            (conversation_id,),
        ).fetchone()
        sequence = int(seq_row[0]) + 1
        message_id = str(uuid.uuid4())
        self.conn.execute(
            "INSERT INTO messages VALUES (?, ?, ?, ?, ?)",
            (message_id, conversation_id, role, sequence, body),
        )
        self.conn.commit()
        return message_id

    def record_tool(
        self,
        conversation_id: str,
        name: str,
        arguments: dict[str, Any],
        idempotency_key: str,
        vendor_call_id: str | None = None,
    ) -> ToolReplay:
        existing = self.conn.execute(
            "SELECT tool_call_id, result_json, status FROM tool_calls WHERE idempotency_key = ?",
            (idempotency_key,),
        ).fetchone()
        if existing and existing[1] is not None:
            return ToolReplay(existing[0], existing[1], replayed=True)
        if existing:
            return ToolReplay(existing[0], "null", replayed=True)
        tool_call_id = str(uuid.uuid4())
        self.conn.execute(
            "INSERT INTO tool_calls VALUES (?, ?, ?, ?, NULL, ?, ?, 'pending')",
            (
                tool_call_id,
                conversation_id,
                name,
                json.dumps(arguments, sort_keys=True),
                idempotency_key,
                vendor_call_id,
            ),
        )
        self.conn.commit()
        return ToolReplay(tool_call_id, "null", replayed=False)

    def complete_tool(self, tool_call_id: str, result: dict[str, Any]) -> None:
        self.conn.execute(
            "UPDATE tool_calls SET result_json = ?, status = 'succeeded' WHERE tool_call_id = ?",
            (json.dumps(result, sort_keys=True), tool_call_id),
        )
        self.conn.commit()

    def prompt_turns(self, conversation_id: str) -> list[dict[str, str]]:
        rows = self.conn.execute(
            "SELECT role, body FROM messages WHERE conversation_id = ? ORDER BY sequence",
            (conversation_id,),
        ).fetchall()
        return [{"role": role, "content": body} for role, body in rows]

You mint idempotency_key from tenant, conversation, tool name, and a canonical argument digest that stays stable across hosts. A free server that restarted mid-call must hash to the same key, or you will double-create tickets while believing you built a careful store. Do not include wall-clock time or a random nonce in that key, because retries would then look like new work.

# worked example: test_conversation_store.py
from conversation_store import ConversationStore

def test_vendor_thread_maps_once():
    store = ConversationStore()
    first = store.open_from_vendor("tenant-a", "thr_paid_1")
    second = store.open_from_vendor("tenant-a", "thr_paid_1")
    assert first == second

def test_duplicate_tool_does_not_reenter_the_world():
    store = ConversationStore()
    cid = store.open_from_vendor("tenant-a", "thr_paid_1")
    store.append(cid, "user", "Create a support ticket for order 9.")
    first = store.record_tool(
        cid, "create_ticket", {"order_id": "9"}, idempotency_key="t:a:c:create_ticket:9"
    )
    store.complete_tool(first.tool_call_id, {"ticket_id": "T-100"})
    replay = store.record_tool(
        cid, "create_ticket", {"order_id": "9"}, idempotency_key="t:a:c:create_ticket:9"
    )
    assert replay.replayed is True
    assert "T-100" in replay.result_json

Run the checks with a plain interpreter so the cutover checklist is executable instead of a slide. If either test fails, you are not ready to dual-read, let alone drain the paid thread API.

python -m pip install pytest
python -m pytest test_conversation_store.py -q

Shadow comparison you can actually score

Do not score free-text similarity during phase two, because wording will drift as soon as the model or temperature changes. Score the envelope: selected tool name, argument keys, and whether your store would have replayed a side effect. Textual drift is not a cutover defect in this diary; an unblocked duplicate ticket is. Keep the comparator small enough that a reviewer can read it in one sitting.

# worked example: envelope score, not prose score
def envelope_score(vendor_tool: dict, local_tool: dict, replayed: bool) -> dict:
    return {
        "same_name": vendor_tool.get("name") == local_tool.get("name"),
        "same_arg_keys": set(vendor_tool.get("arguments", {}))
        == set(local_tool.get("arguments", {})),
        "replay_blocked_side_effect": replayed,
    }

A proposed gate for the internal cohort is envelope equality on comparable turns, zero unblocked duplicate side effects, and zero conversations that cannot be rebuilt from prompt_turns. You can tighten wording later, after the store is the source of truth and the vendor thread is only an archive. If argument values disagree only by JSON types, canonicalize numbers and strings before you call that a mismatch.

Leftovers that remain after a clean drain

Even a careful drain leaves debris, and you should budget support macros for it instead of pretending the cutover ends at SDK removal. Old bookmarks still deep-link to vendor thread URLs that will 404 once the account is closed. Webhook signatures from the paid runtime will arrive late and must be ignored without creating duplicate messages. Partial assistant streams that never received a terminal event need an interrupted status so the UI does not pretend the answer finished.

Attachments are the quiet leftover that breaks multimodal turns after you leave. If images still live on the vendor, your new host cannot see them, and the model will invent around the missing file. Copy blobs into storage you control during dual-write, or strip multimodal turns from the first cohort on purpose. Do not discover this from a customer screenshot of an empty image slot.

Who should not use this cutover

Skip this store if your product is a single-turn classifier with no tools and no user-visible history worth replaying. A request log plus a correlation id is enough there, and a conversation table would only add operational noise. Also skip it if a compliance team requires the vendor to remain the system of record, because in that case you are wrapping the thread API rather than leaving it. This diary assumes you are actually cutting over, not building a decorative mirror of a host you must keep.

This approach also assumes you can fail a write when dual-write is on, including a 500 that the mobile client already knows how to retry. If your current API cannot add that latency without breaking clients, fix that contract before you invent a conversation schema. A store that sometimes drops the user turn is worse than the paid thread you already have, because you will debug ghosts in both systems at once.

Limitations you should state in the design review

The worked example does not encrypt message bodies, does not shard by tenant, and does not expire old conversations on a retention clock. Sequence numbers are local integers, so two writers without a lock will collide and look like a missing turn. Shadow scoring ignores argument value equivalence until you canonicalize types, so "9" and 9 can fail a gate that humans would call equal. Treat those gaps as follow-up tickets, not as reasons to stay on an unowned vendor document.

Free model access and a free server also do not remove product risk around availability, context limits, or output shape. They only make the dual-run cheaper while you prove the store and the idempotency keys. That is why the gate is envelope equality rather than full-text equality, and why the paid thread remains the rollback until drain is done. If you need a cheap host for that shadow path while the paid API is still draining, use MonkeyCode only as the host and keep the conversation contract in your own repository.