Every time I connect an application to a new LLM provider, I run into the same problem: how much provider-specific code do I have to add? One SDK e...
For further actions, you may consider blocking this person and/or reporting abuse
Interesting exploration! One aspect of LLM gateways that I think deserves more attention is the distinction between exposing governance controls and demonstrating that those controls actually hold under real-world conditions.
Things like routing, budget enforcement, and tool access can look straightforward in configuration, but the interesting questions emerge when providers fail, requests are retried, or multiple policies interact.
Model discovery and error handling are particularly interesting here. For example, when a request fails, can a developer reliably distinguish between an invalid model identifier, a model their account can't access, and a gateway configuration problem? Those distinctions matter quite a bit when you're abstracting multiple providers behind a common interface.
I'd be interested to see how these behaviors hold up in the hands-on testing you mentioned. Looking forward to what you uncover!
Thanks, this is a distinction I wish I’d spelled out in the post. A governance feature that looks right in
gateway.tomlis not the same as one that holds up when things go wrong.I should be upfront that I wrote this from the docs and the launch thread and haven’t run the gateway myself. The hands-on results are still ahead of me.
On your error-handling question, I didn’t find anything in the docs that says how the gateway tells these three cases apart:
provider/modelprefix.base_url, a missing key, or a route pointing at a provider that isn’t defined.The docs do give some tools that might help.
respan-gateway checkvalidates the config, and--onlinealso calls each provider./ui/api/routes/explainshows where a model name would go and why. My guess is that neither would catch a model your account can’t access until a real request fails, but that’s an assumption I haven’t verified.The case I’d most want to test is how these failures interact with retries and fallback. An invalid model or a 403 shouldn’t be retried, while a 5xx or timeout should. If a permission error gets retried with backoff, or silently falls through to the next candidate, you could end up with a “successful” response from a model you never meant to use, and the client wouldn’t know why.
My test plan is to send three requests, one per failure type, then compare:
triesor fallback.I’d repeat this on a single model and on a multi-candidate route, since that’s where policies start to interact. If you’ve already tried it, I’d like to hear what error shapes you got, and I’ll include the results in the follow-up post.
That's a solid test plan, particularly the distinction between single-model requests and multi-candidate routes. The interaction between error classification, retries, and fallback is where things can get especially interesting.
I've encountered some ambiguity around model discovery and error reporting in gateway evaluations, particularly when trying to distinguish an invalid model identifier from one that exists but isn't available to the requesting account. The challenge isn't necessarily that the request fails, but whether the resulting error gives the developer enough information to understand why it failed and what to do next.
I'd also be interested in whether the gateway preserves enough context to distinguish a provider-originated failure from one introduced by the gateway itself. A normalized error interface is useful, but losing the underlying failure semantics can make troubleshooting considerably harder.
Your proposed comparison of client-visible responses against request logs should be revealing. Looking forward to seeing what you find!
Thanks, the point about losing failure semantics is the one I'll weigh most heavily. A normalized error is only a net win if it keeps enough of the original for a developer to act on. At a minimum I'd want to see:
If those are missing from both the client response and the request log, you're left guessing, and that's especially painful when a fallback chain hides which hop actually broke.
I'll add a column to the comparison for "origin of failure" so it's judged on its own, separate from whether the status code looks right. I'll also test an upstream 401/403 against a gateway-side misconfiguration to see whether they come back looking the same.
I'll share the results in the follow-up post, including any cases where the gateway behaves better than I expect. If you can share what error shapes you've seen elsewhere, I'd happily use them as a baseline.
The nginx-style small core is the detail I keep coming back to. I have built API gateway middleware where routing, auth, and transformation logic all lived in one codebase, and the moment a third provider needed a special case, every change became a negotiation with the rest of the stack. Making even providers plugins means a special case stays a special case instead of contaminating the core. The honest warning is the compiled-Rust plugin trade: you get isolation of concerns but lose hot-swappability, so your CI pipeline quietly becomes part of the gateway's operational story — a plugin bug can no longer be fixed with a config edit and a reload. I would want the routes-explain endpoint and respan-gateway check wired into that same pipeline as gate checks. One thing the docs did not make clear to me: can two versions of one plugin coexist on different routes, or is it strictly one version per build?
Thanks, this is a sharp read. The “a special case stays a special case” point is the best argument for the design I’ve seen in this thread.
On your question: I can’t give you a documented answer. I read the docs and the launch post and haven’t built a custom plugin, so nothing I found says whether two versions of one plugin can coexist on different routes. My expectation, which is reasoning and not something the docs state, is one version per build. Plugins are compiled-in crates, and routes pick among plugins registered in that one binary. Cargo can technically link two semver-incompatible versions of a crate, but that doesn’t mean the gateway’s registration model would let you name both. I’d treat it as unsupported until a maintainer says otherwise.
If it is one version per build, a few things would stand in for per-route versioning:
I agree on wiring
respan-gateway checkand routes-explain into CI as gates. The pattern I’d try is to snapshot the explain output for a handful of representative model names and diff it on every build. That catches a plugin change that quietly shifts routing. One caution:check --onlinecalls real providers and MCP servers, so I’d keep it out of the per-commit path and run it as a separate pre-deploy gate.This question is also worth raising with the maintainers, since the project is pre-alpha and plugin versioning is cheaper to settle now. If you open an issue, I’d like to know what they say, and I’ll add it to the post.
This is the honest answer, and the Cargo-semver caveat is the kind of detail only someone who actually read the docs would give. The one point I'd push back on gently: the gateway-level canary doesn't dissolve the "negotiation with the rest of the stack" problem, it relocates it. Two builds side by side means two binaries, two configs, and a load balancer split, so a rollback touches three systems instead of one — and that is exactly the kind of 2 AM surgery where outages breed. One addition to your CI gate pattern: snapshot the explain output for the canary build as well and diff it against stable, not just against the previous build. That turns the canary from "let's see what breaks" into a property test of routing parity. On the maintainer question — yes, I'd open that issue now, while plugin versioning is still cheap to settle. Have you seen any plugin architecture that actually did per-route versioning well? I'd love to borrow from it.
Fair pushback, and I'll concede it. A canary doesn't remove the coordination cost, it moves it. Two binaries, two configs, and a traffic split mean a rollback touches three things instead of one, and that's the wrong place to be improvising at 2 AM. I'd soften my suggestion: treat the canary as a stopgap for teams that can't wait on per-route versioning, not as an equivalent.
The parity diff is a good addition. Diffing the canary's explain output against stable, as well as against the previous build, makes the canary check something you can assert on before any traffic moves. I'd add one limit: it proves routing parity, not behavioral parity. Two builds can route identically and still differ in retry handling or error normalization, so I'd pair it with the three-failure-type requests from my earlier test plan, run against both builds.
On your question, I can't name a plugin system I know did per-route versioning well, and I'd rather say that than guess. The closest analogs I can reason about are Envoy's filter chains and Kubernetes' versioned APIs, but I haven't verified how either handles side-by-side versions in practice, so I wouldn't lean on them as models. If you've seen one work, I'd like to hear which and what made it hold up.
I'll open the maintainer issue and link it here, and any answer will go into the follow-up post.
The routing-versus-behavioral parity limit is the sharper cut, and it is the one I would test first: two builds can route identically and still differ in retry handling or error normalization, so a parity diff on explain output alone can bless a build that fails differently under load. I also cannot name a plugin system I have seen do per-route versioning cleanly in practice, and I would not guess one either. The closest shape I have actually operated is versioning the route contract rather than the binary: the schema, the error taxonomy, and the retry policy travel together as one version, so a route can be rolled back without touching the plugin's other routes. Would you version the error contract first, or the schema, if you could only pick one to start with?
Error contract first. The schema mostly fails loudly: a mismatch shows up as a validation error or a parse failure, and someone notices quickly. An error contract can drift silently. If the status code, error type, or retry classification changes between versions, clients that branch on those values keep running and just start behaving differently, which is the exact "fails differently under load" case we've been circling.
It also ties back to the failure-origin point from earlier in this thread. If the contract carries which layer produced the error, the original provider status, and the route candidate attempted, then a version bump becomes something you can diff and test, not something you discover in an incident.
I'd add one caveat. Schema changes are more visible, but they're also harder to roll back once clients depend on them, so I'd want a compatibility rule for the schema from day one even if I don't version it formally at the start.
I haven't operated this myself, so this is a ranking based on failure modes, not experience. Since you have, I'd like to know whether you ever found the retry policy needed to be versioned separately from the error taxonomy, or whether they always moved together.
The
/ui/api/routes/explainendpoint is easy to overlook, but it’s one of the more practical parts of this setup. Being able to see where a model name would go and why is the kind of thing that saves hours when a route behaves unexpectedly.I’m curious about the
pick = "cheapest"logic. Weighting input price plus three times output price is a reasonable default, but real cost depends on your prompt-to-completion ratio. For workloads with long outputs, such as code generation, that heuristic could rank providers differently than your actual bill would.Also, since embeddings aren’t supported yet, how are you handling retrieval pipelines alongside it? A second path just for embeddings feels like it undercuts the “one gateway” idea a bit.
Either way, a good read. The “break it on purpose, then decide” advice is the right way to evaluate any pre-alpha tool.
Thanks for reading this closely, and I agree on
/ui/api/routes/explain. It’s easy to skip past, but when a route does something unexpected, it’s the first place I’d look. That’s why I put it in the pre-deployment checklist next torespan-gateway check.You’re right about
pick = "cheapest". Input price plus three times output price is a fixed heuristic, so it assumes a typical prompt-to-completion ratio. For code generation or other long-output work, the ranking can differ from what your bill shows, especially when two providers have different input/output price ratios. For those workloads I’d treatcheapestas a starting point and check it against real token logs. The docs also listweightedandfastest, which may suit you better if you know your traffic shape. I haven’t benchmarked this myself, so I’d be curious whether anyone has compared the heuristic’s ranking with actual spend.On embeddings, I think you’re right. The docs don’t list them, so retrieval pipelines need a separate path for now, and that does dent the “one gateway” pitch. I’d say that’s a reason to scope the gateway to chat, speech and similar traffic today, rather than try to route everything through it. Since providers and features are plugins, embeddings could be added later, but I wouldn’t count on that until it ships.
Glad the “break it on purpose” advice landed. For a pre-alpha tool it’s the only evaluation that tells you much. Thanks again!
Thanks for the straight answer, and for flagging that you haven’t benchmarked the heuristic yourself. That’s rare in gateway write-ups.
On
pick = "cheapest", here’s a quick way to check it. Pull a week of real token logs and compute the actual blended cost per provider using your real input-to-output ratio. Then compare that ranking against what the 1:3 weighting produced. If the two orderings match, the default is fine for your traffic. If they differ, the gap tells you how far your workload sits from the heuristic’s assumption. The break-even is easy to calculate: if your output-to-input ratio is well above 3:1 (code generation, long-form drafts), output price dominates and a provider with cheap input but expensive output will look better on paper than on your invoice. The reverse holds for RAG-style prompts with huge context and short answers, where the heuristic overweights output.That also makes me wonder whether
weightedlets you set your own input/output weights. If it does, it might be the better default for anyone who already knows their ratio, and it would be worth a short follow-up post comparingcheapestvsweightedagainst real spend.On embeddings, scoping the gateway to chat and speech today seems like the honest framing. One practical suggestion for anyone running retrieval alongside it: keep the embeddings client behind a thin interface in your own code. If the gateway adds embedding support later, you can swap the backend without touching the retrieval pipeline, and you avoid coupling your vector index to whichever provider you picked as a stopgap. It’s worth noting that changing embedding models means re-indexing, so that decision is stickier than the chat routing one.
If you do end up running a comparison on real traffic, I’d read that post.
Thanks, Rafid. This is a good way to test it, and I’d run it before trusting
cheapeston a real route.You’re right about the break-even. The post only says the
cheapestscore is input price plus three times output price. It doesn’t say where that 3 comes from. I’d add that the same check works for a RAG-heavy route as well as a code-generation one. If the ranking flips on either, the default is wrong for that traffic.On
weighted, I can’t tell you whether it takes custom input/output weights. The docs I read list it as apickoption, but I didn’t find its parameters, and I haven’t run it. If it does let you set the ratio, I agree it would be the better choice for anyone who already knows their traffic shape. I’ll check the config reference before I say more. If anyone here has used it, I’d like to know.A
cheapestvsweightedvs actual-spend comparison is a good follow-up. I’d cover:pickmode, compared against the provider invoicesOn embeddings, I like the thin-interface suggestion. Re-indexing is what makes that choice sticky, so keeping your own abstraction in front of the embeddings client costs little and protects the vector index. I’ll add a short note to the post.
If I get real numbers, I’ll publish them with the test setup so you can reproduce or challenge them.
The 100k lines of wrapper code story is the part that got me, that's such a familiar way to die. Curious how the plugin sandboxing works though, if a bad provider plugin hangs does the whole core stall or is dispatch isolated per event?
Thanks, glad the 100k-lines story landed. It’s the part that stuck with me too.
On sandboxing, I can’t give you a documented answer. I went through the docs and the launch post, and I haven’t run the gateway myself. What I can say:
To check it, point a route’s first candidate at a provider that accepts the connection and never responds. Then send concurrent requests to a different route and watch latency in
/ui/requests.Also tryhedgeandtrieson that route to see how they behave with a hang versus a clean error.If you test it before I do, I’d like to hear what you find.
Appreciate you separating what the docs say from your own guess, that’s rare in posts like this. The “same process, no sandbox” point answers most of what I was wondering.
The distinction you drew between a plugin that awaits forever and one that blocks a runtime thread is the one I’d worry about most. The first only hurts its own request, but the second can quietly degrade everything on that runtime, and it won’t show up in a happy-path test. A per-plugin timeout or a
spawn_blockingconvention in the plugin guidelines would go a long way here, so it’s worth asking the maintainers whether either is planned.The never-responding provider test is a good one, and I’d add a variant where the connection opens and then stalls mid-stream. A hung stream after the first token is a different failure from a hung connect, and I’d guess
hedgeandtrieshandle them differently.Thanks, Mayur. That's the distinction I'd put the most weight on too. A hung await is a bad day for one request, but a blocked runtime thread degrades everything quietly, and a happy-path test will never show it.
You're right that the mid-stream stall is a separate case from a hung connect. Once the first token has gone out, the gateway can't cleanly retry or fall back without the client seeing a broken or duplicated response. I didn't find anything in the docs saying how
triesorhedgebehave after a stream has started, so I'd test it directly: stall after the first chunk, then see whether the request fails, retries, or hangs until the client gives up.A per-plugin timeout or a
spawn_blockingconvention in the plugin guidelines would help a lot. I haven't seen either mentioned, so I'd open an issue and ask the maintainers whether they're planned. Since the project is pre-alpha, now is probably a good time to raise it.If I run these tests, I'll post the results here. If you get to them first, I'd like to hear what you see.
The plugin-first design is what stood out to me too. Using Nginx’s small-core idea for an LLM gateway makes sense, and a provider being a short file with quirks like Groq’s usage field split into their own tested plugins sounds much easier to maintain than a 100k-line wrapper.
I also liked that you covered the trade-offs. Needing a CI step for custom plugins instead of pulling a container image is a real cost, and the checklist for pre-alpha software is useful: pin versions, break it on purpose, measure time to first token.
The MCP
tool_loopsection was the most interesting part. Keeping credentials on the gateway so the client needs no MCP support is convenient, but it’s good you flagged that any client key can call any tool by default. I’d want deny lists and per-key globs set up before trying it.Have you tried the hedging setup (
hedge = "800ms") under load? I’m curious how much it increases upstream cost compared to plain retries with fallback. Thanks for sharing!Thanks for the thoughtful read! Glad the architecture section came through, since that’s what made this gateway more interesting to me than the usual “one endpoint, many providers” pitch.
On your question: I haven’t load-tested hedging yet. Everything in the post comes from the docs and the launch thread, and I didn’t want to claim results I hadn’t measured. From the docs, though, hedging should cost more than retry-with-fallback. Retries only fire after a failure, while a hedge starts a second request after the delay (800ms in the example) even if the first one is still on its way. For slow-but-healthy providers, you’d often pay for both calls to cut tail latency.
If you try it, I’d compare three setups: plain fallback, retries with
tries = 2, andhedge.Check the request log for cost per request and p95 latency. I’d also love to hear what you find, and I’ll add a follow-up if I get numbers of my own.And agreed on the MCP point: the default of any client key reaching any tool is the first thing I’d lock down.
The part I found most useful was the section on trade-offs. Posts often stop at the setup steps, but listing what to verify before relying on the tool, such as failure behavior, latency, and what gets logged, is what actually helps someone decide. I also appreciated that you separated what the docs claim from what you tested yourself. Do you plan to follow up with real numbers once you’ve run it?
Thanks, that’s really kind of you to say. The trade-offs section was the part I most wanted to get right, because a setup walkthrough only tells you the tool can work. What helps you decide is knowing where it might not, and I’d rather be upfront about what I verified versus what I’m taking on the docs’ word.
And yes, a follow-up with real numbers is the plan. I want to measure added latency under load, what happens when a provider fails or times out (retries, fallback, error shape), and what ends up in the logs by default. I’d rather publish actual measurements, with the test setup described, than vague impressions, so anyone can reproduce or challenge them.
If there’s a specific scenario you’d want covered, like streaming responses or a particular provider mix, let me know and I’ll try to include it. Thanks again for reading so carefully.
The “route to the cheapest model that supports tools” idea is what got my attention. It’s a small config line (
pick = "cheapest"withneeds = ["tools"]) that would normally require custom code in an app.I’m curious how it behaves in practice, though. If a fallback model has a shorter context window or weaker tool calling, would you only notice from the request log? I’d want to test that before trusting it with real traffic.
Also, good call on
/ui/api/routes/explain. Being able to see why a request went where it did is underrated for debugging.You’re right that this is the part I’d most want to see in practice. One caveat: I wrote the post from the docs and launch material and haven’t run it locally, so I can’t tell you how it behaves in practice yet.
From the docs,
needsfilters on declared capabilities (tools, reasoning, vision, pdf). I didn’t see anything about filtering on context length, so a shorter-context fallback could still be picked. The request log shows provider, latency, tokens and cost per request, so a fallback would show up there. A weaker tool-calling model that still lists the “tools” capability would be harder to spot, since the log can’t tell you whether the tool calls were any good.Before trusting it with real traffic, I’d test it like this:
/ui/api/routes/explainis great for the “why did it go here” question, but the quality and context checks are on you.If you try it, I’d love to hear what you find. I’ll add the results to the post if you’re okay with that.
The Nginx-style small core, where even providers are plugins, is a smarter design choice than the usual “one endpoint, many providers” pitch.
The trade-off you flagged is the part I’d think hardest about. Compiled-in Rust plugins mean you get something reviewable, but every custom change becomes a CI and rebuild step instead of a config tweak. For small teams, that could decide whether they adopt it at all. The config-only template plugins should cover most cases, but I’d like to see how far they stretch before someone has to touch Rust.
I also liked the failure-test suggestion: put a bad key first in the route and watch the retry and fallback behavior in the request log. Hedging is the one I’d test carefully, since racing candidates quietly multiplies upstream cost.
Have you tried the
tool_loopMCP setup yet? Having the gateway run the tools and return only the final answer is a neat idea, but the per-keytoolsglobs look essential to keep it safe.Thanks, this is the kind of feedback I was hoping the post would get.
On the trade-off, I agree it could decide adoption for small teams. The config-only template plugins cover adding a vendor or format, but I haven’t pushed them to find where the limit is. Custom routing or request-rewriting logic is where I’d expect someone to hit Rust. If anyone has found that boundary, I’d like to hear about it. Until then, I’d start with the template plugins and only commit to the rebuild pipeline if a real need shows up.
You’re right about hedging. Each racing candidate is a real upstream request, so cost multiplies quietly. I’d test it with a short delay and a cheap model pair first, and compare the request log against the provider’s billing dashboard before putting it on a busy route.
On tool_loop: not yet, honestly. What’s in the post comes from the docs, and I haven’t run it myself. I agree the per-key
toolsglobs look essential. By default any client key can trigger any tool you expose, using the server’s credentials, so I’d start with a narrow allowlist per key and deny lists for anything destructive. If I test it, I’ll post what I find. If you try it first, I’d love to hear how it goes.Appreciate the honesty about tool_loop coming from the docs rather than a test run. That’s useful to know, and it makes the allowlist-first approach easier to trust.
The hedging plan is sensible. One thing I’d add: when you compare the request log against the billing dashboard, run it for a day or two, not a single test. Racing candidates often look cheap in a short burst, but the real cost shows up with long prompts or streaming responses, where the losing request may still run to completion before it’s cancelled. It’s worth checking whether the gateway actually cancels the slower upstream call or just ignores its result.
On finding the limit of the template plugins, I’d probably try a small, deliberately awkward case first: a provider with a non-standard auth header or a slightly different streaming format. If that works in config alone, that’s a good sign. If it needs Rust, you’ve found the boundary cheaply, before committing to anything bigger.
If I get a chance to try tool_loop this week, I’ll start with a read-only tool and a single narrow key, then report back here. Would be great to compare notes if you test it too.
Thanks, Mayur. These are the checks I wouldn’t have thought to run on my own.
Your point about running hedging for a day or two is a good one. A short burst only shows the average case. Long prompts and streaming responses are where a losing request that keeps running would show up on the bill. The docs say the first answer wins, but I didn’t find anything on whether the slower upstream call is actually cancelled. I’d want to confirm that directly. One way would be to point a hedged route at two providers and compare the request log’s token counts with each provider’s own usage page for the same window. If the losing side shows tokens the gateway never returned, the call is running to completion.
I like the awkward-provider test for finding where the template plugins stop. I’d try a non-standard auth header first and a different streaming format second. If the auth header works in config alone, that tells me something. If the streaming format needs Rust, I’ve found the limit without much sunk cost.
Your read-only tool with a single narrow key is the same way I’d start
tool_loop. I’d also add a deny rule for anything that writes, as a second layer in case the key’s glob is wider than intended. If either of us runs it, let’s compare notes here. I’ll report back too if I get to it first.The 100,000+ lines of wrapper code and the rollback after a routine upgrade explain why Respan went plugin-first. It’s a useful reminder that a gateway turns into infrastructure the moment you depend on it.
What I like most is that the post treats it as a proof of concept, not a production recommendation. Pre-alpha, no official Docker image, and config that may still change are all fair reasons to wait. I’d pin the version and read the release notes before every upgrade.
One thing I’d add to your checklist: test what the
respanplugin sends by default. Sincecontentistrueunless you turn it off, anyone routing prompts with customer data should flip it tofalsefirst.Thanks for including the install steps and the
gateway.tomlexample. They make it easy to try in an afternoon.You’re right that “a gateway turns into infrastructure the moment you depend on it” is the point behind the plugin-first design, and I wanted the post to say that without overselling the project. Pre-alpha means pre-alpha, so I kept it as a proof of concept on purpose.
Pinning the version and reading the release notes before each upgrade is good advice, especially while config and APIs can still change. The
content = truedefault on therespanplugin is easy to miss, so thanks for flagging it. Anyone routing prompts that contain customer data should setcontent = falseand check what actually leaves the gateway before pointing real traffic at it. I’ll add a “verify what’s sent by default” item to the privacy part of the checklist.Glad the install steps and the
gateway.tomlexample made it quick to try. If you run it against your own models and find anything that breaks, especially around fallbacks or provider quirks, I’d like to hear about it.Thanks for adding the privacy item to the checklist. One practical way to make it concrete: send a canary string through the gateway, something like
CANARY-12345in a prompt, and check what shows up in the plugin’s destination. That tells you what actually leaves the gateway, rather than what you assume the config says. Setcontent = falsefirst, run it again, and compare the two.It’s also worth checking more than
content. Metadata like user IDs, API key labels, or custom headers can identify customers even when prompt text is excluded, so I’d review every field the plugin sends, not just that flag.The tradeoff is that with
content = falseyou lose prompt and response visibility when debugging a bad output. A middle path is to log only metadata by default (model, token counts, latency, which route fired) and capture redacted content for a small sample of requests when you need to investigate.On upgrades, pinning the version pairs well with a quick smoke test after each bump. Run
respan-gateway checkand compare/ui/api/routes/explainoutput for a few representative models before and after. If the config format shifts between pre-alpha releases, you’d catch it before it reaches production traffic. Keepinggateway.tomlin version control makes that diff easy to review.I’d be curious what you find when you test fallbacks. A provider timing out mid-stream is a different case from a clean error response, and that’s usually where gateways differ most.
The “pre-alpha, treat it as a proof of concept” framing is the right call, and the production checklist at the end is the most useful part of the post.
What stood out to me is the privacy default. The
respanplugin sends message content unless you setcontent = false, and that’s easy to miss in a quick setup. Anyone routing customer prompts through a gateway should check that setting before the first real request goes out.The state note also deserves more attention than it usually gets. Conversation state for translated Responses API clients living in one instance’s memory means multi-instance deployments need sticky routing, and that kind of detail tends to surface only after the first scale-up.
I’d be interested in a follow-up with real numbers: time to first token with and without the gateway, and how retries and fallbacks behave under actual rate limits.
Since I wrote the post from the docs and the launch thread rather than from my own load testing, I’m glad the checklist helped.
You’re right about the
contentdefault. It’s the line I’d most want people to read before sending any real traffic through therespanplugin. Settingcontent = falseshould be part of the first-run config, not something you find later.The state note is a good catch too. It’s the kind of constraint that is invisible on a single instance and then bites you right after the first scale-up. Sticky routing works, but I’d want to see what happens to in-flight conversations when an instance restarts.
A follow-up with real numbers is a good idea, and I’d like to do it properly. My rough plan is:
If you have a setup or scenario you’d want covered, tell me and I’ll fold it in. Thanks for reading so closely.
Very insightful thank you for sharing!