Set up Bedrock model fallback with AgentCore Gateway

Setting up a Bedrock model fallback used to be manual work. If you wanted to react to a ThrottlingException or a 503 from Amazon Bedrock, you wrapped InvokeModel in a try/catch, hard-wired a second model and rebuilt that logic in every service. Since October 22, 2026, AgentCore Gateway can take over this job centrally: it now speaks the Bedrock InvokeModel API and, based on a rule, reroutes requests to a backup model when the primary model is throttled, unavailable or too slow.

This topic is on my mind because LLM calls are moving from experiment to production dependency in many teams right now. A chatbot that simply shows an error message when it hits a 429 during peak load gets noticed. In this article you build the setup from scratch: gateway and Bedrock target with AWS CDK, the fallback rule with the AWS SDK for TypeScript, a client that only changes its endpoint, and a small chaos test that actually triggers the fallback.

In short

AgentCore Gateway accepts Bedrock InvokeModel calls and, on 429, 5XX or timeouts, works through a chain of up to two backup destinations, including a circuit breaker. Your client only changes its endpoint. The biggest pitfall: backup models must understand the same payload schema, and throttling (429) does not trigger a fallback unless you configure it explicitly.

What you will learn in this article

• How rules, fallback chain and circuit breaker work together in AgentCore Gateway
• Set up the gateway and a Bedrock target with AWS CDK in TypeScript
• Create a fallback rule that also kicks in on throttling (429)
• Point the Bedrock client at the gateway without rewriting code
• Trigger the fallback on purpose and avoid typical mistakes

What is new about Bedrock model fallback in AgentCore Gateway?

AgentCore Gateway is a managed entry point from the Bedrock AgentCore family. Originally it was mostly about tools for AI agents: exposing APIs and Lambda functions as MCP tools, securing access, logging calls. By now the gateway can also act as an AI gateway. That means a proxy that accepts LLM requests and forwards them to the right provider based on the requested model. AWS calls these provider connections inference targets.

The announcement from October 22 added two building blocks. First, the inference endpoint now understands the wire format of InvokeModel and InvokeModelWithResponseStream. Existing code using the Bedrock SDK keeps working, it just points at a different URL. Second, a gateway rule can hold an ordered chain of backup destinations, together with the conditions under which the gateway switches over.

What the client notices of the fallback

Practically nothing, and that is the point. Your code sends one request and gets one response. Whether it came from the primary model or the backup model is invisible to the client. Retries happen inside the gateway, and the routing logs record which destination answered. That matters for debugging: if responses suddenly sound different, look there first.

The rule in the gateway decides when a request is rerouted from model A to model B.

The building blocks at a glance

When a model fallback is worth it

Before you start, an honest assessment. A fallback is not a free safety net. The backup model answers differently, often shorter or less precise. For an internal assistant that summarizes tickets, that is perfectly fine: a slightly simpler summary beats no summary. For a pipeline that produces structured data read by a downstream parser, things look different. There you should test the backup model as thoroughly as the primary one, otherwise you trade a visible error for a silent one.

There is also a technical limit that is somewhat hidden in the docs. The gateway translates between API dialects, but it does not rewrite the request content. An InvokeModel body for a Claude model uses the Anthropic Messages format. The backup model has to understand that format too. In practice this means falling back within a model family (large Claude model to smaller Claude model) or to the same provider via a different route. A fallback from Claude to an OpenAI model with the same body fails with a 400.

My take: for most teams, a fallback within the same family is the best starting point. It solves the most common problem, throttling during peak load, and it stays easy to test. If you want more independence, you can later add a second target that connects to the provider directly. That brings API keys, another contract and data protection questions, though, and those need answers first.

Set up Bedrock model fallback step by step

The goal of this section is a working setup: a gateway with IAM authentication, a Bedrock target, a rule with a fallback chain and a TypeScript client. The infrastructure is built with AWS CDK, the rule with a short SDK script. I explain why in step 4.

Prerequisites

• An AWS account with access to Amazon Bedrock and two models of the same family in your region (for example a large and a small Claude model)
• A region where AgentCore Gateway is available. Check the AgentCore region overview first.
• Node.js 22 or newer, AWS CLI v2 and an account prepared with cdk bootstrap
• For step 3, the awscurl tool (pip install awscurl) to send signed requests

Step 1: Create the project

Create a folder and install the dependencies. These are the versions I checked the code with.

mkdir model-fallback && cd model-fallback
npm init -y
npm install [email protected] constructs @aws-sdk/[email protected] @aws-sdk/[email protected]
npm install -D aws-cdk typescript tsx @types/node
mkdir bin lib scripts src

Add a tsconfig.json and a cdk.json that tells CDK how to start the app:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "commonjs",
    "moduleResolution": "node",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "dist"
  },
  "include": ["bin", "lib", "scripts", "src"]
}

{
  "app": "npx tsx bin/app.ts"
}

Step 2: Gateway and Bedrock target with CDK

The stack has three parts. An IAM role that the bedrock-agentcore.amazonaws.com service may assume and that the gateway uses to sign its calls to Bedrock. The gateway itself with AWS_IAM as authorizer. And an inference target named bedrock, connected through the bedrock-runtime connector.

// lib/model-gateway-stack.ts
import { CfnOutput, Stack, StackProps } from "aws-cdk-lib";
import { CfnGateway, CfnGatewayTarget } from "aws-cdk-lib/aws-bedrockagentcore";
import * as iam from "aws-cdk-lib/aws-iam";
import { Construct } from "constructs";

export class ModelGatewayStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // Role the gateway uses to sign its outbound calls to Bedrock
    const gatewayRole = new iam.Role(this, "GatewayRole", {
      assumedBy: new iam.ServicePrincipal("bedrock-agentcore.amazonaws.com", {
        conditions: { StringEquals: { "aws:SourceAccount": this.account } },
      }),
    });

    // Invoke models (including streaming) ...
    gatewayRole.addToPolicy(
      new iam.PolicyStatement({
        actions: ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
        resources: ["*"],
      }),
    );
    // ... and read the model catalog, otherwise the target never becomes "Ready"
    gatewayRole.addToPolicy(
      new iam.PolicyStatement({
        actions: ["bedrock:List*"],
        resources: ["*"],
      }),
    );

    // Gateway without protocolType: MCP would only allow MCP targets
    const gateway = new CfnGateway(this, "ModelGateway", {
      name: "model-gateway",
      roleArn: gatewayRole.roleArn,
      authorizerType: "AWS_IAM",
      description: "One endpoint for all LLM calls, with fallback rules",
    });

    // Inference target "bedrock": the connector discovers the models itself
    const bedrockTarget = new CfnGatewayTarget(this, "BedrockTarget", {
      gatewayIdentifier: gateway.attrGatewayIdentifier,
      name: "bedrock",
      targetConfiguration: {
        inference: {
          connector: { source: { connectorId: "bedrock-runtime" } },
        },
      },
      credentialProviderConfigurations: [
        { credentialProviderType: "GATEWAY_IAM_ROLE" },
      ],
    });
    bedrockTarget.node.addDependency(gatewayRole);

    new CfnOutput(this, "GatewayId", { value: gateway.attrGatewayIdentifier });
    new CfnOutput(this, "InferenceUrl", {
      value: `https://${gateway.attrGatewayIdentifier}.gateway.bedrock-agentcore.${this.region}.amazonaws.com/inference`,
    });
  }
}

Two details are easy to miss. First, the stack deliberately sets no protocolType. With MCP, the gateway would only accept MCP targets. Second, besides bedrock:InvokeModel the role needs read access to the model catalog. Without it, the target cannot discover its models and never becomes “Ready”. I use bedrock:List* here, which the docs name as an alternative to listing every action. For production I would also restrict InvokeModel to the models and inference profiles you actually use.

The app entry point is short:

// bin/app.ts
import { App } from "aws-cdk-lib";
import { ModelGatewayStack } from "../lib/model-gateway-stack";

const app = new App();
new ModelGatewayStack(app, "ModelGatewayStack", {
  env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: process.env.CDK_DEFAULT_REGION ?? "eu-central-1",
  },
});

Step 3: Deploy and list models

npx cdk synth
npx cdk deploy

After the deployment, the output shows GatewayId and InferenceUrl. Store both as environment variables and ask the gateway which models it knows:

export GATEWAY_ID=<GatewayId from the output>
export GATEWAY_INFERENCE_URL=<InferenceUrl from the output>
awscurl --service bedrock-agentcore --region eu-central-1 "$GATEWAY_INFERENCE_URL/v1/models"

The response uses the OpenAI model list format, with each ID prefixed by the target name, so something like bedrock/<model-id>. Pick your primary model and a backup model of the same family from it. For many current models these are inference profiles with a prefix such as eu., which Bedrock spreads across a group of regions. If the list is empty, the List permissions from step 2 are usually missing.

export PRIMARY_MODEL=<ID of the large model>
export FALLBACK_MODEL=<ID of the smaller model in the same family>

Step 4: Create the fallback rule

I would actually prefer to describe the rule in CDK as well. aws-cdk-lib 2.274.0 does include CfnGatewayRule, but its type definitions do not know the fallback block yet. Until CloudFormation catches up, I create the rule with the SDK. The script is short and can later be swapped for a CDK block without restructuring anything.

// scripts/create-fallback-rule.ts
import {
  BedrockAgentCoreControlClient,
  CreateGatewayRuleCommand,
} from "@aws-sdk/client-bedrock-agentcore-control";

const gatewayId = required("GATEWAY_ID");
// Example IDs: GET /inference/v1/models returns the real ones
const primaryModel = required("PRIMARY_MODEL");
const fallbackModel = required("FALLBACK_MODEL");
// For the chaos test: a very small value forces FIRST_BYTE_TIMEOUT
const firstByteTimeoutMs = Number(process.env.FIRST_BYTE_TIMEOUT_MS ?? "8000");

function required(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`Missing environment variable ${name}`);
  return value;
}

async function main(): Promise<void> {
  const client = new BedrockAgentCoreControlClient({});

  const result = await client.send(
    new CreateGatewayRuleCommand({
      gatewayIdentifier: gatewayId,
      priority: 100,
      description: `Fallback ${primaryModel} -> ${fallbackModel}`,
      conditions: [{ matchModels: { anyOf: [{ value: primaryModel }] } }],
      actions: [
        {
          routeToTarget: {
            staticRoute: {
              targetName: "bedrock",
              model: primaryModel,
              fallback: {
                circuitBreaker: { mode: "ENABLED" },
                // List 429 explicitly: the default only covers 5XX
                triggerOn: {
                  statusCodes: ["429", "5XX"],
                  errors: ["CONNECTION_ERROR", "FIRST_BYTE_TIMEOUT", "UNROUTABLE"],
                },
                firstByteTimeoutMilliseconds: firstByteTimeoutMs,
                targetFallback: [{ targetName: "bedrock", model: fallbackModel }],
              },
            },
          },
        },
      ],
    }),
  );

  console.log(JSON.stringify(result, null, 2));
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

npx tsx scripts/create-fallback-rule.ts

The rule matches requests for PRIMARY_MODEL and sends them to the bedrock target. If that target answers with 429 or a 5XX, cannot be reached, or takes longer than eight seconds to send the first byte, the gateway tries the same request with FALLBACK_MODEL. Priority 100 leaves room for later rules before and after it.

Pay attention to the list of status codes. If you leave out triggerOn, the gateway only falls back on 5XX and the three connection errors. Throttling by Bedrock, which is exactly the most common reason for a fallback, would not be covered. That is why the script lists 429 explicitly.

Step 5: Point the client at the gateway

Now comes the pleasant part. The client stays a regular BedrockRuntimeClient, only the endpoint points at the gateway. Requests are still signed with the bedrock signing name, which the gateway accepts according to the docs.

// src/ask.ts
import {
  BedrockRuntimeClient,
  InvokeModelCommand,
} from "@aws-sdk/client-bedrock-runtime";

// Gateway instead of Bedrock: only the endpoint changes
const client = new BedrockRuntimeClient({
  endpoint: process.env.GATEWAY_INFERENCE_URL,
  // The gateway walks the fallback chain itself.
  // Many client retries on top only multiply the attempts.
  maxAttempts: 2,
});

export async function ask(prompt: string): Promise<string> {
  const body = JSON.stringify({
    anthropic_version: "bedrock-2023-05-31",
    max_tokens: 512,
    messages: [{ role: "user", content: prompt }],
  });

  const response = await client.send(
    new InvokeModelCommand({
      modelId: process.env.PRIMARY_MODEL,
      body,
      // Required as soon as the gateway translates to another dialect
      contentType: "application/json",
      accept: "application/json",
    }),
  );

  const payload = JSON.parse(new TextDecoder().decode(response.body));
  return payload.content?.[0]?.text ?? "";
}

if (require.main === module) {
  const started = Date.now();
  ask("Explain in two sentences what a circuit breaker is.")
    .then((text) => {
      console.log(text);
      console.log(`Duration: ${Date.now() - started} ms`);
    })
    .catch((err) => {
      console.error(`${err.name}: ${err.message}`);
      process.exit(1);
    });
}

npx tsx src/ask.ts

Expected result: two sentences about the circuit breaker and a line with the duration. If you would rather not touch the code, you can skip endpoint: set the environment variable AWS_ENDPOINT_URL_BEDROCK_RUNTIME to the inference URL. I checked that the SDK then calls the same path /inference/model/<model>/invoke.

The calling role, for example your Lambda function, no longer needs direct Bedrock access. It only has to be allowed to invoke the gateway:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "bedrock-agentcore:InvokeGateway",
      "Resource": "arn:aws:bedrock-agentcore:eu-central-1:111122223333:gateway/<gateway-id>"
    }
  ]
}

I find this an underrated side effect: which models an application may use is now decided in one place, by the gateway role, instead of separately in every Lambda role.

Step 6: Provoke the fallback and verify it

You should not take a fallback for granted that you have never seen in action. The simplest test without a real outage uses the time to first byte. Delete the rule from step 4 (or use a test environment) and create it again with an absurdly small value:

FIRST_BYTE_TIMEOUT_MS=50 npx tsx scripts/create-fallback-rule.ts
npx tsx src/ask.ts

No model delivers its first byte within 50 milliseconds. So the gateway abandons the attempt at the primary model and asks the backup model. Because the gateway always tries the last remaining destination in the chain, you should still get an answer. The gateway’s routing logs show whether the fallback kicked in. If the API rejects the small value, use the smallest allowed one. Afterwards, create the rule again with the normal timeout.

🎬 Animated flow of a model fallback in AgentCore Gateway – this animated diagram only runs on my site: watch it on sascha-ms.de

How the fallback chain decides in a model fallback

Every destination in the chain follows the same pattern. The gateway checks the circuit breaker, makes exactly one attempt and looks at the result. If the response matches no trigger, it goes to the client unchanged, even if it is an error. A 400 caused by a broken request is therefore not repeated against the backup model, which is the right behavior. If every destination fails, the client receives the last error.

Triggers in detail

• Status codes: single codes such as 429 or classes such as 5XX, but only 4XX and 5XX. Successful responses never trigger a fallback.
• CONNECTION_ERROR: the connection was never established. The provider never saw the request.
• FIRST_BYTE_TIMEOUT: the connection stands, yet the first byte does not arrive within firstByteTimeoutMilliseconds. Without this value there is no timeout.
• UNROUTABLE: the target does not exist or cannot handle the path. This moves to the next destination but does not count toward the circuit breaker.

Each backup destination can have its own triggers. They are not inherited from the route. If you omit them, the defaults apply again, which means no 429.

What the circuit breaker does

A circuit breaker is a protective switch for dependencies. In the closed state, requests go to the primary destination as usual. If it fails repeatedly, the switch opens and the gateway skips that destination, which saves the latency of an attempt that is bound to fail. In the half-open state, it sends a small share of requests there again as a test. If those succeed, the switch closes. The gateway tracks this state per combination of target and model, and it is on by default. With DISABLED, every request walks the chain in the declared order regardless.

Streaming has a hard limit

With InvokeModelWithResponseStream, the fallback only works as long as no byte has reached the client yet. If a stream breaks off midway, you get half an answer plus an error and have to retry yourself. For chat interfaces this means error handling in the frontend is still needed.

Typical mistakes and how to avoid them

The last point deserves one more sentence. By default the SDK retries throttling and server errors itself, and the gateway returns its errors in Bedrock format precisely so that this works. If you stack three client attempts on a chain with three destinations, a single user request can turn into nine gateway calls. I described how this relates to duplicate side effects in my article on idempotency in AWS Lambda.

Costs, security and data protection

Every outbound attempt counts as a separate gateway invocation. A request that goes through three destinations therefore costs three. Bedrock bills model usage on top, and failed attempts at the primary destination are billed as long as the circuit breaker is closed. Skipped destinations cost nothing. You can find current prices on the AgentCore pricing page.

I would not postpone two security points. First, limit tokens per target with a token limit policy. The docs explicitly warn that otherwise the gateway does not cap the length of streaming responses, and a single user can exhaust everyone else’s quota. Second, after the first deployment, add the aws:SourceArn condition with your gateway’s ARN to the gateway role’s trust policy.

And then there is data protection, which in German companies often only comes up during review. If your primary model runs through an EU inference profile and the backup model through a global profile, a fallback can send data to other regions. Design the backup chain so that it keeps the same commitments as the primary destination. If you want to read more about gateways and agent security, my post on connecting MCP servers securely covers further points.

FAQ

Does the gateway replace the retry logic in the SDK?

Partly. The gateway handles switching between destinations, while the SDK still retries throttling and server errors. Keep client attempts low so the two do not multiply unnecessarily.

Can I fall back from Bedrock to the Anthropic API directly?

Yes, through a second inference target using the anthropic connector and an API key from the AgentCore token vault. Because the body stays in Anthropic Messages format, the gateway can translate. Clarify contract and data protection first.

Does the fallback also work with streaming?

Only until the first byte of the response. If a running stream breaks off, the client gets the partial answer and an error and has to retry itself.

How many backup destinations are possible?

Up to two per route. Each combination of target and model may appear only once in the chain, including the primary destination.

Your next step

Find a place in your code where a try/catch around InvokeModel calls a second model today. That is exactly where the change pays off: put the gateway in front, add a rule with 429 in triggerOn, switch the client endpoint and use the 50-millisecond test to check that the fallback really kicks in.

I am curious: which backup model would you choose for your most important LLM use case, and have you ever tested how your users react to the weaker answers?

If your models need current information from the web, take a look at Bedrock web search with a domain filter. There I show how to limit the search to trusted sources.

Sources

• AWS What’s New: AgentCore Gateway now supports Bedrock InvokeModel APIs and model fallback (Oct 22, 2026)
• AgentCore Developer Guide: Fallback in gateway rules
• AgentCore Developer Guide: Use the Amazon Bedrock InvokeModel API with inference targets
• AgentCore Developer Guide: Inference connector targets
• AgentCore Developer Guide: Add rules to a gateway
• AgentCore Developer Guide: Create an AgentCore gateway using the API
• AgentCore Developer Guide: Set up permissions for AgentCore Gateway
• Code checked with Node.js 22, TypeScript 5.9, aws-cdk-lib 2.274.0, CDK CLI 2.1146.1 and AWS SDK for JavaScript 3.1156.0 on Oct 24, 2026 (type check, cdk synth, request construction verified offline, no deployment)

This post was created with AI assistance.

👉 Next step: Working with AWS CDK? Grab my printable AWS CDK cheat sheet with commands, concepts and a stack skeleton.

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en