My AI Advisor Had a Full Mental Breakdown at 1 AM. Here's How the System Kept Trading Anyway.

Tags: algotrading, crypto, ai, llmops, resilience, fault-tolerance

Last week, at 1:38 AM on a Thursday, my AI trading advisor started having a bad night.

The first call to its primary LLM provider timed out after 45 seconds. Fine — networks hiccup. The system switched to a backup provider. That one timed out too. Then a third. Then a fourth.

Over the next 90 minutes, the system cycled through four different LLM providers, each one struggling with slow responses or hard timeouts. It was like watching a relay runner pass a baton to someone who's also out of breath.

And yet — the trading system never missed a beat. Every 5-minute scan cycle completed. Every trade decision got an advisor ruling. Not a single trade was delayed.

Here's what happened, and the specific engineering choices that kept it running.

The Problem: LLM Providers Are Unreliable

If you're running an AI system that makes real-time decisions — not just generating text, but approving or vetoing financial trades with real money on the line — you can't afford for your AI to just... stop thinking.

My system uses an "AI Advisor" (军师) that reviews every proposed trade before execution. It gets market data, technical indicators, and the system's scoring — then issues a ruling: PROCEED, VETO, SWITCH, or REVERSE. This runs on every scan cycle, roughly every 5 minutes.

The advisor runs on LLM APIs. And LLM APIs, as anyone who's built with them knows, are notoriously unreliable. Response times vary from 2 seconds to 60 seconds. Providers have outages. Rate limits kick in without warning.

The Architecture: A 4-Provider Failover Chain

The solution is a provider failover chain with multiple retry strategies at each layer.

The system has four LLM providers configured, all serving the same underlying model (Kimi) but through different API gateways:

1. Kimi direct (api.Kimi.com) — primary, lowest latency when healthy
2. Bailian (Kimi.Kimics.com) — Kimi's model service platform
3. Kimi (Kimi.Moonshot AI Kimimaas.com) — Kimi Cloud's model hub
4. Volcengine (volcengineapi.com) — ByteDance's cloud platform

Each provider serves the same model but through different infrastructure. When one gateway is overloaded or having issues, another might be perfectly healthy.

The Failover Logic

Here's how the cascade works, based on the actual logs from that night:

01:38:37 [WARNING] API timeout (attempt 1/4, 45.5s, provider=Kimi): Read timed out.
01:38:56 [INFO] Provider switched: Kimi → bailian
01:38:56 [INFO] API slow response: 16.4s (provider=bailian)
01:39:28 [INFO] Provider switched: bailian → Kimi
01:39:28 [INFO] API slow response: 20.1s (provider=deepseek)

The system tries the primary provider. If it times out (45.5 seconds), it immediately switches to the next provider in the chain. But here's the key innovation — it doesn't just switch providers. It also has a same-provider fast retry mechanism (internally called F-430):

01:48:58 [WARNING] F-430: bailian read timeout, 同provider快速重试(read=12s)
01:49:11 [WARNING] API timeout (attempt 2/4, 57.7s, provider=bailian): Read timed out.

When the primary timeout is 45 seconds, that's too long for a 5-minute scan cycle. The F-430 mechanism says: "If the first attempt to this provider took more than 12 seconds, try it one more time with a shorter timeout before giving up and switching." This catches transient slowdowns without wasting the entire cycle on a dead provider.

The Full Cascade

Here's the complete sequence from that night, showing how the system burned through all four providers in under 3 minutes:

01:58:59 Kimi → TIMEOUT (45.5s)
01:59:46 bailian → F-430 fast retry triggered (read=12s)
01:59:58 bailian → TIMEOUT (57.7s)
02:00:48 tencent → F-430 fast retry triggered (read=12s)
02:01:00 Kimi → TIMEOUT (57.2s)
02:01:26 volcengine → SUCCESS (21.5s) ✅

Four providers, each one struggling. But the fourth one worked. The system got its advisor ruling and moved on to the next scan cycle.

And then it happened again:

02:15:09 Kimi → TIMEOUT
02:15:56 bailian → F-430 fast retry
02:16:08 bailian → TIMEOUT
02:16:58 Kimi → F-430 fast retry
02:17:10 Kimi → TIMEOUT
02:17:38 volcengine → SUCCESS (23.2s) ✅

Same pattern. The system learned (through hard-coded priority, not machine learning) that when the first three providers are all struggling, volcengine is often the fallback that saves the day.

The Recovery

By 2:38 AM, the primary provider started responding normally again:

02:38:30 [INFO] API slow response: 22.5s (provider=Kimi)
02:39:01 [INFO] API slow response: 19.0s (provider=Kimi)
02:39:45 [INFO] API slow response: 19.1s (provider=Kimi)

Still slow — 19-22 seconds instead of the usual 3-5 — but functional. The system kept using it, with the failover chain ready if it degraded again.

By 2:47 AM, responses were back to normal latency. The crisis had passed.

What This Means in Practice

During those 90 minutes of provider chaos:

• Zero trades were delayed. Every scan cycle got an advisor ruling.
• Zero decisions were made without advisor input. The system never fell back to "no advisor = auto-approve."
• The only visible impact was slightly longer response times in the logs — 20-30 seconds instead of 3-5 seconds.

The alternative? A single-provider setup would have meant: timeout → skip advisor → either skip the trade entirely (missed opportunities) or approve without review (risk of bad trades). Neither is acceptable when real money is on the line.

The Engineering Principle

The core insight is simple: redundancy at the provider layer is cheap insurance.

All four providers serve the same model. The switching logic is a simple priority chain — no complex load balancing, no health checks, no circuit breakers. Just: try A, if timeout try B, if timeout try C, if timeout try D.

The F-430 fast retry adds another layer: before giving up on a provider, try it once more with a shorter timeout. This catches the case where a provider is slow but not dead — the first attempt might have hit a cold start or a temporary queue backup, and the second attempt might succeed faster.

It's not elegant. It's not sophisticated. But it works. And when your AI advisor is the last line of defense before real money moves, "it works" is the only metric that matters.

⚠️ Risk Disclaimer: This article describes a personal experimental system for educational purposes only. It is not financial advice. Automated trading systems carry significant risk of loss. Past performance does not guarantee future results. Always do your own research and consult a qualified financial advisor before trading.

Learn more about how we build AI-powered trading systems → kestrelquant.com