the bomber

Posted on Oct 30

How I Keep My Site’s Chatbot Under a 9‑Second Serverless Timeout

#chatbot #serverless #llm #automation

Overview

I run a personal automation server at home that powers the chatbot on my
Dream Futures Tech website. The serverless platform I use (Cloudflare
Workers) imposes a hard 9‑second execution limit, so every request has to
finish well before that deadline or the user sees a timeout error. This
article walks through the exact steps I took to keep the chatbot under
that limit while still delivering useful AI‑generated replies.

Choosing a Free Model

I wanted a model that is both free to run and small enough to fit in the
~50 MB memory budget of a Worker. After testing a few options locally I
settled on MiniLM‑Chat‑6B from Hugging Face, exported to the
ggml format and run with llama.cpp. The model loads in ~2 seconds and
generates a short answer in under 5 seconds on the free tier of
Cloudflare Workers KV.

Judgement: I chose this model because it balances quality and size.
More powerful models would exceed the timeout, while tiny models would
produce nonsensical replies.

Serverless Constraints

Cloudflare Workers give me:

9 seconds total CPU time per request
128 MB of memory
No persistent filesystem (only KV, Durable Objects, or in‑memory)

Because the model file lives in KV, I have to fetch it on the first
cold start and cache it in a global variable. Subsequent requests reuse
the cached instance, keeping the load time negligible.

The Handler Code

Below is the minimal worker.js that powers the endpoint
/api/chat. It shows model loading, timing, and a fallback path.

// worker.js
import { fetch } from 'undici';
import { Llama } from './llama.js'; // tiny wrapper around llama.cpp WASM

let model = null; // cached across requests
let modelReady = false;

// Load model from KV on first request
async function initModel() {
  if (modelReady) return;
  const kv = await MY_KV.get('minilm-chat-ggml.bin', { type: 'arrayBuffer' });
  model = await Llama.load(kv);
  modelReady = true;
}

// Simple timeout helper
function withTimeout(promise, ms) {
  return Promise.race([
    promise,
    new Promise((_, reject) =>
      setTimeout(() => reject(new Error('timeout')), ms)
    ),
  ]);
}

export default {
  async fetch(request) {
    const url = new URL(request.url);
    if (url.pathname !== '/api/chat') return new Response('Not found', { status: 404 });

    const { prompt } = await request.json();

    // Ensure model is ready
    await initModel();

    // Run inference with a safety margin (6 seconds)
    try {
      const start = Date.now();
      const answer = await withTimeout(
        model.generate(prompt, { maxTokens: 64 }),
        6000
      );
      const duration = Date.now() - start;
      return new Response(
        JSON.stringify({ answer, duration }),
        { headers: { 'Content-Type': 'application/json' } }
      );
    } catch (e) {
      // Fallback to rule‑based reply
      const fallback = simpleFallback(prompt);
      return new Response(
        JSON.stringify({ answer: fallback, fallback: true }),
        { headers: { 'Content-Type': 'application/json' } }
      );
    }
  },
};

function simpleFallback(text) {
  const lower = text.toLowerCase();
  if (lower.includes('price')) return 'Our pricing page is at /pricing.';
  if (lower.includes('hours')) return 'We are open 9 am‑5 pm EST, Monday‑Friday.';
  return "I'm sorry, I couldn't process that. Please try again or email support.";
}

Key Points

Global cache – model lives outside the request handler so the WASM instance stays warm.
withTimeout – caps the inference at 6 seconds, leaving 3 seconds for network overhead and response serialization.
Fallback – a deterministic, instant function that never exceeds the remaining time budget.

Timing the Inference

I added a small logging KV entry to see how long each generation took:

await MY_KV.put(`log-${Date.now()}`, JSON.stringify({ prompt, duration }));

During a week of real traffic the median duration was 4.2 seconds, well
under the 6‑second safety window. The 95th percentile never crossed 5.8 seconds,
so the fallback was rarely triggered.

Fallback Strategy

The fallback is deliberately simple: a few keyword checks and static
answers. Because it runs synchronously in JavaScript, it completes in
under 50 ms. If you need richer fallback content you can store canned
responses in KV and look them up by intent.

Deploying and Testing

Upload the model – wrangler kv:bulk put MY_KV minilm-chat-ggml.bin <path>
Publish – wrangler publish
Smoke test – curl -X POST https://my-site.workers.dev/api/chat -d '{"prompt":"What are your hours?"}' -H "Content-Type: application/json"

The response includes a duration field so you can monitor latency
directly from the client.

Lessons Learned

Cold starts matter. The first request after a deployment takes ~2.5 seconds to load the model. I mitigate this by pinging the endpoint from a cron job every 5 minutes.
Keep the prompt short. Longer prompts increase tokenisation time and push you closer to the timeout.
Never trust the model to finish. The withTimeout guard is essential; without it a runaway generation would cause the whole Worker to be terminated and the user would see a generic 504.

Conclusion

By combining a tiny free model, a global cache, a strict timeout wrapper,
and a deterministic fallback, I can reliably serve AI‑powered chat on a
platform that only gives me 9 seconds per request. The same pattern
works for any serverless environment with tight execution limits.

I build sites and automations for small businesses at dreamfuturestech.com.

DEV Community