DEV Community

Alessio
Alessio

Posted on Edited on Originally published at py-forge.com

Test Retries and Timeouts for Real: an HTTP Fault-Injection Proxy in Python

#chaosengineering #testing #microservices #resilience

Your retry logic, timeouts and circuit breakers only get tested when a dependency actually misbehaves, and that usually happens in production. A fault-injection proxy lets you cause that on purpose. Put it between your service and the dependency, then tell it to add 1.5 s of latency, return a 503 on 30% of requests, or hang up mid-request.

ChaosProxy is a small reverse proxy written in Python on aiohttp that does this. Rules are managed at runtime over a JSON API, so you can turn faults on and off from a test script without restarting anything. Everything below was run locally.

Setup 

pip install aiohttp multidict
python3 chaosproxy.py --upstream http://127.0.0.1:5700 --port 8780 --host 127.0.0.1
Enter fullscreen mode Exit fullscreen mode

Point your client at http://localhost:8780 instead of the real service. With no rules, requests pass straight through, including method, query string, body and headers (hop-by-hop headers are filtered out). For this post the upstream is a tiny API that returns an order:

curl -s localhost:8780/orders/1001
# {"id": "1001", "status": "paid"}
Enter fullscreen mode Exit fullscreen mode

Fault 1: latency 

curl -s -X POST localhost:8780/__chaos/rules -H 'Content-Type: application/json' -d '{
  "rule_id": "slow-orders",
  "path_pattern": "^/orders/",
  "fault_type": "latency",
  "params": {"delay_seconds": 1.5, "jitter_seconds": 0.3}
}'
Enter fullscreen mode Exit fullscreen mode 
{"status": "created", "rule": {"rule_id": "slow-orders", "path_pattern": "^/orders/", "method": "*", "probability": 1.0, "fault_type": "latency", "params": {"delay_seconds": 1.5, "jitter_seconds": 0.3}, "enabled": true}}
Enter fullscreen mode Exit fullscreen mode 
curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" localhost:8780/orders/1001
# 200 in 1.634716s
Enter fullscreen mode Exit fullscreen mode

The request still reaches the upstream and returns 200, just late. That's how you find out whether your client's 1 s timeout really fires, and what the user sees when it does.

Remove the rule when you're done:

curl -s -X DELETE localhost:8780/__chaos/rules/slow-orders
# {"status": "deleted", "rule_id": "slow-orders"}
Enter fullscreen mode Exit fullscreen mode

Fault 2: intermittent errors

probability makes a rule fire on a fraction of matching requests, and method limits it to one HTTP verb:

curl -s -X POST localhost:8780/__chaos/rules -H 'Content-Type: application/json' -d '{
  "rule_id": "flaky", "path_pattern": "^/orders/", "method": "GET",
  "probability": 0.3, "fault_type": "http_error",
  "params": {"status_code": 503, "body": {"error": "upstream unavailable"}}
}'
for i in $(seq 1 20); do curl -s -o /dev/null -w "%{http_code} " localhost:8780/orders/$i; done
Enter fullscreen mode Exit fullscreen mode 
503 200 200 200 200 200 200 503 200 200 200 200 200 200 503 200 503 200 503 200
Enter fullscreen mode Exit fullscreen mode

5 of 20, close to the 30% asked for. This is the realistic case for testing retries with backoff. Failed requests return your JSON body with the status you chose, and they never reach the upstream.

Fault 3: dropped connections and truncated bodies 

curl -s -X POST localhost:8780/__chaos/rules -H 'Content-Type: application/json' \
  -d '{"rule_id":"drop","path_pattern":"^/orders/9$","fault_type":"connection_drop"}'
curl -s localhost:8780/orders/9; echo "curl exit $?"
# curl exit 52   (empty reply from server)
Enter fullscreen mode Exit fullscreen mode

payload_corrupt fetches the real response and then truncates it ("strategy": "truncate") or flips random bytes (any other strategy value):

curl -s -X POST localhost:8780/__chaos/rules -H 'Content-Type: application/json' \
  -d '{"rule_id":"trunc","path_pattern":"^/orders/7$","fault_type":"payload_corrupt","params":{"strategy":"truncate"}}'
curl -s localhost:8780/orders/7
# {"id": "7", "s
Enter fullscreen mode Exit fullscreen mode

That's exactly what a JSON parser hates. Good for checking that your client reports a clear error instead of crashing.

Seeing what happened 

curl -s localhost:8780/__chaos/rules     # active rules
curl -s localhost:8780/__chaos/stats
Enter fullscreen mode Exit fullscreen mode 
{"requests_total": 23, "requests_proxied": 18, "faults_injected": 6,
 "fault_breakdown": {"latency": 1, "http_error": 5, "payload_corrupt": 0, "connection_drop": 0}}
Enter fullscreen mode Exit fullscreen mode

Every injected fault is also logged:

[WARNING] [ChaosProxy] INJECTING FAULT [HTTP_ERROR] via rule [flaky] on GET /orders/1
Enter fullscreen mode Exit fullscreen mode

Using it in an automated test

Because rules are just HTTP calls, a test can switch a fault on, assert on behaviour, and clean up:

import requests

CHAOS = "http://localhost:8780/__chaos/rules"

def test_checkout_survives_slow_payments():
    requests.post(CHAOS, json={"rule_id": "slow-pay", "path_pattern": "^/payments",
                               "fault_type": "latency", "params": {"delay_seconds": 3}})
    try:
        r = requests.post("http://localhost:3000/checkout", json={"cart": 1}, timeout=10)
        assert r.status_code == 202          # we expect async fallback, not a 500
    finally:
        requests.delete(f"{CHAOS}/slow-pay")
Enter fullscreen mode Exit fullscreen mode

How matching works

Each rule has a regex path_pattern matched against the path plus query string, a method (* by default), a probability between 0 and 1, and an enabled flag. Rules are checked in the order they were created, and the first match wins. Paths under /__chaos are reserved for the control API and never forwarded.

Limitations

  • The control API has no authentication. Anyone who can reach the port can add rules. The default bind address is 0.0.0.0, so pass --host 127.0.0.1 (as above) or keep it on an isolated test network. Don't expose it publicly.
  • Rules live in memory. Restarting the proxy clears them, so re-create them from your test setup.
  • One upstream per proxy instance. Run several instances on different ports for several dependencies.
  • payload_corrupt reads the whole upstream response into memory before corrupting it, so it isn't meant for very large downloads. Normal forwarding streams.
  • HTTP only. There's no TCP-level chaos (packet loss, bandwidth limits) and no TLS termination. Put it on the plain-HTTP side of your stack.

FAQ

Does it change responses when no rule matches? No. Status, headers and body are relayed as received, including compressed bodies.

What's the default upstream? https://httpbin.org. Always pass --upstream for real use.

Python version? Python 3 with aiohttp and multidict.

ChaosProxy is a one-time purchase that includes the source and README: get ChaosProxy ($29). More details are on the ChaosProxy product page.

Top comments (0)