Every AI product ends up building the same plumbing.
Teams shipping on LLMs need the same things: one API across providers, fallback when a provider has a bad hour, per-customer limits so one client can't burn the budget, and visibility into cost and latency per model. Switchyard does that as a single Go binary that sits in the request path — so it has to be close to free in latency, and it must never become the reason a request fails.
- OpenAI-compatible API; unknown fields pass through untouched.
- Native Anthropic and Gemini adapters translate requests, responses and streams.
- Fails open: Redis down → limits pause, requests succeed; ClickHouse down → logs drop, requests succeed.
Profiling found 57% of CPU in syscalls — one write per token.
The first streaming benchmark fell over at 5,000 streams per second: p99 rose 80 ms. The CPU profile showed the gateway flushing every server-sent event separately. The fix: flush only when the next complete event isn't already buffered — bursts leave in one write, a lone token is never delayed. A partial next event still triggers a flush, otherwise real providers' 30 ms gaps would hold tokens back. CPU dropped 56% and p99 went from +80 ms to single digits.
“The first streaming benchmark fell over at 5,000 streams per second: p99 rose 80 ms.”
- Overhead is measured from request arrival, including auth, minus upstream time.
- Open-model load generator (no coordinated omission), benchmarks published in BENCHMARKS.md.
- ≤0.13 ms p99 self-measured overhead at every rate up to 7,500 req/s, zero errors.
The circuit breaker that tripped healthy providers.
My first breaker opened after five consecutive failures. In a chaos test where one provider failed 30% of the time, it disabled that provider for the whole run. Sampling the breaker every second showed why: failures return in ~1 ms while successes take ~40 ms, so with 20 requests in flight the failures complete first and look consecutive. The final design compares failures with requests started in a sliding window — independent of completion order. Result: 100% client success with the provider 30% flaky or fully down.
Every request in ClickHouse, without touching the hot path.
Records go into a bounded queue with a non-blocking send and are written in batches over the native protocol; a materialized view keeps a 1-minute rollup. The dashboard's 24-hour overview over a million requests answers in about 15 ms, and the request log uses keyset pagination so page 1,000 costs the same as page 1. With ClickHouse stopped at 5,000 req/s, zero requests failed.
“Records go into a bounded queue with a non-blocking send and are written in batches over the native protocol; a materialized view keeps a 1-minute rollup.”
built by one person · case study written by the same person