Gateway overhead benchmark, October 2026
Added latency per path, throughput, streaming first byte and the LiteLLM head-to-head, from the published benchmark scripts.
On a quiet 8-core host with a mock upstream and 32-way concurrency over 2000 requests, TokenZero adds 1.52 ms at p50 and 3.42 ms at p95 on a cache miss, 0.5 ms on an exact hit and 1.98 ms to the first streamed byte; LiteLLM Proxy on the same box adds 16.25 ms at p50.
· Source: TokenZero pricing table, benchmark result files and public audit ledger; CC BY 4.0 with attribution
At a glance
- Added latency, miss
- 1.52 ms p50 · 3.42 ms p95
- Added latency, exact hit
- 0.5 ms p50
- Added streaming first byte
- 1.98 ms p50
- LiteLLM Proxy, same box
- 16.25 ms p50 (LiteLLM: Current Version = 1.106.4)
- Ratio
- 10.69x (computed)
- Concurrency · requests
- 32 · 2000
- Node
- v24.22.1
- Method
- scripts/bench/overhead.ts and compare-litellm.ts, published with the results
Gateway overhead benchmark, October 2026 (first 9 of 9 rows; full dataset in the downloads)
| host | path | p50_ms | p95_ms | p99_ms | added_p50_ms | rps |
|---|---|---|---|---|---|---|
| quiet host | direct | 0.58 | 1.07 | 3.56 | 0 | 3542 |
| quiet host | miss | 2.1 | 4.49 | 7.24 | 1.52 | 641 |
| quiet host | exact_hit | 1.08 | 1.95 | 4.89 | 0.5 | 1670 |
| quiet host | semantic_hit | 1.22 | 2.08 | 4.7 | 0.64 | — |
| quiet host | streaming first byte (added) | 1.98 | 3.58 | 4.27 | 1.98 | — |
| production host (shared, under load) | direct | 0.66 | 1.59 | 4.21 | 0 | 3666 |
| production host (shared, under load) | miss | 2.83 | 4.85 | 8.22 | 2.17 | 625 |
| production host (shared, under load) | exact_hit | 1.24 | 2.3 | 4.9 | 0.58 | 1453 |
| quiet host | LiteLLM Proxy miss (same box, same mock) | 16.98 | 21.81 | — | 16.25 | — |
Method and licence
Computed at request time from the same data the gateway runs on; the JSON summary carries every number on this page. Reuse under CC BY 4.0 with a link to https://tokenzero.sh/reports/gateway-overhead-benchmark-2026-10.
Questions people ask
How much latency does TokenZero add?
1.52 ms at p50 and 3.42 ms at p95 on a cache miss (quiet host); the production-host rows show the same benchmark under real shared load.
How does it compare with LiteLLM Proxy?
Same box, same mock upstream, same request: LiteLLM Proxy LiteLLM: Current Version = 1.106.4 adds 16.25 ms at p50 against 1.52 ms for TokenZero, a 10.69x difference.
Can I reproduce it?
Yes: the scripts under scripts/bench in the public repository produce exactly these files; /benchmarks shows the method and the raw JSON.