Gateway latency and TTFB benchmark
Measured gateway overhead per path, streaming first byte and the LiteLLM head-to-head, plus the live figure from this gateway.
On a quiet 8-core host TokenZero adds 1.52 ms at p50 and 3.42 ms at p95 to a cache miss, 0.5 ms to an exact hit and 1.98 ms to the first streamed byte; LiteLLM Proxy on the same box adds 16.25 ms at p50. The live figure below is this gateway's own rolling p50 right now.
· Source: pricing table 2026-10-26 and the gateway's own tokenizer and benchmark files; your inputs never leave the browser except token counts, which are not stored
At a glance
- Added latency, miss
- 1.52 ms p50 · 3.42 ms p95 · 3.68 ms p99
- Added latency, exact hit
- 0.5 ms p50
- Added streaming first byte
- 1.98 ms p50
- Throughput, miss / direct
- 641 / 3542 rps
- LiteLLM Proxy, same box
- 16.25 ms p50 (computed)
- Ratio
- 10.69x (computed)
- Production host, miss
- 2.17 ms p50
- Live (this gateway)
- rolling p50/p95 from /v1/status, loaded below
- Not measured
- per-model provider time-to-first-token and tokens/sec
Added latency by path (gateway minus direct, same mock upstream)
| Path | p50 | p95 | p99 |
|---|---|---|---|
| quiet host · miss | 1.52 ms | 3.42 ms | 3.68 ms |
| quiet host · exact hit | 0.5 ms | 0.88 ms | 1.33 ms |
| quiet host · semantic hit | 0.64 ms | 1.01 ms | 1.14 ms |
| quiet host · streaming first byte | 1.98 ms | 3.58 ms | 4.27 ms |
| production host · miss (shared, under load) | 2.17 ms | 3.26 ms | 4.01 ms |
| LiteLLM Proxy miss · same box (v1.106.4) | 16.25 ms | 20.74 ms | — |
Live overhead on this gateway
Loading /v1/status …
The benchmark scripts (scripts/bench/overhead.ts, compare-litellm.ts) and raw JSON are published on /benchmarks.
Questions people ask
Why not per-model TTFT?
Provider time-to-first-token depends on the provider's load and region and TokenZero does not measure it; publishing it would be a claim it cannot stand behind. The gateway's own added latency is what it controls, so that is what is measured and shown.
How is the live number produced?
Every worker keeps a latency reservoir of gateway-only time per request; /v1/status aggregates the rolling p50/p95/p99 and falls back to the published benchmark when the gateway has not served enough requests yet.
How does streaming behave?
The gateway passes chunks through as they arrive and adds 1.98 ms p50 to the first byte; a buffering proxy would add the whole generation time. The final SSE comment carries the signed receipt.
What does TokenZero cost?
Free up to $500 per month of routed spend; then 15% of verified savings with a 3x-ROI-or-$0 rule. Gateway overhead is 1.52 ms p50, measured.