benchmarks.
measured 2026-08-01 · artifact 20260801T190827Z
Every number below comes from a single benchmark session: same machine, same mock upstream, same workload — only the gateway changes. Three runs per gateway, means reported with spreads, zero failed requests anywhere, and the per-request raw output preserved so anyone can re-derive our aggregates. No gateway is containerized; all run as native processes.
Between published sessions, every HeimOS pull request runs a k6 overhead gate in CI —
a latency regression blocks the merge. And every response HeimOS serves carries its own
measured overhead in the X-Gateway-Overhead-Us header, so you can audit the
claim on your own traffic, continuously.
Head-to-head
| Gateway (version · benched 2026-08-01) | Throughput | Median gateway overhead | p95 overhead | Peak memory | Errors |
|---|---|---|---|---|---|
HeimOS d93a14a (Rust) |
4,955 req/s (±4) | 2.2 ms | 11.5 ms | 225 MB | 0 |
Bifrost 1.6.2 / bin v1.5.0-pre6 (Go) |
3,414 req/s (±45) | 14.3 ms | 119.4 ms | 1,139 MB | 0 |
LiteLLM 1.94.1 (Python) |
554 req/s (±8) | 398 ms | 1,416 ms | 251 MB | 0 |
Overhead = client-observed request duration minus the mock upstream's fixed 60 ms latency; it includes each gateway's own code time plus any queueing it introduces. HeimOS additionally self-reports server-measured internal time: 1.3 ms median, 5.9 ms p95, 11.5 ms p99.
Read honestly: at this workload HeimOS serves 45% more throughput than Bifrost in one-fifth of the memory, and 9× LiteLLM's throughput. LiteLLM's large overhead numbers are saturation queueing — the offered load is ~9× beyond its capacity on this machine; that is the point of a fixed-workload comparison, but don't read 398 ms as its unloaded latency. And one number that is not a HeimOS win: LiteLLM's absolute memory footprint (251 MB) is comparable to ours — the Rust advantage over Python here is throughput-per-MB, not footprint.
Methodology
- Hardware: Azure
Standard_D8s_v5(8 vCPU / 32 GB, Ice Lake), Ubuntu 24.04 (6.17.0-1020-azure), single machine — load generator, gateway, and mock upstream all colocated as native processes. - Workload: k6
v2.0.0, OpenAI-shapePOST /v1/chat/completions, progressive ramp to 1,000 VUs over 5m30s. - Upstream: identical Go mock (OpenAI-compatible responses) at a fixed 60 ms latency for every gateway; every request verified to hit the mock, never a real provider.
- Protocol: 3 runs per gateway; mean ± stdev reported. Gateway RSS/CPU sampled every 1 s. Versions pinned; run manifests record kernel, k6, and gateway versions.
- Configuration parity: each gateway runs its vendor-recommended proxy configuration (LiteLLM with 4 uvicorn workers, matching its own published benchmark setup); no auth, quotas, or guardrails enabled on any gateway in this scenario — it isolates pure forwarding overhead.
- Raw data: per-request k6 output (gzipped), summaries, gateway logs, and
RSS samples are preserved verbatim in artifact
20260801T190827Z— available on request while we stand up a public download.
Our comparison policy in one line: pinned competitor versions with dates, same-everything hardware, at least three runs, losing numbers published, raw output kept. A competitor release that plausibly changes performance triggers a re-run before we cite these numbers again.
Questions, corrections, or a configuration you think would change the result? We'll run it — hello@heimos.ai.