Skip to content

6,338 requests a second on two cores.

Kaeso 0.7.0 against LiteLLM, Bifrost and Portkey on BerriAI’s ai-gateway-bench, the benchmark from the company behind LiteLLM. One machine, one mock provider, the benchmark’s own drivers.

BerriAI/ai-gateway-bench at 44365ca, 22 November 2026 (opens in a new tab)
6,338
requests a second on two cores; LiteLLM Rust served 3,211
0.54
ms added at the median with 16 clients
314
µs of CPU per request at full load
0
failed requests, in every run of every gateway

Nearly twice the load on the same cores.

Each gateway got the same two dedicated cores, 64 clients and a mock provider that answers at once, so the gateway itself was the limit.

  1. 01Kaeso0.7.06,3386,086–6,355
  2. 02Kaeso0.7.0, request logs on5,3245,085–5,388
  3. 03LiteLLM Rustmain at d4e8f96, unreleased3,2113,146–3,265
  4. 04Bifrost2.6.02,8062,777–2,819
  5. 05Portkey1.15.2779776–793
  6. 06LiteLLM Python1.110.27474–75

Requests a second, the median of 3 rounds; the thin line runs from the slowest round to the fastest. The same driver reached 27,750 a second straight to the mock, far above any gateway.

GatewayRequests a secondLatency p50 / p99CPU per requestPeak memory
Kaeso0.7.0Requests a second6,338Latency p50 / p9910.0 / 15.6 msCPU per request314 µsPeak memory101 MB, and 271 MB for its databases
Kaeso0.7.0, request logs onRequests a second5,324Latency p50 / p9911.8 / 20.6 msCPU per request373 µsPeak memory450 MB, and 1,163 MB for its databases
LiteLLM Rustmain at d4e8f96, unreleasedRequests a second3,211Latency p50 / p9919.6 / 33.2 msCPU per request573 µsPeak memory86 MB, no database
Bifrost2.6.0Requests a second2,806Latency p50 / p9921.0 / 54.6 msCPU per request708 µsPeak memory405 MB, no database
Portkey1.15.2Requests a second779Latency p50 / p9979.9 / 118.3 msCPU per request1,449 µsPeak memory293 MB, no database
LiteLLM Python1.110.2Requests a second74Latency p50 / p99858.1 / 974.0 msCPU per request13,563 µsPeak memory743 MB, no database

Under a millisecond at the median.

How much longer a request takes through the gateway than straight to the mock, which answers in about 22 ms as a fast provider would.

Gateway1 client4 clients16 clients
Kaeso0.7.01 client+0.73 / +1.66p99 1.36 to 2.054 clients+0.56 / +1.15p99 1.08 to 2.2816 clients+0.54 / +1.47p99 1.44 to 1.57
Kaeso0.7.0, request logs on1 client+0.77 / +2.22p99 2.17 to 2.474 clients+0.60 / +1.90p99 1.83 to 1.9716 clients+0.62 / +1.45p99 1.23 to 3.07
LiteLLM Rustmain at d4e8f96, unreleased1 client+0.87 / +0.98p99 0.91 to 1.014 clients+0.76 / +1.68p99 1.43 to 1.6916 clients+0.69 / +2.26p99 1.59 to 2.82
Bifrost2.6.01 client+1.10 / +1.32p99 1.22 to 1.364 clients+0.85 / +2.41p99 1.98 to 2.5116 clients+0.71 / +4.08p99 3.01 to 4.16
Portkey1.15.21 client+2.04 / +5.26p99 4.12 to 5.474 clients+1.17 / +6.99p99 6.35 to 7.0016 clients+3.1 / +20.4p99 20.1 to 22.6
LiteLLM Python1.110.21 client+16.0 / +17.8p99 17.4 to 18.34 clients+50.9 / +60.5p99 58.2 to 66.416 clients+188 / +216p99 214 to 221

Added latency in ms, p50 and p99, the median of 3 rounds, with the p99’s range under it. Each run was paired with a direct run of the same shape just before it.

No row for 64 clients: there the mock and the drivers share two cores and top out near 2,900 requests a second, so subtracting the direct run would understate every gateway’s time. The capacity figure measures that load instead.

Streams arrive as they were sent.

200 streamed requests at 16 clients, and two coding agents’ recorded sessions replayed turn by turn.

GatewayFirst token, addedGap between chunks, p99Claude Code session, addedCodex session, added
Kaeso0.7.0First token, added+0.87 msGap between chunks, p997.8 msClaude Code session, added+41.5 msCodex session, added+34.0 ms
Kaeso0.7.0, request logs onFirst token, added+0.94 msGap between chunks, p998.1 msClaude Code session, added+66.0 msCodex session, added+65.5 ms
LiteLLM Rustmain at d4e8f96, unreleasedFirst token, added+0.96 msGap between chunks, p9940.4 msClaude Code session, added+41.0 msCodex session, added+44.2 ms
Bifrost2.6.0First token, addedNot comparableGap between chunks, p9920.5 msClaude Code session, added+126.4 msCodex session, added+91.3 ms
Portkey1.15.2First token, added+27.5 msGap between chunks, p998.6 msClaude Code session, added+91.4 msCodex session, added+104.2 ms
LiteLLM Python1.110.2First token, added+119 msGap between chunks, p9934.0 msClaude Code session, added+502.1 msCodex session, added+507.7 ms

Straight to the mock, the gap between chunks is 7.5 ms at p99. A session is 30 recorded turns sent one after another; added is its whole time through the gateway minus straight to the mock. Every streamed session gave valid tool calls through every gateway.

Bifrost sends its first event before the provider answers, and the benchmark counts that as the first token, so its time to first token reads as negative.

Where others are ahead.

Results Kaeso doesn’t win, and what its extra work costs.

Memory01

LiteLLM Rust never went above 86 MB and needs no database. Kaeso reached 164 MB, and its PostgreSQL and Valkey took another 271 MB.

One client02

With one request at a time, LiteLLM Rust’s p99 was +0.98 ms against Kaeso’s +1.66 ms, and no round of Kaeso’s came closer.

A draw03

Time to first token (+0.87 against +0.96 ms) and the Claude Code session (+41.5 against +41.0 ms) fall within each other’s range.

Request logs04

With every request logged to ClickHouse, Kaeso served 5,324 requests a second, 16% fewer, and its databases took 1,163 MB.

What each gateway did.

Every gateway ran as its project ships it, against the mock in Anthropic’s Messages format. Kaeso checked the most on every request.

GatewayVersionChecked on each requestKept
KaesoVersionRelease 0.7.0Checked on each requestA project’s API key and the models the project allowsKeptUsage in PostgreSQL; no request logs
Kaesorequest logs onVersionRelease 0.7.0Checked on each requestA project’s API key and the models the project allowsKeptUsage in PostgreSQL, and every request in ClickHouse
LiteLLM RustVersionmain at d4e8f96, unreleasedChecked on each requestOne master key from its configKeptNothing: no database
BifrostVersionRelease 2.6.0Checked on each requestNo keyKeptNothing: its logging and stores off
PortkeyVersionRelease 1.15.2Checked on each requestNo keyKeptNothing: its logging off
LiteLLM PythonVersionRelease 1.110.2Checked on each requestOne master key from its configKeptNothing: no database

How it ran.

Everything needed to check a number: the machine, the benchmark, every version and how each run was paired.

01
MachineA DigitalOcean CPU-Optimized c-4 in Amsterdam: 4 dedicated vCPUs (Intel Xeon Platinum 8358, 2.6 GHz), 8 GB of memory, Ubuntu 24.04 and Docker.
02
CoresEach gateway ran in Docker on the host network, pinned to two cores. The mock, the load drivers and Kaeso’s databases ran on the other two.
03
BenchmarkBerriAI/ai-gateway-bench at 44365ca, with its mock provider and drivers unchanged: load-probe for requests, run_http_bench.py for streams and session_runner.py for agent sessions.
04
Mock providerAnswers in Anthropic’s Messages format: its first token after 20 ms, then 40 tokens 5 ms apart. In the capacity pass it answers at once.
05
Load1, 4, 16 and 64 clients sending 1,000, 2,000, 4,000 and 16,000 requests a round, 200 streams at 16 clients and four agent sessions of 30 turns.
06
Rounds3 rounds, the gateways in a new random order each round, each one measured right after a direct run of the same shape. Every figure is the median of the rounds, with their range.
07
CPU and memoryEvery process of the gateway’s container, read four times a second, and the same for its databases, reported beside it.
08
Day22 November 2026

Versions

GatewayImage or build
Kaeso0.7.0ghcr.io/logicplanes/kaeso-gateway:0.7.0sha256:3d32fd7827417e49475b4ac9ad7d25f7daed28f02fa489ef4077ff501faaae1a
LiteLLM Rustd4e8f96github.com/BerriAI/litellm@d4e8f96, litellm-gateway built with rust:1.98.0sha256:b80e284d93b67806511e0371456d80083d6f1d932abd723a1b31c1ed578ae122
Bifrost2.6.0maximhq/bifrost:v2.6.0sha256:5968d007ba4f74ee01bc4ceb071decb9927a0b7bb43dbb9072208f75e2d522e3
Portkey1.15.2portkeyai/gateway:1.15.2sha256:97f094d9c8a764cbfaa2a7138c0017b247ca923bb06db1b4c13b7f8a33b5200d
LiteLLM Python1.110.2ghcr.io/berriai/litellm:v1.110.2sha256:ef798594ca3db8d81bad2e1439f536f49b71fe095ee9f20416f5e149b4e22035

Kaeso’s databases: postgres:18.6-alpine, valkey/valkey:9.1.3-alpine and clickhouse/clickhouse-server:26.3.62.3-alpine.

Run it yourself.

On a fresh Linux machine with four cores, as root. The setup installs Docker, builds the benchmark and LiteLLM Rust at the same commits and pulls every image by its digest.

curl -fsSLO https://kaeso.app/benchmarks/2026-11-22/harness.tar.gztar -xzf harness.tar.gzbash kaeso-gateway-bench/setup.shcd /rootai-gateway-bench/.venv/bin/python run.pyai-gateway-bench/.venv/bin/python run_capacity.py

Three rounds of every gateway take about 90 minutes, the capacity pass another 30. Absolute numbers change from machine to machine, so compare gateways measured in the same run.

Measure it onyour own machine.

Install Kaeso next to the gateway you run today and point the same benchmark at both.