Skip to content

How to benchmark an AI gateway without fooling yourself

What we got wrong and fixed while running Kaeso against LiteLLM, Bifrost and Portkey on BerriAI’s benchmark, and the checks that make a gateway benchmark worth reading.

Published
Reading time
5 min
A deep gradient of green and teal, in upright folds like a curtain, with a warm light in one corner

An AI gateway sits between your apps and every model provider, so whatever it adds, every request pays. Measuring that cost is easy to get wrong. Most mistakes flatter the gateway, and the person running the test usually wants it flattered.

We ran Kaeso on LiteLLM’s own benchmark, BerriAI’s ai-gateway-bench, beside LiteLLM, Bifrost and Portkey. On the way we published a claim we had to take back, and found two problems in our own setup only by running it again on a new machine. These are the rules we ended up with.

Subtract a paired direct run

A gateway’s overhead is the time a request takes through the gateway, minus the time the same request takes without it:

overhead = latency(client → gateway → mock) − latency(client → mock)

ai-gateway-bench points every gateway at the same local mock provider, so the model’s time and the network drop out. The direct run has to be fresh, though. Machines drift over an hour of testing, so we ran a direct run of the same shape right before every gateway run, and subtracted that run, not one baseline from the morning.

Make sure the gateway is the bottleneck

Our first table had a column for 64 clients. In it, Kaeso’s p99 overhead was +0.65 ms, lower than its +1.66 ms with a single client. A reviewer pointed out what that meant: latency doesn’t get better under load, so something else was the limit.

It was the test rig. At 64 clients, the mock and the load driver shared two cores and topped out near 2,900 requests a second. Every gateway queued behind them, the direct run’s tail rose too, and subtracting it hid most of the gateway’s own time. Our headline comparison at that load was measuring the mock.

We dropped the column and built a capacity pass instead. The mock answers at once, 64 clients send requests as fast as they can, and each round first measures how fast the same driver goes without a gateway. That ceiling was about 28,000 requests a second, more than four times the fastest gateway, so the gateways really were the limit. If your rig can’t outrun the gateway, you are benchmarking the rig.

Pin the cores, and count what the gateway needs

Each gateway ran in Docker on two cores of its own (--cpuset-cpus 2,3), while the mock, the drivers and the databases ran on the other two. Without that, the gateway competes with the load generator for CPU, and the slower the generator, the better the gateway looks.

Kaeso needs PostgreSQL and Valkey; the other gateways ran without a database. So we sampled the databases too and reported them beside the gateway. They used almost no CPU per request, but 271 MB of memory, which is why LiteLLM Rust wins the memory comparison. Leaving a gateway’s dependencies out of its bill is a quiet way to win.

Run rounds, and let the ranges decide

One run is an anecdote. We ran three rounds, each with the gateways in a new random order, and reported the median with the slowest and fastest round. The ranges then decide what you may claim:

  • With one client, LiteLLM Rust’s p99 was 0.91 to 1.01 ms across rounds and Kaeso’s 1.36 to 2.05 ms. The ranges don’t overlap, so LiteLLM Rust is ahead there, and we say so.
  • Time to first token was 0.68 to 0.94 ms for Kaeso and 0.84 to 1.04 ms for LiteLLM Rust. The ranges overlap, so that is a draw, not a win for whoever has the lower median.

Our first draft also gave Kaeso fewer rounds than the others. Every gateway now gets the same three.

Pin and name every version

“LiteLLM” or “Bifrost” is not a version. Every image ran by its digest, and the digests are published. LiteLLM Rust has no release yet, so it is named as what it is: main at commit d4e8f96, unreleased. Its Dockerfile at that commit can’t build, because it embeds files from outside its build context. We compiled the same commit in rust:1.98.0 instead, and say so. On a second machine, the build produced a binary with the same sha256, so anyone can check that they test the same code.

Say what each gateway did on every request

Gateways differ in how much work a request costs: checking keys, enforcing limits, writing logs. We listed it for every gateway. Kaeso checked a project’s API key and allowed models and recorded usage. Both LiteLLMs compared one static master key. Bifrost and Portkey checked nothing and kept nothing. We also ran Kaeso a second time with every request logged to ClickHouse, which cost it 16% of its capacity. A comparison that leaves logging on for one gateway and off for another is uneven, whichever way it falls.

Check the answers, not only the clock

A fast wrong answer is not fast. ai-gateway-bench replays real Claude Code and Codex sessions and checks that every tool call comes back as valid JSON. Every gateway passed every session, and no run recorded an error.

Read the numbers, too. Bifrost’s time to first token came out negative: it sends its first event before the provider answers, and the benchmark counts that event as the first token. We report it as not comparable rather than as a win for Bifrost.

Run it again on a fresh machine

We published a setup script that rebuilds the whole test from nothing, then ran it on a new machine before calling it done. It failed at once: a just-booted Ubuntu holds the package manager’s lock for its first updates. The script now waits. After that fix, every gateway completed a round without an error.

The numbers on that machine were lower for every gateway: Kaeso 5,287 requests a second, LiteLLM Rust 2,179. Kaeso led again, and Bifrost and LiteLLM Rust swapped places, 3% apart. That is the honest size of “your numbers will differ”. Compare gateways measured in the same run, never numbers from two machines.

Say who ran it

We build Kaeso, and we ran this. BerriAI says the same about its own published results: single-host and vendor-run. No method removes that, so we published everything needed to check us instead. The harness, every raw result and the image digests are on kaeso.app/benchmarks. Kaeso’s setup and this run’s results are also in a pull request to BerriAI’s repository (opens in a new tab), where its maintainers can rerun them.

By Devin Oldenburg (Logicplanes) (opens in a new tab)

On this page

Own the gateway.Own the record.

curl -fsSLhttps://kaeso.app/install.sh | sh