---
title: Kaeso on LiteLLM’s own benchmark
description: "We ran ai-gateway-bench, the benchmark from BerriAI, the company behind LiteLLM, with Kaeso 0.7.0 beside LiteLLM, Bifrost and Portkey. These are the numbers and the evidence behind each one."
url: https://kaeso.app/articles/kaeso-on-litellms-benchmark
markdown: https://kaeso.app/articles/kaeso-on-litellms-benchmark.md
language: en
alternates:
  de: https://kaeso.app/de/articles/kaeso-on-litellms-benchmark.md
  zh-CN: https://kaeso.app/zh-CN/articles/kaeso-on-litellms-benchmark.md
published: 2026-11-22
authors:
  - Devin Oldenburg (Logicplanes)
tags:
  - benchmark
  - performance
  - litellm
image: https://kaeso.app/articles/kaeso-on-litellms-benchmark/cover.jpg
index: https://kaeso.app/llms.txt
---

# Kaeso on LiteLLM’s own benchmark

We ran ai-gateway-bench, the benchmark from BerriAI, the company behind LiteLLM, with Kaeso 0.7.0 beside LiteLLM, Bifrost and Portkey. These are the numbers and the evidence behind each one.

> Every page, article and docs page of Kaeso is listed in https://kaeso.app/llms.txt.

- Published: 22 November 2026
- Reading time: 3 min
- Author: Devin Oldenburg (Logicplanes)

BerriAI, the company behind LiteLLM, publishes [ai-gateway-bench](https://github.com/BerriAI/ai-gateway-bench) to measure what an AI gateway adds on top of the model provider. We ran its local lane with Kaeso 0.7.0 added beside every gateway it compares: LiteLLM Rust, LiteLLM Python, Bifrost and Portkey. Each ran in its newest version on one machine, with the benchmark’s own mock provider and drivers unchanged.

On the same two cores, Kaeso served 6,338 requests a second and LiteLLM Rust served 3,211.

![Requests a second with 64 clients, each gateway on two cores: Kaeso 6,338, Kaeso with request logs 5,324, LiteLLM Rust 3,211, Bifrost 2,806, Portkey 779, LiteLLM Python 74](https://kaeso.app/articles/kaeso-on-litellms-benchmark/ranking-light.png "Requests a second with 64 clients against a mock that answers at once, the median of three rounds. The thin line runs from the slowest round to the fastest.")

## The numbers

Each gateway had the same two dedicated cores. The capacity pass sends 64 clients against a mock that answers at once, so the gateway itself is the limit. The latency sweep uses the benchmark’s default mock, which answers in about 22 ms, like a fast provider. Every figure is the median of three rounds:

| Gateway | Requests a second | Added p50 / p99 at 16 clients | CPU per request |
| --- | --- | --- | --- |
| Kaeso 0.7.0 | 6,338 | +0.54 / +1.47 ms | 314 µs |
| Kaeso 0.7.0, request logs on | 5,324 | +0.62 / +1.45 ms | 373 µs |
| LiteLLM Rust, unreleased `main` | 3,211 | +0.69 / +2.26 ms | 573 µs |
| Bifrost 2.6.0 | 2,806 | +0.71 / +4.08 ms | 708 µs |
| Portkey 1.15.2 | 779 | +3.13 / +20.40 ms | 1,449 µs |
| LiteLLM Python 1.110.2 | 74 | +188 / +216 ms | 13,563 µs |

Streams arrive as sent. Kaeso’s 99th-percentile gap between chunks was 7.8 ms, against 7.5 ms straight to the mock; LiteLLM Rust’s was 40.4 ms. Every streamed Claude Code and Codex session gave valid tool calls through every gateway, and no run of any gateway failed a request.

## Where LiteLLM is ahead

A benchmark that a vendor runs should say where the vendor loses. Here is where Kaeso did:

- **Memory.** LiteLLM Rust never went above 86 MB and needs no database. Kaeso reached 164 MB, and its PostgreSQL and Valkey took another 271 MB.
- **One client.** With one request at a time, LiteLLM Rust’s p99 was +0.98 ms against Kaeso’s +1.66 ms, and no round of Kaeso’s came closer.
- **A draw.** Time to first token (+0.87 against +0.96 ms) and the replayed Claude Code session (+41.5 against +41.0 ms) fall within each other’s range.
- **Request logs cost.** With every request written to ClickHouse, Kaeso served 16% fewer requests a second.

## Kaeso did more on every request

Every gateway ran as its project ships it. Kaeso checked a project’s API key and the models that project allows, and recorded usage in PostgreSQL. Both LiteLLMs compared one static master key, and Bifrost and Portkey checked no key at all. None of the others kept anything. Kaeso doing more work and still serving more requests is the result we set out to check.

## The evidence

A number on a vendor’s website is only as good as what stands behind it. This is what stands behind these:

- **The benchmark is theirs.** It ran on `BerriAI/ai-gateway-bench` at commit `44365ca`, with its mock provider and its three drivers unchanged: `load-probe`, `run_http_bench.py` and `session_runner.py`. Our scripts only start each gateway, pin it to its cores, pair every run with a direct run to the mock, and repeat it in rounds.
- **Every version is pinned.** Every image ran by its digest, listed on [the benchmarks page](https://kaeso.app/benchmarks). LiteLLM Rust has no release yet, so it is named as `main` at `d4e8f96`. Its Dockerfile at that commit can’t build, so we compiled the same commit in `rust:1.98.0`. A second build on a fresh machine produced a binary with the same sha256.
- **Every raw result is public.** Each run’s JSON and every gateway’s log are in [raw.tar.gz](https://kaeso.app/benchmarks/2026-11-22/raw.tar.gz), with [the scripts](https://kaeso.app/benchmarks/2026-11-22/harness.tar.gz), [the figures as JSON](https://kaeso.app/benchmarks/2026-11-22/results.json) and [the sweep as CSV](https://kaeso.app/benchmarks/2026-11-22/results.csv).
- **It runs again on a fresh machine.** We ran the published setup on a new 4-vCPU machine. Every gateway finished one round without an error. Numbers were lower on that host (Kaeso 5,287, LiteLLM Rust 2,179 requests a second), and Kaeso led again.
- **The page can’t drift from the data.** The benchmarks page reads its numbers from the published results. A test fails when they differ, or when a sentence such as “nearly twice the load” stops holding for the numbers.
- **It’s in their repository for review.** [Pull request 7](https://github.com/BerriAI/ai-gateway-bench/pull/7) on `BerriAI/ai-gateway-bench` adds Kaeso as a gateway, with this run’s results, so their maintainers can rerun it.

## Who ran it

Kaeso’s author ran this benchmark, on one machine. BerriAI calls its own published results single-host and vendor-run, and the same holds here. Treat the numbers as a claim to check, and check it. The second article, [how to benchmark an AI gateway without fooling yourself](https://kaeso.app/articles/benchmarking-ai-gateways-honestly), covers the mistakes we caught along the way.

## Run it yourself

On a fresh Ubuntu 24.04 machine with four cores, as root:

```bash
curl -fsSLO https://kaeso.app/benchmarks/2026-11-22/harness.tar.gz
tar -xzf harness.tar.gz
bash kaeso-gateway-bench/setup.sh
cd /root
ai-gateway-bench/.venv/bin/python run.py
ai-gateway-bench/.venv/bin/python run_capacity.py
```

Three rounds take about 90 minutes and the capacity pass another 30. Absolute numbers change from machine to machine, so compare gateways measured in the same run. Every table, the method and every image digest are on [kaeso.app/benchmarks](https://kaeso.app/benchmarks).

By Devin Oldenburg (Logicplanes)

## More articles

- [How to benchmark an AI gateway without fooling yourself](https://kaeso.app/articles/benchmarking-ai-gateways-honestly.md): What we got wrong and fixed while running Kaeso against LiteLLM, Bifrost and Portkey on BerriAI’s benchmark, and the checks that make a gateway benchmark worth reading.
- [Kaeso 0.7: rerank, batches and decisions, 23 more providers, and guardrails](https://kaeso.app/articles/kaeso-0-7.md): Kaeso 0.7.0 is out. Apps rerank, run batches, store Responses and ask decision models through the gateway, 23 more providers are built in, organizations add guardrails, SCIM, webhooks and their own currency, and installs get lighter and run on Kubernetes. Here is what changed and how to update.

[See all articles](https://kaeso.app/articles.md)

## Own the gateway. Own the record

```bash
curl -fsSL https://kaeso.app/install.sh | sh
```

[Install Kaeso](https://docs.kaeso.app/self-hosting/installation) · [Open the live demo](https://demo.kaeso.app)
