---
title: "6,338 requests a second on two cores"
description: "Kaeso 0.7.0 against LiteLLM, Bifrost and Portkey on BerriAI’s ai-gateway-bench, the benchmark from the company behind LiteLLM. One machine, one mock provider, the benchmark’s own drivers."
url: https://kaeso.app/benchmarks
markdown: https://kaeso.app/benchmarks.md
language: en
alternates:
  de: https://kaeso.app/de/benchmarks.md
  zh-CN: https://kaeso.app/zh-CN/benchmarks.md
updated: 2026-11-22
kaeso_version: 0.7.0
index: https://kaeso.app/llms.txt
---

# 6,338 requests a second on two cores

Kaeso 0.7.0 against LiteLLM, Bifrost and Portkey on BerriAI’s ai-gateway-bench, the benchmark from the company behind LiteLLM. One machine, one mock provider, the benchmark’s own drivers.

> Every page, article and docs page of Kaeso is listed in https://kaeso.app/llms.txt.

[BerriAI/ai-gateway-bench at 44365ca, 22 November 2026](https://github.com/BerriAI/ai-gateway-bench/tree/44365ca99307c2fde47c7ca0aa5e17c1665dcf38)

- **6,338** requests a second on two cores; LiteLLM Rust served 3,211
- **0.54** ms added at the median with 16 clients
- **314** µs of CPU per request at full load
- **0** failed requests, in every run of every gateway

## Nearly twice the load on the same cores

Each gateway got the same two dedicated cores, 64 clients and a mock provider that answers at once, so the gateway itself was the limit.

| Gateway | Requests a second | Latency p50 / p99 | CPU per request | Peak memory |
| --- | ---: | ---: | ---: | ---: |
| Kaeso (0.7.0) | 6,338 (6,086–6,355) | 10.0 / 15.6 ms | 314 µs | 101 MB, and 271 MB for its databases |
| Kaeso (0.7.0, request logs on) | 5,324 (5,085–5,388) | 11.8 / 20.6 ms | 373 µs | 450 MB, and 1,163 MB for its databases |
| LiteLLM Rust (main at d4e8f96, unreleased) | 3,211 (3,146–3,265) | 19.6 / 33.2 ms | 573 µs | 86 MB, no database |
| Bifrost (2.6.0) | 2,806 (2,777–2,819) | 21.0 / 54.6 ms | 708 µs | 405 MB, no database |
| Portkey (1.15.2) | 779 (776–793) | 79.9 / 118.3 ms | 1,449 µs | 293 MB, no database |
| LiteLLM Python (1.110.2) | 74 (74–75) | 858.1 / 974.0 ms | 13,563 µs | 743 MB, no database |

Requests a second, the median of 3 rounds; the thin line runs from the slowest round to the fastest. The same driver reached 27,750 a second straight to the mock, far above any gateway.

## Under a millisecond at the median

How much longer a request takes through the gateway than straight to the mock, which answers in about 22 ms as a fast provider would.

| Gateway | 1 client | 4 clients | 16 clients |
| --- | ---: | ---: | ---: |
| Kaeso (0.7.0) | +0.73 / +1.66 (p99 1.36 to 2.05) | +0.56 / +1.15 (p99 1.08 to 2.28) | +0.54 / +1.47 (p99 1.44 to 1.57) |
| Kaeso (0.7.0, request logs on) | +0.77 / +2.22 (p99 2.17 to 2.47) | +0.60 / +1.90 (p99 1.83 to 1.97) | +0.62 / +1.45 (p99 1.23 to 3.07) |
| LiteLLM Rust (main at d4e8f96, unreleased) | +0.87 / +0.98 (p99 0.91 to 1.01) | +0.76 / +1.68 (p99 1.43 to 1.69) | +0.69 / +2.26 (p99 1.59 to 2.82) |
| Bifrost (2.6.0) | +1.10 / +1.32 (p99 1.22 to 1.36) | +0.85 / +2.41 (p99 1.98 to 2.51) | +0.71 / +4.08 (p99 3.01 to 4.16) |
| Portkey (1.15.2) | +2.04 / +5.26 (p99 4.12 to 5.47) | +1.17 / +6.99 (p99 6.35 to 7.00) | +3.1 / +20.4 (p99 20.1 to 22.6) |
| LiteLLM Python (1.110.2) | +16.0 / +17.8 (p99 17.4 to 18.3) | +50.9 / +60.5 (p99 58.2 to 66.4) | +188 / +216 (p99 214 to 221) |

Added latency in ms, p50 and p99, the median of 3 rounds, with the p99’s range under it. Each run was paired with a direct run of the same shape just before it.

No row for 64 clients: there the mock and the drivers share two cores and top out near 2,900 requests a second, so subtracting the direct run would understate every gateway’s time. The capacity figure measures that load instead.

## Streams arrive as they were sent

200 streamed requests at 16 clients, and two coding agents’ recorded sessions replayed turn by turn.

| Gateway | First token, added | Gap between chunks, p99 | Claude Code session, added | Codex session, added |
| --- | ---: | ---: | ---: | ---: |
| Kaeso (0.7.0) | +0.87 ms | 7.8 ms | +41.5 ms | +34.0 ms |
| Kaeso (0.7.0, request logs on) | +0.94 ms | 8.1 ms | +66.0 ms | +65.5 ms |
| LiteLLM Rust (main at d4e8f96, unreleased) | +0.96 ms | 40.4 ms | +41.0 ms | +44.2 ms |
| Bifrost (2.6.0) | Not comparable | 20.5 ms | +126.4 ms | +91.3 ms |
| Portkey (1.15.2) | +27.5 ms | 8.6 ms | +91.4 ms | +104.2 ms |
| LiteLLM Python (1.110.2) | +119 ms | 34.0 ms | +502.1 ms | +507.7 ms |

Straight to the mock, the gap between chunks is 7.5 ms at p99. A session is 30 recorded turns sent one after another; added is its whole time through the gateway minus straight to the mock. Every streamed session gave valid tool calls through every gateway.

Bifrost sends its first event before the provider answers, and the benchmark counts that as the first token, so its time to first token reads as negative.

## Where others are ahead

Results Kaeso doesn’t win, and what its extra work costs.

- **Memory**: LiteLLM Rust never went above 86 MB and needs no database. Kaeso reached 164 MB, and its PostgreSQL and Valkey took another 271 MB.
- **One client**: With one request at a time, LiteLLM Rust’s p99 was +0.98 ms against Kaeso’s +1.66 ms, and no round of Kaeso’s came closer.
- **A draw**: Time to first token (+0.87 against +0.96 ms) and the Claude Code session (+41.5 against +41.0 ms) fall within each other’s range.
- **Request logs**: With every request logged to ClickHouse, Kaeso served 5,324 requests a second, 16% fewer, and its databases took 1,163 MB.

## What each gateway did

Every gateway ran as its project ships it, against the mock in Anthropic’s Messages format. Kaeso checked the most on every request.

| Gateway | Version | Checked on each request | Kept |
| --- | ---: | ---: | ---: |
| Kaeso | Release 0.7.0 | A project’s API key and the models the project allows | Usage in PostgreSQL; no request logs |
| Kaeso (request logs on) | Release 0.7.0 | A project’s API key and the models the project allows | Usage in PostgreSQL, and every request in ClickHouse |
| LiteLLM Rust | main at d4e8f96, unreleased | One master key from its config | Nothing: no database |
| Bifrost | Release 2.6.0 | No key | Nothing: its logging and stores off |
| Portkey | Release 1.15.2 | No key | Nothing: its logging off |
| LiteLLM Python | Release 1.110.2 | One master key from its config | Nothing: no database |

## How it ran

Everything needed to check a number: the machine, the benchmark, every version and how each run was paired.

- **Machine**: A DigitalOcean CPU-Optimized c-4 in Amsterdam: 4 dedicated vCPUs (Intel Xeon Platinum 8358, 2.6 GHz), 8 GB of memory, Ubuntu 24.04 and Docker.
- **Cores**: Each gateway ran in Docker on the host network, pinned to two cores. The mock, the load drivers and Kaeso’s databases ran on the other two.
- **Benchmark**: BerriAI/ai-gateway-bench at 44365ca, with its mock provider and drivers unchanged: load-probe for requests, run_http_bench.py for streams and session_runner.py for agent sessions.
- **Mock provider**: Answers in Anthropic’s Messages format: its first token after 20 ms, then 40 tokens 5 ms apart. In the capacity pass it answers at once.
- **Load**: 1, 4, 16 and 64 clients sending 1,000, 2,000, 4,000 and 16,000 requests a round, 200 streams at 16 clients and four agent sessions of 30 turns.
- **Rounds**: 3 rounds, the gateways in a new random order each round, each one measured right after a direct run of the same shape. Every figure is the median of the rounds, with their range.
- **CPU and memory**: Every process of the gateway’s container, read four times a second, and the same for its databases, reported beside it.
- **Day**: 22 November 2026

### Versions

- Kaeso 0.7.0: `ghcr.io/logicplanes/kaeso-gateway:0.7.0` `sha256:3d32fd7827417e49475b4ac9ad7d25f7daed28f02fa489ef4077ff501faaae1a`
- LiteLLM Rust d4e8f96: `github.com/BerriAI/litellm@d4e8f96, litellm-gateway built with rust:1.98.0` `sha256:b80e284d93b67806511e0371456d80083d6f1d932abd723a1b31c1ed578ae122`
- Bifrost 2.6.0: `maximhq/bifrost:v2.6.0` `sha256:5968d007ba4f74ee01bc4ceb071decb9927a0b7bb43dbb9072208f75e2d522e3`
- Portkey 1.15.2: `portkeyai/gateway:1.15.2` `sha256:97f094d9c8a764cbfaa2a7138c0017b247ca923bb06db1b4c13b7f8a33b5200d`
- LiteLLM Python 1.110.2: `ghcr.io/berriai/litellm:v1.110.2` `sha256:ef798594ca3db8d81bad2e1439f536f49b71fe095ee9f20416f5e149b4e22035`

Kaeso’s databases: postgres:18.6-alpine, valkey/valkey:9.1.3-alpine and clickhouse/clickhouse-server:26.3.62.3-alpine.

### Run by Kaeso’s author

The person who builds Kaeso ran this benchmark: one machine, three rounds. BerriAI calls its own published results single-host and vendor-run; the same holds here. Run it yourself before you rely on it.

## Run it yourself

On a fresh Linux machine with four cores, as root. The setup installs Docker, builds the benchmark and LiteLLM Rust at the same commits and pulls every image by its digest.

```bash
curl -fsSLO https://kaeso.app/benchmarks/2026-11-22/harness.tar.gz
tar -xzf harness.tar.gz
bash kaeso-gateway-bench/setup.sh
cd /root
ai-gateway-bench/.venv/bin/python run.py
ai-gateway-bench/.venv/bin/python run_capacity.py
```

Three rounds of every gateway take about 90 minutes, the capacity pass another 30. Absolute numbers change from machine to machine, so compare gateways measured in the same run.

- [Scripts, configs and setup](https://kaeso.app/benchmarks/2026-11-22/harness.tar.gz)
- [Every raw result and log](https://kaeso.app/benchmarks/2026-11-22/raw.tar.gz)
- [The figures on this page, as JSON](https://kaeso.app/benchmarks/2026-11-22/results.json)
- [The latency sweep, as CSV](https://kaeso.app/benchmarks/2026-11-22/results.csv)
