The Kaeso gateway, one Rust binary between your apps and your providers. It checks every key and limit, translates every format and logs every request.

Step 1 of 8

Key

The API key decides the project and the member or service account it acts for. Keys are stored only as hashes and answered from memory for 30 seconds.

What the app can get

401UnauthorizedThe key is missing, unknown, disabled, archived or expired.
403ForbiddenThe key has platform access only, or its member lacks the right api.request.
429Rate limitedToo many requests with unknown keys came from the same address.

The gateway reads the key from x-api-key, or else from Authorization: Bearer:

x-api-key: sk-kso-…Authorization: Bearer sk-kso-…

Step 2 of 8

Decode

KPP decodes the body from the app's format into Kaeso's canonical format. Anything Kaeso never sends to a provider is refused here.

What the app can get

413Too largeThe body is larger than 8 MiB, the default limit.
400Bad requestKaeso can't read the request, or it asks for something Kaeso never sends, such as more than one choice.
503UnavailableTranslations in progress use their memory, and the request waited 10 seconds.

Both limits are settings; these are their defaults:

[body_limits]model_api = "8MiB"in_flight = "512MiB"

Step 3 of 8

Model

The name resolves in the project's catalogue, and the request is checked against the model's capabilities before anything is sent.

What the app can get

404Not foundThe model isn't in the project's catalogue, or the key may not use it. Both answer alike, so a key can't learn which other models exist.
400Bad requestSeveral providers serve the id, or the model lacks something the request needs, such as tools or images.

The same 404 in each format, so every SDK raises its usual error:

{  "error": {    "message": "The model `gpt-9` does not exist or this key may not use it.",    "type": "invalid_request_error",    "param": "model",    "code": null  }}

Step 4 of 8

Limits

Every rate and budget of the key, the member and the project is checked at once in Valkey. Over a limit, the app gets 429 with Retry-After.

What the app can get

429Rate limitedA rate or a hard budget is reached. Retry-After says when to try again.
200OKA soft budget is passed. The request goes on, and the budget only alerts.
503UnavailableValkey can't be reached and the request falls under a hard budget.

Retry-After gives the wait in whole seconds, rounded up, and retry-after-ms the exact wait:

HTTP/1.1 429 Too Many Requestsretry-after: 13retry-after-ms: 12400kaeso-request-id: req_…

Step 5 of 8

Provider key

The project's key for the provider is chosen and decrypted, and kept only in memory for at most 30 seconds.

What the app can get

400Bad requestThe project has no provider key for the model's provider.
500Server errorKaeso can't decrypt the provider key.

Master keys come from the gateway itself, and rewrap moves every stored secret to the active one:

docker compose exec gateway kaeso master-key generatedocker compose exec gateway kaeso master-key rewrap

Step 6 of 8

Encode and send

KPP encodes the request for the provider's format. What that format can't express is refused before it reaches the provider.

What the app can get

400Bad requestThe provider's format can't express what the request asks for.
502Bad gatewayThe provider can't be reached, or it refused the project's provider key.
504Gateway timeoutThe provider didn't answer in time.

A provider that refuses the project's key gets your app a 502 that asks for an admin, never the provider's 401:

{  "type": "error",  "error": {    "type": "api_error",    "message": "The provider refused this project's provider key. Ask an organization admin to check it under Providers."  },  "request_id": "req_…"}

Step 7 of 8

Reply

The reply is decoded and encoded in the app's format, event by event for a stream, with the kaeso-model and kaeso-provider headers.

What the app can get

200OKThe reply in the app's own format, with the model and provider that served it.
529OverloadedThe provider is overloaded: 529 in the Anthropic format, 503 in the OpenAI formats.

Every answer names the request, and once the model is known, who served it:

kaeso-request-id: req_…kaeso-provider: anthropickaeso-model: claude-haiku-5-5

Step 8 of 8

Log

Usage, the cost at the prices in force when the request was made, and the request log are recorded, with the content when the organization allows it.

What the app can get

200OKLogging never holds a request up: the reply goes out first, and log entries wait in memory.
499Client closedThe app closed the connection. Kaeso stops the provider request at once and charges what was used.

Log rows wait in memory and go to ClickHouse in batches:

[logs]buffer_size = "256MiB"flush_interval = "1s"flush_rows = 5000

What the gateway does with every request.

Keys that stay in the gateway, state in stores you already run, and a service you operate like any other.

Keys stay in the gateway

Provider keys never reach your apps, and every action in the console is checked against one right.

Read about provider keys

What your apps holdsk-kso-AbC1…wxyzInference

A Kaeso key, stored only as a hash, with an expiry and inference or platform access.

What the gateway holdsOpenAIAnthropicAES-256-GCM

Provider keys, envelope-encrypted with AES-256-GCM under a master key ring you hold. Rotate the master key with kaeso master-key rewrap.

Sign-in: Password with an authenticator app and recovery codes, sign-in links, and single sign‑on through each organization's OIDC provider
Rights: Fine-grained rights with built-in roles and up to 50 custom roles per organization
Audit logs: For the instance, every organization and every project

Sign-in: Password with an authenticator app and recovery codes, sign-in links, and single sign‑on through each organization's OIDC provider
Rights: Fine-grained rights with built-in roles and up to 50 custom roles per organization
Audit logs: For the instance, every organization and every project

State in stores you know

Gateway instances keep nothing of their own, so you add more behind your load balancer. They share one PostgreSQL, ClickHouse and Valkey.

Read the architecture

Every gateway instanceStateless

PostgreSQLKeys, catalogue, limits, usage and audit logs

ClickHouseRequest logs and usage rollups

ValkeyRate and budget counters, shared caches

PostgreSQL: Keys, catalogue, limits, usage and audit logs
ClickHouse: Request logs and usage rollups
Valkey: Rate and budget counters, shared caches

PostgreSQL: Keys, catalogue, limits, usage and audit logs
ClickHouse: Request logs and usage rollups
Valkey: Rate and budget counters, shared caches

Run it like any service

Configure it with KAESO_* variables or a TOML file, read every secret from a file, and watch it with the tools you already run.

Read the configuration

kaeso.toml

1listen = "0.0.0.0:8080"2public_url = "https://kaeso.example.com"3 4[providers.anthropic]5base_url = "https://api.anthropic.com"6 7[upstream]8request_timeout = "600s"9 10[logs]11buffer_size = "256MiB"

1KAESO_LISTEN=0.0.0.0:80802KAESO_PUBLIC_URL=https://kaeso.example.com3KAESO_PROVIDERS_ANTHROPIC_BASE_URL=https://api.anthropic.com4KAESO_UPSTREAM_REQUEST_TIMEOUT=600s5KAESO_LOGS_BUFFER_SIZE=256MiB6KAESO_MASTER_KEYS_FILE=/run/secrets/master_keys

Metrics: Prometheus on port 9100, which Caddy doesn't route; keep it private
Health: /healthz while the process runs, and /readyz with the state of PostgreSQL, Valkey and ClickHouse
Logs: Structured JSON, and request logs kept 90 days, content 30 days, by default
Streams: Keepalives every 15 seconds; the provider call is cancelled when the app goes away

Metrics: Prometheus on port 9100, which Caddy doesn't route; keep it private
Health: /healthz while the process runs, and /readyz with the state of PostgreSQL, Valkey and ClickHouse
Logs: Structured JSON, and request logs kept 90 days, content 30 days, by default
Streams: Keepalives every 15 seconds; the provider call is cancelled when the app goes away

One repository, one version.

The gateway, the console, KPP and Kaeso UI ship together, MIT licensed.

Browse the source on GitHub (opens in a new tab)

Put it in front of your apps.

One Compose file with Caddy, the gateway, the console, PostgreSQL, ClickHouse and Valkey, on your own servers.

Self-host Kaeso Read the docs