Skip to content

Images, speech and video through the same gateway as chat

With Kaeso 0.6.0, apps make images, speech, transcripts, video, music and sound effects through the OpenAI and ElevenLabs SDKs they already use, with Kaeso's keys, limits, logs and costs. What works, on which providers, and how it looks in code.

Published
Reading time
3 min
Author
Kaeso Team
A deep gradient of coral and pink, folding diagonally upward

Most teams that put a gateway in front of their language models still send everything else around it. Image generation goes straight to one provider, speech to another, transcription to a third, each with its own key in the app's environment, its own bill and no shared limit. When a key leaks or a budget runs out, nobody sees it in one place.

Kaeso 0.6.0 brings that traffic into the gateway. Apps make images, speech, transcripts, video, music and sound effects through Kaeso, with the official SDKs, and every request gets the same keys, aliases, limits, budgets, logs and costs as a chat completion.

The same base URL, the same key

Nothing new to learn: point the OpenAI SDK at Kaeso, as you would for chat, and call its media methods.

import base64, osfrom openai import OpenAI client = OpenAI(base_url="https://kaeso.example.com/v1", api_key=os.environ["KAESO_API_KEY"]) image = client.images.generate(    model="gpt-image-2",    prompt="A lighthouse at dusk, watercolor",    size="1024x1024",)with open("lighthouse.png", "wb") as f:    f.write(base64.b64decode(image.data[0].b64_json))

Kaeso sends the request to the model's provider in that provider's own form, and the answer comes back in OpenAI's. The model only has to be in the project's catalogue; any model name or alias works, as for every other endpoint.

What apps can make

  • Images
    API
    OpenAI Images: generations, edits, streamed partial images
    Providers
    OpenAI, SpaceX, Google, OpenRouter, Vercel, Together AI, QwenCloud, Zhipu AI, MiniMax, ElevenLabs
  • Speech
    API
    OpenAI Audio: speech, streamed as events when asked
    Providers
    Every built-in provider that offers it
  • Transcripts and translations
    API
    OpenAI Audio: transcriptions, translations
    Providers
    Every built-in provider that offers it
  • Video
    API
    OpenAI Videos: create, poll, download, remix
    Providers
    SpaceX, Google AI Studio, Gemini Enterprise Agent Platform, MiniMax, Zhipu AI, OpenRouter, Together AI, QwenCloud, ElevenLabs
  • Music, sound effects, voice changer
    API
    ElevenLabs' own SDKs
    Providers
    ElevenLabs, and QwenCloud's Qwen-Audio-TTS-Next for sound effects

The APIs by provider (opens in a new tab) page in the docs shows exactly which API reaches which provider, and says why where one is refused. A request a provider can't take is refused before anything is sent.

Speech and transcripts

with client.audio.speech.with_streaming_response.create(    model="gpt-4o-mini-tts",    voice="coral",    input="Your order shipped today.",) as response:    response.stream_to_file("speech.mp3") with open("meeting.mp3", "rb") as audio:    transcript = client.audio.transcriptions.create(model="whisper-1", file=audio)print(transcript.text)

Speech is sent to the app as it is made. Transcripts and the ElevenLabs voice changer are billed by the length their file states when the provider reports none, so a transcript never shows up as free by accident.

Video jobs

Video generation takes minutes, so it runs as a job. Kaeso starts the job at the provider, keeps track of it under an id of its own, asks the provider while it runs, streams the finished file and bills the job once, when it ends.

video = client.videos.create_and_poll(    model="grok-imagine-video-1.5",    prompt="A lighthouse at dusk, waves rolling in",    seconds="8",    size="1280x720",)if video.status == "completed":    client.videos.download_content(video.id).write_to_file("lighthouse.mp4")

A key limited to some models sees only the videos and images made with those models, and video lookups count against its rate limits.

Apps built for ElevenLabs

Apps written with the official elevenlabs SDKs keep their code and change two values: the base URL and the key.

from elevenlabs import ElevenLabs client = ElevenLabs(base_url="https://kaeso.example.com", api_key=os.environ["KAESO_API_KEY"])audio = client.text_to_speech.convert(    voice_id="21m00Tcm4TlvDq8ikWAM",    text="Your order shipped today.",    model_id="eleven_flash_v2_5",)

Speech, voice changer, sound effects, music, transcription and images all work, and so do ElevenLabs' realtime speech and transcription sessions over WebSocket. Because Kaeso sends each request to the model's provider, an ElevenLabs app can reach other providers' speech and transcription models too. If you run Kaeso behind a proxy of your own, route /v2/ to the gateway and pass WebSocket upgrades.

Media inside chat

Chat requests carry media as well. Apps send audio and video to every provider that takes them, from any API format. Chat Completions and Gemini apps get audio replies from each other's models, and images that Gemini models make reach Gemini apps. A format or provider with no place for something refuses before anything is sent, rather than dropping it silently.

Costs you can read

Every media request lands in the request log with what it made: an image and its size, the characters spoken, the seconds of audio or video. Usage and costs count it next to chat. A cost Kaeso can't price shows as Unknown, never as $0, so a missing price doesn't hide spend. Budgets and rate limits apply to media requests exactly as to chat.

By default, request logs keep the type and size of images, audio and files, not the files themselves. Turn on Keep media for an organization or a project, and the logs keep them too, up to 10 MiB per request, and the request inspector shows, plays and downloads them. The organization's setting is the ceiling for its projects.

Try every model in the Playground

The console's Playground tries every kind of model side by side from one prompt: chat, images, speech, sound effects, music, transcription with recorded or attached audio, and video with progress and Cancel. It is the quickest way to compare two image models or two voices before you write any code. The live demo (opens in a new tab) shows the Playground with sample data.

Media is one of the larger additions of Kaeso 0.6. The docs for the Images (opens in a new tab), Audio (opens in a new tab), Videos (opens in a new tab) and ElevenLabs (opens in a new tab) APIs have every endpoint and parameter.

By Kaeso Team

On this page

Own the gateway.Own the record.

curl -fsSLhttps://kaeso.app/install.sh | sh