One streaming format across every provider

For an interactive application, streaming is the difference between a product that feels alive and one that feels stalled. Users perceive first-token latency, not total latency. A response that starts in 200ms and finishes in three seconds feels faster than one that appears complete after two. So streaming isn't a nice-to-have. It's table stakes.

The problem is that every provider does it differently.

Set stream: true and you get server-sent events instead of a single body. But the shape of those events varies: how deltas are framed, how tool calls arrive, how reasoning tokens are distinguished from content, where usage totals live, how the stream signals completion. Consume them naively and you write a bespoke parser per provider, which is exactly the per-provider coupling a gateway is supposed to eliminate.

for await (const chunk of stream) {
  // One shape, regardless of which provider actually answered.
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

The consumer above works across providers because the gateway normalised the stream before it reached the application. The normalisation is unglamorous (a lot of careful mapping of one event vocabulary onto another), and it is precisely the kind of work that is invisible when it's done well.

Normalising the format is the visible benefit. The valuable one is what it enables underneath: a mid-stream provider failure can be handled by the gateway rather than surfacing to your application as a truncated response.

Without a gateway, a provider that dies halfway through a stream hands your user half an answer. With one, the failure is a routing decision the user never sees.

One piece of advice regardless of how you stream: design your consumer to ignore event types it doesn't recognise rather than to assume a fixed sequence. Tool calls and reasoning tokens arrive as distinct event types, and providers add new ones. A consumer that assumes a rigid order is a consumer that breaks on the next capability. A consumer that skips what it doesn't understand keeps working.