OpenAI restructured routing into two layers: a control plane computes global routing weights, while each cluster’s data plane makes per-request decisions from locally cached weights to cut down on synchronous round-trips.

Three protective mechanisms were added to prevent failures: penalties to curb routing toward underperforming engines, dynamic retry budgets to avoid retry storms, and load shedding to degrade gracefully when demand exceeds capacity.

A side effect of frequent traffic shuffling was reduced chat context efficiency, as the KV cache locality depended on directing requests to the same engine within a conversation.

The broader context highlights massive anticipated capital expenditure in AI infrastructure and the high value of efficiency gains, with OpenAI expecting substantial negative cash flow through 2030 even as revenue scales, making routing efficiency critical for capacity and cost savings at scale.

Early routing decisions were hard to explain or debug because they relied on a blended, opaque set of signals rather than clear, understandable rules.

A transferable lesson emerges: separate synchronous local decisions from asynchronous global optimization, and use explicit budgets to prevent feedback loops from magnifying failures.

OpenAI’s initial GPU load-balancing router used an adaptive feedback loop based on per-engine signals, which caused oscillations as cooling improved an engine’s apparent performance, prompting reallocation and breaking KV cache reuse.

The updated architecture adopts a constrained optimization approach that aims to minimize end-to-end latency by weighing four inputs—cluster request volume, network latency, engine capacity/health, and TTFT/TBOT latency profiles—with weights published by the control plane and applied locally by the data plane.