A proxy under load shouldn't fire one uncoordinated upstream call per client request. It should group requests that arrive together and release them in controlled waves. Here's the why, the trade, and the proof.
Same burst of requests. Left: every request hits the upstream the instant it arrives. Right: requests are windowed and released under a concurrency cap. Hit play.
The roadmap said "accumulate requests into one upstream call." Anthropic literally has an API for that. Using it here would have been wrong.
Picking the right primitive over the same-named one is the actual engineering. The Batches API still has a home later, as a separate async endpoint for bulk work. See the Batches API docs vs the realtime Messages API.
Two flush triggers, one shared client, one concurrency gate, hard timeouts so nothing can hang.
BATCH_MAX_SIZE → flush nowBATCH_MAX_WAIT_MS since first item → flushasyncio.Semaphoregather(return_exceptions=True)wait_for timeoutasync def run_one(item):
async with self._sem: # concurrency cap
return await self._dispatch_one(item.params)
results = await asyncio.wait_for( # per-window timeout
asyncio.gather(*(run_one(it) for it in batch), return_exceptions=True),
timeout=self._batch_timeout_s, # one stuck call can't hang the rest
)
Why a concurrency cap is the right lever:
Little's Law says in-flight work
L = λ × W. When upstream latency W spikes, holding L fixed with a
semaphore stops in-flight work from exploding. You pay a little queueing delay to avoid an unbounded meltdown.
vLLM / TGI batch tokens across concurrent requests to keep the GPU busy. Same instinct, lower layer.
Buffer small sends briefly to coalesce them into fuller packets. A size/time window, exactly like ours.
Coalesce many field fetches in one tick into a single backend call. Batch-per-tick.
Many writes share one fsync. Trade a touch of latency for far more throughput.
| Variable | Default | Meaning |
|---|---|---|
BATCH_ENABLED | 0 | Master switch. Off keeps the plain realtime path. |
BATCH_MAX_SIZE | 16 | Flush once the window holds this many requests. |
BATCH_MAX_WAIT_MS | 20 | Max wait for a window to fill before flushing. |
BATCH_MAX_CONCURRENCY | 8 | Max simultaneous in-flight upstream calls. |
BATCH_TIMEOUT_S | 60 | Hard ceiling on one window's dispatch. |
Every term that matters here, with a primary source.