Queues and batching for AI workloads
A GPU reads the model's weights from memory at every generation step, and serving one request or 16 costs roughly the same memory traffic, so batching requests together raises throughput a lot for a small increase in latency. A micro-batcher collects requests from a queue until the batch is full or a short wait expires, then runs them together. Serving engines go further with continuous batching, adding and removing requests at every step, and for slow jobs you put a queue in front so users get a job ID instead of a timeout.
Why batching works
During generation, each step reads every weight once. For one request, most of the GPU's compute sits idle waiting on memory. With a batch, the same weights are read once and used for every request in the batch, so throughput rises with batch size until compute or KV cache memory runs out.
A micro-batcher
import asyncio
import time
class MicroBatcher:
def __init__(self, max_batch: int, max_wait_s: float):
self.max_batch = max_batch
self.max_wait_s = max_wait_s
self.queue: asyncio.Queue = asyncio.Queue()
self.batch_sizes: list[int] = []
async def submit(self, item: str) -> str:
future = asyncio.get_running_loop().create_future()
await self.queue.put((item, future))
return await future
async def run(self) -> None:
while True:
item, future = await self.queue.get()
batch = [(item, future)]
deadline = time.monotonic() + self.max_wait_s
while len(batch) < self.max_batch:
remaining = deadline - time.monotonic()
if remaining <= 0:
break
try:
batch.append(await asyncio.wait_for(self.queue.get(), remaining))
except asyncio.TimeoutError:
break
self.batch_sizes.append(len(batch))
await asyncio.sleep(0.05)
for text, fut in batch:
fut.set_result(text.upper())
async def main() -> None:
batcher = MicroBatcher(max_batch=8, max_wait_s=0.02)
worker = asyncio.create_task(batcher.run())
start = time.monotonic()
results = await asyncio.gather(*(batcher.submit(f"request {i}") for i in range(20)))
elapsed = time.monotonic() - start
worker.cancel()
print(f"{len(results)} results in {elapsed:.2f}s, batch sizes {batcher.batch_sizes}")
asyncio.run(main())
Each simulated model call takes 50 ms whatever the batch size, so 20 requests finish in 3 batches instead of 20 calls.
max_wait_s is the latency you're willing to add to fill a batch.
Continuous batching
Static batching waits for the whole batch to finish, so a short answer waits for the longest one. Continuous batching, used by vLLM, TGI and SGLang, schedules at the level of each generation step: a finished request leaves the batch and a waiting one joins at the next step. You get this by using a serving engine rather than building it. See inference serving.
Queues for slow jobs
Document processing, long agent runs and batch evaluations shouldn't hold an HTTP request open. Put them on a queue, return a job ID, process with workers, and let the client poll or receive a webhook. See the async job queue pattern.
Backpressure
A queue without a limit hides overload until latency explodes. Set a maximum queue length, reject or shed load above it with a clear error, and alert on queue depth and wait time, because queue wait is part of the latency your users see.
Provider batch APIs
For work that can wait hours, like nightly evals or bulk classification, provider batch APIs offer lower prices than real-time calls in exchange for a completion window, so check your provider's current terms.
Check your understanding
Run the batcher with max_wait_s=0.0 and max_wait_s=0.2, and explain the change in batch sizes and total time.
From the floor
In the AI data structures lecture, I used video buffering as the everyday example of data arriving in order and being processed in sequence, and request queues in an AI system work the same way, with the difference that you group what's waiting so the GPU does more work per memory read.