Every production LLM deployment forces the same three-way compromise: pushing throughput up raises latency, clamping latency down inflates GPU cost, and optimizing cost forces uncomfortable concessions on one of the other two dimensions. DigitalOcean's practitioner guide to dedicated LLM hosting calls this the "trilemma" of throughput, latency, and cost, and treats it as the central engineering challenge in serving models at scale. The architecture you choose—dedicated GPU infrastructure or a managed serverless endpoint—determines where you sit on that trade-off surface. This article explains why the trilemma exists, examines the two dominant deployment patterns, and provides a comparison matrix for matching a pattern to your workload's latency, cost, and scaling requirements.
Understanding the LLM Inference Trilemma: Latency, Cost, and Scalability
The trilemma is not a marketing abstraction; it emerges from the physics and economics of GPU serving. As Michael Brenndoerfer explains in his guide to LLM inference serving, lower latency typically requires either more hardware or smaller batch sizes, both of which increase cost. Higher throughput typically requires larger batches, which increases latency for individual requests. And serving many users cheaply requires high GPU utilization—but high utilization leaves no headroom for traffic spikes. These objectives cannot all be maximized simultaneously.
Batching sits at the center of the tension. Grouping requests onto a GPU amortizes its cost across more users, but each request waits while the batch fills, so per-request latency rises. Writing on the economics of LLM inference, Mathias Lechner puts it bluntly: "LLM inference economics come down to three things: how you batch requests determines the latency-throughput trade-off, that trade-off drives tiered pricing, and who can afford to overprovision GPU capacity determines who wins on cost." His analysis attributes model labs' cost advantage to a deeper pool of workloads that keeps expensive hardware utilized around the clock.
IBM's overview of LLM inference reinforces why these trade-offs matter operationally: because inference workloads run continuously and at scale, their efficiency directly affects end-to-end latency, GPU memory consumption, request throughput, infrastructure cost, and overall system scalability. A third dimension—operational complexity—compounds the picture. Architectures that give you finer control over the trilemma generally demand more engineering effort to manage, and that effort is itself a cost. The question for an architect is therefore not "which architecture is best" but "which compromise profile matches my application."
