Size the GPUs, power and cooling for an LLM deployment.

Round synthetic figures chosen to make the arithmetic legible. They are not a benchmark, not a measurement of any product, and not drawn from any deployment. The model-fit worksheet is left at zero to show what the page does when it is skipped.

• Peak requests — 420 a minute
• Average output tokens — 380 per request
• Required peak concurrent sequences — 900 sequences
• Measured sustainable output throughput — 640 tokens a second per GPU
• Planning utilisation ceiling — 70 %
• Measured concurrent sequences — 48 sequences per GPU
• GPUs per server — 8 GPUs
• Reserve GPUs — 4 GPUs
• GPU planning power — 700 W each
• Other server power — 1400 W per server
• Storage, network and management — 3.5 kW
• Power usage effectiveness — 1.35 PUE
• Average load factor — 55 %
• Electricity price — 0.21 per kWh
• Measured model footprint — 160 GiB
• Usable VRAM per GPU — 76 GiB
• Parameters, for the weights-only estimate — 70 billions
• Bits per parameter — 16
• Currency — AUD

Required output throughput

2,660.0

tokens a second

GPUs by throughput

6

GPUs

GPUs by concurrency

19

GPUs

Active GPUs

21

GPUs

Replicas

7

model copies

Binding constraint

Model fit

which of throughput, concurrency and model fit sets the number

Planned GPUs

25

GPUs

Servers

4

servers

IT electrical load

26.60

kW

Facility input power

35.91

kW

IT heat rejection

90,763

BTU an hour

GPUs per replica

3

GPUs

Assumptions register

Every figure below is one the reader supplied, not one this page chose. They travel with the result so a reviewer can argue with them.

• Planning utilisation ceiling — 70 %
• Reserve GPUs — 4 GPUs
• GPU planning power — 700 W each
• Power usage effectiveness — 1.35 PUE
• Average load factor — 55 %
• Bits per parameter — 16

Information gaps

1. The reserve you selected is not a whole number of replicas, so part of it cannot serve traffic on its own.

Two constraints, and why the binding one matters

The optional model-fit worksheet

Getting the two measured figures

1. Benchmark the stack you will run, not a similar one. Sustained output tokens per second per GPU depends on the model, the quantisation, the serving framework, batching strategy, context length and the accelerator. Changing any one of them changes the number materially.
2. Measure sustained, not burst. Run long enough for thermal behaviour, memory fragmentation and scheduler effects to appear. A thirty-second benchmark overstates what a fleet does for a month.
3. Use your own prompt and output lengths. A benchmark with 200-token prompts tells you little about a workload with 4,000-token retrieved context.
4. Find the concurrency point where it degrades, not where it fits. Sequences per GPU should be the level at which latency stays acceptable, not the level at which memory has not yet run out.
5. Ask a supplier in writing, and ask what configuration produced it. A quoted throughput figure without the model, quantisation, context length, batch size and serving stack attached is not usable for planning.
6. Re-measure after any change. A model upgrade, a serving-framework upgrade or a context-length increase invalidates the sizing.

1. It is not an electrical or mechanical design. Final electrical distribution, protection, redundancy, cooling, containment, airflow and floor loading require qualified engineering review. The IT load and facility input figures are an input to that work, not a substitute for it.
2. The refrigeration tonnage is a unit conversion. It accounts for nothing about sensible and latent load, ambient conditions, redundancy or the cooling approach.
3. It sizes for the peak you enter. It does not model the arrival distribution, queueing behaviour, tail latency or what happens above the peak. Requests above capacity queue or fail, and which of those it is depends on the serving stack, not on this page.
4. It sizes decode throughput, not prefill. Output tokens set the throughput requirement. Long input contexts create a prefill load that is sized separately and can dominate with short outputs and large retrieved contexts.
5. It assumes uniform GPUs. One accelerator type, one measured figure, one planning ceiling. Mixed fleets need per-pool arithmetic.
6. It does not handle networking or storage design. Interconnect bandwidth between accelerators and servers can be the real constraint for larger models, and it is not modelled.
7. The weights-only figure is weights only. No KV cache, no activations, no runtime overhead. It is not a memory requirement.
8. It costs nothing. Cost is the companion calculator.

• Output tokens per second per GPU — sustained generation throughput, measured on your model, quantisation, serving stack and hardware. The most important input here and the one most often guessed.
• Concurrent sequences — requests held in flight at once. Limited mainly by memory at your context length.
• Planning utilisation ceiling — the share of measured throughput you are willing to design against, leaving headroom for variance.
• Binding constraint — whichever of throughput and concurrency demands more GPUs. It determines what a change would have to alter to help.
• IT load — electrical draw of the equipment: accelerators, servers, storage, network and management.
• PUE — facility input power divided by IT load.
• KV cache — per-sequence attention state held during generation. Grows with context length and batch size, and is excluded from any weights-only figure.
• Usable VRAM — accelerator memory actually available to the model after the driver, runtime and fragmentation, which is below the nameplate figure.