AI Model Sizing — Infrastructure Quick Reference

Three numbers decide your deployment: how much VRAM the model + cache needs, how many GPUs hit your tokens/sec target, and whether the network can keep up. Worked example: Cisco Foundation-Sec-8B on an NVIDIA H100.

Foundation-Sec-8B params 8 B precision FP16 · 2 B layers 32 d_model 4096 GQA 32/8 = 4 GPU H100 · 80 GB · 3350 GB/s

VRAM — Will it fit?

Model weights (static) + KV cache (grows with users × context)

Footprint on one H100 (80 GB)

16 GB · Model

13 GB · KV Cache ★ the variable cost

51 GB · Spare capacity

0 GB40 GB80 GB

Model weights — fixed

params × precision = size
8 B × 2 B (FP16) = 16 GB

Cross-check by tensor groups: 4.98 + 5 + 4.92 + 1.17 = 16.07 GB ✓

★ KV Cache — how the 13 GB is built

Per-token footprint stored in the cache

(2 × precision × n_layers × d_model) ÷ GQA factor
2 × 2 B × 32 × 40964 (GQA = 32 ÷ 8) = 131 KB / token

The leading 2 = one K + one V tensor per token. GQA divides KV heads, shrinking the cache 4×.

Then scale to the live workload

1
Tokens cached per GPU = 25 users × 4k context = 100k tokens
2
KV VRAM = 100k tokens × 131 KB = 13.1 GB ≈ 13 GB
3
Total = 16 GB model + 13 GB KV = 29 GB → fits in 80 GB

Single max-context (128k) for one user alone ≈ 17 GB of KV. The cache — not the weights — is what you plan capacity around.

GPU Tokens/sec — How many GPUs?

For dense decode the bottleneck is memory bandwidth, not compute

Per-GPU throughput — the two candidate limits

6.3k tps
Memory BW BOTTLENECK

@ 4k ctx

123k tps
Compute (FLOPs) NOT LIMITING

19× headroom

Decode reads the whole KV cache + weights for every single token → bandwidth runs out long before the math units do. Lowest number wins → memory-bound at 6.3k tps.

25k tps needed ÷ 6.3k tps/GPU = 4 GPUs

✓ Primary bound — memory bandwidth

GPU Mem BWKV read bytes = 3350 GB/s (H100)131 KB × 4k tokens = 6.3k tps

At 128k context the read grows 32× → ≈ 197 tps. Longer context ⇒ fewer tokens/sec.

✗ Not the bottleneck — compute (FLOPs)

FLOPs per token = 2 × parameters
1979 TFLOPS (H100)2 × 8 B = 123k tps

Shown for completeness only. Compute caps out ~20× higher than memory, so the 2×params·FLOPs formula does not set throughput for this model. (For MoE, use active param count.)

North / South Communication — Can the fabric keep up?

Per-token transfer size is set by the model's precision — the link is never the limit

Clients / GPUs

produce tokens

400 Gbps

⇄

256 KB / token

Fabric

N/S link

Even at the model's full per-token transfer size, a single 400 Gbps link carries ~190K tps — roughly 30× the 6.3k-tps memory bound. The network has enormous headroom.

Reference points (per token)

450 GB NVLink

1.7M tps

400 Gbps Eth

190K tps

100 Gbps Eth

47K tps

Step 1 — token size, from the precision formula

token size = hidden_size × layers_on_GPU × precision
4096 × 32 × 2 B (FP16) = 256 KB ≈ 2.1 Mbit

Precision is derived from the model (FP16 = 2 bytes) — the same 2 B that set the 16 GB weights and the 131 KB KV token. Bigger precision ⇒ bigger transfer.

Step 2 — divide bandwidth by token size

link bandwidthtoken transfer bytes = 400 Gbps2.1 Mbit / token = ~190K tps

190K ≫ 6.3K → communication is not the bottleneck. (The bare user-facing token is only ~4 B, so client-side N/S has billions of tps of headroom.)

VERDICT

Foundation-Sec-8B on an H100 is Memory-Bandwidth Bound: compute (123k) and network (190k) have huge headroom, so the binding number is 6.3k tps @ 4k context (≈ 197 tps @ 128k). Size GPU count off that one figure.

Putting it together — 100 users · 4k context · 25k tps target

How the three numbers above combine into a deployment

GPUs (from §2)

25k ÷ 6.3k tps

Users / GPU

100 users ÷ 4 GPUs

KV cache / GPU

13 GB

25 × 4k × 131 KB

Spare VRAM / GPU

51 GB

80 − (16 + 13)

N/S traffic

~12.8 Mbps

25k tps · tiny ≪ link

UCS X-Series · 2–8 GPU · H100 · 100 Gbps N/S

C845A · 2–8 GPU · H100 · 200 Gbps N/S

C885A · 8 GPU · H100 · 200 Gbps N/S