Plan a local AI workstation

Memory breakdown

Model weights (Q4_K_M (GGUF, 4-bit))15.0 GB at FP16

4.53 GB

KV cache (8192 tokens x 1)

1.00 GB

Runtime overhead (activations, buffers, CUDA context)

0.90 GB

Display and driver reserve

0.80 GB

Total runtime requirement

7.24 GB

Weight size is the theoretical file size; the total is what the runtime needs. KV cache = 2 x layers x KV heads x head dim x bytes x tokens x concurrency. Engines and versions differ, so treat the overhead as an estimate.

Speed

Measured tokens per second for this GPU from public benchmark pages. Model, quantization and engine are shown because they change the result completely.

• Llama 2 7B Q4_0 · llama.cpp (Vulkan) commit 4fdbc1e · generation (tg128)118 tok/s
• Llama 2 7B Q4_0 · llama.cpp (Vulkan) commit 4fdbc1e · prompt processing (pp512)2017 tok/s
• Llama 2 7B Q4_0 · llama.cpp (ROCm/HIP) commit 00131d6 · generation (tg128)101 tok/s
• Llama 2 7B Q4_0 · llama.cpp (ROCm/HIP) commit 00131d6 · prompt processing (pp512)2152 tok/s

Theoretical single-stream ceiling from memory bandwidth: 128 tok/s (624 GB/s divided by 8.03B active parameters at this quantization). Real results are usually 50 to 80% of this. An upper bound, not a benchmark.