LLM inference and quantization
Forty model configurations, one H200, and no single best format
This suite compares BF16, FP8, AWQ, GPTQ, and NVFP4 variants across six model families. Every configuration is measured on the same H200 MIG profile for serving throughput and latency, then evaluated on standard accuracy tasks. The final tables keep throughput and quality side by side because either metric can change the format choice.
Experimental design
Coverage across model size, format, and request rate
The inventory covers Llama 3.1 and 3.2, DeepSeek R1 Distill Qwen, Qwen 2.5, Qwen 3, and Gemma 4 at sizes from 3B to 32B. Serving runs sweep 1, 2, 4, 8, and 16 requests per second using ShareGPT prompts. Quality runs use HellaSwag, Winogrande, ARC Challenge, GSM8K, and MMLU.
Hardware and software
A fixed serving environment
| GPU | NVIDIA H200 NVL, MIG 3g.71gb |
|---|---|
| Host | Dual Intel Xeon Platinum 8480+, 224 threads |
| Runtime | vLLM OpenAI compatible server in Docker |
| CUDA | 12.8 with driver 570.133.20 |
| Serving data | ShareGPT prompts at five request rates |
| Quality harness | lm eval 0.4.11 with local completions |
Throughput
Small models converged near 3,000 output tokens per second
Qwen3 8B FP8 led the sweep at 3,150.57 output tokens per second at 16 requests per second. DeepSeek R1 Distill Qwen 7B in AWQ, FP8, GPTQ, and BF16 all finished within roughly 40 tokens per second of one another. For 7B and 8B models, quality and latency are better selection criteria than peak throughput alone.
Model size
FP8 mattered much more at 32B
Qwen3 32B FP8 reached 2,146.57 output tokens per second, compared with 776.52 for BF16. DeepSeek R1 Distill Qwen 32B showed a similar split at 2,036.66 for FP8 and 801.95 for BF16. The larger BF16 models also developed severe time to first token queues at high request rates. Quantization changed both capacity and latency stability.
Observed boundary
The 3B to 8B group scaled almost linearly through 16 requests per second. Several 30B to 32B BF16 models saturated near 770 to 800 output tokens per second after 4 requests per second.
Quality
Accuracy retention depended on the family and task
Gemma 4 31B FP8 scored 87.54 percent on full MMLU, slightly above its 87.32 percent BF16 result. Qwen 2.5 32B remained near 82 percent across all four formats. On Llama 3.1 8B, FP8 stayed close to BF16 while AWQ and GPTQ gave up more ground on ARC Challenge, GSM8K, and MMLU.
Harness limitation
Some reasoning model scores were excluded
DeepSeek R1 Distill and Qwen 3 entered extended reasoning mode during GSM8K evaluation. The 2,047 token context limit ended generations before the final answer marker, which made strict matching return zero. Those GSM8K rows are omitted rather than presented as model quality results. The repository records the limitation alongside the final tables.
Decision surface
Throughput and quality must be read together
