MLPerf Inference v5.1 (2025): Results Explained for GPUs, CPUs, and AI Accelerators

Understanding What MLPerf Inference Truly Evaluates

MLPerf Inference benchmarks assess the speed and efficiency of a complete system-encompassing hardware, runtime environments, and serving infrastructure-when running fixed, pre-trained machine learning models. These evaluations are conducted under stringent latency and accuracy requirements to ensure real-world relevance. The results are categorized into Datacenter and Edge suites, each employing standardized request patterns, known as “scenarios,” generated by the LoadGen tool. This approach guarantees fairness across different architectures and reproducibility of results.

Within MLPerf, the Closed division mandates the use of fixed models and preprocessing pipelines, enabling direct, apples-to-apples comparisons. Conversely, the Open division permits modifications to models and preprocessing, which may lead to less comparable outcomes. Additionally, submissions are tagged by availability status-Available for shipping products, Preview for near-release versions, and RDI (Research, Development, Internal) for experimental configurations.

Highlights of the 2025 MLPerf Inference Update (v5.1)

Released on September 9, 2025, the v5.1 update introduces three cutting-edge workloads and expands interactive serving capabilities:

• DeepSeek-R1: The first benchmark focused on reasoning tasks, emphasizing complex control flow and memory access patterns.
• Llama-3.1-8B: A new summarization model replacing the previous GPT-J benchmark, reflecting advancements in language model architectures.
• Whisper Large V3: An automatic speech recognition (ASR) model that broadens modality coverage.

This cycle saw participation from 27 submitters and introduced new hardware platforms such as the AMD Instinct MI355X, Intel Arc Pro B60 48GB Turbo, NVIDIA GB300, RTX 4000 Ada-PCIe-20GB, and RTX Pro 6000 Blackwell Edition. The interactive scenarios, which impose strict Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT) limits, were extended beyond single-model tests to better represent agent and chat workloads.

Decoding MLPerf Scenarios: Four Serving Patterns Aligned with Real-World Applications

MLPerf defines four primary serving scenarios, each simulating different workload characteristics:

• Offline: Focuses on maximizing throughput without latency constraints, ideal for batch processing where scheduling and batching strategies dominate performance.
• Server: Models Poisson-distributed request arrivals with strict p99 latency bounds, closely mirroring backend services for chatbots and agents.
• Single-Stream: Emphasizes strict tail latency per individual stream, typical for edge devices with real-time requirements.
• Multi-Stream: Tests concurrency by maintaining fixed inter-arrival intervals across multiple streams, stressing parallel processing capabilities.

Each scenario is associated with specific performance metrics, such as maximum Poisson throughput for Server and overall throughput for Offline, enabling targeted evaluation based on workload type.

Latency Metrics for Large Language Models: Elevating TTFT and TPOT

In the context of large language models (LLMs), MLPerf now treats TTFT (time-to-first-token) and TPOT (time-per-output-token) as critical latency indicators. The v5.0 release introduced tighter interactive latency thresholds for models like Llama-2-70B, setting p99 TTFT at 450 ms and TPOT at 40 ms to better reflect user experience expectations. For larger models with extended context windows, such as Llama-3.1-405B, more lenient limits apply (p99 TTFT of 6 seconds and TPOT of 175 ms) due to their computational complexity. These latency constraints continue into v5.1, alongside the introduction of new LLM and reasoning benchmarks.

2025 Datacenter Benchmark Suite: Closed Division Targets for Direct Comparison

The v5.1 datacenter benchmarks include the following key workloads, each with defined quality and latency thresholds:

• LLM Q&A – Llama-2-70B (OpenOrca): Conversational mode with 2000 ms TTFT/200 ms TPOT; Interactive mode with 450 ms TTFT/40 ms TPOT; accuracy targets at 99% and 99.9%.
• LLM Summarization – Llama-3.1-8B (CNN/DailyMail): Conversational latency at 2000 ms TTFT/100 ms TPOT; Interactive latency at 500 ms TTFT/30 ms TPOT.
• Reasoning – DeepSeek-R1: TTFT capped at 2000 ms and TPOT at 80 ms; quality measured at 99% of FP16 exact-match baseline.
• ASR – Whisper Large V3 (LibriSpeech): Quality assessed via Word Error Rate (WER) for both datacenter and edge deployments.
• Long-Context – Llama-3.1-405B: TTFT of 6000 ms and TPOT of 175 ms.
• Image Generation – SDXL 1.0: Evaluated using FID and CLIP score ranges; Server scenario enforces a 20-second latency limit.

Legacy computer vision and natural language processing models such as ResNet-50, RetinaNet, BERT-Large, DLRM, and 3D-UNet remain included to maintain continuity with previous benchmark cycles.

Interpreting Power Consumption Data in MLPerf

The optional Power measurement tracks system-level energy consumption during benchmark runs, reporting wall-plug power usage. For Server and Offline scenarios, total system power is recorded, while Single-Stream and Multi-Stream scenarios report energy per stream. Only measured runs are valid for energy efficiency comparisons; theoretical TDP values or vendor estimates are excluded. The v5.1 update includes power submissions for both datacenter and edge systems, encouraging broader participation to better understand energy-performance trade-offs.

Guidelines for Accurate Benchmark Interpretation

• Compare Closed division results exclusively: Open division submissions may involve different models or quantization techniques, making direct comparisons unreliable.
• Align accuracy targets: Higher accuracy thresholds (e.g., 99.9% vs. 99%) typically reduce throughput, so comparisons should consider quality levels.
• Exercise caution when normalizing: MLPerf reports system-level throughput under specific constraints. Dividing throughput by the number of accelerators yields a derived “per-chip” figure, which is not an official metric and should only be used for rough budgeting, not marketing claims.
• Filter by availability status: Prioritize Available systems for procurement decisions and include power metrics when energy efficiency is a concern.

Analyzing 2025 Benchmark Outcomes: GPUs, CPUs, and Emerging Accelerators

GPUs: From rack-scale clusters to single-node setups, new GPU architectures excel in Server-Interactive scenarios with tight TTFT/TPOT constraints and in long-context workloads where efficient scheduling and key-value cache management are critical. High-throughput rack-scale systems, such as the NVIDIA GB300 NVL72 class, lead in aggregate performance. When comparing these to single-node systems, normalize results by both accelerator and host counts, ensuring identical scenarios and accuracy targets.

CPUs: CPU-only benchmarks continue to serve as important baselines, highlighting preprocessing and dispatch overheads that can limit accelerator performance in Server mode. The v5.1 update introduces new Intel Xeon 6 results and mixed CPU+GPU configurations. When comparing systems with similar accelerators, consider host CPU generation and memory configurations.

Alternative accelerators: The 2025 cycle broadens architectural diversity with GPUs from multiple vendors and new workstation/server SKUs. For Open division submissions involving pruned or low-precision models, ensure comparisons maintain consistent division, model, dataset, scenario, and accuracy parameters.

Practical Benchmark-to-SLA Mapping for Informed Procurement

• Interactive chat and agent applications: Use Server-Interactive benchmarks on Llama-2-70B, Llama-3.1-8B, or DeepSeek-R1, focusing on latency and accuracy, especially p99 TTFT and TPOT metrics.
• Batch summarization and ETL workflows: Rely on Offline benchmarks with Llama-3.1-8B, where throughput per rack is a key cost factor.
• ASR front-end processing: Evaluate Whisper V3 in Server mode with tail-latency constraints, paying attention to memory bandwidth and audio pre/post-processing efficiency.
• Long-context analytics: Assess Llama-3.1-405B benchmarks, considering whether your user experience can accommodate a 6-second TTFT and 175 ms TPOT latency.

Key Takeaways from the 2025 MLPerf Inference Cycle

• Interactive LLM serving is now a baseline expectation. The stringent TTFT and TPOT limits in v5.x highlight the importance of scheduling, batching, paged attention, and key-value cache management, shifting leadership compared to pure Offline benchmarks.
• Reasoning workloads are officially benchmarked. DeepSeek-R1 introduces new challenges related to control flow and memory traffic, distinct from traditional next-token generation tasks.
• Expanded modality coverage enhances realism. Models like Whisper V3 and SDXL test beyond token decoding, exposing I/O and bandwidth bottlenecks in real-world pipelines.

Conclusion

MLPerf Inference v5.1 delivers a comprehensive and actionable framework for evaluating machine learning inference performance. To derive meaningful insights, it is essential to focus on the Closed division, align on specific scenarios and accuracy targets-including LLM latency metrics like TTFT and TPOT-and prioritize Available systems with measured power data for efficiency considerations. The 2025 update enriches the benchmark suite with DeepSeek-R1, Llama-3.1-8B, and Whisper Large V3, alongside increased hardware diversity. Procurement decisions should filter results to match production SLAs-favoring Server-Interactive for chat and agent workloads, and Offline for batch processing-and verify claims through the official MLCommons result repository and power measurement methodologies.