Intel Demonstrates AI Inference Gains in MLPerf Inference v6.1 Benchmarks

Intel has detailed performance enhancements across its enterprise compute portfolio in the latest MLPerf Inference v6.1 benchmark release from MLCommons. The benchmarking round highlighted AI inference throughput increases driven by software optimization rather than hardware revisions, demonstrated on Intel Xeon 6 central processing units and Intel Arc Pro B-series graphics processing units. The cycle also registered expanded customer and hardware partner participation validating Intel platforms across production enterprise models and heterogeneous workload architectures.

Xeon 6 Server Throughput and Partner Deployments
Utilizing the identical socket count and silicon configuration evaluated in MLPerf v6.0, Intel Xeon 6980P processors achieved a 2.4-fold increase in Llama 3.1 8B Server scenario throughput alongside a 56% increase in Offline scenario processing. These gains were achieved entirely through software enhancements, allowing operators to scale existing bare-metal capacity without hardware reconfigurations.

Third-party validations for Intel-based infrastructure grew from 29 benchmarked submissions in v6.0 to 39 in the v6.1 evaluation round. Third-party testing contributions included the first Intel-based benchmark submission from Oracle, the first Xeon CPU inference submission from Red Hat, and initial partner submissions leveraging the Intel Arc Pro B70 GPU delivered by Supermicro and Quanta Cloud Technology. Intel also broadened its Xeon 6 portfolio evaluation from two SKUs in v6.0 to five in v6.1, increasing total CPU inference configurations from 24 to 35, where Xeon remains the sole standalone server CPU tested in MLPerf Inference.

Arc Pro B70 Graphics Benchmarks and End-to-End RAG Workloads
Evaluating the discrete graphics pipeline, a single-node rack system equipped with four Intel Arc Pro B70 GPUs totaling 128 GB of onboard video memory (VRAM) registered benchmark submissions across Llama 3.1 8B, Llama 2 70B, gpt-oss-120B, Whisper speech-to-text, and end-to-end retrieval-augmented generation (E2E-RAG). Evaluating the identical four-GPU hardware profile utilized in round v6.0, gpt-oss-120B Server throughput climbed 36%, and Offline processing rose 27%.

Intel also participated in the co-development and benchmarking of the new MLPerf Inference v6.1 E2E-RAG benchmark. Executed on a unified system pairing an Intel Xeon 6787P processor with four Intel Arc Pro B70 discrete GPUs, the multi-stage pipeline was segmented across processor types. The host Xeon CPU executed document embedding, vector database search, semantic reranking, and Small Language Model (SLM) routines, while the Arc Pro B70 GPUs handled Large Language Model (LLM) token generation. Performance enhancements engineered for Xeon are upstreamed directly into open-source enterprise AI frameworks, while Arc Pro B70 development continues to refine compiler backends, compute kernels, and model-serving software stacks.

Additional Context
This section details technical specifications not included in the original news release.

Scaling generative AI inference on host x86 CPU architectures without hardware changes relies on low-precision vectorization kernels, notably Intel Advanced Matrix Extensions (AMX). Embedded within Granite Rapids-based Xeon 6 P-core architectures, AMX units execute matrix multiply-accumulate operations in hardware across bfloat16 and INT8 data types via dedicated tile registers, achieving up to 2,048 INT8 operations per cycle per core. Framework upstreaming into platforms like PyTorch, vLLM, and Hugging Face integrates low-level Intel oneAPI Deep Neural Network Library (oneDNN) primitives alongside PagedAttention algorithms, FlashAttention-2 CPU implementations, and weight-only INT4/INT8 quantization to alleviate memory bandwidth bottlenecks during auto-regressive decoding phases. In heterogeneous E2E-RAG pipelines, offloading dense similarity searches (such as HNSW index traversal) and embedding generation to host CPUs eliminates PCIe bus contention, freeing GPU high-bandwidth memory (HBM/GDDR) exclusively for key-value (KV) caching and continuous batched multi-head attention computations across distributed tensor-parallel GPU arrays.

Edited by Romila DSilva, Induportals Editor, with AI assistance.