November 12, 2026 · 6 min read

Enterprise speech-to-text on Intel Xeon 6 CPUs, on premise and on cloud

On Intel Xeon 6 CPUs, the optimized model roughly doubles throughput and returns the first token 48% faster on an HPE ProLiant DL380 Gen12 on premise and 86% faster on AWS

Multiverse Computing

Multiverse Computing and Intel benchmarked Whisper Large-v3 Turbo, a widely used speech-to-text model, in its original form and in our optimized version, on Intel Xeon 6 processors with P-cores, both on premise (HPE ProLiant DL380 Gen12) and in the cloud (AWS EC2 c8i.32xlarge). The optimized model roughly doubles throughput with half the parameters and half the memory, at a cost of 0.61 points of Word Error Rate, and runs without GPUs.

Automatic Speech Recognition (ASR) is a core workload for enterprises with large voice operations, such as contact centers, where hundreds of thousands of calls must be transcribed in near real time. Today that work usually runs on GPU infrastructure, which performs well but comes with a significant footprint.

The question we tested with Intel is whether an optimized speech-to-text model can serve production traffic on CPUs alone, and how much capacity that setup sustains.

Whisper Large-v3 Turbo, optimized with CompactifAI

We ran Whisper Large-v3 Turbo, a widely used ASR model, in two forms: the original model and Multiverse Computing’s optimized, more efficient version, which delivers virtually the same accuracy with fewer parameters and a smaller memory footprint. We built it with CompactifAI, our proprietary model optimization technology.

Whisper Large-v3 Turbo is a distilled version of Whisper Large-v3, released in October 2024. WER measured on about 2,600 samples in float16.

That is about 51% fewer parameters and about 52% less storage, for a 0.61-point increase in WER.

Intel Xeon 6 with P-cores and AMX

Intel Advanced Matrix Extensions (AMX) is built into Intel Xeon 6 processors with Performance-cores (P-cores). It provides dedicated tile-based matrix multiplication hardware for BF16, FP16 and INT8, the operations that dominate the compute cost of transformer inference. That makes Xeon 6 with P-cores, and its built-in AMX, a key differentiator for running speech models such as Whisper on CPU.

Both benchmarks used AMX-accelerated BF16 inference served with vLLM on the CPU backend, with inference threads bound to physical cores and one instance per NUMA node behind an nginx round-robin load balancer. On the on-premise server, enabling the AMX-accelerated kernels completed the same load 1.84× faster (38.8 s against 71.5 s for 1,000 requests at 128 concurrent users).

Two deployments:

Results

We compared both models at the operating point of each deployment: 128 concurrent users on premise and 96 on AWS. In both cases the optimized model roughly doubles throughput and returns the first token faster.

RTFx is the real-time throughput factor: how many times faster than real time the system processes audio.

In practice, 711× real time means about 711 hours of audio transcribed per hour on a single on-premise server, roughly 17,000 hours a day. On AWS, 317× real time is roughly 7,600 hours a day on one instance.

Latency stays usable at those loads. On premise, mean time to first token is 1.10 s at 128 concurrent users, against 2.11 s for the original. On AWS it stays under one second up to 96 concurrent requests (0.86 s), where the original reaches 6.33 s. Past those points latency rises for both models, so the whitepapers include the full curves from 1 to 192 concurrent users.

We used vLLM’s built-in load-testing client against an OpenAI-compatible audio transcription endpoint, with 1,000 LibriSpeech samples normalized to about 30 seconds, at concurrency levels from 1 to 192. WER came from the LibriSpeech Clean-Test subset. Hardware details, configuration flags and charts are in the whitepapers.

Download the whitepapers

An optimized speech model on Intel Xeon 6 with P-cores roughly doubles the audio a CPU-only server handles, on premise and on AWS. Each whitepaper covers its deployment in full.

  • ON PREMISE: Whisper Large-v3 Turbo on Intel Xeon 6 with HPE ProLiant DL380 Gen12 (PDF):
Download Publication

  • ON CLOUD: Whisper Large-v3 Turbo on Intel Xeon 6 with AWS EC2 c8i.32xlarge (PDF):
Download Publication

Want to know more?

Reach out to us at business@multiversecomputing.com