This blog has been co-authored by Jon Shelley, Principal PM Manager, Azure Compute.

Microsoft Azure is committed to providing its customers with industry-leading real-world AI capabilities. In December 2021, Microsoft Azure debuted its leadership performance with the MLPerf training v1.1 results. Azure debuted at number one among cloud providers and number two overall at scale among all submitters. Azure’s supercomputer's building blocks were used to generate the results in our v2.0 submissions for the MLPerf inferencing results published on April 6, 2022.

These industry-leading results are driven by Microsoft’s publicly available supercomputing capabilities designed for real-world AI inferencing workloads. Microsoft enables customers of all scales to deploy powerful AI solutions, whether at a focused local scale or at the scale of the largest supercomputers in the world.

Microsoft Azure’s publicly available AI inferencing capabilities are led by the NDm A100 v4, ND A100 v4, and NC A100 v4 virtual machines (VMs) that are powered by NVIDIA A100 SXM and PCIe Tensor Core graphics processing units (GPUs). These results showcase Azure’s commitment to making AI inferencing available to all in the most accessible way—while raising the bar for AI inferencing in Azure.

In our quest to continually provide the best technology for our customers, Azure has recently announced the preview for the NC A100 v4. With this introduction of the NC A100 v4 series, we have provided our customers with three different VM sizes ranging from one to four GPUs. From our benchmarking, we have seen more than two times performance over the previous generation. Azure’s customers can get access to these new systems today by signing up for the preview program.

Some highlights for this round of MLPerf inferencing submissions can be seen in the following tables.

Highlights from the results

ND96amsr A100 v4 powered by NVIDIA A100 80G SXM Tensor Core GPU

Benchmark	Samples/second	Queries/second	Scenarios
bert-99	27,500 plus	~22,500 plus	Offline and server
resnet	300,000 plus	~200,000 plus	Offline and server
3d-unet	24.87		Offline

NC96ads A100 v4 powered by NVIDIA A100 80G PCIe Tensor Core GPU

Benchmark	Samples/second	Queries/second	Scenarios
bert-99	~6,300	~5,300	Offline and server
resnet	144,000	~119,600	Offline and server
3d-unet	11.7		Offline

The above tables showcase three of the six benchmarks the team ran using NVIDIA A100 SXM and PCIe Tensor Core GPUs for offline and server scenarios respectively. Take a look at the full list of results for the various divisions.

Azure works closely with NVIDIA

The results were generated by deploying the environment using the VM offerings and Azure’s Ubuntu 18.04-HPC marketplace image. We worked closely with NVIDIA to quickly deploy the environment and perform benchmarks with industry-leading results in performance and scalability.

These results are a testament to Azure’s focus on offering scalable supercomputing for any workload while enabling our customers to utilize “on-demand” supercomputing capabilities in the cloud to solve their most complex problems. Visit the Azure Tech Community blog to read the steps to reproduce the results.

More about MLPerf

MLPerf is a consortium of AI leaders from academia, research labs, and industry where the mission is to “build fair and useful benchmarks” that provide unbiased evaluations of training and inference performance for hardware, software, and services—all conducted under prescribed conditions. To stay on the cutting edge of industry trends, MLPerf continues to evolve, holding new tests at regular intervals and adding new workloads that represent state-of-the-art AI. MLPerf’s tests are transparent and objective, so users can rely on the results to make informed buying decisions. The industry benchmarking group, formed in May 2018, is backed by dozens of industry leaders. The benchmark tests across inferencing are increasingly becoming the key tests that hardware and software vendors use to demonstrate performance. Take a look at the full list of results for MLPerf Inference v2.0.