{"id":30284,"date":"2023-11-08T09:00:00","date_gmt":"2023-11-08T17:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/en-us\/blog\/?p=30284"},"modified":"2025-06-12T00:34:07","modified_gmt":"2025-06-12T07:34:07","slug":"azure-sets-a-scale-record-in-large-language-model-training","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/azure-sets-a-scale-record-in-large-language-model-training\/","title":{"rendered":"Azure sets a scale record in large language model training"},"content":{"rendered":"\n

Azure<\/a> empowers intelligent services like Microsoft Copilot<\/a>, Bing<\/a>, and Azure OpenAI Service<\/a> that have captured our imagination in recent days. These services, facilitating various applications like Microsoft Office 365, chatbots, and search engines with generative AI, owe their magic to large language models (LLMs). While the latest LLMs are transcendental, bringing a generational change in how we apply artificial intelligence in our daily lives and reason about its evolution, we have merely scratched the surface. Creating more capable, fair, foundational LLMs that consume and present information more accurately is necessary.  <\/p>\n\n\n\n

How Microsoft maximizes the power of LLMs<\/h2>\n\n\n\n

However, creating new LLMs or improving the accuracy of existing ones is no easy feat. To create and train improved versions of LLMs, supercomputers with massive computational capabilities are required. It is paramount that both the hardware and software in these supercomputers are utilized efficiently at scale, not leaving performance on the table. This is where the sheer scale of the supercomputing infrastructure in Azure cloud shines and setting a new scale record in LLM training matters. <\/p>\n\n\n\n

\"Scale
Figure 1: Scale records on the model GPT-3 (175 billion parameters) from MLPerf Training v3.0 in June 2023 (3.0-2003) and Azure on MLPerf Training v3.1 in November 2023 (3.1-2002).<\/em> <\/figcaption><\/figure>\n\n\n\n

Customers need reliable and performant infrastructure to bring the most sophisticated AI use cases to market in record time. Our objective is to build state-of-the-art infrastructure and meet these demands. The latest MLPerf\u2122 3.1 Training results<\/a>1<\/sup> are a testament to our unwavering commitment to building high-quality and high-performance systems in the cloud to achieve unparalleled efficiency in training LLMs at scale. The idea here is to use massive workloads to stress every component of the system and accelerate our build process to achieve high quality.<\/p>\n\n\n\n

The GPT-3 LLM model and its 175 billion parameters were trained to completion in four minutes on 1,344 ND H100 v5<\/a> virtual machines (VMs), which represent 10,752 NVIDIA H100 Tensor Core GPUs<\/a>, connected by the NVIDIA Quantum-2 InfiniBand networking platform (as shown in Figure 1). This training workload uses close to real-world datasets and restarts from 2.4 terabytes of checkpoints acting closely a production LLM training scenario. The workload stresses the H100 GPUs Tensor Cores, direct-attached Non-Volatile Memory Express disks, and the NVLink interconnect that provides fast communication to the high-bandwidth memory in the GPUs and cross-node 400Gb\/s InfiniBand fabric. <\/p>\n\n\n\n

\n

Azure\u2019s submission, the largest in the history of MLPerf Training, demonstrates the extraordinary progress we have made in optimizing the scale of training. MLCommons\u2019 benchmarks showcase the prowess of modern AI infrastructure and software, underlining the continuous advancements that have been achieved, ultimately propelling us toward even more powerful and efficient AI systems<\/p>\nDavid Kanter, Executive Director of MLCommons<\/cite><\/blockquote>\n\n\n\n

Microsoft’s commitment to performance<\/h2>\n\n\n\n

In March 2023, Microsoft introduced the ND H100 v5-series which completed training a 350 million parameter Bidirectional Encoder Representations from Transformers (BERT) language model in 5.4 minutes, beating our existing record. This resulted in a four times improvement in time to train BERT within just 18 months, highlighting our continuous endeavor to bring the best performance to our users.<\/p>\n\n\n\n

\"chart\"
Figure 2: Relative size of the models BERT (350 million parameters) and GPT-3 (175 billion parameters) from MLPerf Training v3.1.\u00a0\u00a0<\/figcaption><\/figure>\n\n\n\n

Today\u2019s results are with GPT-3, a large language model in the MLPerf Training benchmarking suite, featuring 175 billion parameters, a remarkable 500 times larger than the previously benchmarked BERT model (figure 2). The latest training time from Azure reached a 2.7x improvement compared to the previous record from MLPerf Training v3.0. The v3.1 submission underscores the ability to decrease training time and cost by optimizing a model that accurately represents current AI workloads.<\/p>\n\n\n\n

The power of virtualization<\/h2>\n\n\n\n

NVIDIA\u2019s submission to the MLPerf Training v3.1 LLM benchmark on 10,752 NVIDIA H100 Tensor Core GPUs achieved a training time of 3.92 minutes. This amounts to just a 2 percent increase in the training time in Azure VMs compared to the NVIDIA bare-metal submission, which has the best-in-class performance of virtual machines across all offerings of HPC instances in the cloud (figure 3).<\/p>\n\n\n\n

\"chart,
Figure 3: Relative training times on the model GPT-3 (175 billion parameters) from MLPerf Training v3.1 between the NVIDIA submission on the bare-metal platform (3.1-2007) and Azure on virtual machines (3.1-2002).<\/figcaption><\/figure>\n\n\n\n

The latest results in AI Inferencing on Azure ND H100 v5 VMs show leadership results as well, as shown in MLPerf Inference v3.1<\/a>. The ND H100 v5-series delivered 0.99x-1.05x relative performance compared to the bare-metal submissions on the same NVIDIA H100 Tensor Core GPUs (figure 4), echoing the efficiency of virtual machines.<\/p>\n\n\n\n

\"Performance
Figure 4: Performance of the ND H100 v5-series (3.1-0003) compared to on-premises and bare metal offerings of the same NVIDIA H100 Tensor Core GPUs (3.1-0107 and 3.1-0121). All the results were obtained with the GPT-J benchmark from MLPerf Inference v3.1, scenarios: Offline and Server, accuracy: 99<\/em> percent.<\/figcaption><\/figure>\n\n\n\n

In conclusion, created for performance, scalability, and adaptability, the Azure ND H100 v5-series offers exceptional throughput and minimal latency for both training and inferencing tasks in the cloud and offers the highest quality infrastructure for AI.<\/p>\n\n\n\n

Learn more about Azure AI Infrastructure<\/h2>\n\n\n\n