{"id":1142,"date":"2019-07-17T00:00:00","date_gmt":"2019-07-17T07:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/microsoft-makes-it-easier-to-build-popular-language-representation-model-bert-at-large-scale"},"modified":"2025-06-25T02:32:29","modified_gmt":"2025-06-25T09:32:29","slug":"microsoft-makes-it-easier-to-build-popular-language-representation-model-bert-at-large-scale","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/microsoft-makes-it-easier-to-build-popular-language-representation-model-bert-at-large-scale\/","title":{"rendered":"Microsoft makes it easier to build popular language representation model BERT at large scale"},"content":{"rendered":"\n

This post is co-authored by Rangan Majumder, Group Program Manager, Bing and Maxim Lukiyanov, Principal Program Manager, Azure Machine Learning. <\/em><\/p>\n\n\n\n

Today we are announcing the open sourcing of our recipe to pre-train BERT (Bidirectional Encoder Representations from Transformers) built by the Bing team, including code that works on Azure Machine Learning<\/a>, so that customers can unlock the power of training custom versions of BERT-large models using their own data. This will enable developers and data scientists to build their own general-purpose language representation beyond BERT.<\/p>\n\n\n\n

The area of natural language processing has seen an incredible amount of innovation over the past few years with one of the most recent being BERT. BERT<\/a>, a language representation created by Google AI language research, made significant advancements in the ability to capture the intricacies of language and improved the state of the art for many natural language applications, such as text classification, extraction, and question answering. The creation of this new language representation enables developers and data scientists to use BERT as a stepping-stone to solve specialized language tasks and get much better results than when building natural language processing systems from scratch.<\/p>\n\n\n\n

The broad applicability of BERT means that most developers and data scientists are able to use a pre-trained variant of BERT rather than building a new version from the ground up with new data. While this is a reasonable solution if the domain\u2019s data is similar to the original model\u2019s data, it will not deliver best-in-class accuracy when crossing over to a new problem space. For example, training a model for the analysis of medical notes requires a deep understanding of the medical domain, providing career recommendations depend on insights from a large corpus of text about jobs and candidates, and legal document processing requires training on legal domain data. In these cases, to maximize the accuracy of the Natural Language Processing (NLP) algorithms one needs to go beyond fine-tuning to pre-training the BERT model.<\/p>\n\n\n\n

Additionally, to advance language representation beyond BERT\u2019s accuracy, users will need to change the model architecture, training data, cost function, tasks, and optimization routines. All these changes need to be explored at large parameter and training data sizes. In the case of BERT-large, this can be quite substantial as it has 340 million parameters and trained over 2.5 billion Wikipedia and 800 million BookCorpus words. To support this with Graphical Processing Units (GPUs), the most common hardware used to train deep learning-based NLP models, machine learning engineers will need distributed training support to train these large models. However, due to the complexity and fragility of configuring these distributed environments, even expert tweaking can end up with inferior results from the trained models.<\/p>\n\n\n\n

To address these issues, Microsoft is open sourcing a first of a kind, end-to-end recipe for training custom versions of BERT-large models on Azure. Overall this is a stable, predictable recipe that converges to a good optimum for developers and data scientists to try explorations on their own.<\/p>\n\n\n\n

\u201cFine-tuning BERT was really helpful to improve the quality of various tasks important for Bing search relevance,\u201d says Rangan Majumder, Group Program Manager at Bing, who led the open sourcing of this work.  \u201cBut there were some tasks where the underlying data was different from the original corpus BERT was pre-trained on, and we wanted to experiment with modifying the tasks and model architecture.  In order to enable these explorations, our team of scientists and researchers worked hard to solve how to pre-train BERT on GPUs. We could then build improved representations leading to significantly better accuracy on our internal tasks over BERT.  We are excited to open source the work we did at Bing to empower the community to replicate our experiences and extend it in new directions that meet their needs.\u201d<\/em><\/p>\n\n\n\n

\u201cTo get the training to converge to the same quality as the original BERT release on GPUs was non-trivial,\u201d says Saurabh Tiwary, Applied Science Manager at Bing.  \u201cTo pre-train BERT we need massive computation and memory, which means we had to distribute the computation across multiple GPUs. However, doing that in a cost effective and efficient way with predictable behaviors in terms of convergence and quality of the final resulting model was quite challenging. We\u2019re releasing the work that we did to simplify the distributed training process so others can benefit from our efforts.\u201d<\/em><\/p>\n\n\n\n

Results<\/h2>\n\n\n\n

To test the code, we trained BERT-large model on a standard dataset and reproduced the results of the original paper on a set of GLUE tasks, as shown in Table 1. To give you estimate of the compute required, in our case we ran training on Azure ML cluster of 8xND40_v2 nodes (64 NVidia V100 GPUs total) for 6 days to reach listed accuracy in the table. The actual numbers you will see will vary based on your dataset and your choice of BERT model checkpoint to use for the upstream tasks.<\/p>\n\n\n\n

\"clip_image002\"<\/a><\/figure>\n\n\n\n

Table1. GLUE development set results. Google BERT results are evaluated by using published BERT models on development set. The \u201caverage\u201d column is simple average over the table results. F1 scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks. The results for tasks with smaller dataset sizes have significant variation and may require multiple fine-tuning runs to reproduce the results.<\/em><\/p>\n\n\n\n

The code is available in open source on the Azure Machine Learning BERT GitHub repo<\/a>. Included in the repo is:<\/p>\n\n\n\n