{"id":4285,"date":"2016-11-29T00:00:00","date_gmt":"2016-11-29T00:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/from-a-pc-on-every-desktop-to-deep-learning-in-every-software"},"modified":"2025-06-12T09:27:55","modified_gmt":"2025-06-12T16:27:55","slug":"from-a-pc-on-every-desktop-to-deep-learning-in-every-software","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/from-a-pc-on-every-desktop-to-deep-learning-in-every-software\/","title":{"rendered":"From \u201cA PC on every desktop\u201d to \u201cDeep Learning in every software\u201d"},"content":{"rendered":"\n

Deep learning is behind many recent breakthroughs in Artificial Intelligence, including speech recognition, language understanding and computer vision. At Microsoft, it is changing customer experience in many of our applications and services, including Cortana, Bing, Office 365, SwiftKey, Skype Translate, Dynamics 365, and HoloLens. Deep learning-based language translation<\/a> in Skype was recently named one of the 7 greatest software innovations of the year<\/a> by Popular Science, and the technology helped us achieve human-level parity in conversational speech recognition. Deep learning is now a core feature of development platforms such as the Microsoft Cognitive Toolkit, Cortana Intelligence Suite, Microsoft Cognitive Services, Azure Machine Learning, Bot Framework, and the Azure Bot Service. I believe that the applications of this technology are so far reaching  that \u201cDeep Learning in every software\u201d will be a reality within this decade.<\/p>\n\n\n\n

We\u2019re working very hard to empower developers with AI and Deep Learning, so that they can make smarter products and solve some of the most challenging computing tasks. By vigorously improving algorithms, infrastructure and collaborating closely with our partners like NVIDIA, OpenAI and others and harnessing the power of GPU-accelerated systems, we\u2019re making Microsoft Azure the fastest, most versatile AI platform \u2013 a truly intelligent cloud.<\/p>\n\n\n\n

Production-Ready Deep Learning Toolkit for Anyone<\/h2>\n\n\n\n

The Microsoft Cognitive Toolkit<\/strong><\/a> (formerly CNTK<\/strong>) is our open-source, cross-platform toolkit for learning and evaluating deep neural networks. The Cognitive Toolkit expresses arbitrary neural networks by composing simple building blocks into complex computational networks, supporting all relevant network types and applications. With the state-of-the art accuracy and efficiency, it scales to multi-GPU\/multi-server environments. According to both internal and external benchmarks, the Cognitive Toolkit continues to outperform other Deep Learning frameworks in most tests and unsurprisingly, the latest version is faster than the previous releases, especially when working on massively big data sets and on Pascal GPUs from NVIDIA. That\u2019s true for single-GPU performance, but what really matters is that Cognitive Toolkit can already scale up to using a massive number of GPUs. In the latest release, we\u2019ve extended Cognitive Toolkit to natively support Python in addition to C++. Furthermore, the Cognitive Toolkit also now allows developers to use reinforcement learning to train their models. Finally, Cognitive Toolkit isn\u2019t bound to the cloud in any way. You can train models on the cloud but run them on premises or with other hosters. Our goal is to empower anyone to take advantage of this powerful technology.<\/p>\n\n\n\n

To quickly get up to speed on the Toolkit, we\u2019ve published Azure Notebooks<\/strong> with numerous  tutorials and we\u2019ve also assembled a DNN Model Gallery<\/strong><\/a> with dozens of code samples, recipes and tutorials across scenarios working with a variety of datasets: images, numeric, speech and text.<\/p>\n\n\n\n

What Others Are Saying<\/h2>\n\n\n\n

In the \u201cBenchmarking State-of-the-Art Deep Learning Software Tools\u201d<\/strong><\/a> paper published in September 2016, academic researchers have run a comparative study of the state-of-the-art GPU-accelerated deep learning software tools, including Caffe, Cognitive Toolkit (CNTK), TensorFlow, and Torch. They\u2019ve benchmarked the running performance of these tools with three popular types of neural networks on two CPU platforms and three GPU platforms. Our Cognitive Toolkit outperformed other deep learning toolkits on nearly every workload.<\/p>\n\n\n\n

Furthermore, Nvidia recently has also run a benchmark comparing all the popular Deep Learning toolkits with their latest hardware. The results show that the Cognitive Toolkit trains and evaluates deep learning algorithms faster than other available toolkits, scaling efficiently in a range of environments\u2014from a CPU, to GPUs, to multiple machines\u2014while maintaining accuracy. Specifically, it\u2019s 1.7 times faster<\/strong> than our previous release and 3x faster than TensorFlow<\/strong> on Pascal GPUs (as presented at SuperComputing\u201916 conference).<\/p>\n\n\n\n

End users of deep learning software tools can use these benchmarking results as a guide to selecting appropriate hardware platforms and software tools. Second, for developers of deep learning software tools, the in-depth analysis points out possible future directions to further optimize performance.<\/p>\n\n\n\n

Real-world Deep Learning Workloads<\/h2>\n\n\n\n

We at Microsoft use Deep Learning and the Cognitive Toolkit in many of our internal services, ranging from digital agents to core infrastructure in Azure.<\/p>\n\n\n\n

1.<\/strong> Agents (Cortana):<\/strong> Cortana is a digital agent that knows who you are and knows your work and life preferences across all your devices. Cortana has more 133 million users and has intelligently answered more than 12 billion questions. From speech recognition to computer vision in Cortana \u2013 all these capabilities are powered by Deep Learning and Cognitive Toolkit. We have recently made a major breakthrough in speech recognition<\/a>, creating a technology that recognizes the words in a conversation and makes the same or fewer errors than professional transcriptionists. The researchers reported a word error rate (WER) of 5.9 percent, down from the 6.3 percent, the lowest error rate ever recorded against the industry standard Switchboard speech recognition task. Reaching human parity using Deep Learning is a truly historic achievement.<\/p>\n\n\n\n

\"DeepLearning\"<\/figure>\n\n\n\n

Our approach to image recognition<\/a> also placed first in several major categories of the ImageNet and the Microsoft Common Objects in Context challenges. The DNNs built with our tools won first place in all three categories we competed in: classification, localization and detection. The system won by a strong margin, because we were able to accurately train extremely deep neural nets, 152 layers \u2013 much more than in the past \u2013 and it used a new \u201cresidual learning\u201d principle. Residual learning reformulates the learning procedure and redirects the information flow in deep neural networks. That helped solve the accuracy problem that has traditionally dogged attempts to build extremely deep neural networks.<\/p>\n\n\n\n

2.<\/strong> Applications:<\/strong> Our applications, from Office 365, Outlook, PowerPoint, Word and Dynamics 365 can use deep learning to provide new customer experiences. One excellent example of a deep learning application the bot used by Microsoft Customer Support and Services<\/strong>. Using Deep Neural Nets and the Cognitive Toolkit, it can intelligently understand the problems that a customer is asking about, and recommend the best solution to resolve those problems. The bot provides a quick self-service experience for many common customer problems and helps our technical staff focus on the harder and more challenging customer issues.<\/p>\n\n\n\n

Another example of an application using Deep Learning is the Connected Drone<\/strong><\/a> application built for powerline inspection by one of our customers eSmart (to see the Connected Drone in action, please watch this video<\/a>). eSmart Systems<\/strong> began developing the Connected Drone out of a strong conviction that drones combined with cloud intelligence<\/a> could bring great efficiencies to the power industry. The objective of the Connected Drone is to support and automate the inspection and monitoring of power grid infrastructure instead of the currently expensive, risky, and extremely time consuming inspections performed by ground crews and helicopters. To do this, they use Deep Learning<\/a> to analyze video data feeds streamed from the drones. Their analytics software recognizes individual objects, such as insulators on power poles, and directly links the new information with the component registry, so that inspectors can quickly become aware of potential problems. eSmart applies a range of deep learning technologies to analyze data from the Connected Drone, from the very deep Faster R-CNN to Single Shot Multibox Detectors and more.<\/p>\n\n\n\n

3. Cloud Services (Cortana Intelligence Suite):<\/strong> On Azure, we offer a suite for Machine Learning and Advanced Analytics, including Cognitive Services<\/strong><\/a> (Vision, Speech, Language, Knowledge, Search, etc.), Bot Framework, Azure Machine Learning, Azure Data Lake, Azure SQL Data Warehouse and PowerBI, called the Cortana Intelligence Suite. You can use these services along with the Cognitive Toolkit or any other deep learning framework of your choice to deploy intelligent applications. For instance, you can now massively parallelize scoring using a pre-trained DNN machine learning model on an HDInsight Apache Spark<\/strong> cluster in Azure. We are seeing a growing number of scenarios that involve the scoring of pre-trained DNNs on a large number of images, such as our customer Liebherr that runs DNNs to visually recognize objects inside a refrigerator<\/a>. Developers can implement such a processing architecture with just a few steps (see instructions here<\/a>).<\/p>\n\n\n\n

A typical large-scale image scoring scenario may require very high I\/O throughput and\/or large file storage capacity, for which the Azure Data Lake Store<\/strong><\/a> (ADLS) provides a high performance and scalable analytical storage. Furthermore, ADLS imposes data schema on read, which allows the user to not worry about the schema until the data is needed. From the user\u2019s perspective, ADLS functions like any other HDFS storage account through the supplied HDFS connector. Training can take place on an Azure N-series NC24 GPU-enabled Virtual Machine<\/strong> or using recipes from the Azure Batch Shipyard<\/strong>, which allows training of our DNNs with bare-metal GPU hardware acceleration in the public cloud using as many as four NVIDIA Tesla K80 GPUs. For scoring, one can use HDInsight Spark Cluster or Azure Data Lake Analytics<\/strong><\/a> to massively parallelize the scoring of a large collection of images with the rxExec function in Microsoft R Server<\/strong> (MRS) by distributing the workload across the worker nodes. The scoring workload is orchestrated from a single instance of MRS and each worker node can read and write data to ADLS independently, in parallel.<\/p>\n\n\n\n

SQL Server<\/strong><\/a>, our premier database engine, is \u201cbecoming deep\u201d as well. This is now possible for the first time with R and ML built into SQL Server<\/a>. Pushing deep learning models inside SQL Server, our customers now get throughput, parallelism, security, reliability, compliance certifications and manageability, all in one. It\u2019s a big win for data scientists and developers \u2013 you don\u2019t have to separately build the management layer for operational deployment of ML models. Furthermore, just like data in databases can be shared across multiple applications, you can now share the deep learning models.  Models and intelligence become \u201cyet another type of data\u201d, managed by the SQL Server 2016. With these capabilities, developers can now build a new breed of applications that marry the latest transaction processing advancements in databases with deep learning.<\/p>\n\n\n\n

4. Infrastructure (Azure):<\/strong> Deep Learning<\/a> requires a new breed of high performance infrastructure that is able to support the computationally intensive nature of deep learning training. Azure now enables these scenarios with its N-Series Virtual machines that are powered by NVIDIA’s Tesla K80 GPUs<\/a> that are best in class for single and double precision workloads in the public cloud today. These GPUs are exposed via a hardware pass-through mechanism called Discreet Device Assignment that allows us to provide near bare-metal performance. Additionally, as data grows for these workloads, data scientists have the need to distribute the training not just across multiple GPUs in a single server, but to a number of GPUs across nodes. To enable this distributed learning need across tens or hundreds of GPUs, Azure has invested in high-end networking infrastructure for the N-Series using a Mellanox’s InfiniBand fabric which allows for high bandwidth communication between VMs with less than 2 microseconds latency. This networking capability allows for libraries such as Microsoft’s own Cognitive Toolkit (CNTK) to use MPI for communication between nodes and efficiently train with a larger number of layers and great performance.<\/p>\n\n\n\n

We are also working with NVIDIA on a best in class roadmap for Azure with the current N-Series as the first iteration of that roadmap. These Virtual Machines are currently in preview and recently announced General Availability of this offering starting on 1st of December.<\/p>\n\n\n\n

It is easy to get started with deep learning on Azure. The Data Science Virtual Machine (DSVM)<\/strong><\/a> is available in the Azure Marketplace, and comes pre-loaded with a range of deep learning frameworks and tools for Linux and Windows. To easily run many training jobs in parallel or launch a distributed job across more than one server, Azure Batch \u201cShipyard\u201d<\/strong><\/a> templates are available for the top frameworks. Shipyard takes care of configuring the GPU and InfiniBand drivers, and uses Docker containers to setup your software environment.<\/p>\n\n\n\n

Lastly, our team of our engineers and researchers has created a system that uses a reprogrammable computer chip called a field programmable gate array, or FPGA<\/strong><\/a>, to accelerate Bing and Azure. Utilizing the FPGA chips, we can now write Deep Learning algorithms directly onto the hardware, instead of using potentially less efficient software as the middle man. What\u2019s more, an FPGA can be reprogrammed at a moment\u2019s notice to respond to new advances in AI\/Deep Learning or meet another type of unexpected need in a datacenter. Traditionally, engineers might wait two years or longer for hardware with different specifications to be designed and deployed. This is a moonshot project that\u2019s succeeded and we are bringing this now to our customers.<\/p>\n\n\n\n

Join Us in Shaping the Future of AI<\/h2>\n\n\n\n

Our focus on innovation in Deep Learning is across the entire stack of infrastructure, development tools, PaaS services and end user applications. Here are a few of the benefits our products bring:<\/p>\n\n\n\n