{"id":1919,"date":"2018-12-04T00:00:00","date_gmt":"2018-12-04T00:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/onnx-runtime-is-now-open-source"},"modified":"2023-05-11T15:35:39","modified_gmt":"2023-05-11T22:35:39","slug":"onnx-runtime-is-now-open-source","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/onnx-runtime-is-now-open-source\/","title":{"rendered":"ONNX Runtime is now open source"},"content":{"rendered":"

\"ONNXToday we are announcing we have open sourced Open Neural Network Exchange\t(ONNX) Runtime\ton\tGitHub<\/a>. ONNX Runtime is a high-performance inference engine for machine learning models in the ONNX format on Linux, Windows, and Mac.<\/p>\n

ONNX<\/a>\tis an open format for deep learning and traditional machine learning models that Microsoft co-developed with Facebook and AWS. The ONNX format is the basis of an open ecosystem that makes AI more accessible and valuable to all: developers can choose the right framework for their task, framework authors can focus on innovative enhancements, and hardware vendors can streamline optimizations for neural network computations.\t<\/p>\n

Microsoft has been conducting research in AI for more than two decades and incorporating machine learning and deep neural networks in a plethora of products and services. With teams using many different training frameworks and targeting different deployment options, there was a real need to unify these scattered solutions to make it quick and simple to operationalize models. ONNX Runtime provides that solution. It gives data scientists the flexibility to train and tune models in the framework of their choice and productionize these models with high performance in products spanning both cloud and edge.<\/p>\n

Why use ONNX Runtime<\/h2>\n

ONNX Runtime is the first publicly available inference engine with full support for ONNX 1.2 and higher including the ONNX-ML profile. This means it is advancing directly alongside the ONNX standard to support an evolving set of AI models and technological breakthroughs.<\/p>\n

At Microsoft, teams are using ONNX Runtime to improve the scoring latency and efficiency for many of our models used in core scenarios in Bing Search, Bing Ads, Office productivity services, and more. For models we’ve converted to ONNX, we’ve seen average performance improve by 2X compared to scoring in their existing solutions. ONNX Runtime is also incorporated in other Microsoft offerings including Windows ML and ML.net.<\/p>\n

ONNX Runtime is lightweight and modular in design, with the CPU build only a few megabytes in size. The extensible architecture<\/a>\tenables optimizers and hardware accelerators to provide low latency and high efficiency for computations by registering as \u201cexecution providers.\u201d The result is smoother end-to-end user experiences with lower perceived latency, as well as cost savings from decreased machine utilization and higher throughput.\t<\/p>\n

Deep support from industry partners<\/h2>\n

Leading companies in the ONNX community are actively working or planning\tto integrate their technology with ONNX Runtime. This enables them to support the full ONNX specification while achieving the best performance.<\/p>\n

Microsoft and Intel are working together to integrate the\tnGraph\tCompiler as an execution provider for the ONNX Runtime. The\tnGraph\tCompiler is capable of accelerating both existing and upcoming hardware targets by applying both non-device specific and device specific optimizations.\tUsing the\tnGraph\tCompiler for CPU inference achieves\tup to 45x performance boost<\/a>\tas compared to native frameworks.\t\t<\/p>\n

NVIDIA is helping integrate\tTensorRT\twith ONNX Runtime to offer an easy workflow for deploying a rapidly growing set of models and apps on NVIDIA GPUs while achieving the best performance possible.\tNVIDIA\tTensorRT\tincludes a high-performance inference optimizer and runtime that delivers dramatically higher throughput at minimal latency across applications such as recommenders, natural language processing,\tand image\/video processing.\t<\/p>\n

Qualcomm, another\tearly advocate<\/a>\tof ONNX, has also expressed support for ONNX Runtime.\t\u201cThe introduction of ONNX Runtime is a positive next step in further driving framework interoperability, standardization, and performance optimization across multiple device categories and we expect developers to welcome support for ONNX Runtime on Snapdragon mobile platforms,”\tsays Gary Brotman, senior director of AI product management at Qualcomm Technologies, Inc.\t\t<\/p>\n

After joining ONNX recently,\tleading IoT chip maker NXP also announced support for ONNX Runtime.\t\u201cWhen it comes to choosing from among the many machine learning frameworks, we want our customers to have maximum flexibility and freedom,\u201d says Markus Levy, head of the AI Technology Center at NXP.\t“We\u2019re happy to bring the ONNX benefits to our customer community of ML developers by supporting the ONNX Runtime released by Microsoft in our platform.\u201d<\/p>\n

In addition to hardware partners, framework provider Preferred Networks is also leveraging ONNX Runtime. “Preferred Networks, in addition to developing the deep learning framework\tChainer<\/a>, has created\tMenoh<\/a>, an ONNX inference engine wrapper library for multiple programming languages,” says Toru Nishikawa, President and CEO of Preferred Networks, Inc. “Menoh\twill use ONNX Runtime as its main backend, and\tChainer\tcurrently uses ONNX Runtime to test its ONNX export features. Preferred Networks is delighted that Microsoft has made ONNX Runtime and looks forward to working on ONNX with Microsoft in the future.”\t\t<\/p>\n

How to use ONNX Runtime<\/h2>\n

First, you’ll need an ONNX model. Don’t have an ONNX model? No problem. The beauty of ONNX is the framework interoperability enabled through a\tmultitude of tools<\/a>.<\/p>\n