• 5 min read

Spark + AI Summit – Developing for the intelligent cloud and intelligent edge

We are truly at a unique tipping point in the history of technology. The pace of growth is more rapid than ever before, with estimates of more than 150B connected devices and data growth up to 175 Zettabytes by 2025.

We are truly at a unique tipping point in the history of technology. The pace of growth is more rapid than ever before, with estimates of more than 150B connected devices and data growth up to 175 Zettabytes by 20251. With the dramatic acceleration of digitization, the primary question we now face is how to take advantage of the data and capabilities at our fingertips to help our companies and communities transform.

We see a massive opportunity powered by the intelligent cloud and intelligent edge. The intelligent cloud is ubiquitous computing at massive scale, enabled by the public cloud and powered by AI, for every type of application one can envision. The intelligent edge is a continually expanding set of connected systems and devices that gather and analyze data—close to your users, the data, or both.

Enabling intelligent cloud and intelligent edge solutions requires a new class of distributed, connected applications. Fundamentally, a cloud/edge application must be developed and run as a single environment. Azure provides a unified, comprehensive platform from the cloud to the edge, with a consistent app platform, holistic security, single identity controls, and simplified cloud and edge management.

We are also bringing the latest innovations in machine learning and artificial intelligence to the intelligent cloud and intelligent edge. Azure enables you to combine data in the cloud and at the edge to develop ML models and distribute them across a massive set of certified devices. Tapping into contextual insights from the edge provides ML models more robust data and thus better results.

Microsoft Azure services

Investing in open source to fuel innovation

Today, at the Spark + AI Summit in San Francisco, I had the opportunity to share several exciting announcements on how we are building upon these cloud/edge capabilities and are deeply investing in the open source community.

Azure Machine Learning support for MLflow

First, we are excited to join the open source MLflow project as an active contributor. Azure Machine Learning—a popular machine learning service enabling Azure customers to build, train, and deploy machine learning models—will support open source MLflow to provide customers with maximum flexibility. This means that developers can use the standard MLflow tracking API to track runs and deploy models directly into Azure Machine Learning service.

Managed MLflow and Managed Delta Lake in Azure Databricks

Furthermore, we are excited to announce that managed MLflow is generally available on Azure Databricks and will use Azure Machine Learning to track the full ML lifecycle. This approach enables organizations to develop and maintain their machine learning lifecycle using a single model registry on Azure. The combination of Azure Databricks and Azure Machine Learning makes Azure the best cloud for machine learning. In addition to being able to deploy models from the cloud to the edge, customers benefit from an optimized, autoscaling Apache Spark based environment, collaborative workspace, automated machine learning, and end-to-end Machine Learning Lifecycle management.

Additionally, today, Databricks open sourced Databricks Delta, now known as Delta Lake. Delta Lake is an engine built on top of Apache Spark for optimizing data pipelines. With Delta Lake, Azure Databricks customers get greater reliability, improved performance, and the ability to simplify their data pipelines.

Azure Databricks customers have been experiencing the benefits of the Delta engine in general availability since February, and they will continue to enjoy the innovations from the community going forward with the open source Delta Lake project.

.NET for Apache Spark™

We are excited to be making Apache Spark accessible to the .NET developer ecosystem with .NET for Apache Spark. We’re seeing incredible growth with .NET, which is actively used by millions of developers, with over 1 million new developers coming to the platform in the last year. .NET for Apache Spark is free, open source, and .NET Standard compliant, which means you can use it anywhere you write .NET code.

.NET for Apache Spark provides high performance DataFrame-level APIs for using Apache Spark from C# and F#. With these .NET APIs, you can access all aspects of Apache Spark including Spark SQL, for working with structured data, and Spark Streaming. Additionally, .NET for Apache Spark allows you to register and call user-defined functions written in .NET at scale. With .NET for Apache Spark, you can reuse all the knowledge, skills, code, and libraries you already have as a .NET developer.

Microsoft is committed to engaging closely with the Apache Spark community to grow this project and enable more developers to use Apache Spark.

Bringing the intelligent cloud and the intelligent edge to life

Many of our customers are already bringing the power of the intelligent cloud and the intelligent edge to life in innovative new ways.

Anheuser-Busch InBev is brewing up game-changing business solutions with Azure IoT and AI services. With its RFID program, AB InBev tracks pallets of beer from the brewery to the wholesaler to the retailer in order to optimize inventory, reduce out-of-stock scenarios, and better forecast future retailer consumption trends.

“With the Azure platform and services, we’re transforming how we do business, interact with our suppliers, and connect with our customers.”

— Chetan Kundavaram, Global Director, Anheuser-Busch InBev

Schneider Electric is breaking new ground with new methods for proactively identifying pump problems in real-time through predictive edge analytics. By deploying predictive models to edge devices with Azure Machine Learning and Azure IoT Edge, they can shut down pumps before damages occur, protecting machinery and preventing potential environmental damage.

“In some critical systems—whether at an oil pump or in a manufacturing plant—you may have to make a decision in a matter of milliseconds. By building machine learning algorithms into our applications and deploying analytics at the edge, we reduce any communication latency to the cloud or a central system, and that critical decision can happen right away.”

— Matt Boujonnier, Analytics Application Architect, Schneider Electric

AccuWeather, a leading global provider of weather forecasts, leverages Azure AI services to create custom weather-impact predictions to help people plan their lives, protect their businesses, and stay safe. To develop highly accurate forecasts, AccuWeather depends on Azure’s machine learning tools, customizable with R and Python code.

“Azure really stands out from other clouds by providing out-of-the-box machine learning capabilities that are powerful yet customizable using R and Python. It’s very important to our data scientists to have a cloud platform that plays well with open source, and that was one of the things that attracted us to Azure.”

— Rosemary Yeilding Radich, Director of Data Science, AccuWeather

It has been incredible to see our customers leverage AI to drive digital transformation across industries. But even beyond transforming businesses, what has truly inspired us is determining how the power of AI can be used toward creating a more sustainable and accessible world. We developed a program called AI for Accessibility focused on leveraging the power of AI to amplify human capability for the more than one billion people globally who are differently abled. Today at the Spark + AI Summit, we were excited to showcase Seeing AI, a Microsoft research project designed for the low vision community that harnesses the power of AI to describe people, text, and objects. We truly believe that technology can empower everyone to achieve more, and look forward to the new wave of possibilities that AI brings to life.

We are committed to delivering the latest innovations across cloud and edge into the hands of every developer and data scientist to enable new possibilities with data. We can’t wait to see what you build next.


1IDC White Paper, Data Age 2025. The Digitization of the World: From Edge to Core.