Apache Kafka for HDInsight

Managed high-throughput, low-latency service for real-time data

Kafka for HDInsight is an enterprise-grade, open-source, streaming ingestion service that is cost-effective and easy to provision, manage and use. This service enables you to build real-time solutions such as IoT, fraud detection, click-stream analysis, financial alerts and social analytics.

Stream millions of events per second

Process massive amounts of data produced by your real-time applications with Kafka for HDInsight. Apache Kafka is the most popular open-source stream-ingestion broker and can handle large numbers of reads and writes per second from thousands of clients. Architect powerful streaming pipelines to drive intelligent real-time actions using out-of-the-box integration with Apache Storm for HDInsight or Apache Spark for HDInsight.


Apache Hadoop® and associated open-source project names are trademarks of the Apache Software Foundation.

Data comes in from various event sources (applications, devices, sensors, web, social) and is collected in the cloud through web APIs or field gateways. The data stream is ingested by Kafka for HDInsight for processing and analytics with services such as Azure Machine Learning, Spark for HDInsight, Storm for HDInsight and storage adapters. The data moves to long-term storage with services such as Apache HBase on HDInsight, DocumentDB, MonoDB SQL, Solr Azure, Data Lake store and Azure Search. You can then run your dashboards, queries and analytics in real time, or send data to devices to take action.

Enterprise-grade management and control

Gain visibility and control over your real-time solution with threat detection, monitoring and management through Microsoft Operations Management Suite. Capture log, event and JMX metrics to define pre-emptive actions, and set alerts with Azure Automation runbooks. Plus, get the power to perform statistical aggregations and build rich visualisations for reporting and monitoring.

Peace of mind and open-source support

Kafka for HDInsight is managed and supported by Microsoft with 24/7 enterprise support and cluster monitoring. At general availability, HDInsight will guarantee a 99.9% uptime for your Kafka clusters.

Easy to set up, with quick results

With Kafka for HDInsight, there's no time-consuming installation or setup – Azure does it for you. Deploy a managed Kafka cluster of your configuration using the full-featured portal or through simple JSON templates. Your cluster will be up and running in minutes, ingesting low-latency, high-throughput data. You only pay for the compute and storage you use, with no need to buy new hardware or incur other upfront costs.

Try Kafka for HDInsight