Apache Kafka for HDInsight

Managed high-throughput, low-latency service for real-time data

Kafka for HDInsight is an enterprise-grade, open-source, streaming ingestion service that’s cost-effective and easy to set up, manage and use. Build real-time solutions such as Internet of Things (IoT), fraud detection, clickstream analysis, financial alerts and social analytics.

Stream millions of events per second

Process massive amounts of data produced by your real-time applications with Kafka for HDInsight. Apache Kafka is a popular open-source stream-ingestion broker. It can handle large numbers of reads and writes per second from thousands of clients. Design powerful streaming pipelines to drive intelligent real-time actions using out-of-the-box integration with Apache Storm for HDInsight or Apache Spark for HDInsight.


Apache Hadoop® and associated open-source project names are trademarks of The Apache Software Foundation.

Data comes in from various event sources (applications, devices, sensors, web, social) and is collected in the cloud through web APIs or field gateways. The data stream is ingested by Kafka for HDInsight for processing and analytics with services such as Azure Machine Learning, Spark for HDInsight, Storm for HDInsight and storage adapters. The data moves to long-term storage with services such as Apache HBase on HDInsight, DocumentDB, MonoDB SQL, Solr Azure, Data Lake store and Azure Search. You can then run your dashboards, queries and analytics in real time, or send data to devices to take action.

Enterprise-grade management and control

Get visibility and control over your real-time solution with threat detection, monitoring and management via Microsoft Operations Management Suite. Capture log, event and Java Management Extensions (JMX) metrics to define preemptive actions, and set alerts with Azure Automation runbooks. Get the power to perform statistical aggregations and build rich visualisations for reporting and monitoring.

Peace of mind and open-source support

Kafka for HDInsight is managed and supported by Microsoft with 24/7 enterprise support and cluster monitoring. At general availability, HDInsight will provide 99.9% up time for your Kafka clusters.

Easy to set up, with quick results

There’s no time-consuming installation or setup with Kafka for HDInsight. Azure does it for you. Deploy a managed Kafka cluster of your configuration using the full-featured portal or through simple JSON templates. Your cluster will be up and running in minutes, ingesting low-latency, high-throughput data. You only pay for the compute and storage that you use, with no need to buy new hardware or pay other up-front costs.

Try Kafka for HDInsight