Apache Kafka for HDInsight

Managed high-throughput, low-latency service for real-time data

Kafka for HDInsight is an enterprise-grade, open-source, streaming ingestion service that’s cost-effective and easy to set up, manage, and use. Build real-time solutions such as Internet of Things (IoT), fraud detection, clickstream analysis, financial alerts, and social analytics.

Stream millions of events per second

Process massive amounts of data produced by your real-time applications with Kafka for HDInsight. Apache Kafka is a popular open-source stream-ingestion broker. It can handle large numbers of reads and writes per second from thousands of clients. Design powerful streaming pipelines to drive intelligent real-time actions using out-of-the-box integration with Apache Storm for HDInsight or Apache Spark for HDInsight.

Apache Hadoop® and associated open source project names are trademarks of The Apache Software Foundation.

Data comes in from various event sources (applications, devices, sensors, web, social) and is collected in the cloud through web APIs or field gateways. The data stream is ingested by Kafka for HDInsight for processing and analytics with services like Azure Machine Learning, Spark for HDInsight, Storm for HDInsight, and storage adapters. The data moves to long-term storage with services like Apache HBase on HDInsight, DocumentDB, MonoDB SQL, Solr Azure, Data Lake store, and Azure Search. Then you can run your real-time dashboards, queries, and analytics, or send data to devices to take action.

Enterprise-grade management and control

Get visibility and control over your real-time solution with threat detection, monitoring, and management through Microsoft Operations Management Suite. Capture log, event, and Java Management Extensions (JMX) metrics to define preemptive actions, and set alerts with Azure Automation runbooks. Get the power to perform statistical aggregations and build rich visualizations for reporting and monitoring.

Peace of mind and support for open source

Kafka for HDInsight is managed and supported by Microsoft with 24x7 enterprise support and cluster monitoring. At general availability, HDInsight will provide 99.9% up time for your Kafka clusters.

Easy to set up, fast results

There’s no time-consuming installation or setup with Kafka for HDInsight. Azure does it for you. Deploy a managed Kafka cluster of your configuration using the full-featured portal or through simple JSON templates. Your cluster will be up and running in minutes, ingesting low-latency, high-throughput data. You only pay for the compute and storage that you use, with no need to buy new hardware or pay other up-front costs.

Try Kafka for HDInsight