With the newly released support for Kafka streams in Event Hubs, it is now possible for Azure IoT Hub customers to easily feed their IoT device telemetry data into Kafka-based applications for further downstream processing or analysis. This gives customers with existing Kafka-based applications the added flexibility of faster adoption of Azure IoT Hub, without the need to rewrite any parts of their applications upfront. This means that customers can start using IoT Hub's native support for messaging, device and configuration management early, and defer the decision to migrate their telemetry processing applications to natively use Event Hubs at a later time.
Applicable customer scenarios
The ability to feed IoT device telemetry data into Kafka-based processing applications is valuable in several scenarios:
- A primary scenario involves a prospective Azure IoT Hub customer with legacy data processing applications that are already written to interface with Kafka clusters. With the support for Kafka in Event Hubs, the customer can defer the need to make upfront changes to such applications as part of onboarding to Azure IoT Hub. The new feature enables a faster IoT Hub adoption cycle for the customer with lower upfront development costs.
- A secondary scenario involves a customer who would like to keep an existing Kafka-based telemetry processing application as is, perhaps to use it for processing telemetry data emitted as Kafka streams by other disparate sources (e.g., non-Azure-IoT-Hub-managed devices). Similarly, the new feature benefits the customer by reducing churn in existing Kafka-based applications.
- A final scenario involves a customer who is simply evaluating Azure IoT Hub's capabilities for future adoption and would typically like to undertake a minimal set of changes needed to validate IoT Hub's capabilities without making significant changes in any external periphery data processing system.
Using Kafka-based applications with IoT Hub telemetry
As shown in the diagram below, IoT Hub acts as a conduit for the device telemetry data. This data is persisted in Event Hubs, and ultimately consumed by downstream applications.
To enable your Kafka-based application to retrieve this data, you need to add a Event Hub as an endpoint/route to IoT Hub and configure your Kafka-based application with the connection string of your Event Hub where the data is stored. Use the links below on how to complete these steps:
- Create a new Kafka-enabled Event Hubs guide (ensure the "Enable Kafka" option is selected in the creation process).
- Add the Event Hub as a new endpoint in your IoT Hub, and add a route to route device messages;
- Configure your Kafka consumer with the connection string of your Event Hubs (you can use QuickStart code as a sample Kafka consumer)
For a detailed demonstration of the steps involved, see the recorded video below: