Organizations need data driven strategies to increase competitive advantage. Customers want to stream data or analyze in real-time to get valuable insights faster. To meet these big data needs, you need a massively scalable distributed event driven messaging platform with multiple producers and consumers Apache Kafka and Azure Event Hubs provide such distributed platforms.
How is Event Hubs different from Kafka?
Kafka and Event Hubs are both designed to handle large scale stream ingestion driven by real-time events. Conceptually, both are a distributed, partitioned, and replicated commit log service. Both use partitioned consumer model offering huge scalability for concurrent consumers. Both use a client side cursor concept and scale very high workloads.
Apache Kafka is a software that is installed and run. Azure Event Hubs is a fully managed service in the cloud. While Kafka is popular with its wide eco system and its on-premises and cloud presence, Event Hubs offers you the freedom of not having to manage servers or networks or worry about configuring brokers.
Talk to Event Hubs, like you would with Kafka and unleash the power of PaaS!
Today we are happy to marry both these powerful distributed streaming platforms to offer you Event Hubs for Kafka Ecosystem.
With this integration, you are provided with a Kafka endpoint. This endpoint enables you to configure your existing Kafka applications to talk to Azure Event Hubs, an alternative to running your own Kafka clusters. Azure Event Hubs for Kafka Ecosystem supports Apache Kafka 1.0 and later.
This integration not only allows you to talk to Azure Event Hubs without changing your Kafka applications, also allows you to work with some of the most demanding features of Event Hubs like Capture, Auto-Inflate, and Geo Disaster-Recovery.
For those of you new to Event Hubs, conceptually they map as below:
|
Kafka Concept |
Event Hubs Concept |
|
Cluster |
Namespace |
|
Topic |
Event Hub |
|
Partition |
Partition |
|
Consumer Group |
Consumer Group |
|
Offset |
Sequence Number |
What to expect for this preview?
For the public preview of this new integration, the supported Kafka features are limited and the following are not currently supported.
- Idempotent producer
- Transaction
- Compression
- Size based retention
- Log compaction
- Adding partitions to an existing topic
- HTTP Kafka API support
- Kafka Connect
- Kafka Streams
Next steps
- Want to learn more? Check out Event Hubs for Kafka Ecosystems.
- Learn more about Event Hubs.
Enjoyed this blog? Follow us as we update the features list we will start supporting. Leave us your valuable feedback, questions, or comments below.
Happy event-ing!