HDInsight Apache Kafka REST Proxy is now generally available
Published date: March 09, 2021
HDInsight Kafka REST Proxy is now generally available. Kafka REST proxy greatly simplifies messaging architecture patterns by enabling producers and consumers to be located outside the virtual network. It also allows viewing of cluster metadata providing the state of the cluster and ability to perform administrative operations. HDInsight Kafka REST proxy combines the advantages of a Kafka REST proxy with the reliability, scalability, monitoring and cost optimization benefits that HDInsight is known for.
Common use cases
- Clients in an unsupported language sending/receiving messages to/from Kafka
- Integrating with legacy applications which natively may not support Kafka
- Clients which may not have network line of sight with Kafka Brokers
Key Features
- One click deployment
- Highly available and scalable
- Secured using AAD authorization and OAuth protocol
- Support for Kafka Admin APIs enabling common Kafka Admin operations
- Support for Kafka Data APIs enabling the production and consumption of records.
Performance and Scalability
Guarantees low millisecond latency and high throughout almost nearing native Kafka Broker performance. The REST Proxy service is backed by scalable management nodes which leads to high availability and cost optimization guarantees without compromising performance.
Security
Integrates with HDInsight Private Link to provide network isolation without any public IP addresses. When integrated with HDInsight Private Link, Kafka REST Proxy endpoints resolve to a private IP that can resolved only within your intranet. All traffic transiting the Kafka REST Proxy traverses the Azure backbone.
Access to the Kafka REST Proxy is secured using Azure Active directory service groups. Services would need to register to the Azure Active directory service and use token based authentication to identify themselves to the service.
Getting started
To get started with Kafka REST Proxy refer to one of our various application samples on the service.
HDInsight Kafka REST Proxy Client application sample