HDInsight Apache Kafka REST Proxy is now generally available
Published date: 09 March, 2021
HDInsight Kafka REST Proxy is now generally available. Kafka REST proxy greatly simplifies messaging architecture patterns by enabling producers and consumers to be located outside the virtual network. It also allows viewing of cluster metadata providing the state of the cluster and ability to perform administrative operations. HDInsight Kafka REST proxy combines the advantages of a Kafka REST proxy with the reliability, scalability, monitoring and cost optimisation benefits that HDInsight is known for.
Common use cases
- Clients in an unsupported language sending/receiving messages to/from Kafka
- Integrating with legacy applications which may not support Kafka natively
- Clients who may not have network line of sight with Kafka Brokers
- One click deployment
- Highly available and scalable
- Secured using AAD authorisation and OAuth protocol
- Support for Kafka Admin APIs enabling common Kafka Admin operations
- Support for Kafka Data APIs enabling the production and consumption of records.
Performance and Scalability
Guarantees low millisecond latency and high throughput almost nearing native Kafka Broker performance. The REST Proxy service is backed by scalable management nodes which leads to high availability and cost optimisation guarantees without compromising performance.
Integrates with HDInsight Private Link to provide network isolation without any public IP addresses. When integrated with HDInsight Private Link, Kafka REST Proxy endpoints resolve to a private IP that can only be resolved within your intranet. All traffic transiting the Kafka REST Proxy traverses the Azure backbone.
Access to the Kafka REST Proxy is secured using Azure Active directory service groups. Services would need to register to the Azure Active directory service and use token-based authentication to identify themselves to the service.
To get started with Kafka REST Proxy refer to one of our various application samples on the service.
HDInsight Kafka REST Proxy Client application sample