Secure incoming traffic to HDInsight clusters in a virtual network with private endpoint

We are excited to announce the general availability of private endpoint in HDInsight clusters deployed in a virtual network. This feature enables enterprises to better isolate access to their HDInsight clusters from the public internet and enhance their security at the networking layer.

Previously, when customers deployed an HDI cluster in a virtual network, there was only one public endpoint available in the form of https://.azurehdinsight.net. This endpoint resolves to a public IP for accessing the cluster. Customers who wanted to restrict the incoming traffic had to use network security group (NSG) rules. Specifically, they had to white-list the IPs of both the HDInsight management traffic as well as the end users who wanted to access the cluster. These end users might have already been located inside the virtual network, but they had to be white-listed to be able to reach the public endpoint. It was hard to identify and white-list these end users’ dynamic IPs, as they would often change.

With the introduction of private endpoint, customers can now use NSG rules to separate access from the public internet and end users that are within the virtual network’s trusted boundary. The virtual network can be extended to the on-premise network, hence traffic coming from on-premise to an HDInsight cluster can also be isolated from the public internet.

Customers can now white-list only the required static IPs needed by the HDInsight management plane to reach the public endpoint and have their end users access the private endpoint inside the virtual network. Each HDI cluster deployed in a virtual network will have a private endpoint in the form of https://-int.azurehdinsight.net as well as a public endpoint. Note the “-int” in this URL, this endpoint will resolve to a private IP in that virtual network and is not accessible from the public Internet.

The isolation of public traffic from traffic inside the virtual network, potentially expanded to the on-premises environment through an express route, is an important security feature to control access at the networking layer and add an additional security layer to HDInsight clusters as a whole. To learn more about HDInsight and virtual networks, please see “Extend Azure HDInsight using an Azure Virtual Network” on the Microsoft Azure documentation page.

Try Azure HDInsight now

We are excited to see what you will build next with Azure HDInsight. Read the HDInsight developer guide and follow the quick start to learn more about implementing open source analytics pipelines on Azure HDInsight. Stay up-to-date on the latest Azure HDInsight news and features by following us on Twitter, #HDInsight and @AzureHDInsight. For questions and feedback, please reach out to AskHDInsight@microsoft.com.

About HDInsight

Azure HDInsight is an easy, cost-effective, enterprise-grade service for open source analytics that enables customers to easily run popular open source frameworks including Apache Hadoop, Spark, Kafka, and others. The service is available in 27 public regions and Azure Government Clouds in the United States and Germany. Azure HDInsight powers mission-critical applications in a wide variety of sectors and enables a wide range of use cases including ETL, streaming, and interactive querying.