Support for Enterprise Security Package–enabled HDInsight clusters in Azure Data Factory

Posted on 26 February 2019

You can now use Azure Data Factory to operationalise your Azure HDInsight Spark and Hadoop workloads against HDInsight clusters with Enterprise Security Package (ESP) that are joined to an Active Directory domain. We are introducing a new property, isEspEnabled, in the HDInsight linked service in Azure Data Factory, which you can specify if the cluster is ESP enabled.

Because ESP clusters are inside a virtual network, you currently have to set up a self-hosted integration runtime inside the same virtual network to submit the jobs from Data Factory. 

Here are the high-level steps to operationalise jobs against an HDInsight ESP cluster in Data Factory:

1. Create a self-hosted integration runtime in the virtual network.

2. Create an HDInsight linked service.

clip_image002

3. Create an Azure Data Factory pipeline to run HDInsight activities.

Note: You cannot create ESP-enabled HDInsight clusters on demand from Azure Data Factory because of security requirements for ESP setup.

For more information, see:

Related feedback