Support for Enterprise Security Package–enabled HDInsight clusters in Azure Data Factory
Published date: February 25, 2019
You can now use Azure Data Factory to operationalize your Azure HDInsight Spark and Hadoop workloads against HDInsight clusters with Enterprise Security Package (ESP) that are joined to an Active Directory domain. We're introducing a new property, isEspEnabled,in the HDInsight linked service in Azure Data Factory, which you can specify if the cluster is ESP enabled.
Because ESP clusters are inside a virtual network, you currently have to set up a self-hosted integration runtime inside the same virtual network to submit the jobs from Data Factory.
Here are the high-level steps to operationalize jobs against an HDInsight ESP cluster in Data Factory:
1. Create a self-hosted integration runtime in the virtual network.
2. Create an HDInsight linked service.
3. Create an Azure Data Factory pipeline to run HDInsight activities.
Note: You can't create ESP-enabled HDInsight clusters on demand from Azure Data Factory because of security requirements for ESP setup.
For more information, see: