Support for Enterprise Security Package–enabled HDInsight clusters in Azure Data Factory
Published date: 25 February, 2019
You can now use Azure Data Factory to operationalise your Azure HDInsight Spark and Hadoop workloads against HDInsight clusters with Enterprise Security Package (ESP) that are joined to an Active Directory domain. We are introducing a new property, isEspEnabled, in the HDInsight linked service in Azure Data Factory, which you can specify if the cluster is ESP enabled.
Because ESP clusters are inside a virtual network, you currently have to set up a self-hosted integration runtime inside the same virtual network to submit the jobs from Data Factory.
Here are the high-level steps to operationalise jobs against an HDInsight ESP cluster in Data Factory:
Note: You cannot create ESP-enabled HDInsight clusters on demand from Azure Data Factory because of security requirements for ESP setup.
For more information, see: