{"id":1883,"date":"2018-12-10T00:00:00","date_gmt":"2018-12-10T00:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/azure-hdinsight-integration-with-data-lake-storage-gen-2-preview-acl-and-security-update"},"modified":"2023-05-11T15:35:29","modified_gmt":"2023-05-11T22:35:29","slug":"azure-hdinsight-integration-with-data-lake-storage-gen-2-preview-acl-and-security-update","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/azure-hdinsight-integration-with-data-lake-storage-gen-2-preview-acl-and-security-update\/","title":{"rendered":"Azure HDInsight integration with Data Lake Storage Gen2 preview – ACL and security update"},"content":{"rendered":"

Today we are sharing an update to the Azure HDInsight integration with Azure Data Lake Storage Gen 2. This integration will enable HDInsight customers to drive analytics from the data stored in Azure Data Lake Storage Gen 2 using popular open source frameworks such as Apache Spark, Hive, MapReduce, Kafka, Storm, and HBase in a secure manner.<\/p>\n

Azure Data Lake Storage Gen2<\/h2>\n

Azure Data Lake Storage Gen2<\/a> is the only data lake designed specifically for enterprises to run large scale analytics workloads in the cloud. It unifies the core capabilities from the first generation of Azure Data Lake with a Hadoop compatible file system endpoint now directly integrated into Azure Blob Storage. This enhancement combines the scale and cost benefits of object storage with the reliability and performance typically associated only with on-premises file systems. This new file system includes a full hierarchical namespace that makes files and folders first class citizens, translating to faster, more reliable analytics job execution.<\/p>\n

Azure Data Lake Storage Gen2 also includes limitless storage ensuring capacity to meet the needs of even the largest, most complex workloads. In addition, Azure Data Lake Storage Gen2 delivers on native integration with Azure Active Directory and support POSIX compliant ACLs to enable granular permission assignments on files and folders.<\/p>\n

Key benefits<\/h2>\n

Hadoop compatible access<\/h3>\n

Azure Data Lake Storage Gen2 allows you to manage and access data just as you would with a Hadoop Distributed File System (HDFS). The ABFS driver is available within all Apache Hadoop environments. File systems are well understood by developers and users alike. There is no need to learn a new storage paradigm when you move to the cloud as the file system interface exposed by Azure Data Lake Storage Gen2 is the same paradigm used by computers, large and small.<\/p>\n

Role based access control<\/h3>\n

The security model for Azure Data Lake Storage Gen2 supports ACL and POSIX permissions.<\/p>\n

These storage ACL capabilities along with fine grain access control via apache Ranger in HDInsight for applications such as Spark, Kafka, Hive, and HBase make it very convenient to open up your data lake for entire organization with appropriate security control and auditing in place.<\/p>\n

SSL only access<\/h3>\n

With this update, ADLS Gen 2 accounts can only be accessed via https protocol ensuring that only encrypted communication is possible between HDInsight and storage.<\/p>\n

Global availability<\/h3>\n

Azure Data Lake Storage Gen 2 and HDInsight are available across the globe, offering the scale needed to bring big data applications closer to users around the world, preserving data residency, and offering comprehensive compliance and resiliency options for customers.<\/p>\n

Atomic directory manipulation<\/h3>\n

Object stores approximate a directory hierarchy by adopting a convention of embedding slashes (\/) in the object name to denote path segments. While this convention works for organizing objects, the convention provides no assistance for actions like moving, renaming, or deleting directories. Without real directories, applications must process potentially millions of individual blobs to achieve directory-level tasks. By contrast, the hierarchical namespace<\/a> processes these tasks by updating a single entry (the parent directory).<\/p>\n

This dramatic optimization is especially significant for many big data analytics frameworks. Tools like Hive and Spark often write output to temporary locations and then rename the location at the conclusion of the job. Without the hierarchical namespace, this rename can often take longer than the analytics process itself. Lower job latency equals lower total cost of ownership (TCO) for analytics workloads.<\/p>\n

Scale<\/h3>\n

HDInsight and Azure Data Lake Storage Gen2 bring new levels of scale for big data workloads. Customers can run workloads that scale at 100\u2019s Gb\/Sec to Petabytes of storage without needing to shard the data across multiple storage accounts.<\/p>\n

Encryption at REST<\/h3>\n

Encryption in Azure Data Lake Storage Gen2 helps you protect your data, implement enterprise security policies, and meet regulatory compliance requirements. Azure Data Lake Storage Gen 2 supports encryption of data both at rest and in transit.<\/p>\n

Network firewall<\/h3>\n

Integrated network firewall capabilities<\/a> allow you to define rules restricting access only to requests originating from specified networks or HDInsight clusters in a specific VNET.<\/p>\n

How does the integration work?<\/h2>\n

HDInsight and Azure Data Lake Storage Gen2 integration is based upon user-assigned managed identity<\/a>. You assign appropriate access to HDInsight with your Azure Data Lake Storage Gen2 accounts. Once configured, your HDInsight cluster is able to use Azure Data Lake Storage Gen2 as its storage.<\/p>\n

1. Create an Azure storage account and enable Data Lake Storage Gen 2 preview.<\/p>\n

\"Storage<\/p>\n

2. Create a user assigned managed identity.<\/p>\n

\"Creating<\/p>\n

3. Assign Storage Blob Data Owner access to the created managed identity on Azure Storage.<\/p>\n

\"Assigning<\/p>\n

4. Now you can proceed to creating HDInsight cluster. In the storage blade, select the storage account, and the associated managed user identity, and proceed with cluster creation workflow.<\/p>\n

\"Selecting<\/p>\n

Getting started<\/h2>\n

Start using Azure Data Lake Storage Gen2 with Azure HDInsight today<\/a>.<\/p>\n

Feedback<\/h2>\n

We look forward to your comments and feedback. If there are any feature requests, customer asks, or suggestions, please contact us at askhdinsight@microsoft.com<\/a>.<\/p>\n

Additional resources<\/h2>\n