- \n
- Azure Data Lake Store<\/strong> \u2013 the first cloud Data Lake for enterprises that is secure, massively scalable and built to the open HDFS standard.\u00a0 With no limits to the size of data and the ability to run massively parallel analytics, you can now unlock value from all your unstructured, semi-structured and structured data.<\/li>\n\n\n\n
- Azure Data Lake Analytics<\/strong> \u2013 the first cloud analytics job service where you can easily develop and run massively parallel data transformation and processing programs in U-SQL, R, Python and .NET over petabytes of data. It has rich built-in cognitive capabilities such as image tagging, emotion detection, face detection, deriving meaning from text, and sentiment analysis with the ability to extend to any type of analytics. With Azure Data Lake Analytics, there is no infrastructure to manage, and you can process data on demand, scale instantly, and only pay per job.<\/li>\n\n\n\n
- Azure HDInsight<\/strong> \u2013 the only fully managed cloud Hadoop offering that provides optimized open source analytic clusters for Spark, Hive, Map Reduce, HBase, Storm, Kafka and R-Server backed by a 99.9% SLA. Today, we are announcing the general availability of R Server for HDInsight to do advanced analytics and predictive modelling with R+Spark. Further, we are introducing the public preview of Kafka for HDInsight, now the first managed cluster solution in the cloud for real-time ingestion with Kafka.<\/li>\n<\/ul>\n\n\n\n
![\"Azure](\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp\" "\\"Azure")
<\/figure>\n\n\n\nData Lake Store – A No Limits Data Lake that powers Big Data Analytics<\/h2>\n\n\n\n
Petabyte sized files and trillions of objects<\/h3>\n\n\n\n
With Azure Data Lake Store your organization can now securely capture and analyze all data in a central location with no artificial constraints to limit the growth of your business. It can manage trillions of files where a single file can be greater than a petabyte in size – this is 200x larger<\/strong> file size than other cloud object stores. Without the limits that constrain other cloud offerings, Data Lake Store is ideal for managing any type of data; including massive datasets like high-resolution video, genomic and seismic datasets, medical data, and data from a wide variety of industries. Data Lake Store is an enterprise data lake that can power your analytics today and in the future.<\/p>\n\n\n\n
![\"icon\"](\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/2e09ca69-dda7-4cac-ae62-9de66eed1cf5.webp\")
<\/figure>\n\n\n\n\n
DS-IQ provides Dynamic Shopper Intelligence by curating data from large amounts of non-relational sources like weather, health, traffic, and economic trends so that we can give our customers actionable insights to drive the most effective marketing and service communications. Azure Data Lake was perfect for us because it could scale elastically, on-demand, to petabytes of data within minutes. This scalability and performance has impressed us, giving us confidence that it can handle the amounts of data we need to process today and, in the future, enable us to provide even more valuable, dynamic, context-aware experiences for our clients.\u201d<\/em><\/p>\n-William Wu, Chief Technology Officer at DS-IQ<\/cite><\/blockquote>\n<\/div>\n\n\n\n
Scalability for massively parallel analytics<\/h3>\n\n\n\n
Data Lake Store provides massive throughput to run analytic jobs with thousands of concurrent executors that read and write hundreds of terabytes of data efficiently. You are no longer forced to redesign your application or repartition your data because Data Lake Store scales throughput to support any size of workload. Multiple services like Data Lake Analytics, HDInsight or HDFS compliant applications can efficiently analyze the same data simultaneously.<\/p>\n\n\n\n
![\"ecolab](\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/ad4c11ff-9c19-464f-ae65-c87f9f755908.webp\")
<\/figure>\n\n\n\n\n
Ecolab has an ambitious mission to find solutions to some of the world\u2019s biggest challenges \u2013 clean water, safe food, abundant energy and healthy environments.\u00a0 \u201cAzure Data Lake has been deployed to our water division where we are collecting real-time data from IoT devices so we can help our customers understand how they can reduce, reuse, and recycle water and at the same address one of the world\u2019s most pressing sustainability issues. We\u2019ve been impressed with Azure Data Lake because it allows us to store any amount of data we require and also lets us use our existing skills to analyze the data. Today, we have both groups who use open source technologies such as Spark in HDInsight to do analytics and other groups that use U-SQL, leveraging the extensibility of C# with the simplicity of SQL.\u201d<\/em><\/p>\n-Kevin Doyle, VP of IT, Global Industrial Solutions at Ecolab<\/cite><\/blockquote>\n<\/div>\n\n\n\n
Data Lake Analytics – A On-Demand Analytics Job Service to power intelligent action<\/h2>\n\n\n\n
Start in seconds, scale instantly, pay per job<\/h3>\n\n\n\n
Our on-demand service will have your data developer and scientist processing big data jobs to power intelligent action in seconds. There is no infrastructure to worry about because there are no servers, VMs, or clusters to wait for, manage or tune. Instantly apply or adjust the analytic units (processing power) from one to hundreds or even thousands for each job. Only pay for the processing used per job, freeing valuable developer time from doing capacity planning and optimizations required in cluster-based systems that can take weeks to months.<\/p>\n\n\n\n
![\"DeviceDesk](\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/320258ea-17cc-499b-997e-e8516f8bf8c1.webp\")
<\/figure>\n\n\n\n\n
Azure Data Lake is instrumental because it helps Insightcentr ingest IoT-scale telemetry from PCs in real-time and gives detailed analytics to our customers without us spending millions of dollars building out big data clusters by scratch. We saw Azure Data Lake as the fastest and most scalable way we can get bring our customers these valuable insights to their business.\u201d<\/em><\/p>\n-Anthony Stevens, CEO Australian start-up Devicedesk<\/cite><\/blockquote>\n<\/div>\n\n\n\n
Develop massively parallel analytic programs with simplicity<\/h3>\n\n\n\n
U-SQL is an easy-to-use, highly expressive, and extensible language that allows you to write code once and automatically have it be parallelized for the scale you need. Instead of writing low-level code dealing with clusters, nodes, mappers, and reducers, etc., a developer writes a simple logical description of how data should be transformed for their business using both declarative and imperative techniques as desired. The U-SQL data processing system automatically parallelizes the code \u2013 enabling developers to control the amount of resources devoted to parallel computation with the simplicity of a slider. The U-SQL language is highly extensible and can reuse existing libraries written in a variety of languages like .NET languages, R, or Python. You can massively parallelize the code to process petabytes of data for diverse workload categories such as ETL, machine learning, feature engineering, image tagging, emotion detection, face detection, deriving meaning from text, and sentiment analysis.<\/p>\n\n\n\n
![\"PureCars](\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/9c20ae55-bb27-4010-addb-a3b5899e257d.webp\")
<\/figure>\n\n\n\n\n
Azure Data Lake allows us to develop quickly on large sets of data with our current developer expertise. We have been able to leverage Azure Data Lake Analytics to capture and process large marketing audiences for our dynamic marketing platforms.”<\/em><\/p>\n-McPherson White, Director of Development at PureCars<\/cite><\/blockquote>\n<\/div>\n\n\n\n
Run Big Cognition at Petabyte Scale<\/h3>\n\n\n\n
Furthermore, we\u2019ve incorporated the technology that sits behind the Cognitive Services API<\/a> inside U-SQL directly. Now you can process any amount of unstructured data, e.g., text, images, and extract emotions, age, and all sorts of other cognitive features using Azure Data Lake and perform query by content. You can join emotions from image content with any other type of data you have and do incredibly powerful analytics and intelligence over it. This is what we call \u2018Big Cognition\u2019<\/em><\/strong>. It\u2019s not just extracting one piece of cognitive information at a time, not just about understanding an emotion or whether there\u2019s an object in an image, but rather it\u2019s about joining all the extracted cognitive data with other types of data, so you can do some really powerful analytics with it. We have demonstrated this capability at Microsoft Ignite<\/a> and PASS Summit, by showing a Big Cognition demo in which we used U-SQL inside Azure Data Lake Analytics to process a million images<\/a> and understand what\u2019s inside those images. You can watch this demo here<\/a> and try it yourself using a sample project on GitHub<\/a>.<\/p>\n\n\n\n
Debug and optimize your Big Data programs with ease<\/h3>\n\n\n\n
With today\u2019s tools, developers face serious challenges debugging distributed programs. Azure Data Lake makes debugging failures in cloud distributed programs as easy as debugging a program in your personal environment using the powerful tools within Visual Studio. Developers no longer need to inspect thousands of logs on each machine searching for failures. When a U-SQL job fails, logs are automatically located, parsed, and filtered to the exact components involved in the failure and available as a visualization. Developers can even debug the specific parts of the U-SQL job that failed to their own local workstation without wasting time and money resubmitting jobs to the cloud. Our service can detect and analyze common performance problems that big data developers encounter such as imbalanced data partitioning and offers suggestions to fix your programs using the intelligence we\u2019ve gathered in the analysis of over a billion jobs in Microsoft\u2019s data lake.<\/p>\n\n\n\n
Developers do a lot of heavy lifting when optimizing big data systems and frequently overpay for unused cluster capacity. Developers must manually optimize their data transformations, requiring them to carefully investigate how their data is transformed step-by-step, often manually ordering steps to gain improvements. Understanding performance and scale bottlenecks is challenging and requires distributed computing and infrastructure experts. For example, to improve performance, developers must carefully account for the time & cost of data movement across a cluster and rewrite their queries or repartition their data. Data Lake\u2019s execution environment actively analyzes your programs as they run and offers recommendations to improve performance and reduce cost. For example, if you requested 1000 AUs for your program and only 50 AUs were needed, the system would recommend that you only use 50 AUs resulting in a 20x cost savings.<\/p>\n\n\n\n
![\"Plexure](\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/ff7c9ed4-ae4f-471c-9fec-75b728af2d46.webp\")
<\/figure>\n\n\n\n\n
We ingest a massive amount of live data from mobile, web, IoT and retail transactions. Data Lake gives us the ability to easily and cost effectively store everything and analyse what we need to, when we need to. The simplicity of ramping up parallel processing on the U-SQL queries removes the technical complexities of fighting with the data and lets the teams focus on the business outcomes. We are now taking this a step further and exposing the powerful Data Lake tools directly to our clients in our software allowing them to more easily explore their data using these tools.\u201d<\/em><\/p>\n-David Inggs, CTO at Plexure<\/cite><\/blockquote>\n<\/div>\n\n\n\n
Today, we are also announcing the availability of this big data productivity environment in Visual Studio Code allowing users to have this type of productivity in a free cross-platform code editor that is available on Windows, Mac OS X, and Linux.<\/p>\n\n\n\n
Azure HDInsight introduces fully managed Kafka for real-time analytics and R Server for advanced analytics<\/h2>\n\n\n\n
At Strata + Hadoop World New York, we announced<\/a> new security, performance and ISV solutions that build on Azure HDInsight\u2019s leadership for enterprise-ready cloud Hadoop. Today, we are announcing the public preview of Kafka for HDInsight. This service lets you ingest massive amounts of real-time data and analyze that data with integration to Storm, Spark, for HDInsight and Azure IoT Hub to build end-to-end IoT, fraud detection, click-stream analysis, financial alerts, or social analytics solutions.<\/p>\n\n\n\n
We are also announcing the general availability of R Server for HDInsight. Running Microsoft R Server as a service on top of Apache Spark, developers can achieve unprecedented scale and performance with code that combines the familiarity of the open source R language and Spark. Multi-threaded math libraries and transparent parallelization in R Server enables handling up to 1000x more data and up to 50x faster speeds than open source R\u2014helping you train more accurate models for better predictions than previously possible. Newly available in GA is the inclusion of R Studio Server Community Edition out-of-the-box making it easy for data scientists to get started quickly.<\/p>\n\n\n\n
![\"Milliman](\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/94f8dff5-e545-4b92-a565-473e8fdfdbaa.webp\")
<\/figure>\n\n\n\n\n
Milliman is among the world\u2019s largest providers of actuarial and related products and services, with offices in major cities around the globe. R Server for HDInsight, offers the ability for our clients to be able to forecast risk over much larger datasets than ever before, improving the accuracy of predictions, in a cost-efficient way. The familiarity of the R Programming language to our users, as well as the ability to spin up Hadoop and Spark clusters within minutes, running at unprecedented scale and performance, is what really gets me excited about R Server for HDInsight.\u201d<\/em><\/p>\n-Paul Maher, Chief Technology Officer of the Life Technology Solutions practice at Milliman<\/cite><\/blockquote>\n<\/div>\n\n\n\n
Enterprise-grade Security, Auditing and Support<\/h2>\n\n\n\n
Enterprise-grade big data solutions must meet uptime guarantees, stringent security, governance & compliance requirements, and integrate with your existing IT investments. Data Lake services (Store, Analytics, and HDInsight) guarantee an industry-leading 99.9% uptime SLA and 24\/7 support for all Data Lake services. They are built with the highest levels of security for authentication, authorization, auditing, and encryption to give you peace-of-mind when storing and analyzing sensitive corporate data and intellectual property. Data is always encrypted; in motion using SSL, and at rest using service or user managed HSM-backed keys in Azure Key Vault. Capabilities such as single sign-on (SSO), multi-factor authentication and seamless management of your on-premises identity & access management is built-in through Azure Active Directory. You can authorize users and groups with fine-grained POSIX-based ACLs for all data in the Store or with Apache Ranger in HDInsight enabling role-based access controls. Every access or configuration change is automatically audited for security and regulatory compliance requirements.<\/p>\n\n\n\n
Supporting open source and open standards<\/h2>\n\n\n\n
Microsoft continues to collaborate with the open source community reflected by our contributions to Apache Hadoop<\/a>, Spark<\/a>, Apache REEF<\/a> and our work with Jupyter notebooks. This is also the case with Azure Data Lake.<\/p>\n\n\n\n
Azure Data Lake Analytics uses Apache YARN, the central part of Apache Hadoop to govern resource management and deliver consistent operations. To lead innovations to YARN, Microsoft has been a primary contributor to improve performance, scale, and made security innovations.<\/p>\n\n\n\n
![\"HortonWorks](\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/fd6df004-1318-4427-8419-a8a171a82331.webp\")
<\/figure>\n\n\n\n\n
Hortonworks and Microsoft have partnered closely for the past 5 years to further the Hadoop platform for big data analytics, including contributions to YARN, Hive, and other Apache projects.\u00a0 Azure Data Lake services, including Azure HDInsight and Azure Data Lake Store, demonstrate our shared commitment to make it easier for everyone to work with big data in an open and collaborative way.\u201d<\/em><\/p>\n-Shaun Connolly, Chief Strategy Officer at Hortonworks<\/cite><\/blockquote>\n<\/div>\n\n\n\n
Data Lake Store supports the open Apache Hadoop Distributed File System (HDFS) standard. Microsoft has also contributed improvements to HDFS such as OAuth 2.0 protocol support.<\/p>\n\n\n\n
![\"Cloudera](\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/aa22628a-9c5a-4539-a463-c8eab2fb8ca9.webp\")
<\/figure>\n\n\n\n\n
Cloudera is working closely with Microsoft to integrate Cloudera Enterprise with the Azure Data Lake Store. Cloudera on Azure will benefit from the Data Lake Store which acts as a cloud-based landing zone for all data in your enterprise data hub.\u00a0 Cloudera will leverage Data Lake and provide customers with a secure and flexible big data solution in the future.\u201d<\/em><\/p>\n-Mike Olson, founder and chief strategy officer at Cloudera<\/cite><\/blockquote>\n<\/div>\n\n\n\n
Leadership<\/h2>\n\n\n\n
Both industry analysts and customers recognize Microsoft\u2019s capabilities in big data. Forrester recently recognized<\/a> Microsoft Azure as a leader in their Big Data Hadoop Cloud Solutions. Forrester notes that leaders have the most comprehensive, scalable, and integrated platforms. Microsoft specifically was called out for having a cloud-first strategy that is paying off.<\/p>\n\n\n\n
Getting started with Data Lake<\/h2>\n\n\n\n
Data Lake Analytics<\/a> and Store<\/a> is generally available today. R Server for HDInsight<\/a> is also generally available today. Kafka for HDInsight<\/a> is in public preview. Try it today individually or as part of Cortana Intelligence Suite<\/a> to transform your data into intelligent action.<\/p>\n\n\n\n
- \n
- Read the overview<\/a>, pricing<\/a> and getting started<\/a> pages of Data Lake Analytics or attend the free course<\/a><\/li>\n\n\n\n
- Read the overview<\/a>, pricing<\/a> and getting started<\/a> pages of Data Lake Store<\/li>\n\n\n\n
- Read the R Server<\/a>, Kafka<\/a> overview and pricing<\/a> pages of HDInsight<\/li>\n<\/ul>\n\n\n\n
@josephsirosh<\/p>\n","protected":false},"excerpt":{"rendered":"
The general availability of Azure Data Lake ushers in a new era of productivity for your big data developers and scientists.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ms_queue_id":[],"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","_alt_title":"","footnotes":"","msx_community_cta_settings":[]},"categories":[1474,1491],"tags":[48],"audience":[3054,3057,3053],"content-type":[1465],"product":[1610],"tech-community":[],"topic":[],"coauthors":[97],"class_list":["post-4326","post","type-post","status-publish","format-standard","hentry","category-analytics","category-storage","tag-big-data","audience-business-decision-makers","audience-data-professionals","audience-it-decision-makers","content-type-announcements","product-azure-data-lake-storage","review-flag-1680286581-295","review-flag-1680286581-56","review-flag-1680286581-364","review-flag-2-1680286581-601","review-flag-24-7-1680286585-656","review-flag-5-1680286581-950","review-flag-7-1680286581-146","review-flag-9-1680286581-259","review-flag-alway-1680286580-106","review-flag-forre-1680286585-445","review-flag-free-1680286579-836","review-flag-ga-1680286584-289","review-flag-gener-1680286584-335","review-flag-integ-1680286579-214","review-flag-iot-1680286585-835","review-flag-lever-1680286579-649","review-flag-machi-1680286585-314","review-flag-microsofts","review-flag-new-1680286579-546","review-flag-publi-1680286584-566"],"yoast_head":"\n
The Intelligent Data Lake | Microsoft Azure Blog<\/title>\n<meta name=\"description\" content=\"The general availability of Azure Data Lake ushers in a new era of productivity for your big data developers and scientists.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The Intelligent Data Lake | Microsoft Azure Blog\" \/>\n<meta property=\"og:description\" content=\"The general availability of Azure Data Lake ushers in a new era of productivity for your big data developers and scientists.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft Azure Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/microsoftazure\" \/>\n<meta property=\"article:published_time\" content=\"2016-11-16T00:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-17T16:50:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp\" \/>\n<meta name=\"author\" content=\"Microsoft Azure\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@azure\" \/>\n<meta name=\"twitter:site\" content=\"@azure\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Microsoft Azure\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/\"},\"author\":[{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/microsoft-azure\/\",\"@type\":\"Person\",\"@name\":\"Microsoft Azure\"}],\"headline\":\"The Intelligent Data Lake\",\"datePublished\":\"2016-11-16T00:00:00+00:00\",\"dateModified\":\"2025-06-17T16:50:10+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/\"},\"wordCount\":2829,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp\",\"keywords\":[\"Big Data\"],\"articleSection\":[\"Analytics\",\"Storage\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/\",\"name\":\"The Intelligent Data Lake | Microsoft Azure Blog\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp\",\"datePublished\":\"2016-11-16T00:00:00+00:00\",\"dateModified\":\"2025-06-17T16:50:10+00:00\",\"description\":\"The general availability of Azure Data Lake ushers in a new era of productivity for your big data developers and scientists.\",\"breadcrumb\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#primaryimage\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog home\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Analytics\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/analytics\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"The Intelligent Data Lake\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"name\":\"Microsoft Azure Blog\",\"description\":\"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.\",\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\",\"name\":\"Microsoft Azure Blog\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"width\":512,\"height\":512,\"caption\":\"Microsoft Azure Blog\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/microsoftazure\",\"https:\/\/x.com\/azure\",\"https:\/\/www.instagram.com\/microsoftdeveloper\/\",\"https:\/\/www.linkedin.com\/company\/16188386\",\"https:\/\/www.youtube.com\/user\/windowsazure\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117\",\"name\":\"shakir\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"caption\":\"shakir\"},\"sameAs\":[\"https:\/\/azure.microsoft.com\"],\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The Intelligent Data Lake | Microsoft Azure Blog","description":"The general availability of Azure Data Lake ushers in a new era of productivity for your big data developers and scientists.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/","og_locale":"en_US","og_type":"article","og_title":"The Intelligent Data Lake | Microsoft Azure Blog","og_description":"The general availability of Azure Data Lake ushers in a new era of productivity for your big data developers and scientists.","og_url":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/","og_site_name":"Microsoft Azure Blog","article_publisher":"https:\/\/www.facebook.com\/microsoftazure","article_published_time":"2016-11-16T00:00:00+00:00","article_modified_time":"2025-06-17T16:50:10+00:00","og_image":[{"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp","type":"","width":"","height":""}],"author":"Microsoft Azure","twitter_card":"summary_large_image","twitter_creator":"@azure","twitter_site":"@azure","twitter_misc":{"Written by":"Microsoft Azure","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#article","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/"},"author":[{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/microsoft-azure\/","@type":"Person","@name":"Microsoft Azure"}],"headline":"The Intelligent Data Lake","datePublished":"2016-11-16T00:00:00+00:00","dateModified":"2025-06-17T16:50:10+00:00","mainEntityOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/"},"wordCount":2829,"commentCount":0,"publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp","keywords":["Big Data"],"articleSection":["Analytics","Storage"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/","name":"The Intelligent Data Lake | Microsoft Azure Blog","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#primaryimage"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp","datePublished":"2016-11-16T00:00:00+00:00","dateModified":"2025-06-17T16:50:10+00:00","description":"The general availability of Azure Data Lake ushers in a new era of productivity for your big data developers and scientists.","breadcrumb":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#primaryimage","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2016\/11\/571f2d03-6e94-43ad-b02c-5ec6b337a3d0.webp"},{"@type":"BreadcrumbList","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-intelligent-data-lake\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog home","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/"},{"@type":"ListItem","position":2,"name":"Analytics","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/analytics\/"},{"@type":"ListItem","position":3,"name":"The Intelligent Data Lake"}]},{"@type":"WebSite","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","name":"Microsoft Azure Blog","description":"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.","publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization","name":"Microsoft Azure Blog","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","width":512,"height":512,"caption":"Microsoft Azure Blog"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/microsoftazure","https:\/\/x.com\/azure","https:\/\/www.instagram.com\/microsoftdeveloper\/","https:\/\/www.linkedin.com\/company\/16188386","https:\/\/www.youtube.com\/user\/windowsazure"]},{"@type":"Person","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117","name":"shakir","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4","url":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","caption":"shakir"},"sameAs":["https:\/\/azure.microsoft.com"],"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/"}]}},"msxcm_display_generated_audio":false,"msxcm_animated_featured_image":null,"distributor_meta":false,"distributor_terms":false,"distributor_media":false,"distributor_original_site_name":"Microsoft Azure Blog","distributor_original_site_url":"https:\/\/azure.microsoft.com\/en-us\/blog","push-errors":false,"_links":{"self":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/4326","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/comments?post=4326"}],"version-history":[{"count":2,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/4326\/revisions"}],"predecessor-version":[{"id":42118,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/4326\/revisions\/42118"}],"wp:attachment":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/media?parent=4326"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/categories?post=4326"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tags?post=4326"},{"taxonomy":"audience","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/audience?post=4326"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/content-type?post=4326"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/product?post=4326"},{"taxonomy":"tech-community","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tech-community?post=4326"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/topic?post=4326"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/coauthors?post=4326"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}} - Read the overview<\/a>, pricing<\/a> and getting started<\/a> pages of Data Lake Store<\/li>\n\n\n\n
- Azure Data Lake Analytics<\/strong> \u2013 the first cloud analytics job service where you can easily develop and run massively parallel data transformation and processing programs in U-SQL, R, Python and .NET over petabytes of data. It has rich built-in cognitive capabilities such as image tagging, emotion detection, face detection, deriving meaning from text, and sentiment analysis with the ability to extend to any type of analytics. With Azure Data Lake Analytics, there is no infrastructure to manage, and you can process data on demand, scale instantly, and only pay per job.<\/li>\n\n\n\n