{"id":1193,"date":"2019-06-26T00:00:00","date_gmt":"2019-06-26T00:00:00","guid":{"rendered":""},"modified":"2025-06-24T23:04:12","modified_gmt":"2025-06-25T06:04:12","slug":"event-driven-analytics-with-azure-data-lake-storage-gen2","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/event-driven-analytics-with-azure-data-lake-storage-gen2\/","title":{"rendered":"Event-driven analytics with Azure Data Lake Storage Gen2"},"content":{"rendered":"\n
On November 4, 2019, Azure Data Lake Storage Gen2 integration with Azure Event Grid became generally available. \n

Most modern-day businesses employ analytics pipelines for real-time and batch processing. A common characteristic of these pipelines is that data arrives at irregular intervals from diverse sources. This adds complexity in terms of having to orchestrate the pipeline such that data gets processed in a timely fashion.<\/p>\n

The answer to these challenges lies in coming up with a decoupled event-driven pipeline using serverless<\/a> components that responds to changes in data as they occur.<\/p>\n

An integral part of any analytics pipeline is the data lake. Azure Data Lake Storage Gen2<\/a> provides secure, cost effective, and scalable storage for the structured, semi-structured, and unstructured data arriving from diverse sources. Azure Data Lake Storage Gen2\u2019s performance, global availability, and partner ecosystem make it the platform of choice for analytics customers and partners around the world. Next comes the event processing aspect. With Azure Event Grid<\/a>, a fully managed event routing service, Azure Functions<\/a>, a serverless compute engine, and Azure Logic Apps<\/a>, a serverless workflow orchestration engine, it is easy to perform event-based processing and workflows responding to the events in real-time.<\/p>\n

Today, we\u2019re very excited to announce that Azure Data Lake Storage Gen2 integration with Azure Event Grid is in preview!<\/strong> This means that Azure Data Lake Storage Gen2 can now generate events that can be consumed by Event Grid and routed to subscribers with webhooks, Azure Event Hubs<\/a>, Azure Functions, and Logic Apps as endpoints. With this capability, individual changes to files and directories in Azure Data Lake Storage Gen2 can automatically be captured and made available to data engineers for creating rich big data analytics platforms that use event-driven architectures.<\/p>\n

\"Modern<\/figure>

<\/p>\n

The diagram above shows a reference architecture for the modern data warehouse<\/a> pipeline built on Azure Data Lake Storage Gen2 and Azure serverless components. Data from various sources lands in Azure Data Lake Storage Gen2 via Azure Data Factory<\/a> and other data movement tools. Azure Data Lake Storage Gen2 generates events for new file creation, updates, renames, or deletes which are routed via Event Grid and Azure Function to Azure Databricks<\/a>. A databricks job processes the file and writes the output back to Azure Data Lake Storage Gen2. When this happens, Azure Data Lake Storage Gen2 publishes a notification to Event Grid which invokes an Azure Function to copy data to Azure SQL Data Warehouse<\/a>. Data is finally served via Azure Analysis Services<\/a> and PowerBI<\/a>.<\/p>\n

The events that will be made available for Azure Data Lake Storage Gen2 are BlobCreated, BlobDeleted, BlobRenamed, DirectoryCreated, DirectoryDeleted, and DirectoryRenamed<\/em>. Details on these events can be found in the documentation \u201cAzure Event Grid event schema for Blob storage<\/a>.\u201d<\/p>\n

Some key benefits include:<\/strong><\/p>\n