{"id":2557,"date":"2018-06-28T00:00:00","date_gmt":"2018-06-28T07:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/the-emerging-big-data-architectural-pattern"},"modified":"2023-08-01T16:53:40","modified_gmt":"2023-08-01T23:53:40","slug":"the-emerging-big-data-architectural-pattern","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-emerging-big-data-architectural-pattern\/","title":{"rendered":"The emerging big data architectural pattern"},"content":{"rendered":"

Why lambda?<\/p>\n

Lambda architecture is a popular pattern in building Big Data pipelines. It is designed to handle massive quantities of data by taking advantage of both a batch<\/a> layer (also called cold layer) and a stream-processing<\/a> layer (also called hot or speed layer).<\/p>\n

The following are some of the reasons that have led to the popularity and success of the lambda architecture, particularly in big data processing pipelines.<\/p>\n

Speed and business challenges<\/h3>\n

The ability to process data at high speed in a streaming context is necessary for operational needs, such as transaction processing and real-time reporting. Some examples are fault\/fraud detection, connected\/smart cars\/factory\/hospitals\/city, sentiment analysis, inventory control, network\/security monitoring, and many more.<\/p>\n

Typically, batch processing, involving massive amounts of data, and related correlation and aggregation is important for business reporting. This is to understand how the business is performing, what the trends are, and what corrective or additive measure can be executed to improve business or customer experience.<\/p>\n

Product challenges<\/h3>\n

One of the triggers that lead to the very existence of lambda architecture was to make the most of the technology and tool set available. Existing batch processing systems, such as data warehouse, data lake, Spark\/Hadoop, and more, could deal with petabyte scale data operations easily but couldn\u2019t do it fast enough that was warranted by the operational needs.<\/p>\n

Similarly, very fast layers such as cache databases, NoSQL, streaming technology allows fast operational analytics on smaller data sets but cannot do massive scale correlation and aggregation and other analytics operations (such as Online Analytical Processing<\/a>) like a batch system can.<\/p>\n

The skills challenge<\/h3>\n

Additionally, in the market you will find people who are highly skilled in batch systems, and often they do not have the same depth of skills in stream processing, and vice versa.<\/p>\n

Lambda on Azure<\/h3>\n

The following is one of the many representative Lambda architecture on Azure for building Big Data pipelines.<\/p>\n

\"Figure
\n\u00a0
\nFigure 1: Lambda architecture for big data processing represented by Azure products and services. Note, other Azure and (or) ISV solutions can be placed in the mix if needed based on specific requirements.<\/em><\/p>\n

What problems do lambda solve vs. what problems does it introduce?<\/h2>\n

As stated in the previous section, lambda architecture resolves some business challenges. Various parts of the business have different needs in terms of speed, level of granularity and mechanism to consume data. It also resolves the challenge of the choice of technology, by using the best of the speed layer and batch layer together, and not stretching one product to do both which it isn\u2019t comfortable in doing. Finally, it ensures people with skills dealing with transaction and speed layer can work in parallel and together with people with skills in batch processing.<\/p>\n

Although immensely successful and widely adopted across many industries and a defacto architectural pattern for big data pipelines, it comes with its own challenges. Here are a few:<\/p>\n