What is a Data Lake?

資料湖定義

This introductory guide explores the many benefits and use cases of a data lake. Learn what a data lake is, why it matters, and discover the difference between data lakes and data warehouses. But first, let's define data lake as a term.

A data lake is a centralized repository that ingests and stores large volumes of data in its original form. The data can then be processed and used as a basis for a variety of analytic needs. Due to its open, scalable architecture, a data lake can accommodate all types of data from any source, from structured (database tables, Excel sheets) to semi-structured (XML files, webpages) to unstructured (images, audio files, tweets), all without sacrificing fidelity. The data files are typically stored in staged zones—raw, cleansed, and curated—so that different types of users may use the data in its various forms to meet their needs. Data lakes provide core data consistency across a variety of applications, powering big data analytics, machine learning, predictive analytics, and other forms of intelligent action.

Why are data lakes important for businesses?

如果沒有資料湖解決方案的出現，今天的高度連結、深入解析驅動的世界將無法實現。這是因為組織依賴如 Azure Data Lake 等全方位的資料湖平台，來保持未經處理資料的合併、整合、安全且可存取。Azure Data Lake Storage 等可調整的儲存工具可將資料保留和保護在中央位置，並以最佳成本消除孤島。這會為使用者執行各種工作負載類別打下基礎，例如巨量資料處理、SQL 查詢、文字採礦、串流分析及機器學習。資料可用來摘要上游資料視覺效果和隨選報表需求。像是 Azure Synapse Analytics 的新式端對端資料平台，可解決以資料湖為中心的巨量資料架構完整需求。

資料湖使用案例

資料湖與資料倉儲

Now you know what a data lake is, why it matters, and how it's used across a variety of organizations. But what's the difference between a data lake and a data warehouse? And when is it appropriate to use one over the other?

While data lakes and data warehouses are similar in that they both store and process data, each have their own specialties, and therefore their own use cases. That's why it's common for an enterprise-level organization to include a data lake and a data warehouse in their analytics ecosystem. Both repositories work together to form a secure, end-to-end system for storage, processing, and faster time to insight.

資料湖會從各種來源擷取關聯式和非關聯式資料—商務應用程式、行動應用程式、IoT 裝置、社交媒體或串流，而不需要定義資料的結構或結構描述，直到讀取為止。Schema-on-read 可確保任何類型的資料都可以儲存在其原始表單中。因此，資料湖可以隨時保存各種資料類型，從結構化到半結構化或非結構化。其彈性且可調整的特性，使其對於使用不同類型的計算處理工具，例如 Apache Spark 或 Azure Machine Learning 來執行複雜資料分析而言非常重要。

相反地，資料倉儲在本質上是關聯式。結構或結構描述是由已針對 SQL 查詢作業策劃、符合及最佳化的商務和產品需求，建立模型或預先定義。當資料湖存放所有結構類型的資料，包括原始和未處理的資料時，資料倉儲會儲存已處理及轉換且具有特定用途的資料，進而可用於來源分析或作業報告。這會讓資料倉儲更適合用於產生更標準化的 BI 分析形式，或服務已定義的商業用途案例。

Data lake vs. data lakehouse

Now you know the difference between a data lake vs. a data warehouse. But what's the difference between a data lake and a data lakehouse? And is it necessary to have both?

儘管傳統資料湖有許多優點，但並非沒有缺點。因為資料湖可以容納各種來源的所有資料類型，所以可能會發生品質控制、資料損毀和不當分割等相關問題。管理不良的資料湖不僅會損害資料完整性，而且也會造成瓶頸、效能緩慢及安全性風險。

That's where the data lakehouse comes into play. A data lakehouse is an open standards-based storage solution that is multifaceted in nature. It can address the needs of data scientists and engineers who conduct deep data analysis and processing, as well as the needs of traditional data warehouse professionals who curate and publish data for business intelligence and reporting purposes. The beauty of the lakehouse is that each workload can seamlessly operate on top of the data lake without having to duplicate the data into another structurally predefined database. This ensures that everyone is working on the most up-to-date data, while also reducing redundancies.

資料湖存放庫透過直接在雲端資料湖之上新增 資料湖儲存層，以解決傳統資料湖的挑戰。儲存層提供彈性的分析架構，可處理 ATOMIC (不可部分完成性、一致性、隔離性和耐用性) 交易，以取得資料可靠性、串流整合，以及資料版本設定和結構描述強制等進階功能。這允許在資料湖上進行一系列的分析活動，而不影響核心資料一致性。雖然湖存放庫的任務取決於您的需求有多複雜，但其彈性與範圍使其成為許多企業組織的最佳解決方案。

什麼是資料湖結構?