Articles

Introduction to Data Lakehouse

4 min read

By Brais Rodríguez Suárez and José Manuel Simón Ramos

In today’s digital era, data has become one of the most valuable resources for organizations across all sectors. From technology companies to financial institutions, the ability to collect, store, and analyze data effectively is essential for strategic decision-making and continuous improvement. However, with the exponential growth in the volume and complexity of generated data, the need arises to manage it efficiently to avoid turning it into data swamps, where valuable information gets lost in large, disorganized systems, making its use and exploitation difficult.

This is where concepts like Data Warehouse and Data Lake come into play—architectures traditionally used for large-scale data exploitation. However, each presents its own limitations and challenges in terms of performance, scalability, and reliability. To overcome these limitations, a new architecture known as Data Lakehouse has emerged.

This article covers the basic fundamentals of the Data Lakehouse architecture, its main components, and the storage format and schema.

What is a Data Lakehouse?

A Data Lakehouse is an integrated data management platform that combines the ability to store unstructured and semi-structured data from a Data Lake with the processing and analytical capabilities of a Data Warehouse. By merging both solutions, a Data Lakehouse offers a complete, efficient, and reliable data architecture without compromising performance or scalability.

This architecture can be deployed both in the cloud and on-premise, although cloud deployment is recommended to leverage benefits such as high availability, shorter implementation time, simplified maintenance, etc.

Key characteristics of a Data Lakehouse

  • Scalability and flexibility: There is full separation between storage and compute. This enables independent or combined scaling depending on data volume and needs.
  • Metadata layer: Helps organize and provide information about stored data: data types, lineage, documentation, etc.
  • Governance: Offers granular access control, limiting which data a user can access or modify, increasing security and data privacy.
  • Low cost: Services follow a pay-per-use model without fixed commitments. Component independence reduces overprovisioning and unnecessary expenses.
  • Batch and streaming processing support. Suitable for various update frequencies (daily, hourly, near real-time, etc.).
  • Portability: Based on open-source technologies (Python, Spark, Delta Lake), eliminating license requirements and vendor lock-in.

The following table shows a comparison between different functionalities and how they fit within each of the systems mentioned above:

  Data Warehouse Data Lake Data Lakehouse
Data structure Structured Unstructured and semi-structured Structured, semi-structured and unstructured
Scheme flexibility No Yes Yes
Cost High Medium Low
Use cases Idea for analytical queries and reports Ideal for data exploration, ingestion, and ML projects Combine the use cases of the Data Warehouse and the Data Lake
Ease of scaling Low High High

Data Lakehouse Architecture

From a cloud architecture perspective, implementing a Data Lakehouse requires:

  • Storage system: Azure Data Lake Gen2, Amazon S3, Google Cloud Storage, etc.
  • Compute resources for ETLs that process and transform the data: Databricks clusters, Synapse, AWS Glue jobs, Google Dataproc, etc.
  • SQL engine for analytical queries: Trino/Presto, AWS Athena, Azure Synapse, Databricks SQL, Google BigQuery.

The following figure shows a concrete example of an architecture in which the Databricks platform is used on the Azure cloud, which can be extrapolated to other cloud providers:

Data Architecture

For storage, Data Lakehouses use transaction-compatible file formats such as Apache Hudi, Apache Iceberg, or Delta Lake. Based on our experience, Delta Lake is the best fit in most scenarios due to its compatibility and robustness.

Delta Lake is an optimized, open-source storage layer that extends Parquet files by adding a transaction log. This combines Parquet’s advantages (partitioning, columnar storage, compression) with the reliability of transactional systems. Key features:

  • Schema evolution: Supports schema changes without needing to rewrite data.
  • Transaction support: Stores a log of all operations, ensuring consistent data views even with concurrent modifications (ACID compliance).
  • Version control: Enables rollback and recovery of previous data states when issues occur.

For storage schema, Data Lakehouse uses the Medallion Architecture (bronze, silver, gold), which simplifies data management and improves structure and quality as data progresses through transformation stages.

Conclusions

This article described the characteristics and advantages of this new paradigm for data management and processing. This approach allows organizations to centralize data by adding a transversal system across all their areas.

At Decide’s Data Engineering team, we are deeply committed to this architecture and have hands-on experience implementing it for various clients. We offer a Databricks-based Data Lakehouse solution with low implementation time, easy maintenance, and secure data access management. Our proposal includes everything needed for agile and scalable development supported by Infrastructure as Code and CI/CD.

For data exploitation, we design the architecture to provide value to all parts of the organization, internally (Data Scientists, Business Intelligence, Reporting) and externally, enabling secure data sharing with third parties when needed.

All of this is complemented by a prior requirements analysis, adjusting the cost of the solution to the organization’s needs.

References

CONTACT US

Our experts can help you

Tell us the constraint you are working against: the regulator, the core system, the data, and we will tell you where we would start.