Delta Live Table

In DLT, Tables are similar to traditional materialized views.

The Delta Live Tables runtime automatically creates tables in the Delta format and ensures those tables are updated with the latest result of the query that creates the table.

In DLT, Views are similar to a temporary view in SQL and are an alias for some computation. A view allows you to break a complicated query into smaller or easier-to-understand queries.

The bronze layer is all about loading the raw data.

Usually it can be done incrementally, so we can create to incremental live table to receive it.

The Silver layer is all about high-quality, diverse, and accessible datasets.

proper use of constraints and comments.

Constraints allow you to define data quality expectations

tblproperties are like tags that can be used for data cataloging

Comment: A string briefly describing the table’s purpose

The gold layer is about aggregated business data

The aggregation sometimes can be done incrementally, e.g. max of the whole table

Otherwise it has to be a complete gold table (create live table, without the incremental keyword)

DLT allows you to choose whether each dataset in a pipeline is complete or incremental

In DLT,

an individual datasets may be Incremental or Complete

a pipeline may be Triggered or Continuous

A triggered pipeline release the cpu resource when it is finished, while a continuous pipeline always keeps the resource.

Data Quality Monitoring (requires Databricks SQL)

Based on the constraints applied on the DLT, you can monitor the quality status by querying the pipeline log data

https://databricks.com/discover/pages/getting-started-with-delta-live-tables