cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta Live Table streaming pipeline

rt-slowth
Contributor

How do I do a simple left join of a static table and a streaming table under catalog in the streaming pipeline of a Delta Live Table?

1 REPLY 1

Priyanka_Biswas
Valued Contributor
Valued Contributor

Hi @rt-slowth I would like to share with you the Databricks documentation, which contains details about stream-static table joins

https://docs.databricks.com/en/delta-live-tables/transform.html#stream-static-joins

Stream-static joins are a good choice when denormalizing a continuous stream of append-only data with a primarily static dimension table.

With each pipeline update, new records from the stream are joined with the most current snapshot of the static table. If records are added or updated in the static table after corresponding data from the streaming table has been processed, the resultant records are not recalculated unless a full refresh is performed.

In pipelines configured for triggered execution, the static table returns results as of the time the update started. In pipelines configured for continuous execution, each time the table processes an update, the most recent version of the static table is queried.

The following is an example of a stream-static join:

@dlt.table
def customer_sales():
  return dlt.read_stream("sales").join(read("customers"), ["customer_id"], "left")
Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.