Delta Live Table streaming pipeline

rt-slowth — Thu, 28 Dec 2023 03:11:43 GMT

How do I do a simple left join of a static table and a streaming table under catalog in the streaming pipeline of a Delta Live Table?

Re: Delta Live Table streaming pipeline

Priyanka_Biswas — Wed, 17 Jan 2024 07:51:27 GMT

Hi @rt-slowth I would like to share with you the Databricks documentation, which contains details about stream-static table joins

https://docs.databricks.com/en/delta-live-tables/transform.html#stream-static-joins

Stream-static joins are a good choice when denormalizing a continuous stream of append-only data with a primarily static dimension table.

With each pipeline update, new records from the stream are joined with the most current snapshot of the static table. If records are added or updated in the static table after corresponding data from the streaming table has been processed, the resultant records are not recalculated unless a full refresh is performed.

In pipelines configured for triggered execution, the static table returns results as of the time the update started. In pipelines configured for continuous execution, each time the table processes an update, the most recent version of the static table is queried.

The following is an example of a stream-static join:

@dlt.table
def customer_sales():
  return dlt.read_stream("sales").join(read("customers"), ["customer_id"], "left")

topic Re: Delta Live Table streaming pipeline in Data Engineering

Delta Live Table streaming pipeline

Re: Delta Live Table streaming pipeline