cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

When using Delta Live Tables, how do I set a table to be incremental vs complete using Python?

User16826992666
Valued Contributor

When using SQL, I can use the Create Live Table command and the Create Incremental Live Table command to set the run type I want the table to use. But I don't seem to have that same syntax for python. How can I set this table type while using Python?

1 REPLY 1

sajith_appukutt
Honored Contributor II

The documentation at https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#mix... has an example

the first two functions load data incrementally and the last one loads it fully

@dlt.view
def incremental_bronze():
  return (
    # Since this is a streaming source, this table is incremental.
    spark.readStream.format("cloudFiles")
      .option("cloudFiles.format", "json")
      .load("s3://path/to/raw/data")
  )
 
@dlt.table
def incremental_silver():
  # Since we read the bronze table as a stream, this silver table is also
  # updated incrementally.
  return dlt.read_stream("incremental_bronze").where(...)
 
@dlt.table
def complete_gold():
  # This table will be recomputed completely by reading the whole silver table
  # when it is updated.
  return dlt.read("incremental_silver").groupBy("user_id").count()

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group