When using Delta Live Tables, how do I set a table to be incremental vs complete using Python?

User16826992666 — Fri, 25 Jun 2021 18:59:12 GMT

When using SQL, I can use the Create Live Table command and the Create Incremental Live Table command to set the run type I want the table to use. But I don't seem to have that same syntax for python. How can I set this table type while using Python?

Re: When using Delta Live Tables, how do I set a table to be incremental vs complete using Python?

sajith_appukutt — Fri, 25 Jun 2021 22:00:20 GMT

The documentation at https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#mixing-complete-tables-and-incremental-tables has an example

the first two functions load data incrementally and the last one loads it fully

@dlt.view
def incremental_bronze():
  return (
    # Since this is a streaming source, this table is incremental.
    spark.readStream.format("cloudFiles")
      .option("cloudFiles.format", "json")
      .load("s3://path/to/raw/data")
  )
 
@dlt.table
def incremental_silver():
  # Since we read the bronze table as a stream, this silver table is also
  # updated incrementally.
  return dlt.read_stream("incremental_bronze").where(...)
 
@dlt.table
def complete_gold():
  # This table will be recomputed completely by reading the whole silver table
  # when it is updated.
  return dlt.read("incremental_silver").groupBy("user_id").count()

topic Re: When using Delta Live Tables, how do I set a table to be incremental vs complete using Python? in Data Engineering

When using Delta Live Tables, how do I set a table to be incremental vs complete using Python?

Re: When using Delta Live Tables, how do I set a table to be incremental vs complete using Python?