cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Why does DLT CDC some time manifests the results table as a table and other times as a view?

Jennifer_Lu
New Contributor III

I have a simple DLT pipeline that reads from an existing table, do some transformations, saves to a view, and then uses dlt.apply_changes() to insert the view into a results table. My question is:

  1. why is my results table a view and not a table like I expected? (in another pipeline where dlt.apply_changes() is used, the target table is manifested as a table)
  2. if I create an empty results table ahead of time, then why does the pipeline complain that the table already exists?

@dlt.view(name = "cool_table")
def transform_uncool_table():
    return (spark.readStream
                           .option("ignoreChanges", "true")
                           .table("uncool_table")
    )
 
dlt.create_streaming_live_table(
    name = "target_table",
    table_properties={ "quality": "gold" }
)
 
dlt.apply_changes(
    target = "target_table",
    source = "cool_table",
    keys = ["pair_hash"],
    sequence_by = "last_seen"
) 

1 REPLY 1

Jfoxyyc
Valued Contributor

I find most of my apply_changes tables are being created as materialized views as well. They do recalculate at runtime, so they're up to date and behave a lot like a table, but they aren't tables in the same sense.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!