cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Why does DLT CDC some time manifests the results table as a table and other times as a view?

Jennifer_Lu
New Contributor III

I have a simple DLT pipeline that reads from an existing table, do some transformations, saves to a view, and then uses dlt.apply_changes() to insert the view into a results table. My question is:

  1. why is my results table a view and not a table like I expected? (in another pipeline where dlt.apply_changes() is used, the target table is manifested as a table)
  2. if I create an empty results table ahead of time, then why does the pipeline complain that the table already exists?

@dlt.view(name = "cool_table")
def transform_uncool_table():
    return (spark.readStream
                           .option("ignoreChanges", "true")
                           .table("uncool_table")
    )
 
dlt.create_streaming_live_table(
    name = "target_table",
    table_properties={ "quality": "gold" }
)
 
dlt.apply_changes(
    target = "target_table",
    source = "cool_table",
    keys = ["pair_hash"],
    sequence_by = "last_seen"
) 

1 REPLY 1

Jfoxyyc
Valued Contributor

I find most of my apply_changes tables are being created as materialized views as well. They do recalculate at runtime, so they're up to date and behave a lot like a table, but they aren't tables in the same sense.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group