cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Why does DLT CDC some time manifests the results table as a table and other times as a view?

Jennifer_Lu
New Contributor III

I have a simple DLT pipeline that reads from an existing table, do some transformations, saves to a view, and then uses dlt.apply_changes() to insert the view into a results table. My question is:

  1. why is my results table a view and not a table like I expected? (in another pipeline where dlt.apply_changes() is used, the target table is manifested as a table)
  2. if I create an empty results table ahead of time, then why does the pipeline complain that the table already exists?

@dlt.view(name = "cool_table")
def transform_uncool_table():
    return (spark.readStream
                           .option("ignoreChanges", "true")
                           .table("uncool_table")
    )
 
dlt.create_streaming_live_table(
    name = "target_table",
    table_properties={ "quality": "gold" }
)
 
dlt.apply_changes(
    target = "target_table",
    source = "cool_table",
    keys = ["pair_hash"],
    sequence_by = "last_seen"
) 

1 REPLY 1

Jfoxyyc
Valued Contributor

I find most of my apply_changes tables are being created as materialized views as well. They do recalculate at runtime, so they're up to date and behave a lot like a table, but they aren't tables in the same sense.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.