cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Why does DLT CDC some time manifests the results table as a table and other times as a view?

Jennifer_Lu
New Contributor III

I have a simple DLT pipeline that reads from an existing table, do some transformations, saves to a view, and then uses dlt.apply_changes() to insert the view into a results table. My question is:

  1. why is my results table a view and not a table like I expected? (in another pipeline where dlt.apply_changes() is used, the target table is manifested as a table)
  2. if I create an empty results table ahead of time, then why does the pipeline complain that the table already exists?

@dlt.view(name = "cool_table")
def transform_uncool_table():
    return (spark.readStream
                           .option("ignoreChanges", "true")
                           .table("uncool_table")
    )
 
dlt.create_streaming_live_table(
    name = "target_table",
    table_properties={ "quality": "gold" }
)
 
dlt.apply_changes(
    target = "target_table",
    source = "cool_table",
    keys = ["pair_hash"],
    sequence_by = "last_seen"
) 

1 REPLY 1

Jfoxyyc
Valued Contributor

I find most of my apply_changes tables are being created as materialized views as well. They do recalculate at runtime, so they're up to date and behave a lot like a table, but they aren't tables in the same sense.