Why does DLT CDC some time manifests the results table as a table and other times as a view?

Jennifer_Lu — Wed, 21 Dec 2022 19:37:30 GMT

I have a simple DLT pipeline that reads from an existing table, do some transformations, saves to a view, and then uses dlt.apply_changes() to insert the view into a results table. My question is:

why is my results table a view and not a table like I expected? (in another pipeline where dlt.apply_changes() is used, the target table is manifested as a table)
if I create an empty results table ahead of time, then why does the pipeline complain that the table already exists?

@dlt.view(name = "cool_table")
def transform_uncool_table():
    return (spark.readStream
                           .option("ignoreChanges", "true")
                           .table("uncool_table")
    )
 
dlt.create_streaming_live_table(
    name = "target_table",
    table_properties={ "quality": "gold" }
)
 
dlt.apply_changes(
    target = "target_table",
    source = "cool_table",
    keys = ["pair_hash"],
    sequence_by = "last_seen"
)

Re: Why does DLT CDC some time manifests the results table as a table and other times as a view?

Jfoxyyc — Thu, 29 Dec 2022 07:36:13 GMT

I find most of my apply_changes tables are being created as materialized views as well. They do recalculate at runtime, so they're up to date and behave a lot like a table, but they aren't tables in the same sense.

topic Why does DLT CDC some time manifests the results table as a table and other times as a view? in Data Engineering

Why does DLT CDC some time manifests the results table as a table and other times as a view?

Re: Why does DLT CDC some time manifests the results table as a table and other times as a view?