I have a simple DLT pipeline that reads from an existing table, do some transformations, saves to a view, and then uses dlt.apply_changes() to insert the view into a results table. My question is:
- why is my results table a view and not a table like I expected? (in another pipeline where dlt.apply_changes() is used, the target table is manifested as a table)
- if I create an empty results table ahead of time, then why does the pipeline complain that the table already exists?
@dlt.view(name = "cool_table")
def transform_uncool_table():
return (spark.readStream
.option("ignoreChanges", "true")
.table("uncool_table")
)
dlt.create_streaming_live_table(
name = "target_table",
table_properties={ "quality": "gold" }
)
dlt.apply_changes(
target = "target_table",
source = "cool_table",
keys = ["pair_hash"],
sequence_by = "last_seen"
)