Hello community :).
I am currently implementing some pipelines using DLT. They are working great for my medalion architecture for landed json in bronze -> silver (using apply_changes) then materialized gold views ontop.
However, I am attempting to create a new silver table by merging into it from an existing silver table. I only want to insert one column (let's call it id1), then I would like to set a second column (id2) equal to my source column id1.
So in essence data would move like so:
SOURCE: id = A
-> MERGE
TARGET: id1 = A, id2 = A
Is this possible with apply_changes?
I have been able to achieve it traditionally using deltaTables and .merge:
silver_match.alias("t").merge(
silver_supporter.alias("s"), "s.id1 = t.source_supporter_id"
).whenMatchedUpdate(set={"source_supporter_id": "s.id1"}).whenNotMatchedInsert(
values={
"source_supporter_id": "s.id1",
"cbi_id": "s.cbi_id",
}
).execute()
if this isn't supported using apply_changes, are there any recommended methods (or just general suggestions) to being able to load this table with my dlt pipelines? If not, does anyone have experience of running hybrid dlt and non-dlt table pipelines.
Any help or suggestions of where I might be able to get more information would be greatly apprecitated!
Thanks,
Robbie