Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2023 08:47 AM
Did you ever get an answer to this? It seems like a huge hole in the DLT feature set. My current DLT pipeline stops at the point where the data being outputted is distinct and REMOVEd records dropped. From there, I'm using a normal Delta pipeline to scan the table twice
- created a table with the source data's proprietary alpha-numeric IDs and an identity column (my new surrogate key, aka SK) as part of the table's DDL, where I append in new alpha IDs from the DLT where the alpha ID doesn't already exist in this target table. This ensures the sk to alpha ID relationship is immortalized
- created a table to warehouse all the data and joined the table above to it so that the output has a SK. The problem with this is that it's not a read_stream() operation so I'm literally doing a full overwrite. Very, very suboptimal...
Did you come up with a better solution for this problem?
The other challenge in this is that the output tables have some dimension data that doesn't natively have IDs. I have been forced to create my own dimension tables with their own SKs, then retroactively write the dimension's SK back into the target table. I don't mind doing this, but again the DLT seems ill-equipped to support this.