TomRenish
New Contributor III

Did you ever get an answer to this? It seems like a huge hole in the DLT feature set. My current DLT pipeline stops at the point where the data being outputted is distinct and REMOVEd records dropped. From there, I'm using a normal Delta pipeline to scan the table twice

  1. created a table with the source data's proprietary alpha-numeric IDs and an identity column (my new surrogate key, aka SK) as part of the table's DDL, where I append in new alpha IDs from the DLT where the alpha ID doesn't already exist in this target table. This ensures the sk to alpha ID relationship is immortalized
  2. created a table to warehouse all the data and joined the table above to it so that the output has a SK. The problem with this is that it's not a read_stream() operation so I'm literally doing a full overwrite. Very, very suboptimal...

Did you come up with a better solution for this problem?

The other challenge in this is that the output tables have some dimension data that doesn't natively have IDs. I have been forced to create my own dimension tables with their own SKs, then retroactively write the dimension's SK back into the target table. I don't mind doing this, but again the DLT seems ill-equipped to support this.