Joining tables across DLT pipelines

[Mkk1]

How can I join a silver table (s1) from a DLT pipeline (D1) to another silver table (S2) from a different DLT pipeline (D2)?

#DLT #DeltaLiveTables

[Kaniz]

Hi @Mkk1, To join a silver table from one Delta Live Tables (DLT) pipeline to another silver table from a different DLT pipeline, you can follow these steps:

1. Read the Silver Tables:

   • In your DLT pipeline code, read the silver tables you want to join. For example:

     @dlt.table(comment="Joining Silver Tables")
     def my_gold_table():
         silver_one = dlt.read("silver_table_one")
         silver_two = dlt.read("silver_table_two")
     

2. Perform the Join:

   • Once you have both silver tables (silver_one and silver_two), you can perform the join operation based on the desired columns. Use appropriate join keys or conditions to combine the data.

3. Write the Resulting Table:

   • After joining the tables, create a new table (e.g., a gold table) that contains the combined data. You can write this resulting table back to your DLT pipeline.

Remember to adjust the table names (silver_table_one and silver_table_two) and any other specifics according to your actual DLT pipeline configurations. Happy joining! 🚀🔗