Hi Shreevani,

How are you doing today?, As per my understanding, you're really close, and it's awesome that you’re testing this out using MERGE for SCD Type 2 in your DLT pipeline. The main issue here is that you’re performing the merge() operation inside the DLT table function, but DLT expects the function to return a DataFrame—not run an action like merge(). So while your existing records are getting updated (because merge() executes), the new records aren't being picked up in the returned result, which is why you don't see the inserted row in your dim_customers table. A better approach would be to move the merge logic outside the DLT table function, or if you want to keep everything inside one function, you can skip using merge() and instead manually filter changed records and use unionByName() to return both old and new versions. That keeps it all within DLT’s expectations and still gets the SCD 2 behavior you want. Let me know if you’d like help writing that version—I’d be happy to help clean it up with you!

Regards,

Brahma