Now that delta live pipeline is GA we are looking to convert our existing processes to leverage it. One thing that remains unclear is how to populate new delta live tables with historical data?
Currently we are looking to use CDC by leveraging create_target_table to apply_changes into a bronze and a silver layer to keep history going forward. When trying to merge into the create_target_table outside of the DLT pipeline I get an error saying it must be a delta table and not a view.
I have also attempted drop view and recreate as a managed delta table. I am able to populate this table with the historical data but cannot use it in the DLT pipeline.
The other option I am considering is having the DLT pipeline execute a different set of code that pulls from the delta tables once and then convert to the daily code afterwards.
We have ~150m rows in a delta table that we would like to incorporate into the DLT pipeline. How can we populate the DLT silver and bronze layer with historical data from a managed delta tables? I would like to avoid running the entire ETL process for all rows. Thanks!
