cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Migrating DLT tables from TEST to PROD catalogs

BF7
Contributor

Why can't we just copy all the DLT tables and materialized views from one UC catalog to another to get the historical data in place and then run the DLT pipelines on those UC tables?

We are migrating many very large tables from our TEST catalog to our PROD catalog. They are all tables generated by and owned by DLT processes. It is expensive to reprocess our entire dataset. Its a big dataset. So we would like to avoid it.

If it is not possible to do this, and the tables do indeed have to have "Full Refresh All" performed, can someone tell me exactly why? I realize it has something to do with how the DLT API interfaces with the underlying UC table metadata, but I need specifics about what exactly is going on under the hood. Can anyone help me?

1 REPLY 1

nikhilj0421
Databricks Employee
Databricks Employee

Hi @BF7, we do not support moving the streaming table yet. If we clone the DLT streaming table, it will get converted into a normal delta table instead of a streaming table. In that condition, we need to go with the "full refresh all" option and indeed it is a costly one. 

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now