cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to migrate Delta live table.

Therdpong
New Contributor III

I need guidance on migrating the Delta live table to another pipeline without a full refresh

1 ACCEPTED SOLUTION

Accepted Solutions

Hkesharwani
Contributor II

Hi, Yes you can use the same checkpoint location in the new pipeline.

Harshit Kesharwani
Data engineer at Rsystema

View solution in original post

5 REPLIES 5

Hkesharwani
Contributor II

Hi delta live tables are not stored in the metastore they are stored in specified storage location,  Changing the "Target" parameter in the  new pipeline settings will allow you to re-register the tables in a new schema without reprocessing the data. The data will remain in the same location specified in the "Storage location" parameter, and only the table registration in the Hive Metastore will be updated to point to the new schema.

Harshit Kesharwani
Data engineer at Rsystema

Therdpong
New Contributor III

Can i migrate the checkpoint to a new pipeline?

Hkesharwani
Contributor II

Hi, Yes you can use the same checkpoint location in the new pipeline.

Harshit Kesharwani
Data engineer at Rsystema

Thank you very much

Again please, I remove the table from the old pipeline. but the new pipeline was showing an error already from the old pipeline.


Therdpong_0-1717772462140.png

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group