topic Re: How to migrate Delta live table. in Data Engineering

How to migrate Delta live table.

Therdpong — Fri, 07 Jun 2024 06:29:32 GMT

I need guidance on migrating the Delta live table to another pipeline without a full refresh

Re: How to migrate Delta live table.

Hkesharwani — Fri, 07 Jun 2024 06:45:33 GMT

Hi delta live tables are not stored in the metastore they are stored in specified storage location, Changing the "Target" parameter in the new pipeline settings will allow you to re-register the tables in a new schema without reprocessing the data. The data will remain in the same location specified in the "Storage location" parameter, and only the table registration in the Hive Metastore will be updated to point to the new schema.

Re: How to migrate Delta live table.

Therdpong — Fri, 07 Jun 2024 06:49:47 GMT

Can i migrate the checkpoint to a new pipeline?

Re: How to migrate Delta live table.

Hkesharwani — Fri, 07 Jun 2024 07:23:48 GMT

Hi, Yes you can use the same checkpoint location in the new pipeline.

Re: How to migrate Delta live table.

Therdpong — Fri, 07 Jun 2024 09:11:30 GMT

Thank you very much

Re: How to migrate Delta live table.

Therdpong — Fri, 07 Jun 2024 15:04:21 GMT

Again please, I remove the table from the old pipeline. but the new pipeline was showing an error already from the old pipeline.