topic Re: How to migrate Git repos with DLT configurations in Data Engineering

How to migrate Git repos with DLT configurations

PrebenOlsen — Mon, 22 Apr 2024 12:52:13 GMT

Hi!
I want to migrate all my databricks related code from one github repo to another. I knew this wouldn't be straight forward. When I copy my code for one DLT, I get the error

org.apache.spark.sql.catalyst.ExtendedAnalysisException: Table 'vessel_battery' is already managed by pipeline <pipeline ID>. A table can only be owned by one pipeline. Concurrent pipeline operations such as maintenance and full refresh will conflict with each other. Please rename the table 'vessel_battery' to proceed.

The tables are multi-terrabytes of data and should not be removed/overwritten. Is it possible to transfer the ownership of these tables? All DLT configs are setup using Databricks Asset Bundles.

Re: How to migrate Git repos with DLT configurations

AmanSehgal — Mon, 22 Apr 2024 13:27:14 GMT

Would cloning the tables help your cause? You could probably try shallow or deep cloning as per your requirement.

Re: How to migrate Git repos with DLT configurations

PrebenOlsen — Mon, 29 Apr 2024 08:24:35 GMT

Does cloning take considerably less time then recreating the tables?

Can I resume append operations to a cloned table?