cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

break production using a shallow clone

oosterhuisf
New Contributor II

Hi,

If you create a shallow clone using the latest LTS, and drop the clone using a SQL warehouse (either current or preview), the source table is broken beyond repair. Data reads and writes still work, but vacuum will remain forever broken. I've attached a notebook that demonstrates this behaviour.

To me it looks like the 'drop table' command in the warehouse does not remove the reference to the source files.

Wanted solution:
* fixed `drop table` in the SQL Warehouse
* working delta table after `repair table`

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @oosterhuisf, When you create a shallow clone using the latest LTS and subsequently drop the clone using a SQL warehouse (either current or preview), the source table becomes irreparably broken. Although data reads and writes continue to function, the vacuum operation remains permanently impaired. It appears that the โ€˜drop tableโ€™ command in the warehouse fails to remove the reference to the source files.

 

Your desired solution involves:

  1. Addressing the issue with the drop table command in the SQL Warehouse.
  2. Ensuring that the delta table remains functional after executing the repair table.

For more detailed information on shallow and deep clones, you can refer to the official documentation for Azure Databricks SQL. Shallow clones copy only the metadata of the source table, while deep clones create an independent c.... Keep in mind that shallow clones are useful for scenarios like data migration, archiving, machine le....

 

If you encounter any further issues or need additional assistance, feel free to ask! ๐Ÿ˜Š

oosterhuisf
New Contributor II

To add to that: the manual does not state that this might happen

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group