cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Can we do a shallow clone on top of a shallow clone!

damodhargandha
New Contributor

Case 1 Can we do a shallow clone on top of a shallow clone? If I do so, What would be the result 
Case 2  when ever the table over writes with new data how does it work.
Case 3  when a the table is dropped and loaded then how does this scenario effect the shallow cloned table.

1 ACCEPTED SOLUTION

Accepted Solutions

Ashwin_DSA
Databricks Employee
Databricks Employee

Hi @damodhargandha,

A good way to think about a shallow clone is that it copies the table’s metadata but still points to the source table's data files rather than copying them. Databricks explains that behaviour in the Clone a table " section of the Databricks docs.

For your first question, no, you can’t create a shallow clone from another shallow clone. That’s explicitly called out as a limitation for Unity Catalog shallow clones in the Shallow clone for Unity Catalog tables documentation, and you’ll also see the same restriction reflected in the public error-class docs.

For the second question, if the source table is overwritten with new data, the existing shallow clone does not automatically "move forward" with it. A clone has its own independent history, and time travel on the clone is separate from the source. In other words, the shallow clone continues to represent the snapshot/version it was created from unless you run clone again. The general clone docs cover that behaviour, including the fact that shallow clones reference source files and that cloned tables have an independent history. One important nuance here is file cleanup... outside Unity Catalog, if old source files are later removed by VACUUM, the shallow clone can break with FileNotFoundException. Databricks recommends rerunning the clone with replace in that scenario. In Unity Catalog, VACUUM is shallow-clone aware and tracks referenced files across the source and its clones, so vacuuming the source does not break the clone the same way. See the general clone docs and the Unity Catalog shallow clone docs.

For the third question, if the base table is dropped, the outcome depends on the table type and catalog. For Unity Catalog managed tables, Databricks blocks the drop by default if shallow clones still reference the base table, unless you explicitly use DROP TABLE ... FORCE. If you force the drop, the referencing shallow clones become broken for operations such as SELECT, INSERT, UPDATE, DESCRIBE HISTORY, and CLONE. That behaviour is documented in both DROP TABLE and Shallow clone for Unity Catalog tables. There is one managed-table nuance. Unity Catalog supports UNDROP TABLE for about 7 days, and during that window, managed shallow clones can continue to work if the source is restored. If it is not restored, the shallow clone stops functioning once garbage collection removes the underlying files.

If the source is a Unity Catalog external table, the story is different because DROP TABLE removes the metadata but does not delete the underlying external files, so shallow clones of external tables are not impacted in the same way. Also worth calling out: for Unity Catalog shallow clones, you can’t use REPLACE or CREATE OR REPLACE to overwrite an existing shallow clone; the documented approach is to drop the shallow clone and create it again.

Trust this answers your queries.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***

View solution in original post

1 REPLY 1

Ashwin_DSA
Databricks Employee
Databricks Employee

Hi @damodhargandha,

A good way to think about a shallow clone is that it copies the table’s metadata but still points to the source table's data files rather than copying them. Databricks explains that behaviour in the Clone a table " section of the Databricks docs.

For your first question, no, you can’t create a shallow clone from another shallow clone. That’s explicitly called out as a limitation for Unity Catalog shallow clones in the Shallow clone for Unity Catalog tables documentation, and you’ll also see the same restriction reflected in the public error-class docs.

For the second question, if the source table is overwritten with new data, the existing shallow clone does not automatically "move forward" with it. A clone has its own independent history, and time travel on the clone is separate from the source. In other words, the shallow clone continues to represent the snapshot/version it was created from unless you run clone again. The general clone docs cover that behaviour, including the fact that shallow clones reference source files and that cloned tables have an independent history. One important nuance here is file cleanup... outside Unity Catalog, if old source files are later removed by VACUUM, the shallow clone can break with FileNotFoundException. Databricks recommends rerunning the clone with replace in that scenario. In Unity Catalog, VACUUM is shallow-clone aware and tracks referenced files across the source and its clones, so vacuuming the source does not break the clone the same way. See the general clone docs and the Unity Catalog shallow clone docs.

For the third question, if the base table is dropped, the outcome depends on the table type and catalog. For Unity Catalog managed tables, Databricks blocks the drop by default if shallow clones still reference the base table, unless you explicitly use DROP TABLE ... FORCE. If you force the drop, the referencing shallow clones become broken for operations such as SELECT, INSERT, UPDATE, DESCRIBE HISTORY, and CLONE. That behaviour is documented in both DROP TABLE and Shallow clone for Unity Catalog tables. There is one managed-table nuance. Unity Catalog supports UNDROP TABLE for about 7 days, and during that window, managed shallow clones can continue to work if the source is restored. If it is not restored, the shallow clone stops functioning once garbage collection removes the underlying files.

If the source is a Unity Catalog external table, the story is different because DROP TABLE removes the metadata but does not delete the underlying external files, so shallow clones of external tables are not impacted in the same way. Also worth calling out: for Unity Catalog shallow clones, you can’t use REPLACE or CREATE OR REPLACE to overwrite an existing shallow clone; the documented approach is to drop the shallow clone and create it again.

Trust this answers your queries.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***