cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta tables background

SaraCorralLou
New Contributor III

Hi,

Looking at the delta tables and how they are stored I have a question.

If the delta tables are stored as parquet files in ADLS why if I copy/paste/rename a folder that corresponds to an existing table in the same location/database this does not generate a copy of the previous table but with another name? What happens in the backgroud? 

Thank you in advanced!

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

@Sara Corral​ :

When you copy/paste/rename a folder that corresponds to an existing delta table in the same location/database, it does not generate a copy of the previous table because delta tables are not just plain parquet files. They have additional metadata that is stored in a hidden folder called "_delta_log". This metadata includes information about the transaction log, version history, and other metadata that allows Delta Lake to perform ACID transactions on the table.

When you copy/paste/rename a folder, you are essentially creating a new folder with a new name, but you are not copying the "_delta_log" folder that contains the table's metadata. Without this metadata, the new folder is not recognized as a Delta table, and you cannot query it or interact with it as a Delta table. In order to create a copy of a Delta table with a different name, you need to use the Delta Lake

COPY command or other Delta-specific tools to ensure that the metadata is also copied and updated appropriately. Simply copying the folder will not work.

View solution in original post

1 REPLY 1

Anonymous
Not applicable

@Sara Corral​ :

When you copy/paste/rename a folder that corresponds to an existing delta table in the same location/database, it does not generate a copy of the previous table because delta tables are not just plain parquet files. They have additional metadata that is stored in a hidden folder called "_delta_log". This metadata includes information about the transaction log, version history, and other metadata that allows Delta Lake to perform ACID transactions on the table.

When you copy/paste/rename a folder, you are essentially creating a new folder with a new name, but you are not copying the "_delta_log" folder that contains the table's metadata. Without this metadata, the new folder is not recognized as a Delta table, and you cannot query it or interact with it as a Delta table. In order to create a copy of a Delta table with a different name, you need to use the Delta Lake

COPY command or other Delta-specific tools to ensure that the metadata is also copied and updated appropriately. Simply copying the folder will not work.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group