cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta tables background

SaraCorralLou
New Contributor III

Hi,

Looking at the delta tables and how they are stored I have a question.

If the delta tables are stored as parquet files in ADLS why if I copy/paste/rename a folder that corresponds to an existing table in the same location/database this does not generate a copy of the previous table but with another name? What happens in the backgroud? 

Thank you in advanced!

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

@Sara Corral​ :

When you copy/paste/rename a folder that corresponds to an existing delta table in the same location/database, it does not generate a copy of the previous table because delta tables are not just plain parquet files. They have additional metadata that is stored in a hidden folder called "_delta_log". This metadata includes information about the transaction log, version history, and other metadata that allows Delta Lake to perform ACID transactions on the table.

When you copy/paste/rename a folder, you are essentially creating a new folder with a new name, but you are not copying the "_delta_log" folder that contains the table's metadata. Without this metadata, the new folder is not recognized as a Delta table, and you cannot query it or interact with it as a Delta table. In order to create a copy of a Delta table with a different name, you need to use the Delta Lake

COPY command or other Delta-specific tools to ensure that the metadata is also copied and updated appropriately. Simply copying the folder will not work.

View solution in original post

1 REPLY 1

Anonymous
Not applicable

@Sara Corral​ :

When you copy/paste/rename a folder that corresponds to an existing delta table in the same location/database, it does not generate a copy of the previous table because delta tables are not just plain parquet files. They have additional metadata that is stored in a hidden folder called "_delta_log". This metadata includes information about the transaction log, version history, and other metadata that allows Delta Lake to perform ACID transactions on the table.

When you copy/paste/rename a folder, you are essentially creating a new folder with a new name, but you are not copying the "_delta_log" folder that contains the table's metadata. Without this metadata, the new folder is not recognized as a Delta table, and you cannot query it or interact with it as a Delta table. In order to create a copy of a Delta table with a different name, you need to use the Delta Lake

COPY command or other Delta-specific tools to ensure that the metadata is also copied and updated appropriately. Simply copying the folder will not work.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.