Delta tables background

SaraCorralLou — Thu, 06 Apr 2023 09:24:14 GMT

Hi,

Looking at the delta tables and how they are stored I have a question.

If the delta tables are stored as parquet files in ADLS why if I copy/paste/rename a folder that corresponds to an existing table in the same location/database this does not generate a copy of the previous table but with another name? What happens in the backgroud?

Thank you in advanced!

Re: Delta tables background

Anonymous — Fri, 07 Apr 2023 02:02:10 GMT

@Sara Corral :

When you copy/paste/rename a folder that corresponds to an existing delta table in the same location/database, it does not generate a copy of the previous table because delta tables are not just plain parquet files. They have additional metadata that is stored in a hidden folder called "_delta_log". This metadata includes information about the transaction log, version history, and other metadata that allows Delta Lake to perform ACID transactions on the table.

When you copy/paste/rename a folder, you are essentially creating a new folder with a new name, but you are not copying the "_delta_log" folder that contains the table's metadata. Without this metadata, the new folder is not recognized as a Delta table, and you cannot query it or interact with it as a Delta table. In order to create a copy of a Delta table with a different name, you need to use the Delta Lake

COPY command or other Delta-specific tools to ensure that the metadata is also copied and updated appropriately. Simply copying the folder will not work.

topic Re: Delta tables background in Data Engineering

Delta tables background

Re: Delta tables background