topic Re: Merge version data files of Delta table in Get Started Discussions

Merge version data files of Delta table

Data_Analytics1 — Thu, 12 Oct 2023 09:27:11 GMT

Hi,

I am having one CDC enabled Delta table. In 256th version, table is having 50 data files. I want all to merge and create a single file. How can I merge all 50 data file and when I query for 256th version, I should get 1 data file? Is there any command which can optimize the file size?

Re: Merge version data files of Delta table

Data_Analytics1 — Mon, 16 Oct 2023 13:22:31 GMT

Hi @Retired_mod ,

Here 256th version data files is of the CDF where I am querying the data for this version by following code:

display(spark.read.format("delta") \ .option("readChangeFeed", "true") \ .option("startingVersion", 256) \ .option("endingVersion", 256) \ .table("catalogName.schemaName.tableName"))

Current version of the table is 300 but I wants to see what are the updates in 256th version. But what I found is that, this version is having 50 data files, reading which my code is displaying the data to me. As it is having 50 files, it is taking time to return the result. How to optimize the 256th version of the table when my current table version is 300?

OPTIMIZE command will optimize the current version (snapshot) of the table and creates the new version with lesser number of file.

Re: Merge version data files of Delta table

Debayan — Tue, 17 Oct 2023 04:44:38 GMT

Hi, ae you talking about merging CSV files? https://community.databricks.com/t5/machine-learning/merge-12-csv-files-in-databricks/td-p/3551#:~:text=Use%20Union()%20method%20to,from%20the%20specified%20set%2Fs.