Databricks Community

Data_Analytics1 · ‎10-12-2023

Hi,

I am having one CDC enabled Delta table. In 256th version, table is having 50 data files. I want all to merge and create a single file. How can I merge all 50 data file and when I query for 256th version, I should get 1 data file? Is there any command which can optimize the file size?

Data_Analytics1 · ‎10-16-2023

Hi @Retired_mod ,

Here 256th version data files is of the CDF where I am querying the data for this version by following code:

display(spark.read.format("delta") \
  .option("readChangeFeed", "true") \
  .option("startingVersion", 256) \
  .option("endingVersion", 256) \
  .table("catalogName.schemaName.tableName"))

Current version of the table is 300 but I wants to see what are the updates in 256th version. But what I found is that, this version is having 50 data files, reading which my code is displaying the data to me. As it is having 50 files, it is taking time to return the result. How to optimize the 256th version of the table when my current table version is 300?

OPTIMIZE command will optimize the current version (snapshot) of the table and creates the new version with lesser number of file.