topic Re: Delta file question in Data Engineering

Delta file question

apiury — Thu, 22 Jun 2023 10:39:47 GMT

Hi! Im using Autoloader to ingest Binary files into delta format. I have 7 binary files but delta generate 3 files and the format is part-0000, part-0001... Why generate this files with format part-000...

Re: Delta file question

Lakshay — Thu, 22 Jun 2023 13:13:43 GMT

Hi @Alejandro Piury Pinzón , The Delta table manages the size of the file being written to the table. The no. of files being written in the Delta table depends upon the total volume of the data being written to the table and not the no. of files at the source location.

The file format part-000 is generated because of the use of a hash algorithm to divide the no. of rows into different files.

Re: Delta file question

apiury — Thu, 22 Jun 2023 13:36:08 GMT

But i don't understand. For example, i have 3 files:

When i upload the files using autoloader, 3 files are generated:

Why databricks doesn't put them all into 1 file?

Re: Delta file question

Lakshay — Thu, 22 Jun 2023 18:05:03 GMT

As spark processes the data by dividing the data into multiple partitions, so when writing the data no. of part files created will be equal to no. of partitions. If you are doing this outside Autoloader, you can use coalesce to control the no. of partitions but in Autolader, I am not sure if we can use coalesce.

However, you can run optimize command on the delta table to compact the file.

Re: Delta file question

Anonymous — Fri, 23 Jun 2023 05:24:32 GMT

Hi @Alejandro Piury Pinzón

We haven't heard from you since the last response from @Lakshay Goel r, and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.