Delta file question

[apiury]

Hi! Im using Autoloader to ingest Binary files into delta format. I have 7 binary files but delta generate 3 files and the format is part-0000, part-0001... Why generate this files with format part-000...

[Lakshay]

Hi @Alejandro Piury Pinzón​ , The Delta table manages the size of the file being written to the table. The no. of files being written in the Delta table depends upon the total volume of the data being written to the table and not the no. of files at the source location.

The file format part-000 is generated because of the use of a hash algorithm to divide the no. of rows into different files.

[apiury]

But i don't understand. For example, i have 3 files:

 

When i upload the files using autoloader, 3 files are generated:

Why databricks doesn't put them all into 1 file?

[Lakshay]

As spark processes the data by dividing the data into multiple partitions, so when writing the data no. of part files created will be equal to no. of partitions. If you are doing this outside Autoloader, you can use coalesce to control the no. of partitions but in Autolader, I am not sure if we can use coalesce.

However, you can run optimize command on the delta table to compact the file.

[Anonymous]

Hi @Alejandro Piury Pinzón​ 

We haven't heard from you since the last response from @Lakshay Goel​ r​, and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others. 

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.