topic Re: Small/big file problem, how do you fix it ? in Data Engineering

Small/big file problem, how do you fix it ?

William_Scardua — Thu, 07 Oct 2021 01:13:12 GMT

How do you work to fixing the small/big file problem ? what you suggest ?

Re: Small/big file problem, how do you fix it ?

jose_gonzalez — Thu, 07 Oct 2021 16:54:26 GMT

Hi @William Scardua ,

I will recommend to use Delta to avoid having small/big files issues. For example, Auto Optimize is an optional set of features that automatically compact small files during individual writes to a Delta table. Paying a small cost during writes offers significant benefits for tables that are queried actively. For more details and examples please check the following link

Auto optimize will create files of 128 MB each. If you would like to compress and optimize further, then I will recommend to use "Optimize" command on your Delta tables. It will compress and create files of 1 GB in size, by default. For more details on this optimize feature, please check the following link

Thank you.

Re: Small/big file problem, how do you fix it ?

-werners- — Fri, 08 Oct 2021 07:01:20 GMT

What Jose said.

If you cannot use delta or do not want to:

the use of coalesce and repartition/partitioning is the way to define the file size.

There is no one ideal file size. It all depends on the use case, available cluster size, data flow downstream etc.

What you do want to avoid is a lot of small files (think only a few megabytes or kilobytes).

But there is nothing wrong with a single file of 2 MB.

That being said: delta lake makes this exercise way easier.

Re: Small/big file problem, how do you fix it ?

William_Scardua — Mon, 11 Oct 2021 21:14:08 GMT

Okay @Jose Gonzalez , I understand .. thank you man

Re: Small/big file problem, how do you fix it ?

William_Scardua — Mon, 11 Oct 2021 21:16:57 GMT

thank you for feedback @Werner Stinckens , that`s a good point