Confused with databricks Tips and Tricks - Optimizations regarding partitining

[eimis_pacheco]

Hello Community,

Today I was in Tips and Tricks - Optimizations webinar and I started being confused, they said:

"Don't partition tables <1TB in size and plan carefully when partitioning

• Partitions should be >=1GB"

Now my confusion is if this recommendation is given for storing the data on disk while writing it at the end of the spark job? or does this apply when we are running a job and we are doing some transformations and we want to split this data into more executors so that this run faster? I mean if I want to partition a table that is close to 1TB to make the job faster by splitting the same let's say in 10 executors should I not do this?

Thank you in advance for the clarification.

Thanks

#dataengineering

[eimis_pacheco]

Hi @Retired_mod ,

Sorry for asking again but this recommendation of "don't partition tables <1TB" but this recommendation applies when this is written on disk or when the job is actually running in memory too? (sorry if the answer is obvious)

Because in clusters we have GBs of memory to allocate to our datasets not TBs.

Thank you once more time

[mohitmanna]

Hi @Retired_mod Thanks for the tricks. 
I have a table where roughly 1.7 TB data is being ingested daily. 
I have partitioned it on date. In s3 I can see the size of files for one day in parquet format as 1.7 TB. 
Should I change the partition values ? Is the partition size too big?

[-werners-]

that is partitions on disk.

Defining the correct amount of partitions is not that easy.  One would think that more partitions is better because you can process more data in parallel.
And that is true if you only have to do local transformations (no shuffle needed).
But that is almost never the case.
When a shuffle is applied, having more partitions means more overhead.  That is why the recommendation is to not partition below 1TB (that is only a recommendation though, you might have cases where partitioning makes sense with smaller data).

Recently there is liquid clustering, which makes things easier (no partitioning necessary), but it only works for delta lake and recent databricks versions.