Data Skewnesss

dyusuf
New Contributor II

I am trying to visualize data skewness through a simple aggregation example by performing groupby operation on a dataframe, the data is skewed highly for one customer, but yet databricks is balancing it automatically when I check spark UI. Is there any configuration I need to disable to review the skewness in spark UI?

Please clarify.

 

Thanks,

Yusuf