topic Re: Data Skewnesss in Data Engineering

Data Skewnesss

dyusuf — Sat, 15 Mar 2025 08:35:29 GMT

I am trying to visualize data skewness through a simple aggregation example by performing groupby operation on a dataframe, the data is skewed highly for one customer, but yet databricks is balancing it automatically when I check spark UI. Is there any configuration I need to disable to review the skewness in spark UI?

Please clarify.

Thanks,

Yusuf

Re: Data Skewnesss

SantoshJoshi — Sat, 15 Mar 2025 14:46:33 GMT

Hi @dyusuf ,

It could be because AQE (Adaptive Query Execution) is enabled.

...AQE, dynamically handles skew...

Please refer below link for more details:

https://docs.databricks.com/aws/en/optimizations/aqe

Can you please disable AQE and check if this works?

spark.conf.set("spark.sql.adaptive.enabled", "false")

HTH

Re: Data Skewnesss

dyusuf — Sun, 16 Mar 2025 04:44:22 GMT

Thankyou for your response. I already tried disabling AQE, yet it doesnt work. Any other way we could see it?