topic Setting shuffle partitions in Databricks SQL Warehouse in Data Engineering

Setting shuffle partitions in Databricks SQL Warehouse

stevewb — Mon, 07 Jul 2025 15:09:22 GMT

I think it used to be possible to set shuffle partitions in databricks sql warehouse through e.g.: SET spark.sql.shuffle.partitions=20000. However, when I run this now, I get the error:
[CONFIG_NOT_AVAILABLE] Configuration spark.sql.shuffle.partitions is not available. SQLSTATE: 42K0I
Has the ability to set shuffle partitions been removed as part of an update, or is there an alternative way of doing this?
I have several queries that would hugely benefit from a larger number of shuffle partitions, as I can see massive amounts of spill when checking the Spark UI.

Re: Setting shuffle partitions in Databricks SQL Warehouse

szymon_dybczak — Mon, 07 Jul 2025 15:29:22 GMT

Hi @stevewb ,

It's not available anymore. According with documentation:

" Databricks SQL allows admins to configure Spark properties for data access in the workspace settings menu. See Data access configurationsOther than data access configurations, Databricks SQL only allows a handful of Spark confs, which have been aliased to shorter names for simplicity. See Configuration parameters."

Set Spark configuration properties on Databricks | Databricks Documentation

So, using spark conf in Databricks SQL is limited.