Setting shuffle partitions in Databricks SQL Warehouse

stevewb
New Contributor III

‎07-07-2025 08:09 AM

I think it used to be possible to set shuffle partitions in databricks sql warehouse through e.g.: SET spark.sql.shuffle.partitions=20000. However, when I run this now, I get the error:
[CONFIG_NOT_AVAILABLE] Configuration spark.sql.shuffle.partitions is not available. SQLSTATE: 42K0I
Has the ability to set shuffle partitions been removed as part of an update, or is there an alternative way of doing this?
I have several queries that would hugely benefit from a larger number of shuffle partitions, as I can see massive amounts of spill when checking the Spark UI.

0 Kudos

szymon_dybczak
Esteemed Contributor III

‎07-07-2025 08:29 AM

Hi @stevewb ,

It's not available anymore. According with documentation:

" Databricks SQL allows admins to configure Spark properties for data access in the workspace settings menu. See Data access configurationsOther than data access configurations, Databricks SQL only allows a handful of Spark confs, which have been aliased to shorter names for simplicity. See Configuration parameters."

Set Spark configuration properties on Databricks | Databricks Documentation

So, using spark conf in Databricks SQL is limited.

0 Kudos