Louis_Frolio
Databricks Employee
Databricks Employee

You will need to leverage cluster-level Spark configurations or global init scripts.  This will allow you to set "spark.scheduler.poo" property automatically for all workloads on the cluster.

You can try navigationg to "Compute", select the cluster you want to modify, click edit then look under "Advanced Options." Expand the "Spark" tab and add "spark.scheduler.pool=<your pool name>".

You can verify your setting by running the following command in a notebook that is attached to your cluster:

spark.sparkContext.setLocalProperty("spark.scheduler.pool", "<your pool name>"

Cheers, Louis.