topic Re: Enabling Fair Scheduler from Databricks Notebook in Data Engineering

Enabling Fair Scheduler from Databricks Notebook

HariharaSam — Tue, 20 Sep 2022 13:15:39 GMT

How to enable fair scheduler from Databricks notebook using python commands?

Re: Enabling Fair Scheduler from Databricks Notebook

Hubert-Dudek — Tue, 20 Sep 2022 15:21:03 GMT

spark.conf.set("spark.scheduler.mode", "FAIR")
sc.setLocalProperty("spark.scheduler.pool", "pool1")

Please use the above conf setting to enable FAIR (round robin) mode instead of default FIFO.

Using the above example, you can also set a separate pool for each job/notebook.

Re: Enabling Fair Scheduler from Databricks Notebook

Anonymous — Sun, 02 Oct 2022 11:38:19 GMT

Hi @Hariharan Sambath

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!