Enabling Fair Scheduler from Databricks Notebook
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2022 06:15 AM
How to enable fair scheduler from Databricks notebook using python commands?
- Labels:
-
Databricks notebook
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2022 08:21 AM
spark.conf.set("spark.scheduler.mode", "FAIR")
sc.setLocalProperty("spark.scheduler.pool", "pool1")Please use the above conf setting to enable FAIR (round robin) mode instead of default FIFO.
Using the above example, you can also set a separate pool for each job/notebook.
My blog: https://databrickster.medium.com/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-02-2022 04:38 AM
Hi @Hariharan Sambath
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!
