Databricks Setting Dynamic Local Configuration Properties
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-21-2024 11:45 AM
It seems that Databricks is somehow setting the properties of local spark configurations for each notebook. Can someone point me to exactly how and where this is being done?
I would like to set the scheduler to utilize a certain pool by default, but it seems that the property is being set to some dynamic integer value - and ends up using FIFO - rather than just utilizing the default pool. The only solution currently seems to be manually setting the property itself in each notebook either through explicit code for it or importing code that does it, but I'd really prefer something I can make automatic through something like a configuration file.