Run more than nr-of-cores concurrent tasks.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2021 05:46 AM
We are using the terraform databricks provier, which is starting a cluster and checking every mount (since there is no mount rest API!). Each mount takes 20 seconds to check, and 99.9% of that time is idle waiting, and it starts a job per mount. If we could run many (more than nr of cores) jobs concurrently we should be able to make it faster, but I cant find how to do this. I have tried setting
`spark.executor.instances` to 2*cores, but it seems to be ignored.
So, is it possible to set databricks to use more spark executors than nr of cores?
Labels:
- Labels:
-
Tasks