Relation between Driver and Executor size
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-17-2023 08:06 AM
Hi
I would like to ask for recommendations regarding the size of the driver and the amount of executors managed by that driver. I am aware of the best practices regarding executor size/number but I have doubts about the number of executors a single driver can manage. Is it worth investing on a bigger sized driver?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-17-2023 08:29 AM
Depends on your use case. The best is to connect Datatog and see driver and workers utilization https://docs.datadoghq.com/integrations/databricks/?tab=driveronly
Just from my experience, Usually, for big datasets, when need autoscale workers between 8 to 32. I set the driver two times bigger
My blog: https://databrickster.medium.com/
