Relation between Driver and Executor size

[alvaro_databric]

Hi

I would like to ask for recommendations regarding the size of the driver and the amount of executors managed by that driver. I am aware of the best practices regarding executor size/number but I have doubts about the number of executors a single driver can manage. Is it worth investing on a bigger sized driver?

[Hubert-Dudek]

Depends on your use case. The best is to connect Datatog and see driver and workers utilization https://docs.datadoghq.com/integrations/databricks/?tab=driveronly

Just from my experience, Usually, for big datasets, when need autoscale workers between 8 to 32. I set the driver two times bigger