afaik that is correct, if the driver dies your job will fail.

Also check this topic.

There are ways to run spark in HA, but I don't think it is possible on Databricks at the moment:

https://gist.github.com/aseigneurin/3af6b228490a8deab519c6aea2c209bc

If you absolutely need HA for the master/driver, I'd reach out to Databricks support.

FWIW: I do not encounter any issues with masters/drivers dying, unless I write bad code (hammering the driver with a lot of data). The spark driver itself is pretty relaxed normally. The workers/executors are stuffed with work.