According to https://docs.databricks.com/jobs.html#jar-job-tips

Job output, such as log output emitted to stdout, is subject to a 20MB size limit. If the total output has a larger size, the run will be canceled and marked as failed.

That was my problem, to "fix it" I've just set the logging level to ERROR

val sc = SparkContext.getOrCreate(conf)

sc.setLogLevel("ERROR")

It was solved

According to https://docs.databricks.com/jobs.html#jar-job-tips:

"Job output, such as log output emitted to stdout, is subject to a 20MB size limit. If the total output has a larger size, the run will be canceled and marked as failed."

That was my problem, to "fix it" I've just set the logging level to ERROR

val sc = SparkContext.getOrCreate(conf)

sc.setLogLevel("ERROR")

This workaround works for me

I am facing the same error but the log output to stdout is not an issue as the log file size turns out to be < 2 MB. So that issue is ruled out. Moreover, our job is dummy for testing purposes and is not doing any memory intensive operation. Its purely running a simple thread that keeps on logging to the stdout every 5 mins.

Still the cluster is getting timed out.

Below is the post i have submitted on stack overflow.

https://stackoverflow.com/questions/59820940/databricks-job-timed-out-with-error-lost-executor-0-on-...