Databricks cluster restarted inconsistantly causing job failure ( all purpose cluster)

SDas1 — Wed, 04 Oct 2023 21:24:32 GMT

"run failed with error message Driver of the cluster (0307-***-gpbwt) was restarted during the run.", "effectiveIntegrationRuntime": "vnet-ir-*-**** (East US) while performing merge operation.

This error is not consistant.

DB runtime: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12)

Summary

1-10 Workers16-160 GB Memory4-40 Cores

1 Driver16 GB Memory, 4 Cores

Runtime11.3.x-scala2.12

Standard_D4s_v5

Spark config:

spark.databricks.delta.preview.enabled true

topic Databricks cluster restarted inconsistantly causing job failure ( all purpose cluster) in Get Started Discussions

Databricks cluster restarted inconsistantly causing job failure ( all purpose cluster)

Summary