"run failed with error message Driver of the cluster (0307-***-gpbwt) was restarted during the run.", "effectiveIntegrationRuntime": "vnet-ir-*-**** (East US) while performing merge operation.
This error is not consistant.
DB runtime: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12)
Summary
1-10 Workers16-160 GB Memory4-40 Cores
1 Driver16 GB Memory, 4 Cores
Runtime11.3.x-scala2.12
Standard_D4s_v5
Spark config:
spark.databricks.delta.preview.enabled true