Hi!

We are using Databricks on Azure on production since about 3 months. A big part of what we use Databricks for is processing data using a workflow with various Python notebooks. We run the workflow on a 'Pools' cluster and on a 'All-purpose compute'. All computes use Databricks Runtime Version 13.3 LTS.

Since about 3 weeks we have been facing regular failures of tasks in our pipeline that all seem to be due to technical and non-reproducible errors. In most cases a repair of the task fixes runs just fine, but obviously our trust in the platform has taken a beating because of this.

Some of the problems we regularly see:

• Cluster 'xxx' was terminated. Reason: COMMUNICATION_LOST (CLOUD_FAILURE)
• Failed to acquire a SAS token for list on /__unitystorage/catalogs/xxx/tables/xxx/_delta_log due to java.util.concurrent.ExecutionException: org.apache.spark.sql.AnalysisException: 403: Invalid Authorization
• Fatal error: The Python kernel is unresponsive.
• run failed with error message Cluster xxx became unusable during the run since the driver became unhealthy
• com.databricks.common.client.DatabricksServiceHttpClientException: 403: Invalid Authorization

Our Databricks instance is hosted in Azure West Europe.

Does anybody have similar experiences? And if so, did you find a way to add more stability?