Unstable workflow runs lately

[FrankTa]

Hi!

We are using Databricks on Azure on production since about 3 months. A big part of what we use Databricks for is processing data using a workflow with various Python notebooks. We run the workflow on a 'Pools' cluster and on a 'All-purpose compute'. All computes use Databricks Runtime Version 13.3 LTS.

Since about 3 weeks we have been facing regular failures of tasks in our pipeline that all seem to be due to technical and non-reproducible errors. In most cases a repair of the task fixes runs just fine, but obviously our trust in the platform has taken a beating because of this.

Some of the problems we regularly see:

• Cluster 'xxx' was terminated. Reason: COMMUNICATION_LOST (CLOUD_FAILURE)
• Failed to acquire a SAS token for list on /__unitystorage/catalogs/xxx/tables/xxx/_delta_log due to java.util.concurrent.ExecutionException: org.apache.spark.sql.AnalysisException: 403: Invalid Authorization
• Fatal error: The Python kernel is unresponsive.
• run failed with error message Cluster xxx became unusable during the run since the driver became unhealthy
• com.databricks.common.client.DatabricksServiceHttpClientException: 403: Invalid Authorization

Our Databricks instance is hosted in Azure West Europe.

Does anybody have similar experiences? And if so, did you find a way to add more stability?

[holly · Accepted solution]

Hi Frank, I completely understand your frustration. A few things:

• 403 errors: there is a known bug that is due to be fixed with 403 errors. If you continue to experience them, please raise an issue through support and someone will walk through your exact scenario.
• 1,3 & 4 are down to Azure's provision and health of VMs provided. If you're experiencing this a lot, I'd suggest raising it with Azure themselves, they may recommend provisioned VMs for you.

[FrankTa]

Hi holly,

Thanks for your reply, good to hear that the 403 errors are on the radar and due to be fixed. I will reach out to support in case of further issues.