Delta Live Table Pipeline

[SB93]

I have a pipeline that has given me no problems up until today with the following error message:

com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster 0307-134831-tgq587us: Attempt to launch cluster with invalid arguments. databricks_error_message: Spark failed to start: Driver unresponsive. Possible reasons: library conflicts, incorrect metastore configuration, and i...
This error is likely due to a misconfiguration in the pipeline.
Check the pipeline cluster configuration and associated cluster policy.

I ran the exact pipeline multiple times previously with the exact pipeline config and no issues have occurred until now. 

[LRALVA]

@SB93 

The error message you are seeing indicates that the cluster failed to launch because the Spark driver was unresponsive, with possible causes being library conflicts, incorrect metastore configuration, or other configuration issues. Given that the pipeline worked previously without issues, here are some steps you can take to troubleshoot and resolve the issue:

Troubleshooting Steps:

• Check for recent configuration changes to the pipeline or cluster.
• Review driver logs and cluster logs for additional error details.
• Check for library conflicts and ensure compatibility between libraries and the runtime version.
• Verify the metastore configuration and access permissions.
• Ensure that cluster policies and resource allocation are correct.
• Restart or recreate the cluster and check if the issue persists.

[SB93]

Hi @LRALVA,

Thank you for your suggestions. After spending all of Friday struggling with the issue, I decided to give up. However, when I tried the pipeline again this morning, it worked perfectly.

I will review the situation to identify any differences between now and Friday.