topic workflow fails when ran using a job cluster but not in shared cluster in Data Engineering

workflow fails when ran using a job cluster but not in shared cluster

ShaliniC — Thu, 24 Oct 2024 07:31:11 GMT

Hi,

We have a workflow which calls 3 notebooks and when we run this workflow using shared cluster it runs fine, but when ran with job cluster , one of the notebooks fail.This notebook uses sql function Lpad and looks like it errors because of it. Has anyone experienced anything similar.

Thanks,

Shalini

Re: workflow fails when ran using a job cluster but not in shared cluster

filipniziol — Thu, 24 Oct 2024 18:26:39 GMT

Hi @ShaliniC ,

The possible causes are:
1. The Databricks Runtime version of the job cluster is different than the one of the the shared cluster.

2. The libraries installed on the shared cluster are missing in the job cluster.

Could you check the what are the above configuration changes and adjust the job cluster to the configuration of the shared cluster?

If it does not help, could you please share the details of the error message?

Re: workflow fails when ran using a job cluster but not in shared cluster

saurabh18cs — Fri, 25 Oct 2024 07:06:21 GMT

notebooks are executing sequentially or parallel in this workflow?

Re: workflow fails when ran using a job cluster but not in shared cluster

ShaliniC — Thu, 31 Oct 2024 09:20:37 GMT

Hi,

The databricks runtime version and the installed libraries are the same on shared and job cluster. We had also raised a case with databricks support for the same. where they identified the error to be a syntax error in lpad function which we have used.

Here is the explanation for reason for difference between clusters , which we have received from databricks

Single access mode compute uses Classic Pyspark and Shared clusters use the secure SparkConnect interface for python API, So likely the behaviour is inconsistent between SparkConnect and Classic PySpark.

Thanks,Shalini

Re: workflow fails when ran using a job cluster but not in shared cluster

ShaliniC — Thu, 31 Oct 2024 09:22:47 GMT

Hi,

the error occured in the parallel run.

We had also raised a case with databricks support for the same. where they identified the error to be a syntax fail in lpad function which we have used.

Here is the explanation for reason for difference between clusters , which we have received from databricks

Thanks,Shalini