Hi,

The databricks runtime version and the installed libraries are the same on shared and job cluster. We had also raised a case with databricks support for the same. where they identified the error to be a syntax error in lpad function which we have used. 

Here is the explanation for reason for difference between clusters , which we have received from databricks

Single access mode compute uses Classic Pyspark and Shared clusters use the secure SparkConnect interface for python API, So likely the behaviour is inconsistent between SparkConnect and Classic PySpark.

Thanks,Shalini