NoSuchMethodError with Delta-spark and Databricks Runtime 16.4 LTS

[dperkins]

I'm running into an exception when trying to run a Java Spark Jar using the delta-spark library as a job on a Databricks Runtime 16.4 LTS cluster on Azure.

I've tried various versions of the Delta Spark library from 3.0.0 to the latest 3.3.1, but always get the same error. I'm using Scala version 2.13.  They all run fine locally using Spark 3.5.2.  

I've searched the documentation and knowledge base and have been unable to find anything related to the exception. 

Here is the exception

org.apache.spark.SparkException: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.delta.sources.DeltaDataSource could not be instantiated Caused by: org.apache.spark.SparkException: java.lang.NoSuchMethodError: 'void org.apache.spark.sql.sources.CreatableRelationProviderShim.$init$(org.apache.spark.sql.sources.CreatableRelationProviderShim)' 

[SP_6721]

Hi @dperkins 

The error is likely due to a mismatch between your delta-spark library and the Databricks Runtime. The runtime uses Spark 3.5.2 and Scala 2.13 (with Delta Lake 3.3.1), so ensure your JAR is built with these versions.
It’s best to remove the Delta dependency from your JAR (or mark it as provided) and avoid shading Delta classes to prevent conflicts.