NoSuchMethodError with Delta-spark and Databricks Runtime 16.4 LTS

dperkins — Thu, 15 May 2025 15:24:24 GMT

I'm running into an exception when trying to run a Java Spark Jar using the delta-spark library as a job on a Databricks Runtime 16.4 LTS cluster on Azure.

I've tried various versions of the Delta Spark library from 3.0.0 to the latest 3.3.1, but always get the same error. I'm using Scala version 2.13. They all run fine locally using Spark 3.5.2.

I've searched the documentation and knowledge base and have been unable to find anything related to the exception.

Here is the exception

org.apache.spark.SparkException: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.delta.sources.DeltaDataSource could not be instantiated Caused by: org.apache.spark.SparkException: java.lang.NoSuchMethodError: 'void org.apache.spark.sql.sources.CreatableRelationProviderShim.$init$(org.apache.spark.sql.sources.CreatableRelationProviderShim)'

Re: NoSuchMethodError with Delta-spark and Databricks Runtime 16.4 LTS

SP_6721 — Wed, 21 May 2025 09:47:43 GMT

Hi @dperkins

The error is likely due to a mismatch between your delta-spark library and the Databricks Runtime. The runtime uses Spark 3.5.2 and Scala 2.13 (with Delta Lake 3.3.1), so ensure your JAR is built with these versions.
It’s best to remove the Delta dependency from your JAR (or mark it as provided) and avoid shading Delta classes to prevent conflicts.

topic Re: NoSuchMethodError with Delta-spark and Databricks Runtime 16.4 LTS in Data Engineering

NoSuchMethodError with Delta-spark and Databricks Runtime 16.4 LTS

Re: NoSuchMethodError with Delta-spark and Databricks Runtime 16.4 LTS