topic Re: DBR 16.4 LTS - Spark 3.5.2 is not compatible with Delta Lake 3.3.1 in Data Engineering

DBR 16.4 LTS - Spark 3.5.2 is not compatible with Delta Lake 3.3.1

leireroman — Mon, 09 Jun 2025 06:45:53 GMT

I'm migrating to Databricks Runtime 16.4 LTS, which is using Spark 3.5.2 and Delta Lake 3.3.1 according to the documentation: Databricks Runtime 16.4 LTS - Azure Databricks | Microsoft Learn

I've upgraded my conda environment to use those versions, but I get this error message when I try to upgrade my environment:

According to Delta Lake releases (Releases · delta-io/delta), the last version compatible with Spark 3.5.2 is 3.2.0, because the next one (3.2.1) is built on Spark 3.5.3.

Is Databricks really using Delta Lake version 3.3.1? How can I check this from a cluster with DBR 16.4 LTS?

Re: DBR 16.4 LTS - Spark 3.5.2 is not compatible with Delta Lake 3.3.1

Renu_ — Tue, 17 Jun 2025 11:50:14 GMT

Hi @leireroman, Databricks Runtime 16.4 LTS includes Delta Lake 3.3.1, paired with Spark 3.5.2. This combination works within Databricks because it’s a custom build. In your Conda environment, the conflict occurs because delta-spark 3.3.1 requires pyspark >=3.5.3, but you’ve set it to 3.5.2.

To resolve this, you can either:

Upgrade pyspark to 3.5.3 to work with delta-spark 3.3.1
Downgrade to delta-spark 3.2.0 to stay compatible with Spark 3.5.2.

Re: DBR 16.4 LTS - Spark 3.5.2 is not compatible with Delta Lake 3.3.1

SamAdams — Thu, 02 Oct 2025 22:21:00 GMT

@leireroman encountered the same and used an override (like a pip constraints.txt file or PDM resolution override specification) to make sure my local development environment matched the runtime.