Serverless Compute - Spark - Jobs failing with Max iterations (1000) reached for batch Resolution

[Ramana]

Hello Community,

We have been trying to migrate our jobs from Classic Compute to Serverless Compute. As part of this process, we face several challenges, and this is one of them.

When we try to execute the existing jobs with Serverless Compute, if the job deals with a small amount of data or a small number of stages, Serverless Compute works great. But when we try to use the Serverless Compute for processing a large amount of data with a large number of intermediate transformations, the job fails with the following error:

Exception: (java.lang.RuntimeException) Max iterations (1000) reached for batch Resolution, please set 'spark.sql.analyzer.maxIterations' to a larger value

This error indicates that the query plan required more than the default 1000 iterations to resolve, likely due to deeply nested logic or complex transformations in our code. However, in serverless environments, the spark.sql.analyzer.maxIterations configuration is not accessible or overridable, as it is not exposed via Spark Connect.

Has anyone faced the similar issue?

Any suggestion or recommendation is greatly appreciated.

Screenshots:

#ServerlessCompute

#DataEngineering

#ClassicCompute-to-ServerlessCompute-Migration

#Migration

[K_Anudeep]

Hello @Ramana ,

The above error occurs when the Spark SQL optimiser is unable to resolve a query within the fixed maximum number of rule-application iterations (default 1000) in its internal logical plan "Resolution" phase. This typically happens with particularly complex queries, especially those that involve:

• Excessively deep or "chained" query plans, often produced when repeatedly applying DataFrame transformations, like many chained .withColumn() calls
• Highly nested views or subqueries, especially those involving multiple self-joins or recursive structures
• Generated query plans with conflicting or redundant attributes

In Databricks serverless and some managed environments, most Spark SQL configs—including spark.sql.analyzer.maxIterations—cannot be changed/Thus, increasing the setting is not possible as a workaround on serverless.

So the only way is to reduce the complexity of the logical plan the analyser generates, and that can be done by optimising your query by breaking it down to smaller steps, materializing each before proceeding to the next transformation.

Please let me know if you have any further questions