Hello @Ramana ,

The above error occurs when the Spark SQL optimiser is unable to resolve a query within the fixed maximum number of rule-application iterations (default 1000) in its internal logical plan "Resolution" phase. This typically happens with particularly complex queries, especially those that involve:

• Excessively deep or "chained" query plans, often produced when repeatedly applying DataFrame transformations, like many chained .withColumn() calls
• Highly nested views or subqueries, especially those involving multiple self-joins or recursive structures
• Generated query plans with conflicting or redundant attributes

In Databricks serverless and some managed environments, most Spark SQL configs—including spark.sql.analyzer.maxIterations—cannot be changed/Thus, increasing the setting is not possible as a workaround on serverless.

So the only way is to reduce the complexity of the logical plan the analyser generates, and that can be done by optimising your query by breaking it down to smaller steps, materializing each before proceeding to the next transformation.

Please let me know if you have any further questions