K_Anudeep
Databricks Employee
Databricks Employee

Hello @Ramana ,

You’re right that data volume doesn’t change the logical plan, but your pattern (Example: SELECT * from a wide table + 10–15 column transforms) can still exceed the analyzer’s fixed iteration cap on Serverless, because each * expansion and chained withColumn/casts/regex adds more alias resolution work, resulting in a huge stack of projections causing the analyzer to break.

I would suggest that you understand what optimiser rules are being applied and why they are exceeding the default max values by setting spark.sql.planChangeLog.level to INFO and then simplify the code as required. 

Anudeep