Thanks for the workaround. But why this particular piece of code fails in 9.0 LTS runtime and run in 8.3 without issues. Any idea. Please see the code below.

from pyspark.sql.functions import lit,col,row_number,floor,trim

df = spark.read.option("header", "true").csv(filePath)

df2 = df.select(col("cc"),col("ac"),col("an"),\

      col("ag"),col("at")).distinct()

lstOfMissingColumns = ['col1', 'col2', 'col3', 'col4', 'col5', 'col6', 'col7', 'col8', 'col8', 'col9','col9', 'col10', 'col11', 'col12', 'col13',

            'col14', 'col15', 'col16', 'col17']

for c in lstOfMissingColumns:

 df2 = df2.withColumn(c,lit(''))

df2.rdd.isEmpty()