I have a parquet dataframe df. I first add a column using df.withColumn("version",lit(currentTimestamp)) and append it a table db.tbl with format parquet and partitioned by the "version" column. I then ran MSCK REPAIR TABLE db.tbl. I have then created a view from this table called db.tbl_v.

Then I run the code below which gives me the error.

val am = spark.table("db.tbl_v").filter(col("col1")>=0.5).drop("col2")

display(am)

What I have tried:

The same code is working on lower environment. I believe that the configurations and settings will remain same in DEV and Prod environment.

I have tried creating a new table. It did not work.

When I run select * from db.tbl_v where col1 >= 0.5. I get an error Error in SQL statement: UndeclaredThrowableException:

When I run select * from db.tbl where col1 >= 0.5, I get the rows.

Thank you for reading my question and appreciate your help.