K_Anudeep
Databricks Employee
Databricks Employee

Hello @Dhruv-22 ,

  • Can you share the schema of the df? Do you have a _corrupt_record column in your dataframe? If yes.. where are you getting it from, because you said its an empty file correct?
  • As per the design ,Spark blocks queries that only reference_corrupt_record  a column from raw JSON/CSV and it throws an error if explicitly accessed. But in your case, you aren't doing that, but it still throws that error, which is why we will need to schema and the explain plan of the dataframe df.explain(true).

By the way, i created an empty file in serverless and it created an empty df as expected 

K_Anudeep_0-1761910255579.png

 

Anudeep