Hey @K_Anudeep

Here are the details you requested.

• Yes, there is a _corrupt_record column in the dataframe. It is coming since spark is treating the file to have some corrupt records. Therefore, it is generating the _corrupt_record column

• The error comes when I try to run display, collect or any such command. Here is the explain

• Also, db.count() is 1

I checked the file size, it was 3 bytes. It doesn't display any character.

But, printing the hexdump it gives the following

I guess this is causing the issue. Can you tell how to deal with it? It runs fine on the all-purpose cluster though