Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-31-2025 04:28 AM - edited 10-31-2025 04:31 AM
Hello @Dhruv-22 ,
- Can you share the schema of the df? Do you have a _corrupt_record column in your dataframe? If yes.. where are you getting it from, because you said its an empty file correct?
- As per the design ,Spark blocks queries that only reference
_corrupt_recorda column from raw JSON/CSV and it throws an error if explicitly accessed. But in your case, you aren't doing that, but it still throws that error, which is why we will need to schema and the explain plan of the dataframe df.explain(true).
By the way, i created an empty file in serverless and it created an empty df as expected
Anudeep