I ran a databricks notebook to do incremental loads from files in raw layer to bronze layer tables. Today, I encountered a case where the delta file was empty. I tried running it manually on the serverless compute and encountered an error.

df = spark.read.json(path)
df.display()

-- Output
Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the referenced columns only 
include the internal corrupt record column (named _corrupt_record by default). For example: 
spark.read.schema(schema).csv(file).filter($"_corrupt_record".isNotNull).count() and 
spark.read.schema(schema).csv(file).select("_corrupt_record").show(). Instead, you can cache or 
save the parsed results and then send the same query. 
For example, val df = spark.read.schema(schema).csv(file).cache() and then 
df.filter($"_corrupt_record".isNotNull).count().

 I tried caching, but it isn't allowed in serverless compute

[NOT_SUPPORTED_WITH_SERVERLESS] PERSIST TABLE is not supported on serverless compute. SQLSTATE: 0A000

I have the following questions:

1. Why does this issue occur only in serverless compute? I tried using All-purpose compute with 15.4LTS and it created an empty dataframe.
2. Is there a way to display the dataframe to see what exactly is the corrupt record? I tried collect, select('*', lit('c')) but it didn't work.
3. Is there a way in the serverless compute to tolerate empty files?