topic Re: FileReadException Error in Get Started Discussions

FileReadException Error

DataBricks_Use1 — Tue, 30 Apr 2024 14:50:49 GMT

Hi,

I am getting FilereadException Error while reading JSON file using REST API Connector.

It comes when data is huge in Json File and it's not able to handle more than 1 Lac records.

Error details:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 45 in stage 42917.0 failed 4 times,
most recent failure: Lost task 45.3 in stage 42917.0 (TID 246862) (10.22.49.200 executor 18):
com.databricks.sql.io.FileReadException: Error while reading file
dbfs:/mnt/data/raw/Test_RESTAPI.json.

Py4JJavaError: An error occurred while calling o13842.json.

Please help.

Thanks

Re: FileReadException Error

jose_gonzalez — Tue, 30 Apr 2024 23:06:52 GMT

Could you share the Executor 18's logs? was this working fine in the past or this is the first time you are running this code?

Re: FileReadException Error

NandiniN — Wed, 01 May 2024 05:19:05 GMT

Hello @DataBricks_Use1 ,

It would great if you could add the entire stack trace, as Jose mentioned. But there should be a "Caused by:" section below which would give you an idea of what's the reason for this failure and then you can work on that.

for eg:

Caused by: java.io.EOFException

Caused by: java.io.IOException: java.util.concurrent.ExecutionException:

Caused by: com.databricks.sql.io.FileReadException: .. could not verify dictionary page integrity, CRC checksum verification failed.