FileReadException Error

DataBricks_Use1
New Contributor

‎04-30-2024 07:50 AM

Hi,

I am getting FilereadException Error while reading JSON file using REST API Connector.

It comes when data is huge in Json File and it's not able to handle more than 1 Lac records.

Error details:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 45 in stage 42917.0 failed 4 times,
most recent failure: Lost task 45.3 in stage 42917.0 (TID 246862) (10.22.49.200 executor 18):
com.databricks.sql.io.FileReadException: Error while reading file
dbfs:/mnt/data/raw/Test_RESTAPI.json.

Py4JJavaError: An error occurred while calling o13842.json.

Please help.

Thanks 

 

0 Kudos

jose_gonzalez
Databricks Employee
Databricks Employee

‎04-30-2024 04:06 PM

Could you share the Executor 18's logs? was this working fine in the past or this is the first time you are running this code?

0 Kudos

NandiniN
Databricks Employee
Databricks Employee

‎04-30-2024 10:19 PM

Hello @DataBricks_Use1 ,

It would great if you could add the entire stack trace, as Jose mentioned. But there should be a "Caused by:" section below which would give you an idea of what's the reason for this failure and then you can work on that. 

for eg: 

Caused by: java.io.EOFException

Caused by: java.io.IOException: java.util.concurrent.ExecutionException:

Caused by: com.databricks.sql.io.FileReadException: ..  could not verify dictionary page integrity, CRC checksum verification failed.

 

 

0 Kudos