FileReadException Error

[DataBricks_Use1]

Hi,

I am getting FilereadException Error while reading JSON file using REST API Connector.

It comes when data is huge in Json File and it's not able to handle more than 1 Lac records.

Error details:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 45 in stage 42917.0 failed 4 times,
most recent failure: Lost task 45.3 in stage 42917.0 (TID 246862) (10.22.49.200 executor 18):
com.databricks.sql.io.FileReadException: Error while reading file
dbfs:/mnt/data/raw/Test_RESTAPI.json.

Py4JJavaError: An error occurred while calling o13842.json.

Please help.

Thanks 

[jose_gonzalez]

Could you share the Executor 18's logs? was this working fine in the past or this is the first time you are running this code?

[NandiniN]

Hello @DataBricks_Use1 ,

It would great if you could add the entire stack trace, as Jose mentioned. But there should be a "Caused by:" section below which would give you an idea of what's the reason for this failure and then you can work on that. 

for eg: 

Caused by: java.io.EOFException

Caused by: java.io.IOException: java.util.concurrent.ExecutionException:

Caused by: com.databricks.sql.io.FileReadException: ..  could not verify dictionary page integrity, CRC checksum verification failed.