cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

FileReadException Error

DataBricks_Use1
New Contributor

Hi,

I am getting FilereadException Error while reading JSON file using REST API Connector.

It comes when data is huge in Json File and it's not able to handle more than 1 Lac records.

Error details:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 45 in stage 42917.0 failed 4 times,
most recent failure: Lost task 45.3 in stage 42917.0 (TID 246862) (10.22.49.200 executor 18):
com.databricks.sql.io.FileReadException: Error while reading file
dbfs:/mnt/data/raw/Test_RESTAPI.json.

Py4JJavaError: An error occurred while calling o13842.json.

Please help.

Thanks 

 

2 REPLIES 2

jose_gonzalez
Databricks Employee
Databricks Employee

Could you share the Executor 18's logs? was this working fine in the past or this is the first time you are running this code?

NandiniN
Databricks Employee
Databricks Employee

Hello @DataBricks_Use1 ,

It would great if you could add the entire stack trace, as Jose mentioned. But there should be a "Caused by:" section below which would give you an idea of what's the reason for this failure and then you can work on that. 

for eg: 

Caused by: java.io.EOFException

Caused by: java.io.IOException: java.util.concurrent.ExecutionException:

Caused by: com.databricks.sql.io.FileReadException: ..  could not verify dictionary page integrity, CRC checksum verification failed.

 

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group