Getting "Job aborted due to stage failure" SparkException when trying to download full result

[Tahseen0354]

I have generated a result using SQL. But whenever I try to download the full result (1 million rows), it is throwing SparkException. I can download the preview result but not the full result. Why ? What happens under the hood when I try to download the full result ?

Here is the exception:

SparkException: Job aborted due to stage failure: Task 0 in stage 133.0 failed 4 times, most recent failure: Lost task 0.3 in stage 133.0 (TID 2644) (192.***.x.x executor 6): com.databricks.sql.io.FileReadException: Error while reading file abfss:REDACTED_LOCAL_PART@someurl. It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved. If Delta cache is stale or the underlying files have been removed, you can invalidate Delta cache manually by restarting the cluster.

Caused by: FileReadException: Error while reading file abfss:REDACTED_LOCAL_PART@someurl. It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved. If Delta cache is stale or the underlying files have been removed, you can invalidate Delta cache manually by restarting the cluster.

Caused by: FileNotFoundException: Operation failed: "The specified path does not exist.", 404, HEAD, https://***.snappy.parquet?upn=false&action=getStatus&timeout=90

Caused by: AbfsRestOperationException: Operation failed: "The specified path does not exist.", 404, HEAD, https://***.snappy.parquet?upn=false&action=getStatus&timeout=90

[Anonymous]

@Md Tahseen Anam​ - Hello! My name is Piper and I'm one of the community moderators. Thanks for your question. Let's give it a bit longer to see what the community has to say. Hang in there!

[Tahseen0354]

Hi, thank you for your reply. Would be great to get some lights in here.

[User16763506477]

Hi @Md Tahseen Anam​ are there any updates happening to the table while you are downloading the results?

[Tahseen0354]

No update. can it be a network issue ?

[jose_gonzalez]

hi @Md Tahseen Anam​ ,

Have you try the following steps to re-run your query and get the full results? docs here

[Tahseen0354 · Accepted solution]

It's working now, I think it was a network issue.

[Anonymous]

@Md Tahseen Anam​ - Thanks for letting us know. I'm glad things are working!

[rpshgupta]

I am also having this issue again and again. I really want to understand what can we do to avoid this?