Dan_Z
Databricks Employee
Databricks Employee

What's happening here is that Spark is reading a file and has a list of parquet file names that it wants to pull data from. Then, for one of the parquet files Spark goes to read in the file, but notices that that file does not actually exist in storage. So it throws this error.

Usually this is caused by some other process updating/deleting the files in this location while the read is taking place. I would look to see what else could be touching this location at the same time.