No space left on device and IllegalStateException: Have already allocated a maximum of 8192 pages

[SaiCharan]

Hello, I'm writing to bring to your attention an issue that we have encountered while working with Data bricks and seek your assistance in resolving it.

Context of the Error : 
When a sql query(1700 lines) is ran, corresponding data bricks job is failing for No space left on device and upgrading the cluster configuration to higher versions and added required spark configurations still which errored out for IllegalStateException: Have already allocated a maximum of 8192 pages. 

Initially runs were completing with configuration as mentioned :  Driver: m5.4xlarge · Workers: m5.4xlarge · 20-50 workers and spark configurations as spark.driver.maxResultSize 8g andspark.sql.autoBroadcastJoinThreshold -1 and now this is failing with Caused by: java.io.IOException: No space left on device.
ERROR Uncaught throwable from user code: org.apache.spark.SparkException: Job aborted due to stage failure: ShuffleMapStage 149 (count at DuplicatesFinder.scala:23) has failed the maximum allowable number of times: 4. Most recent failure reason:
org.apache.spark.shuffle.FetchFailedException.

Based on this error, checked the stages and dag and observed a failed stage for collecting samples for range partitioning in count at DuplicatesFinder.scala consumes most of driver memory to shuffle read & shuffle write.

Have tried upgrading to higher cluster resources with relevant spark configurations scaling each time as : 
spark.sql.shuffle.partitions ( auto, 100, 300, 400 till 4000)
spark.driver.maxResultSize (16g)
spark.default.parallelism 4000
spark.driver.memory 16g
spark.executor.memory 6g

enable-cache: false
view-partitions: 8000

and now it errors out IllegalStateException: Have already allocated a maximum of 8192 pages.

Kindly let us know what points can considered w.r.t this issue.
Any suggestions to resolve this issue will be much helpful.

Thanks for your time and assistance. Appreciate it.

Best regards,

Sai Charan

[Kaniz_Fatma]

Hi @SaiCharan, Here are some suggestions based on the information you provided:

No Space Left on Device Error: This error indicates that the worker’s local disks are filling up. The local disk is used by Spark for intermediate shuffle files, RDD persistence, and other tasks. If these intermediate shuffle files are not removed quickly enough, they can cause the “No space lef.... You can inspect the amount of local disk space before and after your shuffle1. If necessary, you can explicitly remove intermediate shuffle files1. Another solution is to add more storage capacity or use a bootstrap action to dynamically scale up s....

IllegalStateException: Have already allocated a maximum of 8192 pages: This error might be resolved ....

org.apache.spark.shuffle.FetchFailedException: This exception is seen when Spark is unable to shuffl.... Instead of shuffling the entire remote block in memory, it can be fetched to disk. The threshold for fetching the block to disk size can be controlled by the property spark.maxRemoteB....

General Suggestions: Consider using shared all-purpose clusters if your job is configured to use a j.... You can also change the cluster configuration or switch to another available cluster5. Review best practices for cluster configuration to ensure you’re making optimal use of available res....

I hope this helps!

[jose_gonzalez]

are processing Parquet files or what is the format of your tables? 
can you split your sql query instead of having a huge query with 1700 lines

[Kaniz_Fatma]

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀