Hi,

I'm currently facing an issue with task retries on a structured streaming set to have unlimited retries. Our job crashes frequently due to out of memory problems, so we set the task retry limit to -1  (unlimited) as a workaround. This is also the suggested practice for production streaming jobs, according to the Databricks documentation.

However, quite often after a couple of crashes, the cluster fails to find the dependencies. We have specifically setup  a custom one, which loads fine on first runs. This bug does not always happen, so it's not so easy to reproduce.

This is the error I see:

: java.lang.ClassNotFoundException: 
Failed to find data source: solacemqtt. Please find packages at
https://spark.apache.org/third-party-projects.html