Bug - task retry fails to load cluster dependencies

Paolo — Tue, 11 Jul 2023 06:35:15 GMT

Hi,

I'm currently facing an issue with task retries on a structured streaming set to have unlimited retries. Our job crashes frequently due to out of memory problems, so we set the task retry limit to -1 (unlimited) as a workaround. This is also the suggested practice for production streaming jobs, according to the Databricks documentation.

However, quite often after a couple of crashes, the cluster fails to find the dependencies. We have specifically setup a custom one, which loads fine on first runs. This bug does not always happen, so it's not so easy to reproduce.

This is the error I see:

: java.lang.ClassNotFoundException: Failed to find data source: solacemqtt. Please find packages at https://spark.apache.org/third-party-projects.html

topic Bug - task retry fails to load cluster dependencies in Get Started Discussions

Bug - task retry fails to load cluster dependencies