@WiliamRosa Unfortunately, none of that is relevant. The packaging does not change between runs where it works and does not work, and I'm confident the packaging is correct.

Moreover, if you actually read the post more carefully, you'll see that it's the package itself (which I've called "my_package" for demonstration purposes), not dependencies, which it fails to find - even though, to get to the point where the error occurs, it must have found it (otherwise it could not find the .load() method it is calling, which is part of the module code which is packed into the wheel).

It seems something strange and erratic related to the Spark structured streaming source being dispatched into the Spark cluster.

In fact, this is an important detail I failed to mention - we are using a Spark structured streaming class/implementation (extending SimpleDataSourceStreamReader).