We've been trying to launch a spark-submit python task using the parameter "archives", similar to that one used in Yarn.

​

However, we've not been able to successfully make it work in databricks.

​

​

We know that for our OnPrem installation we can use some tutorial such as this: https://conda.github.io/conda-pack/spark.html and it does use Yarn Resource Localization for uncompressing our tar.gz inside the executors.

​

Which is quite interesting bcuz all we need to do is to pack our project inside a wheel file and them submit inside a spark-submit.

Another reason is that if we successfully address this problem, all of our models could be much more easily migrated to Databricks, without much effort.

Does someone needed this already and have managed to solve? How the community handle spark-submit for databricks, focusing on entire projects instead of notebook-based projects?