We got our app working by using the databricks-connect and creating a spark session on a shared compute cluster separate from the cluster my app runs on. I set the env variable DATABRICKS_CLUSTER_ID to the id of the cluster I want to use. Then in tha...
Hey @roshan_robert. I am facing a similar issue. We are trying to launch a streamlit app from databricks apps that requires spark but Im getting the same error. I have yet to find a way to configure spark to work from my apps cluster. Did you ever re...
