How to instantiate Databricks spark context in a python script?

[ae20cg]

I want to run a block of code in a script and not in a notebook on databricks, however I cannot properly instantiate the spark context without some error.

I have tried ` SparkContext.getOrCreate()`, but this does not work.

Is there a simple way to do this I am missing?

[Ajay-Pandey]

Hi @Andrej Erkelens​ ,

Can you please send error which you are getting while using above code?

[ae20cg]

@Kaniz Fatma​ 

Hi, I have tried this but receive an error

`RuntimeError: A master URL must be set in your configuration`

Is there something I am missing to use databricks cluster (AWS backend) in a py script?

Thanks

[Andrei_Radulesc]

I have the same problem, and would be interested in a solution

[Spartan101]

Hi,
I tried doing this and get master URL and app name error. I tried setting those and get an error message that asks not to create a spark session in databricks and use sparkcontext.getOrCreate() method instead.
But that leads to the same error. I used the getActiveSession method to verify the python script does not have access to a spark session. 

[testing3]

Did this ever get addressed? I would like to use a Databricks notebook to launch a python-based child process (os.popen) that itself ultimately needs to use pyspark. When I try this, I either get told to supply a Master URL to the Spark context, or if I apply local[*] as master, I get told in an exception message on Spark interaction that Notebooks should use the shared context available via sc. This code is executing in a standalone python library being run by the subprocess (based on python, but not just a python script) launched from Notebook.

Is it simply disallowed to access Spark outside of the shared context sc? If so, how can we access that shared context from a standalone python library as I describe?

[mikifare]

Thanks for information

[MichTalebzadeh]

Try this pay attention to import

from pyspark.sql import SparkSession
appName = "abc"

# Create a SparkSession
spark = SparkSession.builder \
    .appName(appName) \
    .getOrCreate()

# Your PySpark code blah blah

# Stop the SparkSession when done
spark.stop()

[Spartan101]

Thanks @MichTalebzadeh , but I have tried this to no avail, I get [MASTER_URL_NOT_SET] error, and when I try to set it, I get an error stating I cannot create another spark session. the getActiveSession() returns null from within the script, but returns the session when called from the notebook.

[MichTalebzadeh]

Thanks. Please send the full detail of error you are getting

[Spartan101]

Sadly, I have the errors in my corporate environment and I can't show the exact error details from this account.But it is quite close to that of @testing3 

[Kaizen]

I came across a similar issue. 

Please detail how you are executing the python script. Are you calling it from the web terminal? or from a notebook?

Note: If you are calling it from the web terminal - your spark session wont be passed. You could create a local variable and pass it in if youd like. I have never gotten to that point yet tho

[Spartan101]

I am running the script from databricks notebook : !streamlit run MyScript.py

[MichTalebzadeh]

Hi,

I believe running a Streamlit app directly from a Databricks notebook using !streamlit run <python_code>.py  is not the way to do it,  because Databricks notebooks are not designed to host Streamlit

OK try below

Create a Databricks Runtime with Streamlit Pre-installed. Configure Cluster, ehen creating a new Databricks cluster, select a runtime that includes Streamlit pre-installed. This eliminates the installation step.

Run the script: Within the notebook cell, simply execute the script directly:

!streamlit run MyScript.py

HTH

apps.

[Spartan101]

Hi @MichTalebzadeh ,

Once again, thanks for your replies.
My databricks cluster does come preinstalled with streamlit, and I have been running the script the way you mentioned.
I am going to try using alternatives to spark for the time being, and try with spark session isolation disabled as well.

I appreciate you taking out time to respond to this issue.