topic Re: How to instantiate Databricks spark context in a python script? in Data Engineering

How to instantiate Databricks spark context in a python script?

ae20cg — Wed, 22 Feb 2023 22:14:25 GMT

I want to run a block of code in a script and not in a notebook on databricks, however I cannot properly instantiate the spark context without some error.

I have tried ` SparkContext.getOrCreate()`, but this does not work.

Is there a simple way to do this I am missing?

Re: How to instantiate Databricks spark context in a python script?

Ajay-Pandey — Thu, 23 Feb 2023 04:29:47 GMT

Hi @Andrej Erkelens ,

Can you please send error which you are getting while using above code?

Re: How to instantiate Databricks spark context in a python script?

ae20cg — Thu, 23 Feb 2023 17:11:03 GMT

@Kaniz Fatma

Hi, I have tried this but receive an error

`RuntimeError: A master URL must be set in your configuration`

Is there something I am missing to use databricks cluster (AWS backend) in a py script?

Thanks

Re: How to instantiate Databricks spark context in a python script?

Andrei_Radulesc — Fri, 14 Apr 2023 21:21:10 GMT

I have the same problem, and would be interested in a solution

Re: How to instantiate Databricks spark context in a python script?

testing3 — Fri, 05 Jan 2024 21:06:17 GMT

Did this ever get addressed? I would like to use a Databricks notebook to launch a python-based child process (os.popen) that itself ultimately needs to use pyspark. When I try this, I either get told to supply a Master URL to the Spark context, or if I apply local[*] as master, I get told in an exception message on Spark interaction that Notebooks should use the shared context available via sc. This code is executing in a standalone python library being run by the subprocess (based on python, but not just a python script) launched from Notebook.

Is it simply disallowed to access Spark outside of the shared context sc? If so, how can we access that shared context from a standalone python library as I describe?

Re: How to instantiate Databricks spark context in a python script?

mikifare — Sun, 07 Jan 2024 10:41:11 GMT

Thanks for information

Re: How to instantiate Databricks spark context in a python script?

Spartan101 — Wed, 13 Mar 2024 05:02:17 GMT

Hi,
I tried doing this and get master URL and app name error. I tried setting those and get an error message that asks not to create a spark session in databricks and use sparkcontext.getOrCreate() method instead.
But that leads to the same error. I used the getActiveSession method to verify the python script does not have access to a spark session.

Re: How to instantiate Databricks spark context in a python script?

MichTalebzadeh — Wed, 13 Mar 2024 10:51:54 GMT

Try this pay attention to import

from pyspark.sql import SparkSession appName = "abc" # Create a SparkSession spark = SparkSession.builder \ .appName(appName) \ .getOrCreate() # Your PySpark code blah blah # Stop the SparkSession when done spark.stop()

Re: How to instantiate Databricks spark context in a python script?

Spartan101 — Wed, 13 Mar 2024 14:55:03 GMT

Thanks @MichTalebzadeh , but I have tried this to no avail, I get [MASTER_URL_NOT_SET] error, and when I try to set it, I get an error stating I cannot create another spark session. the getActiveSession() returns null from within the script, but returns the session when called from the notebook.

Re: How to instantiate Databricks spark context in a python script?

MichTalebzadeh — Wed, 13 Mar 2024 15:03:39 GMT

Thanks. Please send the full detail of error you are getting

Re: How to instantiate Databricks spark context in a python script?

Kaizen — Wed, 13 Mar 2024 15:31:18 GMT

I came across a similar issue.

Please detail how you are executing the python script. Are you calling it from the web terminal? or from a notebook?

Note: If you are calling it from the web terminal - your spark session wont be passed. You could create a local variable and pass it in if youd like. I have never gotten to that point yet tho

Re: How to instantiate Databricks spark context in a python script?

Spartan101 — Wed, 13 Mar 2024 17:03:20 GMT

Sadly, I have the errors in my corporate environment and I can't show the exact error details from this account.But it is quite close to that of @testing3

Re: How to instantiate Databricks spark context in a python script?

Spartan101 — Wed, 13 Mar 2024 17:04:09 GMT

I am running the script from databricks notebook : !streamlit run MyScript.py

Re: How to instantiate Databricks spark context in a python script?

MichTalebzadeh — Wed, 13 Mar 2024 20:35:59 GMT

If you hit an issue related to an already existing Spark context, you can only have one SparkContext instance in a single JVM. In such cases, you can try the following approach

from pyspark import SparkContext from pyspark.sql import SparkSession # Check if a Spark context already exists try: sc = SparkContext.getOrCreate() spark = SparkSession(sc) print("Using existing Spark context.") except Exception as e: print("No existing Spark context found. Creating a new one.") sc = SparkContext() spark = SparkSession(sc)

Re: How to instantiate Databricks spark context in a python script?

Spartan101 — Thu, 14 Mar 2024 01:52:51 GMT

Thanks for your responses.
I did try creating a spark context and feed it to create a spark session constructor, but I get an error from databricks which states I should not initialize a new context despite using SparkContext.getOrCreate() Method.
Tried the following as well:

SparkSession.getActiveSession() (returns null)
getOrCreate method on SparkContext and SparkSession ( Asks for Master URL and app name, after which it states the databricks error: Should not intialize session or context in databricks if one already exists.)

Re: How to instantiate Databricks spark context in a python script?

MichTalebzadeh — Thu, 14 Mar 2024 14:47:44 GMT

Hi,

I believe running a Streamlit app directly from a Databricks notebook using !streamlit run <python_code>.py is not the way to do it, because Databricks notebooks are not designed to host Streamlit

OK try below

Create a Databricks Runtime with Streamlit Pre-installed. Configure Cluster, ehen creating a new Databricks cluster, select a runtime that includes Streamlit pre-installed. This eliminates the installation step.

Run the script: Within the notebook cell, simply execute the script directly:

!streamlit run MyScript.py

HTH

apps.

Re: How to instantiate Databricks spark context in a python script?

Spartan101 — Thu, 14 Mar 2024 17:57:19 GMT

Hi @MichTalebzadeh ,

Once again, thanks for your replies.
My databricks cluster does come preinstalled with streamlit, and I have been running the script the way you mentioned.
I am going to try using alternatives to spark for the time being, and try with spark session isolation disabled as well.

I appreciate you taking out time to respond to this issue.

Re: How to instantiate Databricks spark context in a python script?

ayush007 — Thu, 03 Oct 2024 14:31:05 GMT

Is there some solution for this.We got struck where a cluster having unity catalog is not able to get spark context.This is not allowing to use distributed nature of spark in databricks.