cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Not able to run spark context by running python file in a cell using %sh

deficiant_codge
Contributor II

Hi Everyone,

I am trying to get spark context by running SparkSession builder command in a .py file and when i try to run it directl it works but if i got to a notebook and in a cell if i use %sh command to run python "myfilename" it gives an error that i am trying to create a new session. How to solve this?

Any help would be appreciated.

 

deficiant_codge_1-1689000063074.pngdeficiant_codge_2-1689000103143.png

 

 

Thanks

6 REPLIES 6

Hemant
Valued Contributor II

Hi @deficiant_codge, Databricks already provide variables for sparksession and sparkcontext.

Use spark for sparksession and sc for sparkcontext.
I think this would help you.

Hemant Soni

Actually i have some use case in which i have to mandatorily run the python file using %sh and in that i can't able to access those variables. I am aware of these variables but the problem is they just don't work.
see the below images:

deficiant_codge_0-1689001870121.pngdeficiant_codge_1-1689001889304.png

 

Hemant
Valued Contributor II

@deficiant_codge  Can you try with below:

_spark_session = SparkSession \
    .Builder() \
    .appName("myapp") \
    .getOrCreate()
Hemant Soni

I tried it also attached the image attached with the post, but it didn't worked

Hemant
Valued Contributor II

@deficiant_codge  there is minor difference can you try it.

 

Hemant Soni

No Success 😞

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!