Not able to run spark context by running python file in a cell using %sh

[deficiant_codge]

Hi Everyone,

I am trying to get spark context by running SparkSession builder command in a .py file and when i try to run it directl it works but if i got to a notebook and in a cell if i use %sh command to run python "myfilename" it gives an error that i am trying to create a new session. How to solve this?

Any help would be appreciated.

Thanks

[Hemant]

Hi @deficiant_codge, Databricks already provide variables for sparksession and sparkcontext.

Use spark for sparksession and sc for sparkcontext.
I think this would help you.

[deficiant_codge]

Actually i have some use case in which i have to mandatorily run the python file using %sh and in that i can't able to access those variables. I am aware of these variables but the problem is they just don't work.
see the below images:

[Hemant]

@deficiant_codge  Can you try with below:

_spark_session = SparkSession \

    .Builder() \

    .appName("myapp") \

    .getOrCreate()

[deficiant_codge]

I tried it also attached the image attached with the post, but it didn't worked

[Hemant]

@deficiant_codge  there is minor difference can you try it.

[deficiant_codge]

No Success 😞