topic Not able to run spark context by running python file in a cell using %sh in Get Started Discussions

Not able to run spark context by running python file in a cell using %sh

deficiant_codge — Mon, 10 Jul 2023 14:45:54 GMT

Hi Everyone,

I am trying to get spark context by running SparkSession builder command in a .py file and when i try to run it directl it works but if i got to a notebook and in a cell if i use %sh command to run python "myfilename" it gives an error that i am trying to create a new session. How to solve this?

Any help would be appreciated.

Thanks

Re: Not able to run spark context by running python file in a cell using %sh

Hemant — Mon, 10 Jul 2023 14:53:08 GMT

Hi @deficiant_codge, Databricks already provide variables for sparksession and sparkcontext.

Use spark for sparksession and sc for sparkcontext.
I think this would help you.

Re: Not able to run spark context by running python file in a cell using %sh

deficiant_codge — Mon, 10 Jul 2023 15:11:38 GMT

Actually i have some use case in which i have to mandatorily run the python file using %sh and in that i can't able to access those variables. I am aware of these variables but the problem is they just don't work.
see the below images:

Re: Not able to run spark context by running python file in a cell using %sh

Hemant — Mon, 10 Jul 2023 15:42:09 GMT

@deficiant_codge Can you try with below:

_spark_session = SparkSession \

.Builder() \

.appName("myapp") \

.getOrCreate()

Re: Not able to run spark context by running python file in a cell using %sh

deficiant_codge — Mon, 10 Jul 2023 15:44:29 GMT

I tried it also attached the image attached with the post, but it didn't worked

Re: Not able to run spark context by running python file in a cell using %sh

Hemant — Mon, 10 Jul 2023 15:45:55 GMT

@deficiant_codge there is minor difference can you try it.

Re: Not able to run spark context by running python file in a cell using %sh

deficiant_codge — Mon, 10 Jul 2023 17:04:58 GMT

No Success 😞