cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How can i use the same spark session from onenotebook to another notebook in databricks

Data_Engineer3
Contributor II

I want to use the same spark session which created in one notebook and need to be used in another notebook in across same environment,

Example, if some of the (variable)object got initialized in the first notebook, i need to use the same object in the another notebook.

I am aware of, we can use the other notebook function by using %run(magic command), likewise if i want to use sparksession of onenotebook in another notebook?

4 REPLIES 4

User16752242622
Valued Contributor

Hello, Have you tried dbutils.notebook.run(). I believe this API will let you pass parameters within itself. I am not entirely sure if this uses the same spark session. from a different notebook

https://docs.databricks.com/notebooks/notebook-workflows.html#comparison-of-run-and-dbutilsnotebookr...

No, here we are running second notebook in first notebook with same spark session, I would like to run code in second notebook with spark session which have created in first notebook.

#[Azure databricks]​ #[Databricks notebook]​ #Spark​ #Pyspark​ 

Anonymous
Not applicable

@KARTHICK N​ :

Yes, you can use the same SparkSession instance across multiple notebooks in Databricks. Here's how you can do it:

In the first notebook where you create the SparkSession instance, assign it to a variable:

spark = SparkSession.builder.appName("my_app").getOrCreate()

In the second notebook where you want to use the same SparkSession, you can access the variable that contains the SparkSession instance by using the %run command:

%run "/path/to/first/notebook"

This will execute the first notebook and make all the variables defined in it available in the current notebook. So if you defined a variable called spark in the first notebook, you can access it in the second notebook after running the %run command.

Note that you need to provide the full path to the first notebook in the %run command, including the file extension (e.g. ipynb).Once you have access to the spark variable in the second notebook, you can use it just like you would in the first notebook:

df = spark.read.csv("/path/to/data.csv")

This will create a DataFrame using the same SparkSession instance that was created in the first notebook.

Keep in mind that when you use the %run command to access variables from another notebook, you are essentially importing those variables into the current notebook. So if you modify a variable in the second notebook, it will not affect the original variable in the first notebook. If you need to share data between notebooks in a way that allows you to modify it in one notebook and have those changes reflected in another notebook, you may want to consider using a shared database or file system instead.

Manoj12421
Valued Contributor II

You can use %run and then use the location of the notebook -

%run "/folder/notebookname"

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!