cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Unable to access Scala and python variables in-between shells in same notebook.

Data_Engineer3
Contributor III

I am facing issue in while accessing python data frame in Scala shell and vice versa. I am getting error variable not defined.

2 REPLIES 2

tomasz
Databricks Employee
Databricks Employee

The context is not shared between Scala and Python so you won't be able to access the same variables directly. However you can use createOrReplaceTempView to create a temporary view of your dataframe and read it in the other language with read_df = spark.sql("<sql query reading from view>").

Thanks @Tomasz Bacewiczโ€‹ ,

you mean, even if we use it in same notebook in databricks, it can't share among it, it maintain two different spark context for python and scala.

But I am getting another question like if we create TempView on dataframe, this tempView will maintain on which context level? Could you explain on it.

Thanks

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now