topic Re: Unable to access Scala and python variables in-between shells in same notebook. in Data Engineering

Unable to access Scala and python variables in-between shells in same notebook.

Data_Engineer3 — Mon, 11 Jul 2022 18:26:22 GMT

I am facing issue in while accessing python data frame in Scala shell and vice versa. I am getting error variable not defined.

Re: Unable to access Scala and python variables in-between shells in same notebook.

tomasz — Mon, 11 Jul 2022 18:44:11 GMT

The context is not shared between Scala and Python so you won't be able to access the same variables directly. However you can use createOrReplaceTempView to create a temporary view of your dataframe and read it in the other language with read_df = spark.sql("<sql query reading from view>").

Re: Unable to access Scala and python variables in-between shells in same notebook.

Data_Engineer3 — Thu, 21 Jul 2022 14:54:04 GMT

Thanks @Tomasz Bacewicz ,

you mean, even if we use it in same notebook in databricks, it can't share among it, it maintain two different spark context for python and scala.

But I am getting another question like if we create TempView on dataframe, this tempView will maintain on which context level? Could you explain on it.

Thanks