cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Can we use/import python notebooks in Scala notebooks and use any functions written in Python, vice versa as well?

Srikanth_Gupta_
Databricks Employee
Databricks Employee
 
1 ACCEPTED SOLUTION

Accepted Solutions

sean_owen
Databricks Employee
Databricks Employee

Yes and no. Yes only in the sense that you can mix Python and Scala code in a notebook. But no you can't directly call Python code from Scala or vice versa - they are just entirely separate languages.

What you can do is share data across languages via DataFrames. Register one as a temp view and it becomes available to other interpreters. You can also register a UDF in Scala and call it via Spark SQL statements from Python. This works because Python ultimately is wrapping execution in an underlying JVM.

I don't believe the reverse is true - can't call Python UDFs from Scala.

View solution in original post

2 REPLIES 2

sean_owen
Databricks Employee
Databricks Employee

Yes and no. Yes only in the sense that you can mix Python and Scala code in a notebook. But no you can't directly call Python code from Scala or vice versa - they are just entirely separate languages.

What you can do is share data across languages via DataFrames. Register one as a temp view and it becomes available to other interpreters. You can also register a UDF in Scala and call it via Spark SQL statements from Python. This works because Python ultimately is wrapping execution in an underlying JVM.

I don't believe the reverse is true - can't call Python UDFs from Scala.

aladda
Databricks Employee
Databricks Employee

Temp Views and Global Temp Views are the most common way of sharing data across languages within a Notebook/Cluster

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group