Can we use/import python notebooks in Scala notebooks and use any functions written in Python, vice versa as well?

[sean_owen · Accepted solution]

Yes and no. Yes only in the sense that you can mix Python and Scala code in a notebook. But no you can't directly call Python code from Scala or vice versa - they are just entirely separate languages.

What you can do is share data across languages via DataFrames. Register one as a temp view and it becomes available to other interpreters. You can also register a UDF in Scala and call it via Spark SQL statements from Python. This works because Python ultimately is wrapping execution in an underlying JVM.

I don't believe the reverse is true - can't call Python UDFs from Scala.

[aladda]

Temp Views and Global Temp Views are the most common way of sharing data across languages within a Notebook/Cluster