topic Re: Can we use/import python notebooks in Scala notebooks and use any functions written in Python, vice versa as well? in Data Engineering

Can we use/import python notebooks in Scala notebooks and use any functions written in Python, vice versa as well?

Srikanth_Gupta_ — Mon, 21 Jun 2021 02:10:59 GMT

Re: Can we use/import python notebooks in Scala notebooks and use any functions written in Python, vice versa as well?

sean_owen — Tue, 22 Jun 2021 10:05:13 GMT

Yes and no. Yes only in the sense that you can mix Python and Scala code in a notebook. But no you can't directly call Python code from Scala or vice versa - they are just entirely separate languages.

What you can do is share data across languages via DataFrames. Register one as a temp view and it becomes available to other interpreters. You can also register a UDF in Scala and call it via Spark SQL statements from Python. This works because Python ultimately is wrapping execution in an underlying JVM.

I don't believe the reverse is true - can't call Python UDFs from Scala.

Re: Can we use/import python notebooks in Scala notebooks and use any functions written in Python, vice versa as well?

aladda — Wed, 23 Jun 2021 04:13:02 GMT

Temp Views and Global Temp Views are the most common way of sharing data across languages within a Notebook/Cluster