- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2021 07:10 PM
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2021 03:05 AM
Yes and no. Yes only in the sense that you can mix Python and Scala code in a notebook. But no you can't directly call Python code from Scala or vice versa - they are just entirely separate languages.
What you can do is share data across languages via DataFrames. Register one as a temp view and it becomes available to other interpreters. You can also register a UDF in Scala and call it via Spark SQL statements from Python. This works because Python ultimately is wrapping execution in an underlying JVM.
I don't believe the reverse is true - can't call Python UDFs from Scala.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2021 03:05 AM
Yes and no. Yes only in the sense that you can mix Python and Scala code in a notebook. But no you can't directly call Python code from Scala or vice versa - they are just entirely separate languages.
What you can do is share data across languages via DataFrames. Register one as a temp view and it becomes available to other interpreters. You can also register a UDF in Scala and call it via Spark SQL statements from Python. This works because Python ultimately is wrapping execution in an underlying JVM.
I don't believe the reverse is true - can't call Python UDFs from Scala.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2021 09:13 PM
Temp Views and Global Temp Views are the most common way of sharing data across languages within a Notebook/Cluster