topic Re: Can we access the variables created in Python in Scala's code or notebook ? in Data Engineering

Can we access the variables created in Python in Scala's code or notebook ?

Zircoz — Sat, 11 Sep 2021 09:31:52 GMT

If I have a dict created in python on a Scala notebook (using magic word ofcourse):

%python
d1 = {1: "a", 2:"b", 3:"c"}

Can I access this d1 in Scala ?

I tried the following and it returns d1 not found:

%scala
println(d1)

Re: Can we access the variables created in Python in Scala's code or notebook ?

MartinB — Sat, 11 Sep 2021 10:44:00 GMT

Variables are not shared between language contexts.

Ugly workaround: you could do something like this to pass your python variable to the spark context:

%python
d1 = {1: "a", 2:"b", 3:"c"}
spark.conf.set('d1', str(d1))

%scala
println( spark.conf.get("d1") )

However, you lose variable type. You only can pass on strings.

Re: Can we access the variables created in Python in Scala's code or notebook ?

cpm1 — Sat, 11 Sep 2021 14:38:36 GMT

Martin is correct. We could only access the external files and objects. In most of our cases, we just use temporary views to pass data between R & Python.

https://docs.databricks.com/notebooks/notebooks-use.html#mix-languages