cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Can we access the variables created in Python in Scala's code or notebook ?

Zircoz
New Contributor II

If I have a dict created in python on a Scala notebook (using magic word ofcourse):

%python
d1 = {1: "a", 2:"b", 3:"c"}

Can I access this d1 in Scala ?

I tried the following and it returns d1 not found:

%scala
println(d1)

1 ACCEPTED SOLUTION

Accepted Solutions

MartinB
Contributor III

Variables are not shared between language contexts.

Ugly workaround: you could do something like this to pass your python variable to the spark context:

%python
d1 = {1: "a", 2:"b", 3:"c"}
spark.conf.set('d1', str(d1))
%scala
println( spark.conf.get("d1") )

However, you lose variable type. You only can pass on strings.

View solution in original post

2 REPLIES 2

MartinB
Contributor III

Variables are not shared between language contexts.

Ugly workaround: you could do something like this to pass your python variable to the spark context:

%python
d1 = {1: "a", 2:"b", 3:"c"}
spark.conf.set('d1', str(d1))
%scala
println( spark.conf.get("d1") )

However, you lose variable type. You only can pass on strings.

cpm1
New Contributor II

Martin is correct. We could only access the external files and objects. In most of our cases, we just use temporary views to pass data between R & Python.

https://docs.databricks.com/notebooks/notebooks-use.html#mix-languages

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.