Variables can't be accessed between API's this way.

See: Switch between %python variables and %sql variables in Databricks - Stack Overflow

In my opinion, the best way is to use the recommended answer above and create/update a tempview, or just run the query in sqlContext.sql()/spark.sql(). Creating a tempview from pyspark or capturing the result of a sql query in a dataframe isn't the same as accessing a python variable in SQL. The link above has some pretty good info in it.