Use of Python variable in SQL cell

jar
Contributor

If using spark.conf.set(<variable_name>, <variable_value>), or just referring a widget value directly, in a Python cell and then referring to it in a SQL cell with ${variable_name} one gets the warning: "SQL query contains a dollar sign parameter, $param, which is deprecated. Please migrate to the parameter marker syntax, :param."

If one does so in a cell which creates a temporary view one gets the error "[UNSUPPORTED_FEATURE.PARAMETER_MARKER_IN_UNEXPECTED_STATEMENT] The feature is not supported: Parameter markers are not allowed in the query of CREATE VIEW. SQLSTATE: 0A000".

I'm using a > 15.2 runtime cluster.

So, what does one do?