spark.sql not supporting kwargs as documented

brianbraunstein — Thu, 02 May 2024 14:51:39 GMT

This documentation https://api-docs.databricks.com/python/pyspark/latest/pyspark.sql/api/pyspark.sql.SparkSession.sql.html#pyspark.sql.SparkSession.sql claims that spark.sql() should be able to take kwargs, such that the following should work:

display(spark.sql('SELECT 9+{amount}', amount=88))

However, instead it produces this error:

TypeError: SparkSession.sql() got an unexpected keyword argument 'amount'

Ultimately I would like to use the kwargs feature to be able to SQL over a dataframe, like the documented example on the page linked above:

mydf = spark.range(10)
spark.sql(
"SELECT {col} FROM {mydf} WHERE id IN {x}",
col=mydf.id, mydf=mydf, x=tuple(range(4))).show()

Obviously the example gives the same error. I just provided the first code as a simplified demonstration of the problem.

My databricks-sdk is up to date:

%pip install --upgrade databricks-sdk
dbutils.library.restartPython()
%pip show databricks-sdk

...
Name: databricks-sdk
Version: 0.26.0
Summary: Databricks SDK for Python (Beta)
Home-page: https://databricks-sdk-py.readthedocs.io
Author: Serge Smertin
Author-email: serge.smertin@databricks.com
License: UNKNOWN
Location: /local_disk0/.ephemeral_nfs/envs/pythonEnv-07bec08e-988d-4ca2-8229-83561254d3c0/lib/python3.10/site-packages
Requires: google-auth, requests
Required-by:

My spark version is up to date:

display(spark.sql('SELECT version()'))

3.5.0 0000000000000000000000000000000000000000

Re: spark.sql not supporting kwargs as documented

brianbraunstein — Thu, 02 May 2024 16:23:38 GMT

Ok, it looks like Databricks might have broken this functionality shortly after it came out: https://community.databricks.com/t5/data-engineering/parameterized-spark-sql-not-working/m-p/57969/highlight/true#M30972

Re: spark.sql not supporting kwargs as documented

adriennn — Wed, 18 Sep 2024 15:53:50 GMT

It's working again in 15.4 LTS

topic Re: spark.sql not supporting kwargs as documented in Data Engineering

spark.sql not supporting kwargs as documented

Re: spark.sql not supporting kwargs as documented

Re: spark.sql not supporting kwargs as documented