cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

spark.sql not supporting kwargs as documented

brianbraunstein
New Contributor II
This documentation https://api-docs.databricks.com/python/pyspark/latest/pyspark.sql/api/pyspark.sql.SparkSession.sql.h... claims that spark.sql() should be able to take kwargs, such that the following should work:
display(spark.sql('SELECT 9+{amount}', amount=88))
However, instead it produces this error:
TypeError: SparkSession.sql() got an unexpected keyword argument 'amount'
 
 
Ultimately I would like to use the kwargs feature to be able to SQL over a dataframe, like the documented example on the page linked above:
mydf = spark.range(10)
spark.sql(
"SELECT {col} FROM {mydf} WHERE id IN {x}",
col=mydf.id, mydf=mydf, x=tuple(range(4))).show()

Obviously the example gives the same error.  I just provided the first code as a simplified demonstration of the problem.

 
My databricks-sdk is up to date:
%pip install --upgrade databricks-sdk
dbutils.library.restartPython()
%pip show databricks-sdk

...
Name: databricks-sdk
Version: 0.26.0
Summary: Databricks SDK for Python (Beta)
Home-page: https://databricks-sdk-py.readthedocs.io
Author: Serge Smertin
Author-email: serge.smertin@databricks.com
License: UNKNOWN
Location: /local_disk0/.ephemeral_nfs/envs/pythonEnv-07bec08e-988d-4ca2-8229-83561254d3c0/lib/python3.10/site-packages
Requires: google-auth, requests
Required-by:
 
My spark version is up to date:
display(spark.sql('SELECT version()'))

3.5.0 0000000000000000000000000000000000000000
 

 

2 REPLIES 2

brianbraunstein
New Contributor II

Ok, it looks like Databricks might have broken this functionality shortly after it came out: https://community.databricks.com/t5/data-engineering/parameterized-spark-sql-not-working/m-p/57969/h...

adriennn
Valued Contributor

It's working again in 15.4 LTS


Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now