Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-09-2025 06:19 AM
Hello Alberto,
thanks for the quick answer! Actually I want to pass a dataframe to the function, like:
@Dlt.table(
name="test"
)
def create_table():
test_df = spark.createDataFrame(["9","10","11","13"], "string").toDF("id")
final_df = spark.sql("SELECT * FROM {df} WHERE id > 9", df=test_df)
return final_df
TypeError: _override_spark_functions.<locals>._dlt_sql_fn() got an unexpected keyword argument 'df',
And I get the same error when trying to only pass an integer for testing purposes, like
@Dlt.table(
name="test"
)
def create_table():
test_df = spark.sql("SELECT * FROM range(10) WHERE id > {bound1} AND id < {bound2}", bound1=7, bound2=9)
return test_df
TypeError: _override_spark_functions.<locals>._dlt_sql_fn() got an unexpected keyword argument 'bound1',
Both spark.sql() statements work outside a DLT pipeline. So it seems like this is an DLT specific issue.