Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-01-2024 08:22 PM
I can see someone has asked a very similar question with the same error message:
https://community.databricks.com/t5/data-engineering/unable-to-use-sql-udf/td-p/61957
The OP hasn't yet provided sufficient details about his/her function so no proper response has appeared so far. I have gone through the 4 listed points to make sure I have narrowed down the root cause of the error. And I have.
See below an even more simplified function definition (to rule out the possibility if the cluster has access to the storage location) that fails with the same NameError: name 'spark' is not defined error
CREATE OR REPLACE FUNCTION dev_fusion.log.WatermarkRead_UC(ADLSLocation STRING, WatermarkAttribute STRING)
RETURNS STRING
LANGUAGE PYTHON
AS $$
WatermarkValue = 'spark.sql(f"SELECT 'value'").collect()[0][0]'
return WatermarkValue
$$And one that works:
CREATE OR REPLACE FUNCTION dev_fusion.log.WatermarkRead_UC(ADLSLocation STRING, WatermarkAttribute STRING)
RETURNS STRING
LANGUAGE PYTHON
AS $$
WatermarkValue = 'Value'
return WatermarkValue
$$The main difference being the spark.sql part.