MartinIsti
Databricks Partner

I can see someone has asked a very similar question with the same error message:

https://community.databricks.com/t5/data-engineering/unable-to-use-sql-udf/td-p/61957

The OP hasn't yet provided sufficient details about his/her function so no proper response has appeared so far. I have gone through the 4 listed points to make sure I have narrowed down the root cause of the error. And I have.

See below an even more simplified function definition (to rule out the possibility if the cluster has access to the storage location) that fails with the same NameError: name 'spark' is not defined error

CREATE OR REPLACE FUNCTION dev_fusion.log.WatermarkRead_UC(ADLSLocation STRING, WatermarkAttribute STRING)
RETURNS STRING
LANGUAGE PYTHON
AS $$

    WatermarkValue = 'spark.sql(f"SELECT 'value'").collect()[0][0]'

    return WatermarkValue

$$

And one that works:

CREATE OR REPLACE FUNCTION dev_fusion.log.WatermarkRead_UC(ADLSLocation STRING, WatermarkAttribute STRING)
RETURNS STRING
LANGUAGE PYTHON
AS $$

    WatermarkValue = 'Value'

    return WatermarkValue

$$

The main difference being the spark.sql part.