UDF already defined error when using it into a DLT pipeline

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

I'm using Unity catalog and defined some UDFs in my catalog.database, as reported by

show functions in main.default
main.default.getgender
main.default.tointlist
main.default.tostrlist

I can use them from a start warehouse pro:

SELECT main.default.get_gender(10,80,10)

The above statement works.

However, from inside a pipeline:

@dlt.view(
name="vudf"
,comment="Extra data collected from sensor tower APIs required to build the silver table"
)
def test():
   return SQL("""
SELECT ${env.database}.get_gender(10, 80 ,20)
""")

I got:

pyspark.errors.exceptions.AnalysisException: [ROUTINE_ALREADY_EXISTS] Cannot create the function `main`.`default`.`get_gender` because it already exists.
Choose a different name, drop or replace the existing function, or add the IF NOT EXISTS clause to tolerate a pre-existing function.; line 2 pos 15