UDF already defined error when using it into a DLT pipeline

6502 — Tue, 19 Dec 2023 17:27:52 GMT

I'm using Unity catalog and defined some UDFs in my catalog.database, as reported by

show functions in main.default main.default.getgender main.default.tointlist main.default.tostrlist

I can use them from a start warehouse pro:

SELECT main.default.get_gender(10,80,10)

The above statement works.

However, from inside a pipeline:

@dlt.view( name="vudf" ,comment="Extra data collected from sensor tower APIs required to build the silver table" ) def test(): return SQL(""" SELECT ${env.database}.get_gender(10, 80 ,20) """)

I got:

pyspark.errors.exceptions.AnalysisException: [ROUTINE_ALREADY_EXISTS] Cannot create the function `main`.`default`.`get_gender` because it already exists. Choose a different name, drop or replace the existing function, or add the IF NOT EXISTS clause to tolerate a pre-existing function.; line 2 pos 15

Re: UDF already defined error when using it into a DLT pipeline

Mounika_Tarigop — Sat, 07 Dec 2024 06:02:32 GMT

Now the Python UDF support is currently in public preview, and it is supported in the current channel as well. Could you please try to re-run the code.

https://docs.databricks.com/en/delta-live-tables/unity-catalog.html

topic Re: UDF already defined error when using it into a DLT pipeline in Data Engineering

UDF already defined error when using it into a DLT pipeline

Re: UDF already defined error when using it into a DLT pipeline