I'm using Unity catalog and defined some UDFs in my catalog.database, as reported by
show functions in main.default
main.default.getgender
main.default.tointlist
main.default.tostrlist
I can use them from a start warehouse pro:
SELECT main.default.get_gender(10,80,10)
The above statement works.
However, from inside a pipeline:
@dlt.view(
name="vudf"
,comment="Extra data collected from sensor tower APIs required to build the silver table"
)
def test():
return SQL("""
SELECT ${env.database}.get_gender(10, 80 ,20)
""")
I got:
pyspark.errors.exceptions.AnalysisException: [ROUTINE_ALREADY_EXISTS] Cannot create the function `main`.`default`.`get_gender` because it already exists.
Choose a different name, drop or replace the existing function, or add the IF NOT EXISTS clause to tolerate a pre-existing function.; line 2 pos 15
