topic Re: Catalogue global UDF's in Data Engineering

Catalogue global UDF's

AlexVB — Tue, 21 Jan 2025 16:35:28 GMT

The current UDF implementation stores UDFs in a catalogue.schema location. This requires reference/call to said udf location; example `select my_catalogue.my_schema.my_udf()`. Or have the sql execute from that schema.

In Snowflake, UDFs are globally available. Say i define an Uber H3 UDF or a UUID5 UDF. Then I want to use this everywhere, without specifying some_catalogue.some_schema.uuid5. I just want to do `select uuid5(name)`.

Alternatively, if we can allow functions to be stored on the catalogue level.

In my opinion, the UDF functionality is very useless at a larger scale with the current implementation.

Re: Catalogue global UDF's

Sidhant07 — Thu, 30 Jan 2025 08:30:26 GMT

Hi @AlexVB ,

The current UDF implementation in Databricks requires referencing the UDF location with select my_catalogue.my_schema.my_udf() or executing SQL from that schema because Databricks organizes database objects using a three-tier hierarchy: Catalog, Schema (or Database), and Data Objects (such as Functions). This structure mandates specifying the catalog and schema to locate the UDF unless it is a temporary function scoped at the session level, which does not require such specification.

Re: Catalogue global UDF's

Sidhant07 — Thu, 30 Jan 2025 09:11:02 GMT

Hi @AlexVB ,

Thanks for the feedback. Since you are comparing databricks with snowflake its ideal you discuss this with your sales rep.

Your suggestion to allow functions to be stored at the catalog level or to make them globally available within Databricks is valuable. I can raise a feature request on your behalf so that our product team can prioritise it in future.