topic Re: How to use SQL UDFs for Delta Live Table pipelines? in Data Engineering

How to use SQL UDFs for Delta Live Table pipelines?

Gim — Wed, 18 Jan 2023 14:31:02 GMT

I've been searching for a way to use a SQL UDF for our DLT pipeline. In this case it is to convert a time duration string into INT seconds. How exactly do we use/apply UDFs in this case?

Re: How to use SQL UDFs for Delta Live Table pipelines?

daniel_sahal — Thu, 19 Jan 2023 07:02:15 GMT

@Gim

You can create Python UDF and then use it in SQL.

https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-cookbook.html#use-python-udfs-in-sql

Re: How to use SQL UDFs for Delta Live Table pipelines?

Gim — Thu, 19 Jan 2023 22:06:19 GMT

Thanks! It wasn't initially clear to me exactly how to actually link separate notebooks in a DLT pipeline. I assume SQL UDF works as well with that same setup? The current transformations are not as complex - just needed to shorten them for reusability.