11-08-2023 06:19 PM
Hello,
I am trying to generate a DLT but need to use a UDF Table Function in the process.
This is what I have so far, everything works (without e CREATE OR REFRESH LIVE TABLE wrapper)
```sql
```
However, this results in the following error:
```
4 weeks ago
Hi all,
I have figured out my error.
The function creation should NOT be in the same notebook as the one used by the DLT pipeline.
Instead, create the function separately and register it in Unity Catalog, then reference the function in the DLT pipeline.
As a side note, UDF support for DLT is only available in DBR 13.3 and above. At the time of writing this, you will need to configure the runtime to 'preview' over 'current' to enable UDFs in DLT.
4 weeks ago
@alexiswl - could you please use CREATE OR REPLACE function instead of CREATE OR REFRESH LIVE table?
4 weeks ago
Hi @shan_chandra, unfortunately this is in the context of creating a DLT, so if I update as suggested, I get the following error:
```
org.apache.spark.sql.AnalysisException: The following code at '/Repos/[email protected]/data-portal-analytics/delta_live_tables/silver/silver_portal_dlt' is not supported in a DLT pipeline: CREATE OR REPLACE FUNCTION silver.portal.get_workf... Please correct your code and make sure you only have DLT statements. For reference: DLT currently accepts 'CREATE MATERIALIZED VIEW', 'CREATE STREAMING TABLE', 'APPLY CHANGES INTO', and 'SET' statements.
```
4 weeks ago
Hi all,
I have figured out my error.
The function creation should NOT be in the same notebook as the one used by the DLT pipeline.
Instead, create the function separately and register it in Unity Catalog, then reference the function in the DLT pipeline.
As a side note, UDF support for DLT is only available in DBR 13.3 and above. At the time of writing this, you will need to configure the runtime to 'preview' over 'current' to enable UDFs in DLT.
3 weeks ago
Hi @alexiswl , I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.