topic Re: Views in DLT with Private Preview feature Direct Publish in Data Engineering

Views in DLT with Private Preview feature Direct Publish

susanne — Fri, 24 Jan 2025 12:59:13 GMT

Hi everyone,

I am building a dlt Pipeline and there I am using the Direct Publish feature which is as of now still under Private Preview.
While it works well to create streaming tables and write them to another schema than the dlt default schema, I get the following error message, when I try to create a View and write it to a different schema than the default schema:
AnalysisException: View with multipart name 'gold.fact_customer' is not supported.
I used the following code to create the View:

@dlt.table(

name = target_table,

)

def get_gold_table():

df = spark.sql(f"""{gold_selection}""")

return df

Does Direct Publish in general not support publishing of views to another schema?
Any help would be much appreciated.
Thank you 🙂
Susanne

Re: Views in DLT with Private Preview feature Direct Publish

Sidhant07 — Thu, 30 Jan 2025 07:59:07 GMT

Hi @susanne ,

Use Tables Instead of Views: If possible, convert your views to tables.

@dlt.table( name=target_table, ) def get_gold_table(): df = spark.sql(f"""{gold_selection}""") return df

Direct Publish Feature: The Direct Publish feature does not support publishing views to another schema. It is intended for tables and materialized views, which can be published to the catalog.

Solution: To publish data to a different schema, consider using a table or materialized view instead of a view, as these can be written to the catalog.

Re: Views in DLT with Private Preview feature Direct Publish

susanne — Thu, 30 Jan 2025 09:13:11 GMT

Hi Sidhan,
thanks a lot for your reply, it works very well to write materialized views to a different schema than the default schema.
Thanks for your guidance!

Best regards
Susanne