topic Re: DLT Direct Publishing Mode in Data Engineering

DLT Direct Publishing Mode

804082 — Fri, 27 Sep 2024 16:25:52 GMT

Hello,

I'm working on a DLT pipeline and have a block of SQL that runs...

USE CATALOG catalog_a; USE SCHEMA schema_a; CREATE OR REFRESH MATERIALIZED VIEW table_a AS SELECT ... FROM catalog_b.schema_b.table_b;

Executing this block returns the following...

com.databricks.pipelines.common.errors.DLTSparkException: [UNSUPPORTED_COMMAND_IN_NON_DPM_PIPELINE] USE CATALOG only supported in direct publishing mode enabled DLT pipeline

I can't find any documentation or mention of this error, or anything about enabling "direct publishing mode". Any idea as to how to do that? Or is there another recommended/possible way to create a materialized view using data from another catalog?

Thanks in advance.

Re: DLT Direct Publishing Mode

gchandra — Sun, 29 Sep 2024 18:41:34 GMT

Direct Publishing Mode is a Private Preview feature.

Till it becomes Public Preview or GA, please remove the USE Catalog and Use Schema and try catalog.schema.view

Re: DLT Direct Publishing Mode

804082 — Mon, 30 Sep 2024 13:20:10 GMT

Ah, thank you! I couldn't find anything about Direct Publishing Mode through my Google search (in fact, if you search it now, this post is the top result).

I believe I'll be waiting until it goes into public preview/GA. Removing USE CATALOG and USE SCHEMA, and instead using catalog.schema.view, returns...

org.apache.spark.sql.catalyst.ExtendedAnalysisException: Unsupported SQL statement for table 'catalog.schema.view': Multipart table names is not supported.

Re: DLT Direct Publishing Mode

kuldeep-in — Sat, 01 Feb 2025 11:39:30 GMT

'Direct Publishing Mode' Public Preview is now live on all production regions. Start exploring today and share your valuable feedback with us.

cc: @804082 @gchandra

Re: DLT Direct Publishing Mode

Dorsey — Tue, 04 Feb 2025 19:06:08 GMT

I'm still running into issues like the op, getting the same error and have confirmed Direct Publishing Mode is set to True in the pipeline definition

Re: DLT Direct Publishing Mode

kuldeep-in — Tue, 04 Feb 2025 21:33:47 GMT

@Dorsey Make sure to enable DPM from the Preview page. Can you confirm the region of your workspace?

Re: DLT Direct Publishing Mode

Dorsey — Tue, 04 Feb 2025 21:39:12 GMT

I'm in EastUS and i don't have that option on my previews page. Also it only works with serverless?

Re: DLT Direct Publishing Mode

shorouk-talat — Wed, 05 Mar 2025 13:39:13 GMT

how can i enable it ? it is require workspace admin role ?
also why i can't find any documentation for this new feature ?

Re: DLT Direct Publishing Mode

mohanaveluk — Thu, 08 May 2025 19:31:52 GMT

when i try creating streaming table as below
dlt.create_streaming_table("dtlk_dev.transform_schme.invoices"), it throws error stating to remove the database qualifier name and try again