DLT Direct Publishing Mode

[804082]

Hello,

I'm working on a DLT pipeline and have a block of SQL that runs...

USE CATALOG catalog_a;
USE SCHEMA schema_a;
CREATE OR REFRESH MATERIALIZED VIEW table_a AS
SELECT ... FROM catalog_b.schema_b.table_b;

Executing this block returns the following...

com.databricks.pipelines.common.errors.DLTSparkException: [UNSUPPORTED_COMMAND_IN_NON_DPM_PIPELINE] USE CATALOG only supported in direct publishing mode enabled DLT pipeline

I can't find any documentation or mention of this error, or anything about enabling "direct publishing mode". Any idea as to how to do that? Or is there another recommended/possible way to create a materialized view using data from another catalog?

Thanks in advance.

[gchandra]

Direct Publishing Mode is a Private Preview feature. 

Till it becomes Public Preview or GA, please remove the USE Catalog and Use Schema and try catalog.schema.view  

[mohanaveluk]

when i try creating streaming table as below
dlt.create_streaming_table("dtlk_dev.transform_schme.invoices"), it throws error stating to remove the database qualifier name and try again

[804082]

Ah, thank you! I couldn't find anything about Direct Publishing Mode through my Google search (in fact, if you search it now, this post is the top result).

I believe I'll be waiting until it goes into public preview/GA. Removing USE CATALOG and USE SCHEMA, and instead using catalog.schema.view, returns...

org.apache.spark.sql.catalyst.ExtendedAnalysisException: Unsupported SQL statement for table 'catalog.schema.view': Multipart table names is not supported.

[kuldeep-in · Accepted solution]

'Direct Publishing Mode' Public Preview is now live on all production regions. Start exploring today and share your valuable feedback with us.

cc: @804082 @gchandra 

[Dorsey]

I'm still running into issues like the op, getting the same error and have confirmed Direct Publishing Mode is set to True in the pipeline definition

[kuldeep-in]

@Dorsey Make sure to enable DPM from the Preview page. Can you confirm the region of your workspace?

[shorouk-talat]

how can i enable it ? it is require  workspace admin role ?
also why i can't find any documentation for this new feature ?

[Dorsey]

I'm in EastUS and i don't have that option on my previews page. Also it only works with serverless?